投影片 1

Size: px

Start display at page:

Download "投影片 1"

史婉跃贝
4 years ago
Views:

Map-Reduce Programming 王耀聰陳威宇 jazz@nchc.org.

1 Map-Reduce Programming 王耀聰陳威宇國家高速網路與計算中心 (NCHC) 自由軟體實驗室

2 Outline 概念程式基本框架及執行步驟方法範例一 : Hadoop 的 Hello World => Word Count 說明動手做範例二 : 進階版 => Word Count 2 說明動手做 2

3 程式基本框架 Program Prototype (v 0.18) Map 區 Reduce 區設定區 Class MR{ Class Mapper { Map 程式碼 } Class Reducer { } Reduce 程式碼 main(){ JobConf conf = new JobConf( MR.class ); conf.setmapperclass(mapper.class); conf.setreduceclass(reducer.class); }} FileInputFormat.setInputPaths(conf, new Path(args[0])); FileOutputFormat.setOutputPath(conf, new Path(args[1])); 其他的設定參數程式碼 JobClient.runJob(conf); 3

4 式基本框架 Class Mapper class MyMap extends MapReduceBase implements Mapper <,, OUTPUT KEY, > { // 全域變數區 INPUT KEY INPUT KEY public void map ( key, value, OutputCollector <, OUTPUT > output, OUTPUT KEY INPUT VALUE VALUE Reporter reporter) throws IOException { // 區域變數與程式邏輯區 output.collect( NewKey, NewValue); } } INPUT VALUE OUTPUT VALUE 4

5 式基本框架 Class Reducer class MyRed extends MapReduceBase implements Reducer <,, OUTPUT KEY, > { // 全域變數區 INPUT KEY INPUT KEY public void reduce ( key, Iterator< > values, OutputCollector<, OUTPUT > output, OUTPUT KEY INPUT VALUE Reporter reporter) throws IOException { // 區域變數與程式邏輯區 output.collect( NewKey, NewValue); } } VALUE INPUT VALUE OUTPUT VALUE 5

6 式基本框架 Class Combiner 指定一個 combiner, 它負責對中間過程的輸出進行聚集, 這會有助於降低從 Mapper 到 Reducer 數據傳輸量可不用設定交由 Hadoop 預設也可不實做此程式, 引用 Reducer 設定 JobConf.setCombinerClass(Class) 6

7 式基本框架 Run Job runjob(jobconf ) 提交作業, 僅當作業完成時返回 submitjob(jobconf ) 只提交作業, 之後需要你輪詢它返回的 RunningJob 句柄的狀態, 並根據情況調度 JobConf.setJobEndNotificationURI(String) 設置一個作業完成通知, 可避免輪詢 7

8 範例程式一 Word Count Sample (1) class MapClass extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map( LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { String line = ((Text) value).tostring(); StringTokenizer itr = new StringTokenizer(line); while (itr.hasmoretokens()) { word.set(itr.nexttoken()); output.collect(word, one); }}} Input key /user/hadooper/input/a.txt. No news is a good news. Input value line itr no news is a good news itr itr itr itr itr itr <word,one> < no, 1 > < news, 1 > < is, 1 > < a, 1 > < good, 1 > 8 < news, 1 >

9 範例程式一 Word Count Sample (2) class ReduceClass extends MapReduceBase implements Reducer< Text, IntWritable, Text, IntWritable> { IntWritable SumValue = new IntWritable(); public void reduce( Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { int sum = 0; while (values.hasnext()) sum += values.next().get(); SumValue.set(sum); output.collect(key, SumValue); }} <word,one> < no, 1 > < news, 1 > < is, 1 > < a, 1 > < good, 1 > < news, 1 > news 1 1 <key,sunvalue> < news, 2 > 9

10 範例程式一 Word Count Sample (3) Class WordCount{ main() JobConf conf = new JobConf(WordCount.class); conf.setjobname("wordcount"); // set path FileInputFormat.setInputPaths(new Path(args[0])); FileOutputFormat.setOutputPath(new Path(args[1])); // set map reduce conf.setmapperclass(mapclass.class); }} conf.setcombinerclass(reduce.class); conf.setreducerclass(reduceclass.class); // run JobClient.runJob(conf); 10

11 執行流程基本步驟編譯與執行 1. 編譯 javac Δ -classpath Δ hadoop-*-core.jar Δ -d Δ MyJava Δ MyCode.java 2. 封裝 jar Δ -cvf Δ MyJar.jar Δ -C Δ MyJava Δ. 3. 執行 bin/hadoop Δ jar Δ MyJar.jar Δ MyCode Δ HDFS_Input/ Δ HDFS_Output/ 所在的執行目錄為 Hadoop_Home./MyJava = 編譯後程式碼目錄 Myjar.jar = 封裝後的編譯檔先放些文件檔到 HDFS 上的 input 目錄./input = hdfs 的輸入目錄./ouput = hdfs 的輸出目錄 11

12 範例一動手做 WordCount1 練習 (I) 1. cd $HADOOP_HOME 2. bin/hadoop dfs -mkdir input 3. echo "I like NCHC Cloud Course." > inputwc/input1 4. echo "I like nchc Cloud Course, and we enjoy this crouse." > inputwc/input2 5. bin/hadoop dfs -put inputwc inputwc 6. bin/hadoop dfs -ls input 12

13 範例一動手做 WordCount1 練習 (II) 編輯 WordCount.java mkdir MyJava javac -classpath hadoop-*-core.jar -d MyJava WordCount.java jar -cvf wordcount.jar -C MyJava. bin/hadoop jar wordcount.jar WordCount input/ output/ 所在的執行目錄為 Hadoop_Home ( 因為 hadoop-*-core.jar ) javac 編譯時需要 classpath, 但 hadoop jar 時不用 wordcount.jar = 封裝後的編譯檔, 但執行時需告知 class name Hadoop 進行運算時, 只有 input 檔要放到 hdfs 上, 以便 hadoop 分析運算 ; 執行檔 ( wordcount.jar ) 不需上傳, 也不需每個 node 都放, 程式的載入交由 java 處理 13

14 範例一動手做 WordCount1 練習 (III) 14

15 範例一動手做 WordCount1 練習 (IV) 15

16 範例二動手做 WordCount 進階版 WordCount2 功能不計標點符號不管大小寫步驟 ( 接續 WordCount 的環境 ) 1. echo "\." >pattern.txt && echo "\," >>pattern.txt 2. bin/hadoop dfs -put pattern.txt./ 3. mkdir MyJava2 4. javac -classpath hadoop-*-core.jar -d MyJava2 WordCount2.java 5. jar -cvf wordcount2.jar -C MyJava2. 16

17 範例二動手做不計標點符號執行 bin/hadoop jar wordcount2.jar WordCount2 input output2 -skip pattern.txt dfs -cat output2/part

18 範例二動手做不管大小寫執行 bin/hadoop jar wordcount2.jar WordCount2 -Dwordcount.case.sensitive=false input output3 -skip pattern.txt 18

19 範例二補充說明 Tool 處理 Hadoop 命令執行的選項 -conf <configuration file> -D <property=value> -fs <local namenode:port> -jt <local jobtracker:port> 透過介面交由程式處理 ToolRunner.run(Tool, String[]) 19

20 範例二補充說明 DistributedCache 設定特定有應用到相關的超大檔案或只用來參考卻不加入到分析目錄的檔案如 pattern.txt 檔 DistributedCache.addCacheFile(URI,conf) URI = hdfs://host:port/filepath 20

雲端 Cloud Computing 技術指南運算應用平台與架構 10/04/15 11:55:46 INFO 10/04/15 11:55:53 INFO 10/04/15 11:55:56 INFO 10/04/15 11:56:05 INFO 10/04/15 11:56:07 INFO

雲端 Cloud Computing 技術指南運算應用平台與架構 10/04/15 11:55:46 INFO 10/04/15 11:55:53 INFO 10/04/15 11:55:56 INFO 10/04/15 11:56:05 INFO 10/04/15 11:56:07 INFO CHAPTER 使用 Hadoop 打造自己的雲 8 8.3 測試 Hadoop 雲端系統 4 Nodes Hadoop Map Reduce Hadoop WordCount 4 Nodes Hadoop Map/Reduce $HADOOP_HOME /home/ hadoop/hadoop-0.20.2 wordcount echo $ mkdir wordcount $ cd wordcount