(Big Data: Challenges and opportunities) 2014/11/04 Copyright 2013 ITRI In God we trust; all other show data! 1

Size: px

Start display at page:

财雕前伏
4 years ago
Views:

1 (Big Data: Challenges and opportunities) 2014/11/04 1

2 vs. 2

3 (Data Driven Decision Making) 3

4 vs. vs. 4

5 Walmart Walmart Why 40% Walmart 5 5

6 7-ELEVEN 7-ELEVEN 30% 6

7 Source : Solutions Big Data IBM,

8 Transaction Data Government Data Traffic Real Estate Scientific Data Genome Climate User Generated Content (UGC) Text Multimedia (Video, Image, Audio) Sensor Data GPS Gyroscope Accelerometer Thermometer Log Data 8

9 50 Blue-ray Disc : 25G, 1.2mm ~50,000 Taipei 101 ~800,000 Taipei 101 Taipei 101 : 508m = 423,333 BDs Exabyte=

( Structured data) Data that reside in fixed fields. ( ) E.g., relational databases or data in spreadsheets. (Unstructured data) Data that do not reside in fixed fields. E.g., free-form text (e.g., books, articles, body of e-mail messages), untagged audio, image and video data.

10 ( Structured data) Data that reside in fixed fields. ( ) E.g., relational databases or data in spreadsheets. (Unstructured data) Data that do not reside in fixed fields. E.g., free-form text (e.g., books, articles, body of messages), untagged audio, image and video data. Aseervatham, S. & Bennani, Y. Semi-structured document categorization with a semantic kernel Pattern Recogn., Elsevier Science Inc., 2009, 42,

11 Big Data: vs. 11

12 Big Data Analysis DoIT Forum- : ( ) IC

13 13

1997 Netflix: Los Gatos, California, America 1,500 20077.

14 1997 Netflix: Los Gatos, California, America 1, Netflix 2,917 HBO 85,000 DVD 5,000 3,000 () Long-tail (vs. BLOCKBUSTER) 14

15 Netflix(1/2) (Accuracy and diversity) 1/3 ( in 2012) 15

16 Netflix(2/2) Awareness. Explain the recommendation, i.e., from your explicit taste preferences and ratings, your viewing history, or even your friends recommendations 16

Netflix Amazon + Open Source + Amazon S3, AWS(EMR) Python/Java, Apache Hive/Pig Open Source Genie PaaS Online( ) Offline() Nearline() : Linear Regression Logistic Regression Elastic Nets Singular

17 Netflix Amazon + Open Source + Amazon S3, AWS(EMR) Python/Java, Apache Hive/Pig Open Source Genie PaaS Online( ) Offline() Nearline() : Linear Regression Logistic Regression Elastic Nets Singular Value Decomposition Restricted Boltzmann Machine (2009) Markov Chains Latent Dirichlet Allocation Association Rules Gradient Boosted Decision Trees Radom Forests Clustering Techniques Matrix Factorization (2009 ) 17

18 Netflix House of Cards NetflixBig Data Netflix1990 House of Cards (David Fincher) (Kevin Spacey) Netflix IMDB 104, (Stop, Fast Forward, Rewind ) 18

19 2012 Project Narwhal ( ): Web + Social + Big Data Call list? 19

20 » 35» 23» Obama % 20

21 PB 20 TB ( ) (//100 8) ( ) Hadoop» MyFOX ( MySQL)» Prom ( HDFS)» glider ()» Garuda ( )» OceanBase ( Open Source )» : 2010» : 2011 API 21

22 - : ++ Personalized, click/purchase behavior, real-time recommendation : ItemCF + Content-based ( Matrix Factorization ) Relevance Recommendation () Complementary Recommendation () Buy-after-View Recommendation ( ) 22

23 / Korea Telecom 2014/11/

24 + vs vs., AB, C, or D? ( ) Obama s Data Driven Champagne 24

25 25

27 : (5 99.9%) (8.7/ ) / : Source: ITIS, ( ) ( ) 27

28 Source: BOST,

29 Open Data () 29

PAKDD 2014 ASUS Malfunctioned Components Prediction

established and leading international conferences in

30 PAKDD 2014 ASUS Malfunctioned Components Prediction 614 : 1. (Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD), is one of the longest established and leading international conferences in the areas of data mining and knowledge discover (Hybrid Intelligent Prediction Technology; HIPT) 30

31 Portal Category Item 31

32 32

33 2011 (Border Risk Identification System, BRIS) ( ) (50 60 ) + + visa + 64% 33

34 Automated Targeting System (ATS) / Analytical Framework for Intelligence (AFI) ATS FALCON Data Analysis and Research for Trade Transparency System (FALCON-DARTTS) 34

35 35

36 LED 36

37 -MOCVD US$ 2M~4M (recipe) run 8 (tool data) 6 MOCVD (Metal Organic Chemical Vapor Deposition) SourceE 37

38 Susceptor Before Epitaxy Process R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 38

39 PL Mapping LED450nm Std. Dev. uniformity Run1-1 Run1-2 Run1-3 Run2-1 Run2-2 Run2-3 39

MOCVD Data Recipe 1,160 10K measurement data

= 1160; variable Bake_Time = 3000; variabl e

Measurement data: equipment logs (structured

Temperature 800 "Heating up Reactor to bake

40 MOCVD Data Recipe 1,160 10K measurement data Wafer # Hydrogen, brushed ShowerHead variable Tcal_H2_BSH_Factor = 1.155; variable Tcal_H2_BSH_Offset = 20; Data type Recipe (excel like, script) # High-Temperature H2 Bake-out variable Bake_Temp = 1160; variable Bake_Time = 3000; variabl e Bake_Press = 150; MOCVD logs (structured data) Measurement data: equipment logs (structured data) and documents (ugc) #07 Reactor to Bake Temperature 800 "Heating up Reactor to bake temperature", Reactor.press to Bake_Press, Heater.temp to (Bake_Temp*Tcal_H2_BSH_Factor+Tcal_H2_BSH_Offset), Temperature 200 x 400 steps-parameters 800 s 6M records ( 29 MB) Source: EOL, CITC Time 40

41 ?? (?)»»,» E.g., vs.»» or E.g.,, Make or buy?! 41

42 Three Kind of Lies 42

43 (correlation) vs. vs. () vs. 43

Big data analytics E.g., Customer behavior clustering by scaling up clustering algorithms with MapReduce Unstructured data analytics E.g., Feature level sentiment analysis, local event detection by semantic analysis and/or Web mining Uncertain data analytics E.

44 Big data analytics E.g., Customer behavior clustering by scaling up clustering algorithms with MapReduce Unstructured data analytics E.g., Feature level sentiment analysis, local event detection by semantic analysis and/or Web mining Uncertain data analytics E.g., Predictive error tolerance on weather forecasts by uncertainty handling decision process Fused data analytics E.g., Integrated analysis of structured transaction data and unstructured online comments for sales prediction Logical knowledge inferences Merchandise ontology (such as GoodRelations) 44

45 45

46 (Opinion Mining) ()»» /» Android Ios Win phone firefox 46

47 statistics and computer science (particularly machine learning) Machine learning» Unsupervised learning, Supervised learning, Association rule learning, Sparse coding, support vector machine Natural language processing, Signal processing Pattern recognition» Classification, Cluster analysis, Genetic algorithms, Neural networks Statistics» Regression, Spatial analysis, Time series analysis» A/B testing, Optimization Database and data warehouse Data fusion and data integration, Data mining Open Source! 47

48 Big Data Venn Diagram 48

49 (Data Scientist) Computer science and mathematic background is preferred Natural Language Processing: the interactions between computers and humans; Machine learning: using computers to improve as well as develop algorithms; Conceptual modelling: to be able to share and articulate modelling; Statistical analysis: to understand and work around possible limitations in models; Predictive modelling: most of the big data problems are towards being able to predict future outcomes; Hypothesis testing: being able to develop hypothesis and test them with careful experiments. Personality and Soft skill be able to communicate their findings, orally and visually. be very curious people, who enjoy diving deep into the material to find an answer to a yet unknown question. They need to be thinkers who can ask the right (business) questions. A big data scientist understands how to integrate multiple systems and data sets. They need to be able to link and mash up distinctive data sets to discover new insights. Programming skill be able to program, preferably in different programming languages such as Python, R, Java, Ruby, Clojure, Matlab, Pig or SQL. They need to have an understanding of Hadoop, Hive and/or MapReduce. Understand Business Strategy The exact background of a big data scientist is of less importance. Great big data scientists can have different backgrounds such as econometrics, physics, biostatistics, computer science, applied mathematics or engineering. Most of the time the background is a Master s Degree or even PhD. 49

51 IoT, Health care E.g., vs. or E.g.,, (Data Scientist) (e.g. Statistics, Machine learning ( clustering, classification, association rule, neural networks, ), Hadoop )domain Team work! 51

52 52

Microsoft PowerPoint - Big Data rc Sharing掃盲時間.ppt [相容模式]

Microsoft PowerPoint - Big Data rc Sharing掃盲時間.ppt [相容模式] Big Data RC Sharing 大數據掃盲 Service Planner of Enterprise Big Data 大數據服務規劃師企業大數據課程規劃依照企業資料流程的特殊性, 安排合適的課程協助企業導入應用大數據案例 :Etu 資策會平安保險湖南國防科技大學等