Presentation title goes here

Size: px

Start display at page:

Download "Presentation title goes here"

俦宣
5 years ago
Views:

1 ACP- 如何在微软 Azure HDInsight 优化 Hadoop 董乃文 Nevin Dong 资深技术顾问开发工具及平台事业部 (DX) 微软公司朱晓勇 Xiaoyong Zhu 产品经理云计算与企业事业部 (C&E) 微软公司

2 Hadoop, HDInsight 及关键能力 HDInsight 性能及调优典型应用场景

3 HDInsight 概述及关键能力

5 Hadoop as a Service, 由微软运营 100% 支持 Apache Hadoop 全面兼容.Net 及 Java 工具基于最新 Hadoop 版本 (v2.6) 无需硬件部署, 立等可用支持 Windows 及 Linux 操作系统按需使用, 保证数据安全微软专业团队技术支持

7 HDFS 分布式文件系统节点数按需伸缩文件可跨节点存储 HDFS 存储海量数据支持超大规模文件的存储 ( 跨节点 ) HDFS 支持非结构化数据的存储

8 数据处理与存储结合分布式计算, 实现数据与计算的本地化紧密结合随节点数增长而线性增长执行关键步骤 Map: 开发者编写 map functions Shuffle / Distributes: 框架自动将数据分发到节点 (networking, synchronization, recovery, scheduling) Reduce: 开发者编写 reduce functions, 只返回计算结果

9 Hadoop 2.0 Microsoft contribution to Apache code

10 HDInsight 上的 NoSQL 数据库运行在 Hadoop Distributed File System (HDFS) 上方便灵活, 可在任何时间增添新的数据列 Coordination HMaster Name Node Region Server Region Server Region Server Region Server Job Tracker Data Node Data Node Data Node Data Node Task Tracker Task Tracker Task Tracker Task Tracker

11 实时的流处理分析百万级实时事件的处理, 支持 event broker (i.e.: Apache Kafka, Azure Event Hub) 处理结果输出到持久化存储仪表盘 (dashboards) 或设备

12 机器学习 (Machine learning) 库可在 HDFS 上执行的机器学习算法算法不受数据规模限制, 具备良好的跨节点伸缩性库主要包括 : Collaborative Filtering, Classification, Clustering, Dimensionality Reduction, Topic Models

13 Cloud

14 HDInsight 性能及调优

16 Hive ODBC JDBC Thrift server 查询语句 Console Visual Studio Command line interface (CLI) Metastore Compiler, Optimizer, Executor Hadoop Head node Name node Data nodes/task nodes

17 Demo 提升 Hive 的性能

$FIELDS TERMINATED BY '\t' STORED AS$

21 CREATE EXTERNAL TABLE ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TEXTFILE

22 100 Latency (minutes) TPCH1 Latency 优化措施 (min) 16 node A3 89 Speedup node A3

26 100 Latency (minutes) 何种优化措施 TPCH1 Latency (min) Speedup 16 node A3 89 Increase Scale (32 node D13) 39-56% node A3 32 node D13

29 MapReduce Tez Group t1 by a Group t2 by a Group t1 by a Group t2 by a Mapper Mapper Mapper Mapper Mapper Mapper Mapper Mapper Mapper Mapper Reducer HDFS Reducer Reducer HDFS Reducer Join t1 t2 Mapper Reducer HDFS Mapper Join t1 t2 Reducer Reducer Order by Mapper Order by Reducer Reducer

30 set hive.execution.engine=tez;

31 优化 TPCH1 Latency (min) Speedup 16 node A3 89 Increase Scale 39-56% Enable Tez 27-70% Latency (minutes)

36 PARTITION (L_SHIPDATE = 5/23/ :00:00 AM ) 12:00:00 AM )) ADD PARTITION (L_SHIPDATE = 5/23/1996

37 SET hive.exec.dynamic.partition = true; SET hive.exec.dynamic.partition.mode = nonstrict; PARTITION (L_SHIPDATE)

39 优化 Latency (min) Speedup 16 node A3 89 Increased Scale (32 node D13) 39-56% with Tez 27-70% With Tez and partitions 24-73% Latency (minutes)

43 INSERT INTO TABLE lineitem_orc

46 优化 16 node A3 Increased Scale (32 node D13) with Tez With Tez and partitions With ORC Latency (min) Speedup 89-56% 39-70% 27-73% 24-94% Latency (minutes)

48 set hive.vectorized.execution.enabled = true;

49 优化 Latency (min) Speedup 16 node A3 89 Increased Scale 39-56% with Tez 27-70% With Tez and partitions 24-73% With ORC 5-94% With vectorizations 4-96% Latency (minutes)

51 Latency (minutes)

53 HDInsight 典型应用及性能

54 使用 Hadoop 来完成 ETL 预处理使用 Hadoop 群集 (clusters) 来降低数据仓库的成本

55 将超大规模的历史数据下架并存入冷存储设施中使用数据仓库存储热数据, 为商业智能 BI 及分析提供支撑可将冷存储中数据转入数据仓库以备使用热数据准备冷数据

56 使用数据仓库进行商业智能 BI 及分析帮助数据科学家 (data scientists) 针对原始数据 ( 无格式 / 结构 ) 进行新的挖掘将挖掘结果整理并存放到数据仓库中准备

57 Applications RabbitMQ / ActiveMQ Web/thick client dashboards Devices Cloud gateways (web APIs) Stream processin g Search and 查询语句 Data analytics (Excel) Sensors Web and Social Field gateways Storage adapters Devices to take action

58 HDFS/WebHDFS API Azure Data Lake managed clusters Hadoop Cluster Azure Data Lake store

59 课后提醒

60 Microsoft-Ignite-China

Reducing Client Incidents through Big Data Predictive Analytics

Reducing Client Incidents through Big Data Predictive Analytics IT@lntel 白皮书英特尔 IT 部门 IT 最佳实践大数据预测分析 2013 年 12 月通过大数据预测分析减少客户端事故总体概述相比过去的被动反应, 我们现在能够更容易主动找出客户端问题, 并及时将其修复以免问题扩大, 从而为企业节约