方圆并济：基于 Spark on Angel 的高性能机器学习_V3

Size: px

Start display at page:

Download "方圆并济：基于 Spark on Angel 的高性能机器学习_V3"

庄褚
4 years ago
Views:

1 方圆并济 : 基于 Spark on Angel 的高性能分布式机器学习

3 源起

4 腾讯的产品需求 d d d d Small Model Big Model Sparse Big Data n Big Data d<<n d n 寻找满足十亿级维度的工业级的分布式机器学习平台

5 Spark 机器学习的瓶颈 Executor Executor Executor Executor Driver Model Executor Driver Model Executor Executor Executor Executor

6 One Issue A Prototype of Parameter Server

7 Glint & Yahoo 2016

8 理念方圆并济 Angel PS PS PS mutable Spark Worker Worker Worker Worker Worker immutable

9 Spark on Angel

10 核心抽象 Mapper Reducer RDD PSModel

11 RDD vs PSModel epoch-1 epoch-2 epoch-3 epoch-4 epoch-5 RDD-1 RDD-2 RDD-3 RDD-4 RDD-5 epoch PSModel

12 RDD 的核心抽象 RDD Partition-1 RDD Partition-2 RDD Dependencies Partition-3 Partition-4 Partitioners Partition-n Compute Func (Transformation or Action ) Preferred locations Node Memory Memory Block -n Node Disk Disk Block -n

13 PSModel 的核心抽象 PSModel M pull Clock MatrixContext PSPartitioner Partition1 PSClient Shard Partition2 PSServer Partition3 push Sync Partition- ΔM

14 Spark on Angel 的架构 Parameter Server Shard Shard Shard PSServer PSServer PSServer PSAgent PSAgent PSAgent Angel Context Spark Driver PSModel PSModel SPARK RDD T A S K T A S K T A S K T A S K T A S K T A S K Executor Executor

15 Parameter Server Shard Shard Shard PSServer PSServer PSServer PSAgent PSAgent PSAgent Model M pull syncprotocol Model Partitioner psfunc push ΔM PsClient PsClient Worker T a s k DataBlock T a s k DataBlock

16 工业级别可用的参数服务器丰富的机器学习及数学计算库友好的用户编程接口

17 Angel 和 Glint 的比较 PSPartitioner Partition1 Partition2 Partition3 Partition- 更丰富的模型切分更灵活的异步模式更强大的 psfunc

18 Angel 的定位

19 Spark on Angel 的开发

20 Angel 的 API 设计 MLRunner 3.runTask Angel Client 1. Start PS 2. Load Model 4.parse & preprocess HDFS TrainTask DataBlock LabledData 5.train LabledData LabledData 6.learn MLLearner MLModel PSModel PSModel 7.push & pull Model PSServer 8.Save Model HDFS PSModel

$PSClient Shard PSServer RDD3 BreezePSVector CachedPSVector { RDD_PS_Functions } Spark Spark on Angel$

21 Spark on Angel 的 API 设计 SparkPSContext RDD RDD1 MLModel PSModel AngelClient RDD2 PSVector PSMartrix PSClient Shard PSServer RDD3 BreezePSVector CachedPSVector { RDD_PS_Functions } Spark Spark on Angel Angel

22 Spark on Angel 的基础写法

23 Vector 的透明替换混入相同特征进行透明替换 <<class>> BreezeVector def round(t: T):T def dot(t: T):T def max(t: T):T <<trait>> NumericOps[T] def round(t: T):T def dot(t: T):T def max(t: T):T <<class>> BreezePSVector def round(t: T):T def dot(t: T):T def max(t: T):T Executor Task BreezePSVector BreezePSVector BreezePSVector PSClient PS Agent Angle PS

25 Angel 的算法 Sparkon Angel Available

26 LR on Angel PS PS PS PS Worker Worker Worker HDFS HDFS HDFS Pull parameters from PS Push update value to PS

27 [Spark on Angel] LR {Breeze Ops} Angel wps gradient PS DenseVector Array Spark samplerdd mappartitions [spark_on_angel_quick_start.md]

28 优化方法

29 [Spark on Angel] LR with Optimizer Angel wps stateps DenseVector DiffFunction(BreezePSVector) : (Double, BreezePSVector) Breeze.optimizer Spark samplerdd SGD OWLQN LBFGS mappartitions [spark_on_angel_optimizer.md]

30 GBDT: 树模型 +Boosting tree 1 tree 2 Y Age<30 N Y Is Male? N Wage<10K Y N A B C D E predict( ) 5+0.5=5.5 predict( ) =11.5 predict( ) 1+1.5=2.5 predict( ) 1+0.5=1.5 predict( ) 1+1.5=2.5

31 GBDT on Angel: 模型存储 PS1 PS2 PS3 feature ID feature ID feature ID feature value feature value feature value leaf prediction leaf prediction leaf prediction grad histogram hess histogram

32 GBDT on Angel(1): 构建森林 PS1 PS2 PS3 Worker1 Worker2 Worker3

33 GBDT on Angel(2): 分裂树节点 find split feature & value [gbdt_on_angel.md]

34 [Spark on Angel] GBDT Angel Instance Layout PS Grad Histogram PS Split Feature PS Split Value PS Leaf Weight PS Spark zip zip Instance RDD Gradient RDD Prediction RDD map [spark_on_angel_gbdt.md]

36 (Spark on Angel)vs Spark LR

37 Angel vs XGBoost GBDT

38 Angel vs Spark LDA

39 Angel vs Spark GD-LR

40 Angel vs Spark ADMM-LR

41 Spark on Angel 的特点

42 OpenSource & Perspective

43 Angel 开源 [GBDT] The purposes of using parameter server in GBDT #7 (PR 60)

44 学术创新国际顶级会议Paper CCF A类 -%" " 3PCVTU BOE -BSHF TDBMF 5PQJD.PEFMJOH 4ZTUFN 7-%# )FUFSPHFOFJUZ BXBSF %JTUSJCVUFE 1BSBNFUFS 4FSWFST 4*(.0% "OHFM B OFX MBSHF TDBMF NBDIJOF MFBSOJOH TZTUFN /BUJPOBM 4DJFODF 3FWJFX /43 5FODFOU#PPTU " (SBEJFOU #PPTUJOH 5SFF 4ZTUFN XJUI 1BSBNFUFS 4FSWFS *$%&

45 版本展望 What is Next 1ZUIPO "1* V1.3 4QBSL 4USFBNJOH PO "OHFM V1.5 %FFQ -FBSOJOH 'SBNFXPSL 4VQQPSU V2.0

46 Q & A 微博 :@ 明风喜欢记得给个 Star 噢 andymhuang@tenent.com 机器学习系统 & 算法工程师 We are Hiring

进击的巨人：基于Angel的高维度Online Learning_V3

进击的巨人：基于Angel的高维度Online Learning_V3 进击的巨人基于 Angel 和 Spark Streaming 的高维度 Online Learning Andymhuang( 黄明 ) 腾讯数据平台部 www.top100summit.com https://github.com/tencent/angel 源起模型网络并行局部性分布式机器学习同步系统调度评估优化容错 MapReduce MPI Parameter