进击的巨人：基于Angel的高维度Online Learning_V3

Size: px

Start display at page:

Download "进击的巨人：基于Angel的高维度Online Learning_V3"

贵纪
4 years ago
Views:

1 进击的巨人基于 Angel 和 Spark Streaming 的高维度 Online Learning Andymhuang( 黄明 ) 腾讯数据平台部

4 源起

5 模型网络并行局部性分布式机器学习同步系统调度评估优化容错

7 MapReduce MPI Parameter Server

8 d d d d Small Model Big Model n Big Data Sparse Big Data d<<n d n 寻找满足十亿级维度的工业级的分布式机器学习平台

9 Executor Executor Executor Executor Driver Model Executor Driver Model Executor Executor Executor Executor

10 DistBelief

11 2015 V 工业级别可用的参数服务器丰富的机器学习及数学计算库友好的用户编程接口

13 Parameter Server Shard Shard Shard PSServer PSServer PSServer PSAgent PSAgent PSAgent Model M pull syncprotocol Model Partitioner psfunc push ΔM PsClient PsClient Worker T a s k DataBlock T a s k DataBlock

14 Mapper Reducer RDD PSModel

15 epoch-1 epoch-2 epoch-3 epoch-4 epoch-5 RDD-1 RDD- 2 RDD- 3 RDD- 4 RDD- 5 epoch PSModel

16 RDD Partition-1 RDD RDD Dependencies Partition-2 Partition-3 Partition-4 Partitioners Compute Func (Transformation or Action ) Preferred locations Partition-n Node Memory Memory Block -n Node Disk Disk Block -n

17 PSPartitioner Partition1 Partition2 Partition3 PSModel M pull PSClient push Clock Sync MatrixContext Shard PSServer Partition- ΔM

18 Model Partitioner PartitionKey PSModel pkey-1 pkey-2 pkey-3 将模型切分成多个分区, 存储在不同 PS 节点上 PKey1 Shard Shard PKey2 PKey3 Shard PSServer PSServer PSServer 优点保证 Server 负载均衡避免 Server 单点性能瓶颈支撑 10 亿 ~100 亿级别维度参数 Angel 的模型分区默认将模型分成大小相等的块可以指定分区块大小支持横切和纵切自定义矩阵分区, 量身定制区块分布方式

19 BSP SSP ASP

20 PSServer1 PSServer2 Vector clock Vector clock Worker1 Vector clock Worker2 Vector clock Worker3 Vector clock

21 Shard Shard Shard PSServer PSServer PSServer UpdateFunc GetFunc MUpdate Func Push Aggr Func Pull add sub mul pow log div sum max min

22 核心流程和类 MLRunner 3.runTask Angel Client 1. Start PS 2. Load Model TrainTask 4.parse & preprocess DataBlock LabledData HDFS LabledData LabledData 5.train MLLearner 6.learn MLModel PSModel PSModel 7.push & pull Model PSServer 8.Save Model HDFS PSModel

26 PS PS PS PS Worker Worker Worker HDFS HDFS HDFS Push update value to PS Pull parameters from PS

29 Angel PS PS PS mutable Spark Worker Worker Worker Worker Worker immutable

30 Parameter Server Shard Shard Shard PSServer PSServer PSServer PSAgent PSAgent PSAgent Angel Context Spark Driver PSMatrix/PSVector PSMatrix/PSVector SPARK RDD T A S K T A S K T A S K T A S K T A S K T A S K Executor Executor

32 混入相同特征进行透明替换 <<trait>> NumericOps[T] Executor Angle PS def round(t: T):T def dot(t: T):T def max(t: T):T Task BreezePSVector <<class>> BreezeVector def round(t: T):T def dot(t: T):T def max(t: T):T <<class>> BreezePSVector def round(t: T):T def dot(t: T):T def max(t: T):T BreezePSVector BreezePSVector PSClient PS Agent

33 Spark Spark on Angel

$PSServer RDD3 BreezePSVector CachedPSVector { RDD_PS_Functions } Spark Spark on Angel$

34 SparkPSContext RDD MLModel AngelClient RDD1 PSModel RDD2 PSVector PSMartrix PSClient Shard PSServer RDD3 BreezePSVector CachedPSVector { RDD_PS_Functions } Spark Spark on Angel Angel

35 A Better Way of Online Learning

36 Online Learning 36

37 t =1, 2,...,T x t Î X pt = ft ( xt ) ÎY pt y t ÎY V p t, y ) ( t R ( H ) = T T å t= 1 V ( p, y ) - min t t f ÎH T å t= 1 V ( f ( x ), y ) t t

39 Angel FTRL.streamingTrain (Z, N) Z N w Executor t-1 t t+1 t+2 t+3 Receiver RDD RDD RDD RDD RDD featuredstream mappartitionswithindex <SparseLRWithFTRL.scala>

40 Angel Z N w pull Z and N Push ΔZ Push ΔN ΔZ ΔZ ΔZ ΔN ΔN 1 11 ΔZ 2 ΔN 2 ΔZ n ΔN n Dz = -s i g i iw t, i i 2 i Dn = g RDD σ t g t w t... Feature t Partition 0 Partition 1 Partition n Spark Streaming

41 FTRL 的整体代码框架 val zps: SparsePSVector = PSVector.sparse(dim) val nps: SparsePSVector = PSVector.sparse(dim) SparseLRWithFTRL.train(zPS, nps, featureds) ssc.start() ssc.awaittermination()

43 性能比对生产数据, 现网环境, 尽量公平

46 Angel vs Spark LDA

47 Angel vs Spark LR

49 开源和展望 OpenSource & Perspective

50 [GBDT] The purposes of using parameter server in GBDT #7 (PR 98)

51 㷖助倝国际顶级会议Paper CCF A类 LDA*: A Robust and Large-scale Topic Modeling System VLDB, 2017 Heterogeneity-aware Distributed Parameter Servers. SIGMOD, 2017 Angel: a new large-scale machine learning system. National Science Review (NSR), 2017 TencentBoost: A Gradient Boosting Tree System with Parameter Server. ICDE, 2017

52 Distributed Serving V1.6 Deep Learning Framework Support V2.0

方圆并济：基于 Spark on Angel 的高性能机器学习_V3

方圆并济：基于 Spark on Angel 的高性能机器学习_V3 方圆并济 : 基于 Spark on Angel 的高性能分布式机器学习源起腾讯的产品需求 d d d d Small Model Big Model Sparse Big Data n Big Data d