Fas Saisical Relaionship Discover in Massive Monioring Daa Hui Zhang Haifeng Chen Guofei Jiang Xiaoqiao Meng Kenji Yoshihira EC Laboraories America Princeon, J EC Laboraories America, Inc. 1
Technical challenges in service nework managemen Difficul o characerize/model ssems Heerogenei: Mixure of various sofware, hardware, neworking ec. Dnamici: User behavior, frequen ssem upgrade and changes, uncerainies such as caching ec.. Scale: Large number of sofware &hardware componens. Widel disribued. Segmened adminisraions. Difficul o characerize/model fauls Diversi: operaor fauls, sofware fauls, hardware fauls and nework fauls ec. A faul ofen manifess iself differenl. Difficul o generalize common knowledge across differen ssems EC Laboraories America, Inc. 2
Invarians of dnamic ssems User raffic m i+2... m i+1 Targe Ssem m n m i... an consan relaionship???... m 1 m 2 m 4 m 3 m n Flow inensi: he inensi wih which inernal monioring daa reacs o he volume of user raffic. User raffic flow hrough ssem endlessl and man inernal monioring daa reac o he volume of user raffic accordingl. We search he relaionships among hese inernal measuremens colleced a various poins. If modeled relaionships coninue o hold all he ime, he can be regarded as invarians of he ssem. EC Laboraories America, Inc. 3
Modeling local properies Difficul o characerize a large, dnamic & complex ssem in a holisic wa!! Raher han modeling he whole ssem, we model man local relaionships among monioring daa Divide & Conquer. 1. Each invarian is able o capure some local properies of is relaed componens. 2. Discovering large number of invarians enable use o characerize he whole ssem from differen perspecives. 3. Inerpre ssem operaional saus b racking an changes of invarians. EC Laboraories America, Inc. 4
EC Laboraories America, Inc. 5 One example in model librar We use an AuoRegressive model wih exogenous ARX o learn he relaionship beween wo flow inensi measuremens. Define Given a sequence of real observaions Using LSM, we learn he model b minimizing....... 1 0 1 m k u b k u b n a a m n + + + + + T n b m b b a a a ],...,,,,...,, [ 1 0 2 1 θ T m k u k u n ],...,, 1,..., [ ϕ ϕ T θ }, 1,..., 1, { n n u u O T O E 1 2 1 2 1 ˆ 1, θ ϕ θ θ θ, O E θ T 1 1 1. ] [ ˆ ϕ ϕ ϕ θ
Finess score of a model A finess funcion can be used o evaluae how well he learned model fis he real daa. F θ [1 1 1 θ ] 100 Man oher relaionships can be modeled in he similar wa. ARX model is onl an example of models ha can be used o describe he relaionship beween flow inensi measuremens. Muliple inpu and oupu relaionship can be described b ARX Oher linear and nonlinear models o describe a dnamic relaionship beween inpus and oupus. ˆ The goal is o find a model ha capure he dnamic relaionship well, i.e. he model ha bes fi he inpu-oupu observaion daa under all scenarios. 2 2 EC Laboraories America, Inc. 6
Confidence score Compue he finess score o evaluae how well he model fis he daa of each ime window. Define a funcion and a hreshold o deermine wheher he model fis he daa or nor. ~ 1, if Fi θ > F. f Fi θ ~ 0, if Fi θ F. Compue he confidence score p k θ prob F θ ~ F 1 > f F θ Sop esing hose models wih afer a period of ime. The score p k θ represens how robus a relaionship is. I can be inegraed o he deecion and diagnosis process well. k k p k θ P i EC Laboraories America, Inc. 7
Faul deecion and localizaion Approach 1. Collec he monioring daa from he arge ssem. 2. Model he normal behavior of he underling ssem. 3. Use he learned models o deec anomalies. 4. Correlae he anomalies o locae he faul componens. 2. Build models Model librar 1. Collec raw daa x x Componen * f x * * R - * A B.. X X A B.. X 4. Correlae Targe Ssem Updaing models Compare raw daa 3. Use models Failure! Deecor failure or no? Localizer Faul Componens Anomalous Behavior Residual Correlaion Curren Pahs Curren Comp. Ineracions faul EC Laboraories America, Inc. 8
A graph model of he relaionship discover process Discover he se E in a graph GV,E When V increases, he compuaion overhead increases quickl! -1 ess. Soluion 1: providing enough compuing resource. Soluion 2: appling domain knowledge o cluser he measuremen daa ses and discover relaionships onl wihin individual clusers. Soluion 3: guided discover wih fas indexing EC Laboraories America, Inc. 9
The Guided Relaionship Discover 1. Indexing phase: appl one fas indexing algorihm o generae an approximae verex rank based on he degree. decide he compuaion budge for he rank esimaion process which is eiher specified direcl b he adminisraor or calculaed based on he esimaion accurac requiremens. 2. Discover phase: repea picking one verex in he order of he rank, esing is relaionship wih he remaining verices, and removing i from he graph unil eiher he overall compuaion budge is run ou or he graph runs ou of verices. a greed heurisic o he verex cover problem, which is Pcomplee. EC Laboraories America, Inc. 10
Rank esimaion algorihm 1: Uniform Sampling 1. Keep one couner for each verex. Iniiall all se o 0. 2. Randoml pick wo differen verices x and from he graph wih uniform probabili, appl a relaionship esing. If he pair x, has been picked before, he es is skipped and he cached resul will be used for he nex sep. 3. If he es resul is posiive, increase he couners of x and b 1. 4. Go back o 2 unil k imes. k: compuaion budge. 5. Oupu a rank on all verices based on he couner values; a ie is broken wih a random choice. EC Laboraories America, Inc. 11
Correc Ranking Probabili CRP Measuring he ranking accurac beween verex x and verex in he esimaion. For he US algorihm, we have EC Laboraories America, Inc. 12
Rank esimaion algorihm 2: Adapive Sampling 1. Keep one couner for each verex. Iniiall all se o 1. 2. Randoml pick wo differen verices x and from he graph wih probabili proporional o heir couner values, appl a relaionship esing. If he pair x, has been picked before, he es is skipped and he cached resul will be used for he nex sep. 3. If he es resul is posiive, increase he couners of x 4. and b 1. if he couner value of x is larger han a hreshold e.g., half of he verex se size, we remove x from he verex se in he following sampling process. 5. Go back o 2 unil k imes. k: compuaion budge. 6. Oupu a rank on all verices based on he couner values; a ie is broken wih a random choice. EC Laboraories America, Inc. 13
Evaluaion Daa ses a collecion of monioring daa from an operaional UTRA UMTS Terresrial Radio Access ework ssem. 129 Ke Performance Indicaors KPI daa ses for each monioring period. The ARX linear regression model was used o es he correlaion relaionships beween hose KPI daa ses. B choosing differen correlaion significance hresholds, differen relaionship graphs were generaed on he same daa se collecion. EC Laboraories America, Inc. 14
Evaluaion The guided relaionship discover on he generaed graphs was ran as following: The Indexing Phase had he compuaion budge of kn ess k 4, 8 and n 129 Four rank esimaion schemes were compared Uniform sampling, adapive sampling, opimal ranking, and random ranking. The Discover Phase followed he oupu rank and was sopped when he graph ran ou of verices. EC Laboraories America, Inc. 15
A dense relaionship graph esed avg degree 32.6 EC Laboraories America, Inc. 16
Uniform Sampling algorihm, UTRA dense graph. wih 8n esimaion ess, he guided discover wih US algorihm covered 80% 90% of he edges wihin 64 85; he opimal discover required 57 81 verices o cover 80% 90% edges; he random discover required 110 119 verices o cover 80% 90% edges. EC Laboraories America, Inc. 17
A sparse relaionship graph esed avg degree 2.3 EC Laboraories America, Inc. 18
Adapive sampling algorihm, UTRA sparse graph. wih 8n esimaion ess, he guided discover wih AS algorihm covered 80% 89% of he edges wihin 24 30; he opimal discover required 22 28 verices o cover 80% 90% edges; he random discover required 92 106 verices o cover 80% 90% edges. EC Laboraories America, Inc. 19
Conclusions & fuure work We sudied he problem of fas saisics relaionship discover in massive monioring daa. a guided discover scheme wih wo simple sampling based rank esimaion algorihms was proposed o enable fas and parial relaionship discover. Fuure work he proper of he adapive sampling algorihm in erms of he correc ranking probabili. analsis and experimens on more real-world daa and snheic opologies. he opimal allocaion beween esimaion ime and discover ime in he guided discover process. he guided discover wih hbrid indexing and discover process. EC Laboraories America, Inc. 20