ApacheEagle-陈浩-GOPS2016深圳大会_v1.0

Size: px

Start display at page:

Download "ApacheEagle-陈浩-GOPS2016深圳大会_v1.0"

轼昝
5 years ago
Views:

2 Apache Eagle: 分布式实时监控预警框架陈浩 ebay Inc.

3 关于我 Tech Lead, Sr. Software ebay Cloud Platform hchen9@ebay.com Co- creator, Committer and Apache Eagle hao@apache.org Qcon / Hadoop Summit (SJC, SHA, BJ)

4 Agenda Introduction Architecture Ecosystem Q & A

5 What is Apache Eagle Apache Eagle' is a distributed real- time monitoring and alerting engine for hadoop from ebay Open sourced as Apache Incubator Project on Oct 26 th 2015 Secure Hadoop in Realtime a data activity monitoring solution to instantly identify access to sensitive data, recognize attacks/ malicious activity and block access in real time. See or

6 Apache Eagle History Donated to Apache Software Foundation (ASF) from ebay at Oct 26 th, 2015 Hadoop Eagle Project Initiative Hadoop Eagle Production Release Apache Incubator eagle.incubator.apache.org Dec 2013 May 2014 Oct Oct Github Open Source github.com/apache/incubator- eagle

7 Why build Eagle Eagle was initialized by end of 2013 for hadoop ecosystem monitoring as any existing tool like zabbix, ganglia can not handle the huge volume of metrics/logs generated by hadoop system in ebay. ebay Inc 100+ nodes cores 1 PB nodes nodes nodes 10,000+ cores 10+ PB nodes 10,000+ cores 50+ PB / ,000 nodes 200,000+ cores 2000 PB user Hadoop Data Security Activity Hadoop Platform Heath Availability Performance

8 Challendges Functional Requirements Flexible Threshold Slide Window Multiple- factors Correlation Real- time or historical data Join Complex monitoring cases Unfunctional Requirements Scalable: Distributed Real- time: Streaming Maintainability: Hot- deploy Intelligent: Machine Learning Ops (80%) DevOps (15%) Dev (10%)

9 Agenda What s Eagle Architecture Ecosystem Q & A

10 Eagle Architecture Overview Scalable Scales to monitor thousands of policies and billions of access events Extensible Eagle can be easily extended to monitor other data sources Real- time Generates alerts in real time and blocks users with malicious intent Machine Learning Create dynamic user profiles based on user behavior

Eagle Architecture Overview Real Time Alert Dashboard Policy Analyst STREAM PROCESSING ENGINE Insights Policy Engine MACHINE LEARNING MODULE HDFS, Audit, Security Data Collector Kafka Actionable

11 Eagle Architecture Overview Real Time Alert Dashboard Policy Analyst STREAM PROCESSING ENGINE Insights Policy Engine MACHINE LEARNING MODULE HDFS, Audit, Security Data Collector Kafka Actionable Alerts Actionable Alerts User properties METADATA MANAGER REMEDIATION ENGINE Apache Ranger Thresholds Policy Custom module ML Thresholds DATA STORES Metadata Management Admin Console Ops Engineer Activities MACHINE LEARNING TRAINING MODULE

12 Eagle Features Highlight Real- time Data Collection Distributed Policy Engine Stream Processing DSL Scalable Data Storage & Query Machine Learning Intergration Module Management MultipleTenant Support NOTE {NAME}- {NUMBER} like HDFS means open source project ticket id contributed by us

13 Eagle - Data Collection Decoupling with Message Bus Apache Kafka: high- throughput distributed messaging Partition: balance between logic and throughput Cross- Platform Integration Community Kafka Client (18+) Python/Go/C/C++/JAVA.. Enhanced Log4j- kafka KAFKA- 2041: Extensible Partition Key KAFKA- 2077: Advanced Topic Selector

14 Eagle - Data Collection Availability: Filebeat + Logstash Resource consumption balance Message throughput balance ( LOGSTASH- 179) Logstash- 1 Shuffle Grouping Logstash- 2 Logstash- Field Grouping Kafka Logstash- Ligh- weight collector (golang) with daemon Logstash instances cluster Distributed Message Bus

15 Eagle - Distributed Real-time Policy Engine Highlights AlertExecutor_{1} Real Time Alerts Real- time Usability Scalability Extensibility Metadata- driven Real- time Event Stream Stream_{1} Stream_{*} Dynamical Stream Schema Stream Processing Distributed Streaming Cluster Environment AlertExecutor_{2} AlertExecutor_{N} Dynamical Policy Deployment Alerts Policy Management METADATA MANAGER Policy

16 Eagle - Distributed Real-time Policy Engine Real- time Real Time Alerts Kafka- based Distributed Message Bus (Extensible) Storm- based Real- time Execution Environment (Extensible) Stream events are processed and alerts are evaluated during streaming Event Stream (Kafka) Dynamical Stream Schema Dynamical Policy Deployment METADATA MANAGER Alerts Policy Policy Management

17 Eagle - Distributed Real-time Policy Engine Usability Real Time Alerts Powerful SQL- Like CEP CQL for Policy Definition Dynamical Poilcy Metadata Lifecycle Management (Deployment/Update) Easy- to- use Policy management and Alert analytics UI from metricstream[(name == 'ReplLag') and (value > 1000)] select * insert into outputstream; Distributed Streaming Cluster Environment Dynamical Policy Deployment METADATA MANAGER Alerts Policy Policy Management

18 Eagle - Distributed Real-time Policy Engine Full- function Streaming CEP CQL: Siddhi on Storm by default hdfsauditlogeventstream[(src == '/tmp/private')]#window.externaltime(timestamp,10 min) select user, count(timestamp) as aggvalue group by user having aggvalue >= 5 insert into outputstream; Filter Join Aggregation: Avg, Sum, Min, Max, etc Group by Having Stream handlers for window: TimeWindow, Batch Window, Length Window Conditions and Expressions: and, or, not, ==,!=, >=, >, <=, <, and arithmetic operations Pattern processing Sequence processing Event Tables: intergrate historical data in realtime processing SQL- Like Query: Query, Stream Definition and Query Plan compilation

19 Eagle - Distributed Real-time Policy Engine Scalability: dynamic policy partition by {event} * {policy} N Users with 3 partitions, M policies with 2 partitions, then 3*2 physical tasks Physical partition + policy- level partition AlertExecutor_{1} Stream_{1} Stream_{*} Stream Processing AlertExecutor_{2} AlertExecutor_{N} Distributed Streaming Cluster Environment

20 Eagle - Distributed Real-time Policy Engine Distributed Streaming Partition Problem m groupby[ GreedyStrategy ]((_.key1,_.key2 )) Realtime Partition Strategy Key Distribution Statistics (Online/Offline) Strategy Greedy (Online/Offline) PoTC PKG Hashing Key Statistics Cache Async HBase

21 Extensibility Eagle - Distributed Real-time Policy Engine Support WSO2 Siddhi CEP as first class Extensible policy engine implementation public interface PolicyEvaluatorServiceProvider { public String getpolicytype(); // literal string to identify one type of policy public Class getpolicyevaluator(); // get policy evaluator implementation public List getbindingmodules(); // policy text with json format to object mapping } Extensible policy lifecycle management Siddhi CEP Policy Evaluator Machine Learning Policy Evaluator Distributed Real-time Policy Engine Policy/Metadata METADATA MANAGER Extensible Policy Evaluator public interface PolicyEvaluator { public void evaluate(valuesarray input) throws Exception; // evaluate input event public void onpolicyupdate(alertdefinitionapientity newalertdef);// policy update public void onpolicydelete(); // invoked when policy is deleted }

22 Eagle - Distributed Real-time Policy Engine Metadata- Driven Stream Schema: AlertStreamSchemaEntity Policy Definition: AlertDefinitionAPIEntity Central metadata management Dynamic metadata deployment Distributed Real- time Policy Engine Dynamic Metadata Loading @JsonIgnoreProperties(ignoreUnknown "datasource", "alertexecutorid", columns = { "alertexecutorid" }, unique = true), }) public class AlertDefinitionAPIEntity extends private String private String private String dedupedef;

23 Eagle - Fluent Stream Processing DSL AlertExecutor_{1} Real- time Event Stream Stream_{1} env.execute(); Stream_{*} Stream Processing AlertExecutor_{2} Alerts AlertExecutor_{N} Distributed Streaming Cluster Environment val env = ExecutionEnvironment.getStorm() env.fromkafka(kafkaconfig).flatmap(auditlogtransformer).groupby(_.user).flatmap(userprofileaggregator);.alert.persistand env.execute()

24 Eagle - Fluent Stream Processing DSL Optimizer 1. Development 2. Optimization 3. Compile to native app Physical execution platform independent Easily assemble data transformation, filtering, join and alerting DAG in fluent way DAG rewrite and optimization StreamUnionExpansion StreamGroupbyExpansion StreamNameExpansion StreamAlertExpansion StreamParallelismConfigExpansion trait StreamProducer{ filter flatmap map{1,2,3,4} groupby streamunion // stream join is hard, not implemented for storm alertwithconsumer } StormExecutionEnvironment env = ExecutionEnvironmentFactory.getStorm(config); env.newsource(new KafkaSourcedSpoutProvider().getSpout(config)).renameOutputFields(1).flatMap(new AuditLogTransformer()).groupBy(0).flatMap(new UserProfileAggregatorExecutor());.alertWithConsumer( useractivity, userprofileexecutor ) env.execute();

25 Eagle - Scalable Data Storage and Query Entity Metadata on large- scale NoSQL storage like HBase Full- function SQL- Like REST Query Optimized rowkey design for time- series monitoring data HBase Coprocessor @JsonIgnoreProperties(ignoreUnknown "datasource", "alertexecutorid", columns = { "alertexecutorid" }, unique = true), }) public class AlertDefinitionAPIEntity extends private String private String private String dedupedef; query= AlertDefinitionService[@dataSource="hiveQueryLog"]{@policyDef}

26 Eagle Uniform HBase Rowkey Design Uniform rowkey design Rowkey ::= Prefix Partition Keys timestamp tagname tagvalue Metric Entity Log Rowkey::= Metric Name Partition Keys timestamp tagname tagvalue Rowkey::= Default Prefix Partition Keys timestamp tagname tagvalue Rowkey::= Log Type Partition Keys timestamp tagname tagvalue Rowvalue ::= Log Content

27 Eagle - Machine Learning Intergration

28 Eagle User/System Activity Profiling User Activity Profiling Offline: Determine bandwidth from training dataset the kernel density function parameters (KDE) Online: If a test data point lies outside the trained bandwidth, it is anomaly (Policy) Kernel Density Function PCs(Principle Components) in EVD (Eigenvalue Value Decomposition)

29 Eagle - Anomaly Metric Predictive Detection Anomaly Metric Predictive Detection Offline: Analyzing and combining 500+ metrics together for causal anomaly detections (IG - > PCA - > GMM - > MCC) Online: Predictively alert for anomaly metrics PCA (Principal Component Analysis) Normal (Green) and Abnormal (Red) Data and Probability Distribution and Threshold Selection Anomaly Metric Predictive Detection Case Study

30 Eagle - Module Management

31 Eagle Multi Tenant Support Model Driven Hot Deploy Isolation Scalability Disaster Recovery

32 Agenda Introduction Architecture Ecosystem Q & A

Eagle Ecosystem Eagle Framework Distributed real- time framework for efficiently developing highly scalable monitoring applications Eagle Apps Securiy / Hadoop / Cloud / Database Eagle Interface REST

33 Eagle Ecosystem Eagle Framework Distributed real- time framework for efficiently developing highly scalable monitoring applications Eagle Apps Securiy / Hadoop / Cloud / Database Eagle Interface REST Service / Management UI / Customizable Analytics Visualization Integration à à à à Ambari Docker Ranger Dataguise Eagle Framework Apps à à à à Security Hadoop Cloud Database Eagle Integration Ambari / Docker / Ranger / Dataguise Open Source Community- driven and Cross- community cooperation Interface à à à Web Portal REST Services Analytics Visualization

34 Eagle Ecosystem Open Source If you want to go fast, go alone. If you want to go far, go together. - - African Proverb

35 Q&A

36 谢谢

<4D6963726F736F667420506F776572506F696E74202D20C8EDBCFEBCDCB9B9CAA6D1D0D0DEBDB2D7F92E707074>

<4D6963726F736F667420506F776572506F696E74202D20C8EDBCFEBCDCB9B9CAA6D1D0D0DEBDB2D7F92E707074> 软件架构师研修讲座胡协刚软件架构师 UML/RUP 专家 szjinco@public.szptt.net.cn 中国软件架构师网东软培训中心小故事 : 七人分粥当前软件团队的开发现状和面临的问题软件项目的特点解决之道 : 从瀑布模型到迭代模型解决项