<4D F736F F F696E74202D20C8E7BACED2C0CDD F54A3ACB4D3496F54D3A6D3C3D6D0BBF1C8A1C9CCD2B5BCDBD6B5>

Size: px

Start display at page:

Download "<4D F736F F F696E74202D20C8E7BACED2C0CDD F54A3ACB4D3496F54D3A6D3C3D6D0BBF1C8A1C9CCD2B5BCDBD6B5>"

雪犹充
5 years ago
Views:

1 如何依托 AWS IoT, 从 IoT 应用中获取商业价值张羽 (Alfred Zhang),AWS 解决方案架构师 Alfred Zhang, Solutions Architect, AWS China 2016 年 6 月 20 日 June 20th, 2017

2 议程理解 AWS 平台提供的不同类型的 IoT 相关的数据服务如何使用 AWS 平台将数据转化成有效的决策和行动如何选择不同类型的数据服务与 AWS IoT 进行集成

3 数据驱动 IoT 应用

4 从 10 到 500 亿到亿, 万物互联势不可挡

5 2025 年,IoT 有潜力创造 4 万亿到 11 万亿的经济价值

6 跟随万物互联的步伐, 踏入大数据的时代人, 物和服务在不断生成数据, 包括 : 状态信息传感器读数用户交互信息执行器控制信息运营事件

7 IoT 所面临的最大挑战采集 -> 存储 -> 处理 / 分析 -> 使用 / 展现 -> 行动采集设备的数据, 获得可行的决策, 帮助业务的优化 AWS IoT Greengrass

8 三类数据分析模式 Retrospective analysis and reporting 面向过去

9 三类数据分析模式 Retrospective analysis and reporting Here-and-now real-time processing and dashboards 面向过去面向现在

10 三类数据分析模式 Retrospective analysis and reporting Here-and-now real-time processing and dashboards Predictions to enable smart applications 面向过去面向现在面向未来

11 三类数据分析模式 Retrospective analysis and reporting Here-and-now real-time processing and dashboards Predictions to enable smart applications Amazon Redshift Amazon RDS Amazon S3 Amazon EMR Amazon Kinesis AWS Lambda Amazon DynamoDB Amazon EC2 Amazon Machine Learning Apache MXNet on AWS

12 IoT 应用的主要特性

13 IoT 需要实时快速的处理响应在实时传感数据中侦测数据模式针对多个事件发现关联关系丰富实时数据的相关信息 Here-and-now real-time processing and dashboards 面向现在用途触发快速响应优化业务运营改善用户体验

14 IoT 需要以历史数据为依据为当前事件提供历史数据分析依据基于历史数据进行趋势分析 Retrospective analysis and reporting 面向过去用途从历史数据中发现知识和经验支撑高效的分析和报表, 精确掌握业务运营状况支撑海量数据高精度的计量计费

15 IoT 需要实时快速的处理响应依据事件进行模式侦测匹配通过数据的分布, 发现潜在的规律, 帮助自动化建立规则 Predictions to enable smart applications 面向未来用途 : 预测未来可能发生的事件发现潜在问题期望用户干预制定可行决策 : 下一步做什么

16 样例

17 气温 / 环境传感器若干空调控制单元, 每个控制单元 3 个传感器持续秒级传输温度湿度气压更新数据无缝接入云端消息传输的可靠性需求适中

18 来源于设备的样本消息 { } "temperature" : "100", "humidity" : "92", "pressure" : "8

19 所有设备启用证书认证采用 MQTT 协议传输消息一个设备一个主体 : rooms/ac/${deviceid}

20 有了传感数据, 我们能做些什么呢? - 温度随着时间是如何变化的? - 气压和温度之间是怎样的关系? - 当前有多少传感器处于活动状态? - 当前的温度和昨天或者去年相同时间的温度基本相符? - 当前传感器读数是否可疑? - 当前传感器是否存在损坏状况? - 今天我们是否需要更换着装?

21 AWS IoT

22 AWS IoT: 安全的设备接入多种协议支持的消息网关数以 10 亿计的设备或应用可以同时接入, 支持 MQTT HTTP 或 WebSockets 协议接口, 并且还会融入更多主流接口高度弹性化的 Pub Sub Broker 以零配置的方式, 高度弹性化的, 完成从 1 个设备到 10 亿个设备的链接开通安全为先设备和 AWS IoT 平台之间通过 X509 Certs 和 TLS v1.2 建议双向认证的安全通道

23 AWS IoT: AWS 服务的一大门户 Device Registry Cloud alter-ego of a physical device. Persists metadata about the device. Device Shadows Apps and devices can access RESTful Shadow (state) that is in sync with the device Rules and Actions Match patterns and take actions to send data to other AWS services or republish

24 AWS IoT 规则引擎 Rules Engine evaluates inbound messages published into AWS IoT, transforms and delivers to the appropriate endpoint based on business rules. External Endpoints can be reached via AWS Lambda and Amazon SNS, etc.

25 规则引擎的灵活性 SQL-like syntax Where operators Inline functions Actions

26 用于回溯性分析的数据处理

27 数据的采集 Devices set up as Things in Device Registry Each device sends data as JSON via MQTT One MQTT topic per device: rooms/ac/{deviceid} Each device has a certificate and access rights to use its topic

28 数据的存储 Move all (?) incoming data into permanent storage Make data available for later analysis: Reporting Billing / Metering Explorative Analysis Machine Learning

29 方法 : Set up a Rule to Capture & Transform Incoming Data Define an Action to Store the Data Query & Analyze the Stored Data

30 1) Set up a Rule to capture all sensor readings

31 2) Define an Action to Store the Data 应该选择哪一种存储方案?

32 Storage Options

Storage Options: Amazon S3 Actions can directly write into (JSON) files on S3 Very simple to configure, just provide bucket name Results in 1 file per event Lots of small files can be hard

33 Storage Options: Amazon S3 Actions can directly write into (JSON) files on S3 Very simple to configure, just provide bucket name Results in 1 file per event Lots of small files can be hard to handle Inefficient when processing with Hadoop/ Amazon EMR or when importing into Redshift Useful when you have a very low frequency of events, e.g. when you only want to log outliers to S3

Storage Options: Amazon S3 Buffer data using Amazon Kinesis or Amazon Kinesis Firehose to get fewer, larger files Buffering, compression & output to S3 is built into Firehose no other infrastructure

34 Storage Options: Amazon S3 Buffer data using Amazon Kinesis or Amazon Kinesis Firehose to get fewer, larger files Buffering, compression & output to S3 is built into Firehose no other infrastructure needed! Kinesis Connector Library can be extended to perform transformation, filter or serialize data Additional Control over Buffering & Output Formats Added complexity: Requires Amazon EC2 workers running Kinesis Connector Library

Storage Options: Amazon Redshift Actions can forward data Amazon Kinesis Firehose Buffering & output to Redshift is built into Firehose Very easy to setup Fully managed Use Amazon Kinesis as

35 Storage Options: Amazon Redshift Actions can forward data Amazon Kinesis Firehose Buffering & output to Redshift is built into Firehose Very easy to setup Fully managed Use Amazon Kinesis as an alternative More control: Use Kinesis Connector Library to perform transformation, filter or serialize data Added complexity: Requires Kinesis Connector Library etc. to execute on Amazon EC2

36 Storage Options: Amazon DynamoDB Actions can directly write into Amazon DynamoDB Creates one row per event, can define: Hash Key, Range Key and attributes to store Hash Key = deviceid, Range Key=timestamp Very simple to configure, just provide table & field names Adding GSIs and LSIs provides additional flexibility and enables different queries

37 Storage Options: Amazon DynamoDB

Storage Options: Amazon DynamoDB AWS Lambda function provides additional flexibility: Transform data Write into different/multiple tables Enrich data with contextual

38 Storage Options: Amazon DynamoDB AWS Lambda function provides additional flexibility: Transform data Write into different/multiple tables Enrich data with contextual information pulled in from other sources Only able to process one event at a time! (i.e., AWS Lambda when called from AWS IoT cannot aggregate events before writing to DynamoDB)

39 3) Query & Analyze the Stored Data 如何处理分析数据?

40 3) Query & Analyze the Stored Data

41 3) Query & Analyze the Stored Data

42 Recommendations Want to run a lot of queries constantly? Use Kinesis Firehose to write into Amazon Redshift Need fast lookups, e.g., in Rules or Lambda functions? write into DynamoDB, add indices if necessary Have a need for heavy queries but not always-on? Use Kinesis Firehose & S3, process with Amazon EMR.

43 样例

44 1) Set up a Rule to capture all sensor readings

45 2) Pump Data through Firehose into Redshift

46 3) Analyze Data using Amazon QuickSight

47 实时监控和实时响应

48 目标当有大幅气温变化的时, 发出报警消息采集当前传感读数, 并进行可视化分析

49 1) 设定规则, 对传感读数做出响应

50 1) 设定规则, 对传感读数做出响应 AWS IoT Rules only have access to the current event cannot take contextual information into account Consider passing all the data to the Action for evaluation.

51 2) Process the Data 处理数据的最佳实践?

52 Processing Options

53 Processing Options 一次调用处理一个单一事件 ( 非批量操作 ) 结合上下文信息, 借助其他数据源丰富消息数据执行各类转换操作可任意运行 Node.js 或 Java 程序无需管理底层基础设施!

Processing Options Great for alerts: Sends push notifications, emails and SMS Call other systems via Http

54 Processing Options Great for alerts: Sends push notifications, s and SMS Call other systems via Http Post or Webhooks(AWS or On-Premises) SNS Topics support multiple subscribers, incl. AWS Lambda and Amazon SQS

55 Processing Options Great when events arrive with varying frequency Buffer data for asynchronous processing Ensure that no event data is lost SNS Topics support multiple subscribers, incl. AWS Lambda and Amazon SQS Easily deploy SQS workers on AWS Elastic Beanstalk (or Amazon EC2)

different rules / topics / devices Supports many

56 Processing Options Provides access to a "rolling window" of event data Scalable, can consume events from a multitude of different rules / topics / devices Supports many independent, concurrent readers (&writers) Multiple processing options:

57 Processing Options Scalable way to connect many different systems to the stream of events, e.g., custom KCL code, Complex Event Processing (CEP) products Amazon Kinesis is a hub for all stream processing needs

58 Example: 1. Read last N events from stream 2. Determine maximum and rate of increase since beginning 3. Decide if alert should be sent

59 Recommendations Only care about individual events? Invoke an AWS Lambda Function via Rule / Action For sliding window analysis and more flexibility Stream into Kinesis and Run AWS Lambda function Use Amazon Kinesis as a Hub for all incoming events.

60 3) Visualize the Current Metrics Managed Amazon Elasticsearchas a service Easy & fast indexing of data, well suited for lookups on streaming data Easy to use visualization / dashboards using Kibana

61 智能预测应用

62 Machine learning and smart devices Machine learning is the technology that automatically finds patterns in your data and uses them to make predictions for new data points as they become available

63 Machine learning and smart devices Machine learning is the technology that automatically finds patterns in your data and uses them to make predictions for new data points as they become available Your devices + machine learning = smart devices

64 IoT Use Cases for Machine Learning Find potential problems by looking for patterns Identify engines that are about to break down Predict when supplies will run out Spot sensors that report implausible data Predict next movement / direction of a connected vehicle Based on driving parameters & observations from other cars Predict traffic jams before they occur

65 Amazon Machine Learning Real-time predictions (and batch) Training & evaluation of machine learning models Picks the right model & parameters, helps build training data

66 Basic Approach 1. Collect / build training data Take past data for sensor readings (temperature, humidity, pressure), not the deviceidor timestamp, as input Target: we define which readings are 'correct' or incorrect and add the target variable's value to the training data.

67 Basic Approach 2. Train a Machine Learning Model

68 Basic Approach 3. Create a real-time prediction endpoint for the model

69 Basic Approach 4. Get predictions for events as they come in

70 Basic Approach 1. Collect / build training data Determine input variables & target Evaluate the data to pick the target value for each set of inputs in the data 2. Train a Machine Learning Model Builds a model based on the information in the training data 3. Create a real-time prediction endpoint for the model Outputs a prediction based on the input variables provided 4. Get predictions for events as they come in

71 Example Use Case: Filter out bad readings 1. Create a training data set based on past data & human evaluation of the data i.e., manually review the data and mark incorrect values 2. Train a Amazon ML model on this data to predict which combinations are (in)correct 3. Invoke ML model on incoming data to predict correctness 4. Alert staff via Amazon SNS push notification

72 Lambda Function

73 Lambda Function

74 Lambda Function

75 Recommendations Rely on past data / context rather than defining 'rules' Use Amazon Machine Learning for an easy start Let real-time predictions drive reaction to patterns in events

76 总结和展望

77 总结 :AWS IoT + 大数据 + 机器学习

78 展望 : 未来的技术方向? 自动化事件处理响应 : 结合历史数据的分析, 比对实时数据的特征, 进行预测性分析, 并基于预测性分析进行智能化干预 Semi-supervised learning,unsupervised learning, Reinforcement learning,deep learning

79 问答

80 Thank You!

PowerPoint Presentation

PowerPoint Presentation 巧妙利用 AWS 进行物联网解决方案开发牛付强, AWS 解决方案架构师议程 AWS IoT 组成单元利用 AWS IoT Device SDK 进行 AWS IoT 的开发演示 AWS IoT 的作用如果你知道每一个事物的状态并能在数据上面推理你会解决什么问题 AWS IoT 架构 ( 设备到云端 ) IoT 设备类型 IoT 设备类型 Amazon FreeRTOS Amazon