QCon2015_Shanghai_XuLi

Size: px

Start display at page:

Download "QCon2015_Shanghai_XuLi"

奋委叶
6 years ago
Views:

2 基于连接与组合的微服务架构剖析

3 Qiniu 1.0 POST GET Object Storage

4 Qiniu 2.0 POST GET Object Storage File Operation Process

5 Qiniu 2.0 / POST GET Object Storage Raw File Processed File File Operation Process (File Operation Process)

6 early Bandwidth LB More LB? 6

7 early Bottlenecks Network IO Data Flow Gate Bandwidth OMG! Daemon Disk IO 7

8 early Bottlenecks Fix Network IO Control Router Disk IO Data Control Agent Data 8

9 nanoservices Request with URL Load Balancer Pull raw File Storage AGENT Local AGENT IMAGE IMAGE IMAGE 9

10 API Gate (Logic Server) Supervisor Router [ACTION] fop2 Qiniu Architecture 配置管理 / 服务发现 / 负载均衡 HOST-1 HOST-2 Agent Agent Docker Daemon Docker Daemon 图像处理 Container1 音视频转码 Container2 图像处理 Container1 音视频转码 Container2 文档转换 Container3 自定义处理 Container4 文档转换 Container3 自定义处理 Container4 Object Cache Object Storage 10

11 How Works? API Gate [GET] (Logic Server) Object Cache 5 Agent NO 4 Supervisor 6 7 Object Storage Router Data IO Daemon CONTAINER 11

12 Questions» 离线计算怎么做» 同一个并发请求怎么办» 多个组合工作流如何处理» 程序 IMAGE 可以自定义吗 12

13 How Offline Works? [POST] Callback API Gate [POST] (Logic Server) Object Cache 5 Agent NO 4 Supervisor 6 7 Object Storage Router Data IO Daemon CONTAINER 13

14 How Concurrency Works? 1 [GET] [GET] [GET] Object Cache API Gate (Logic Server) NO 7 Object Storage 2 Router Agent Memcached Data IO Supervisor 6 Daemon CONTAINER 14

15 Waiting for results done := make(chan Result, 1) go func() {... done <- Result{} }()... select { case result := <-done: do(result) case <- time.after(3 * time.second): fmt.println("timeout") } 15

16 How Multi- Works? [GET] fop2 1 API Gate (Logic Server) Object Cache 5 Agent NO 4 Supervisor 6 7 Object Storage Router Data IO Daemon CONTAINER 16

17 What is Multi-s? [ACTION] fop2 fop

18 从 Unix 谈起» Unix 的连接和组合» app1 params1 app2 params2» App 接口» 输入 stdin, params» 输出 stdout» 协议 text (data stream)» Pipeline» 将一个 app 的输出(stdout) 转为另一个 app 的输入 (stdin) 18

19 论 awk 编程与 Unix 管道艺术 200 GET POST GET GET... HTTP CODE COUNTING awk '{h[$1]++} END {for(k in h) print h[k], k}' test.log cut -d' ' -f1 test.log sort uniq -c 19

20 Unix Pipelines awk - pattern-directed scanning and processing language cut -- cut out selected portions of each line of a file sort - sort lines of text files uniq -- report or filter out repeated lines in a file 20

21 Pipeline 关键点» 多个 app 是并行执行的» 上游每产生一段output 会立即交由下游处理» app 间的协议是松散耦合的» 上游 app 的 ouput 是 xml 还是 json 下游 app 需要知晓但是属于一种松散的耦合关系并无任何强制的约束 21

22 Go 对 Unix Pipeline 的仿真 App func(in io.reader, out io.writer, args []string) app1 params1 app2 params2 bind(app1, params1) bind(app2, params2) pipe( bind(app1, params1), bind(app2, params2) ) 22

23 Go 对 Unix Pipeline 的仿真 func bind( app func(in io.reader, out io.writer, args []string), args []string ) func(in io.reader, out io.writer) { return func(in io.reader, out io.writer) { app(in, out, args) } } 23

24 Go 对 Unix Pipeline 的仿真 func pipe( app1 func(in io.reader, out io.writer), app2 func(in io.reader, out io.writer) ) func(in io.reader, out io.writer) { return func(in io.reader, out io.writer) { pr, pw := io.pipe() defer pw.close() go func() { defer pr.close() app2(pr, out) }() app1(in, pw) } } 24

25 Go 对 Unix Pipeline(s) 的仿真 func pipe(apps...func(in io.reader, out io.writer)) func(in io.reader, out io.writer) { if len(apps) == 0 { return nil } app := apps[0] for i := 1; i < len(apps); i++ { app1, app2 := app, apps[i] app = func(in io.reader, out io.writer) { pr, pw := io.pipe() defer pw.close() go func() { defer pr.close() app2(pr, out) }() app1(in, pw) } } return app 难以想象的优雅 } 25

26 Multi- Works Pipelines pipe(bind(fop1),bind(fop2)) [GET] fop2 1 API Gate (Logic Server) Object Cache 5 Agent NO 4 Supervisor 6 7 Object Storage Router Data IO Daemon CONTAINER 26

27 程序可以自定义吗» Pipelined services» User-defined service (Docker) 27

28 自定义数据处理 U» U 用户自定义的数据处理进程» 基于 Docker» 面向对象存储做就近计算处理» U 程序可以是自行部署也可以选用七牛 U 公共平台上已有的服务程序» Docker 为部署自定义 Image 提供了可能性 28

29 U 案例 29

30 U 不可以做什么» 不可以作为网站主机因为 U 没有 IP:PORT 只能通过 URL?ufop_api_args 形式调用其中 ufop_api_args 由 U 程序自定义» 只要数据存在七牛针对该数据的各种特征计算处理都可以使用 U 实现» U 也可与七牛云平台上的其他无缝集成管道链式处理: fop1 fop2 app3 appn 30

31 Why Docker Product Compose Software Code Process Architecture Modularization Tool RevisionControl (git) Microservices Container (Docker) 31

32 软件架构变迁 Microservices Monolithic Application Introduction to Microservices, Chris Richardson 32

33 Why not Docker» 容器技术提供的隔离性物理边界机制和微服务的理念是契合的» 拆细/封装了环境和实现不同的技术不需要知道里面是怎样的只关心接口» 然而还不够没有解决服务运维问题是以出现官方的工具链» 没有运维的解决方案如同空中楼阁» Kubernetes, 谷歌去年公开的容器编排系统领先十年 omega/borg 论文分量» 旨在解决容器运维问题定义状态而非关心过程» 其他问题存储网络硬伤 33

34 Kubernetes»»»»»» Master Minion(Kubelet) Pods Replication Controller Labels - Proxy 34

35 as-nanoservices Storage DocFormat U Image Transcoding Customized HLS CDN Log & Data 35

实践课堂成都站-0609.key

实践课堂成都站-0609.key 青云 QingCloud 容器器及 Kubernetes 实践王渊命青云 QingCloud 容器器平台负责人当我们谈论容器器时我们在谈论什什么? 容器器的两个视角资源隔离应用封装容器器生态圈现状容器器技术一家独大百花齐放 Docker, Rocket, Mesos Universal container, LXC, Hyper Container 调度系统三足鼎立