PowerPoint 演示文稿

Size: px

Start display at page:

Download "PowerPoint 演示文稿"

昀白艾
5 years ago
Views:

1 A Brief Introduction to HBase & Cassandra 曲直

2 HDFS OpenSource GFS Typically 64M block NameNode: In memory metadata 3 replication Immutable

3 Hadoop database, a distributed, scalable, big data store.

4 HBASE 定位 HBASE 是存储基于 HDFS 实时随机读写

5 HBASE 特性 KV 存储 Get Put Delete Scan 水平扩展行操作的强一致性自动分表支持 MapReduce 基于 HDFS Java,Thrift,REST-ful 接口

6 谁在用 Yahoo! 960 nodes Facebook 180B/month message

7 HBase 数据模型 Table Region ColumnFamily Row Column Version CELL Value

8 SortedMap<RowKey, List<SortedMap<Column, List<Value, Timestamp>>>>

9 物理结构 Challenge 可扩展性 : 一个 Table 可能很大很大, 怎么办? Region Split 字典序排序

10 HRegion Challenge 负载均衡 : 适应服务器的水平扩展? HRegion 是 Hbase 中分布式存储和负载均衡的最小单元

11 Region 定位 Challenge 如何查找某个 key? 谁来负责 Region 管理? 三级游走 -ROOT-.META. Region 增加 Master 节点

12 Dive into HRegion Challenge HDFS 文件只追加, 不可修改, 如何在这样的系统上实现增删查改? 写到一半服务器崩溃了怎么办? 三级结构 HLog: 保证操作不会丢失 MemStore: 写操作缓存 StoreFile(HFile) : 当 MemStore 达到一定大小, 一次刷新到 HDFS( 排序 -> 去重 ->Write)

13 Dive into HRegion

14 Dive into HRegion Challenge 一直追加写入, 一个 KV 在多个 HFile 里会有多个版本 API Put Delete Get Get 的性能问题定期 Compact: 归并排序 BloomFilter BlockCache 直接追加写入取各 StoreFile 和 MemStore 的最新版本

15 HBase as a Cluster 服务器是会被玩坏的 Master 多 Master, 谁是 Active 的? RegionServer 如何判断 RegionServer 是否活着? MemStore 里的数据怎么办? Solution ZooKeeper 分布式一致性服务 Master 负责迁移 Region,HLog 回放恢复 MemStore Hlog 必须放在 HDFS 上!

16 总体结构 Master Region 之上的操作 Put/Get 不经过 Master RegionServer Region 之下的操作 HDFS HFile HLog ZooKeeper 状态信息

17 总体结构

18 HFile

19 HLog

20 HRegion Server

21 A highly scalable, eventually consistent, distributed, structured key-value store.

22 Users <<enter Cassandra The largest production cluster has over 100 TB of data in over 150 machines.

23 Dynamo/Bigtable (1/3) <<related concepts & techniques 23/63

24 Dynamo/Bigtable (2/3) <<related concepts & techniques Dynamo-like features Symmetric,P2P architecture No Special nodes, No Single Point Of Failure(SPF) Gossip Based cluster management Distributed hash table for data placement Pluggable partitioning Pluggable topology discovery Pluggable placement strategies Tunable, Eventual Consistency 24/63

25 Highlights High availability Incremental scalability Eventually consistent Tunable tradeoffs between consistency and latency Minimal administration No Single Point of Failure(SPF) 25/63

26 Super column <<data model { name: address, value: { street: {name: street,value: north zhongshan road,timestamp: }, city:{name: city,value: Shanghai,timestamp: }, zip:{name: zip,value: ,timestamp: } } Value of a super column is a map of columns while value of a column is a string. No timestamp for super column itself. Keyed by the column s name supercolumn Byte[] name list[column] columns 26/63

27 Super Column family <<data model super1 super2 key1 column column column column column key3 super1 column column column super3 column column column No super column is allowed in a standard column family while no standard column is allowed in a super column family. Columns are attributes of super 27/63

28 Storage model(3/4) Key (CF1, CF2, CF3) Commit Log Binary serialized Key ( CF1, CF2, CF3 ) Memtable ( CF1) Memtable ( CF2) Memtable ( CF2) out of space too many keys(128 is default) timeduration(client provided,no cluster clock) Data file on disk Dedicated Disk K 128 K 256 K 384 Offset Offset Offset Bloom Filter (Index in memory) <Key name><size of key Data><Index of columns/supercolumns>< Serialized column family> BLOCK Index <Key Name> Offset, <Key Name> Offset <Key name><size of key Data><Index of columns/supercolumns>< Serialized column family> 31/63

Storage model(4/4) Sorted K2 < Serialized data > K1 < Serialized data > K10 < Serialized data > K2 < Serialized data > K30 < Serialized data > K3 < Serialized data > -- D E L -- E T E D Sorted --

29 Storage model(4/4) Sorted K2 < Serialized data > K1 < Serialized data > K10 < Serialized data > K2 < Serialized data > K30 < Serialized data > K3 < Serialized data > -- D E L -- E T E D Sorted -- Sorted K4 < Serialized data > K5 < Serialized data > K10 < Serialized data > MERGE SORT Index File Loaded in memory K1 Offset K5 Offset K30 Offset Bloom Filter Sorted K1 < Serialized data > K2 < Serialized data > K3 < Serialized data > K4 < Serialized data > K5 < Serialized data > K10 < Serialized data > K30 < Serialized data > Data File 29/63

30 CAP(2/3) <<related concepts & techniques 30/50

31 CAP(3/3) <<related concepts & techniques Why AP? CA-corruption possible if live nodes can t communicate CP-completely inaccessible if any nodes are dead AP-always available, but may not always read most recent Cassandra trade-off strong C in favor of high A choose AP but allows them to be tunable to have more C. eventual consistency (weak consistency) when no updates occur for a long period of time, eventually all updates will propagate through the system and all the replicas will be consistent 31/63

32 partitioning & replication E 1 0 h(key1) C A N=3, rackunaware h(key2) F D B 1/2 32/63

33 Consistency level (1/2) <<related concepts & techniques write Level ZERO ANY ONE QUORUM DCQUORUM ALL Behavior Ensure nothing. A write happens asynchronously in background. Until CASSANDRA-685 is fixed: If too many of these queue up, buffers will explode and bad things will happen. (Requires 0.6) Ensure that the write has been written to at least 1 node, including HintedHandoff recipients. Ensure that the write has been written to at least 1 replica's commit log and memory table before responding to the client. Ensure that the write has been written to N / replicas before responding to the client. As above but takes into account the rack aware placement strategy. See Ensure that the write is written to all N replicas before responding to the client. Any unresponsive replicas will fail the operation. 33/63

34 Write operation <<storage model 34/63

35 Consistency level (2/2) read Level ZERO /ANY ONE QUORUM DCQUORUM ALL Behavior Not supported, because it doesn't make sense. <<related concepts & techniques Will return the record returned by the first replica to respond. A consistency check is always done in a background thread to fix any consistency issues when ConsistencyLevel.ONE is used. This means subsequent calls will have correct data even if the initial read gets an older value. (ReadRepair) Will query all replicas and return the record with the most recent timestamp once it has at least a majority of replicas (N / 2 + 1) reported. Again, the remaining replicas will be checked in the background. When using rack aware placement strategy reads are keept within a data center. See Will query all replicas and return the record with the most recent timestamp once all replicas have replied. Any unresponsive replicas will fail the operation. 35/63

36 Read operation <<storage model Query Client Result Single read Cassandra Cluster Closest replica Result Replica A Read repair if digests differ Digest Response Digest Query Digest Response Replica B Replica C 36/63

37 Cassandra HBase 一致性 Quorum NRW 策略, 最终一致性单节点, 无复制, 强一致性可用性伸缩性负载均衡 1, 基于 Consistent Hash 相邻节点复制数据, 数据存在于多个节点, 无单点故障 2, 某节点宕机,hash 到该节点的新数据自动路由到下一节点做 hinted handoff, 源节点恢复后, 推送回源节点 3, 通过 Gossip 协议维护集群所有节点的健康状态, 并发送同步请求, 维护数据一致性 4,SSTable, 纯文件, 单机可靠性一般 1, 存在单点故障,Region Server 宕机后, 短时间内该 Server 维护的 Region 无法访问, 等待 failover 生效 2, 通过 Master 维护各 Region Server 健康状况和 Region 分布 3, 多个 Master,Master 宕机有 zookeeper 的 Paxos 投票机制选取下一任 Master Master 就算全宕机, 也不影响 Region 读写 4,HDFS, 一备三, 高可靠,0 数据丢失 5,HDFS 的 Namenode 是一个 SPOF 1,Consistent Hash, 快速定位数据所在节点 1, 通过 Zookeeper 定位目标 Region Server, 最后定位 2, 扩容需在 Hash Ring 上多个节点间调整数据 Region 分布 2,Region Server 扩容, 由 Master 均匀分布请求 Zookeeper 取得整个集群地址, 然后根据请求 Zookeeper 取读写数据路由表定位 Region Server Consistent Hash 选择合适的节点 client 会缓,Master 会修改这个路由表 Client 自身也会缓存一存集群地址部分路由信息数据差异 Merkle Tree, Bloom Filter 比较算法锁与事务 Client Timestap(Dynamo 使用 vector lock) 读写性能数据读写定位非常快 1, 最终一致性, 数据可能丢失 CAP 点评 2, 可用性高 3, 扩容方便 Bloom Filter Optimistic Concurrency Control 数据读写定位可能要通过最多 6 次的网络 RPC 1, 强一致性,0 数据丢失 2, 可用性低 3, 扩容方便

38 YCSB(3/8) << Yahoo! Cloud Serving Benchmark Workload a -- update heavy 50/50 Read/update Cassandra is optimized for writes, and achieves higher throughput and lower latency. Sherpa and MySQL achieve roughly comparable performance, as both are limited by MySQL s capabilities. HBase has good write latency, because of commits to memory, and somewhat higher read latency, because of the need to reconstruct records. 38/63

39 YCSB(4/8) << Yahoo! Cloud Serving Benchmark Workload b -- read heavy 95/5 Read/update Sherpa does very well here, with better read latency only one lookup into a Btree is needed for reads, unlike log-structured systems where records must be reconstructed. Cassandra also performs well, matching Sherpa until high throughputs. HBase does well also, although read time is higher. 39/63

40 YCSB(5/8) << Yahoo! Cloud Serving Benchmark Workload e -- short scans Scans of records of size 1KB HBase and Sherpa are roughly equivalent for latency and peak throughput, even though HBase is meant for scans. Cassandra s performance is poor, but the development team notes that many optimizations still need to be done. 40/63

41 YCSB(6/8) << Yahoo! Cloud Serving Benchmark Scale up -Read heavy workload with varying hardware Sherpa and Casandra scale well, with flat latency as system size increases. HBase is very unstable; 3 servers or less performs very poorly. 41/63

42 Thank you!

PowerPoint 演示文稿

PowerPoint 演示文稿 Hadoop 生态技术在阿里全网商品搜索实战阿里巴巴 - 王峰自我介绍真名 : 王峰淘宝花名 : 莫问微博 : 淘莫问 2006 年硕士毕业后加入阿里巴巴集团淘及搜索事业部 ( 高级技术与家 ) 目前负责搜索离线系统团队技术方向 : 分布式计算