2005 3

Size: px

Start display at page:

Download "2005 3"

溆哥仇
7 years ago
Views:

1 Text

2 2005 3

3 2.8M 1/4 20M / 500~600/sec 23 PC (1U*15/2U*8) 12 38G memcached

4 1U (frodo) AMD Athlon GHz 1G 160G SATA*2 Gentoo Linux MySQL 5 Quixote (a Python web framework) Lighttpd + SCGI (shire) Memcached (!)

5 Internet Lighttpd SCGI App FS MySQL Memcache Static Files

6 Gentoo Linux emerge mysql ebuild patch GLSA(Gentoo Linux Security Advisories)

7 MySQL The world s most popular open source database / ==> MyISAM ==> InnoDB Replicate for backup

8 Python Battery Included CPUG:

9 Quixote REST URL Django, TurboGears, Pylons ZOPE # luz/subject/ init.py def _q_lookup(request, name): subject = get_subject(name) return lambda req: subject_ui(req, subject) # luz/subject/subject_ui.ptl def subject_ui [html] (request, subject): site_header(request) <h1>%s</h1> % subject.title site_footer(request)

10 Lighttpd SCGI SCGI: FastCGI Quixote 80 lighttpd SCGI localhost Quixote

11 Memcache MySQL libmemcache python Pyrex python 3x+ def get_subject(subject_id): subject = mc.get( s: +subject_id) if subject is None: store.farm.execute( select xxx, xxx from subject where id=%s, subject_id) subject = Subject(*store.farm.fetchone()) mc.set( s: +subject_id, subject) return subject

12 1.2M / IO

13 1U pippin meriadoc ( merry), 4G 250G SATA*3 IP DNS IP -_-b frodo (subversion, trac, etc...)

14 Internet Lighttpd (#$) SCGI HTTP Proxy DNS App Lighttpd (!") FS Static Files Memcache MySQL

15 innodb_buffer_pool_size IP

16 1.5M / IP

17 IP(BGP) (arwen) 74G 1w SATA * 3

18 Internet Lighttpd Data Mining SCGI write read App Replicate MySQL Master MySQL Slave Static Files Memcache

19 2M / IO, etc...

20 4G 250G SATA* mod_rewrite URL lighttpd mod_memcache IO web

21 Internet App store.farm SCGI MySQL Master Lighttpd Memcache store.farmr Replicate HTTP Proxy WebDAV Web Service Lighttpd (w/ mod_memcache) Replicate Lighttpd WebDAV Spiders MySQL Slave write Static Files Memcache!"#$% MySQL Slave read Data Mining

22 store farmr replicate delay cache... but it works

23 replicate delay def get_subject(sid): sbj = mc.get( s: +sid) if sbj is None: sbj = flush_subject(sid, store.farmr) return sbj def flush_subject(sid, cursor=none): cursor = cursor or store.farm cursor.execute( select... from subject ) subject = Subject(*cursor.fetchone()) mc.set( s: +sid, subject) return subject def update_subject(subject, props): store.farm.execute( update subject... ) store.farm.connection.commit() flush_subject(subject.id, store.farm)

24 2.5M / / SATA

25 Scale Up 1U 16G 147G SCSI * G SATA SCSI RAID-0 MySQL Slave memcached MyISAM InnoDB Sphinx

26 Internet Sphinx Web Service store.farm MySQL Master Lighttpd Replicate SCGI HTTP Proxy App Lighttpd (w/ mod_memcache) WebDAV Memcache store.farmr Memcache Lighttpd WebDAV Web Service Static Files MySQL Slave Spiders Memcache Memcache

27 5.2M / Web IO

28 :) 3 1U 4 32G 1T SATA * 3 otho.douban.com lotho.douban.com lighttpd 1.5 with aio support LVS Scale Up: 4G -> 8G

29 Internet otho.douban.com Lighttpd HTTP Proxy LVS LB (Master) Lighttpd 1.5 (w/ mod_cache) Lighttpd Keepalived Lighttpd WebDAV LVS LB (backup) Lighttpd 1.5 (w/ mod_cache) Static Files

30 write MySQL Master replicate!"#$% MySQL Slave read Data Mining Replicate write read MySQL Slave!"#$% Data Mining

31 6.4M / (5M PV) CPU memcache /

32 lighttpd mod_scgi round-robin lighttpd 1.5 mod_proxy proxy.balance = fair (load based, passive balancing) spread

33 Internet Lighttpd HTTP Proxy SCGI Lighttpd App Memcache HTTP Proxy Lighttpd HTTP Proxy spread Static Files Log Aggregator Lighttpd WebDAV spread Lighttpd SCGI App Memcache

34 11M / 3 Sphinx load

35 Sphinx -> Xapian MogileFS

36 libmemcache -> libmemcached consistent hash memcache libmemcached consistent hash bug CPU libmemcached failover bug nginx lighttpd load balance spread nginx

37 !" Master replicate replicate #$ Master replicate %&'()!" Slave #$ Slave read read Data Mining write!" Slave *+,- Master replicate *+,- Slave write replicate #$ Slave %&'() Data Mining

38 store.farm[r] -> store.get_cursor(table= xxx, ro=true/false) def flush_subject(sid, ro=false): cursor = store.get_cursor(table= subject, ro=ro) cursor.execute( select... from subject ) subject = Subject(*cursor.fetchone()) mc.set( s: +sid, subject) return subject

39 Internet upload.douban.com otho.douban.com signed POST form Uploader Lighttpd 1.5 (w/ mod_cache) App HTTP redirect MogileFS Tracker FileStorage Gateway MogileFS Master MogileFS Node MogileFS Node

40 Internet Lighttpd SCGI App Lighttpd HTTP Proxy HTTP Proxy HTTP Proxy Nginx HTTP Proxy Lighttpd SCGI App Lighttpd HTTP Proxy Static Files Lighttpd WebDAV Lighttpd SCGI App

41 13M / MogileFS Tracker DB

42 8 32G CPU (300G SCSI 2 + 1T SATA) 3 (1T SATA 3) DoubanFS

43 DoubanFS hash hash hash Merkle Tree consistent hash WebDAV MogileFS 3 50

44 Consistent Hash

45 Merkle Tree

46 Internet upload.douban.com otho.douban.com signed POST form Uploader Lighttpd 1.5 (w/ mod_cache) App HTTP redirect FileStorage Gateway DoubanFS Node DoubanFS Node

47 16M / DoubanFS IO

48 DoubanDB MySQL MySQL Master failover replicate delay

49 DoubanDB Key-Value Amazon Dynamo set(key, value), get(key), delete(key) memcache Merkle Tree Consistent Hash TokyoCabinet DoubanDB DoubanFS 2.0 DoubanDB

51 !" Master1 replicate replicate #$ Master1 replicate %&'()!" Slave #$ Slave read read Data Mining write!" Master2 *+,- Master replicate *+,- Slave write replicate #$ Master2 %&'() Data Mining

52 DoubanFS ngnix LVS RabbitMQ spread

53 profile memcache cache join

A API Application Programming Interface 见应用程序编程接口 ARP Address Resolution Protocol 地址解析协议为 IP 地址到对应的硬件地址之间提供动态映射阿里云内

A API Application Programming Interface 见应用程序编程接口 ARP Address Resolution Protocol 地址解析协议为 IP 地址到对应的硬件地址之间提供动态映射阿里云内容分发网络 Alibaba Cloud Content Delivery Network 一