Text 2009.4 hongqn@douban.com
2005 3
2.8M 1/4 20M / 500~600/sec 23 PC (1U*15/2U*8) 12 38G memcached
1U (frodo) AMD Athlon 64 1.8GHz 1G 160G SATA*2 Gentoo Linux MySQL 5 Quixote (a Python web framework) Lighttpd + SCGI (shire) Memcached (!)
Internet Lighttpd SCGI App FS MySQL Memcache Static Files
Gentoo Linux emerge mysql ebuild patch GLSA(Gentoo Linux Security Advisories)
MySQL The world s most popular open source database / ==> MyISAM ==> InnoDB Replicate for backup
Python Battery Included CPUG: http://python.cn/
Quixote REST URL Django, TurboGears, Pylons ZOPE http://www.douban.com/subject/1000001 # luz/subject/ init.py def _q_lookup(request, name): subject = get_subject(name) return lambda req: subject_ui(req, subject) # luz/subject/subject_ui.ptl def subject_ui [html] (request, subject): site_header(request) <h1>%s</h1> % subject.title site_footer(request)
Lighttpd SCGI SCGI: FastCGI Quixote 80 lighttpd SCGI localhost Quixote
Memcache MySQL libmemcache python Pyrex python 3x+ def get_subject(subject_id): subject = mc.get( s: +subject_id) if subject is None: store.farm.execute( select xxx, xxx from subject where id=%s, subject_id) subject = Subject(*store.farm.fetchone()) mc.set( s: +subject_id, subject) return subject
1.2M / IO
1U pippin meriadoc ( merry), 4G 250G SATA*3 IP DNS IP -_-b frodo (subversion, trac, etc...)
Internet Lighttpd (#$) SCGI HTTP Proxy DNS App Lighttpd (!") FS Static Files Memcache MySQL
innodb_buffer_pool_size IP
1.5M / IP
IP(BGP) (arwen) 74G 1w SATA * 3
Internet Lighttpd Data Mining SCGI write read App Replicate MySQL Master MySQL Slave Static Files Memcache
2M / IO, etc...
4G 250G SATA*3 10000 mod_rewrite URL lighttpd mod_memcache IO web
Internet App store.farm SCGI MySQL Master Lighttpd Memcache store.farmr Replicate HTTP Proxy WebDAV Web Service Lighttpd (w/ mod_memcache) Replicate Lighttpd WebDAV Spiders MySQL Slave write Static Files Memcache!"#$% MySQL Slave read Data Mining
store farmr replicate delay cache... but it works
replicate delay def get_subject(sid): sbj = mc.get( s: +sid) if sbj is None: sbj = flush_subject(sid, store.farmr) return sbj def flush_subject(sid, cursor=none): cursor = cursor or store.farm cursor.execute( select... from subject ) subject = Subject(*cursor.fetchone()) mc.set( s: +sid, subject) return subject def update_subject(subject, props): store.farm.execute( update subject... ) store.farm.connection.commit() flush_subject(subject.id, store.farm)
2.5M / / SATA
Scale Up 1U 16G 147G SCSI *2 + 500G SATA SCSI RAID-0 MySQL Slave memcached MyISAM InnoDB Sphinx
Internet Sphinx Web Service store.farm MySQL Master Lighttpd Replicate SCGI HTTP Proxy App Lighttpd (w/ mod_memcache) WebDAV Memcache store.farmr Memcache Lighttpd WebDAV Web Service Static Files MySQL Slave Spiders Memcache Memcache
5.2M / Web IO
:) 3 1U 4 32G 1T SATA * 3 otho.douban.com lotho.douban.com lighttpd 1.5 with aio support LVS Scale Up: 4G -> 8G
Internet www.douban.com otho.douban.com Lighttpd HTTP Proxy LVS LB (Master) Lighttpd 1.5 (w/ mod_cache) Lighttpd Keepalived Lighttpd WebDAV LVS LB (backup) Lighttpd 1.5 (w/ mod_cache) Static Files
write MySQL Master replicate!"#$% MySQL Slave read Data Mining Replicate write read MySQL Slave!"#$% Data Mining
6.4M / (5M PV) CPU memcache /
lighttpd mod_scgi round-robin lighttpd 1.5 mod_proxy proxy.balance = fair (load based, passive balancing) spread
Internet Lighttpd HTTP Proxy SCGI Lighttpd App Memcache HTTP Proxy Lighttpd HTTP Proxy spread Static Files Log Aggregator Lighttpd WebDAV spread Lighttpd SCGI App Memcache
11M / 3 Sphinx load
Sphinx -> Xapian MogileFS
libmemcache -> libmemcached consistent hash memcache libmemcached consistent hash bug CPU libmemcached failover bug nginx lighttpd load balance spread nginx
!" Master replicate replicate #$ Master replicate %&'()!" Slave #$ Slave read read Data Mining write!" Slave *+,- Master replicate *+,- Slave write replicate #$ Slave %&'() Data Mining
store.farm[r] -> store.get_cursor(table= xxx, ro=true/false) def flush_subject(sid, ro=false): cursor = store.get_cursor(table= subject, ro=ro) cursor.execute( select... from subject ) subject = Subject(*cursor.fetchone()) mc.set( s: +sid, subject) return subject
Internet upload.douban.com otho.douban.com signed POST form Uploader Lighttpd 1.5 (w/ mod_cache) App HTTP redirect MogileFS Tracker FileStorage Gateway MogileFS Master MogileFS Node MogileFS Node
Internet Lighttpd SCGI App Lighttpd HTTP Proxy HTTP Proxy HTTP Proxy Nginx HTTP Proxy Lighttpd SCGI App Lighttpd HTTP Proxy Static Files Lighttpd WebDAV Lighttpd SCGI App
13M / MogileFS Tracker DB
8 32G CPU (300G SCSI 2 + 1T SATA) 3 (1T SATA 3) 5 6 2 DoubanFS
DoubanFS hash hash hash Merkle Tree consistent hash WebDAV MogileFS 3 50
Consistent Hash
Merkle Tree
Internet upload.douban.com otho.douban.com signed POST form Uploader Lighttpd 1.5 (w/ mod_cache) App HTTP redirect FileStorage Gateway DoubanFS Node DoubanFS Node
16M / DoubanFS IO
DoubanDB MySQL MySQL Master failover replicate delay
DoubanDB Key-Value Amazon Dynamo set(key, value), get(key), delete(key) memcache Merkle Tree Consistent Hash TokyoCabinet DoubanDB DoubanFS 2.0 DoubanDB
!" Master1 replicate replicate #$ Master1 replicate %&'()!" Slave #$ Slave read read Data Mining write!" Master2 *+,- Master replicate *+,- Slave write replicate #$ Master2 %&'() Data Mining
DoubanFS ngnix www.douban.com LVS RabbitMQ spread
profile memcache cache join