W. Richard Stevens UNIX Sockets API echo Sockets TCP OOB IO C struct C/C++ UNIX fork() select(2)/poll(2)/epoll(4) IO IO CPU 100% libevent IO UNIX CPU

Linux muduo C++ (giantchen@gmail.com) C++ TCP C++ x86-64 Linux TCP one loop per thread Linux native muduo C++ IT 5 C++ muduo 2 C++ C++ Primer 4

W. Richard Stevens UNIX Sockets API echo Sockets TCP OOB IO C struct C/C++ UNIX fork() select(2)/poll(2)/epoll(4) IO IO CPU 100% libevent IO UNIX CPU IO UNIX 2 IPC 7 24 C++ STL C++ Linux muduo C++

C++ x86-64 Linux TCP 5 IO one loop per thread Linux native muduo muduo IO C++ one loop per thread IO muduo Linux A C++ C++11 2005 TR1 C++ C++ C++ 1 449 C++ UNIX UNIX C++ Primer Pthreads Sockets API C++11 2012 Linux g++ 4.4 C++11 x86-64 CPU GB 5 IO 1 C++ C++ 2 muduo muduo 3 C++ 4 C++ UDP iii

iv Observer Reactor Singleton class heap event loop STL algorithm C++ instance resolution instantiation override dereference class 6 syscall (s) fd file descriptor CPU core kb MB GB 10 3 10 6 10 9 KiB MiB GiB 2 10 2 20 2 30 11.5 11.5 L42 42 [JCP] [CC2e] mutex socket C++ class this mutable class fork(2) muduo::eventloop fork(2) fork() manpage 2 man 2 fork - EventLoop- ThreadPool URL muduo/base/date.h http://chenshuo.com GitHub http://github.com/ chenshuo/recipes/ http://github.com/chenshuo/muduo/ recipes/thread GitHub URL muduo/base/types.h 1 muduo::string typedef 15 namespace muduo 16 { 17 18 #ifdef MUDUO_STD_STRING 19 using std::string; 20 #else //!MUDUO_STD_STRING 21 typedef gnu_cxx:: sso_string string; 22 #endif muduo/base/types.h muduo/base/types.h 1 6 7 muduo muduo/examples/xxx examples/xxx 8 recipes/reactor/xxx reactor/xxx

v diff -u giantchen@gmail.com http://chenshuo.com/book URL http://weibo.com/giantchen http://blog.csdn.net/solstice http://github.com/chenshuo http://book.douban.com/subject/20471211

1 C++ 1 1...................... 3 2.............................. 31 3.............. 59 4 C++........................ 83 5............................ 107 2 muduo 123 6 muduo........................... 125 7 muduo............................. 177 8 muduo....................... 277 3 337 9.......................... 339 10 C++......................... 391 11 C++...................... 429 12 C++................................ 501 4 559 A........................ 561 B C++ Primer 4 C++............. 579 C Boost............................ 591 D TCP............... 593....................................... 599 vi

1 C++ 1 1 3 1.1............................ 3 1.1.1............................ 4 1.1.2 MutexLock MutexLockGuard.................... 4 1.1.3 Counter.................... 4 1.2............................... 5 1.3.................................... 7 1.3.1 mutex............................ 7 1.3.2 mutex............... 8 1.4 Observer......................... 8 1.5............................... 11 1.6 shared_ptr/weak_ptr.......................... 13 1.7....................... 14 1.8 Observer.............................. 16 1.9 shared_ptr......................... 17 1.10 shared_ptr............................ 19 1.11..................................... 21 1.11.1 enable_shared_from_this...................... 23 1.11.2................................. 24 1.12.................................... 26 1.13................................... 26 1.14 Observer................................. 28 2 31 2.1 mutex............................... 32 vii

viii 2.1.1 mutex........................ 33 2.1.2.................................. 35 2.2 condition variable....................... 40 2.3............................ 43 2.4 MutexLock MutexLockGuard Condition.............. 44 2.5 Singleton.......................... 48 2.6 sleep(3)............................ 50 2.7................................... 51 2.8 shared_ptr copy-on-write...................... 52 3 59 3.1................................... 59 3.2........................ 61 3.3........................ 62 3.3.1 one loop per thread.......................... 62 3.3.2................................. 63 3.3.3................................ 64 3.4 TCP............................. 65 3.5........................... 67 3.5.1......................... 69 3.5.2......................... 70 3.5.3........................ 71 3.6.................. 74 4 C++ 83 4.1............................. 84 4.2 C/C++......................... 85 4.3 Linux.............................. 89 4.4........................... 91 4.4.1 pthread_cancel C++....................... 94 4.4.2 exit(3) C++................. 94 4.5 thread............................. 96 4.6 IO.................................. 98

ix 4.7 RAII........................... 99 4.8 RAII fork()................................. 101 4.9 fork()............................... 102 4.10 signal................................ 103 4.11 Linux.......................... 105 5 107 5.1.................................... 109 5.2.................................... 112 5.3................................ 114 5.4.................................... 120 2 muduo 123 6 muduo 125 6.1....................................... 125 6.2....................................... 127 6.3.................................... 129 6.3.1................................ 131 6.3.2.................................. 134 6.3.3................................ 135 6.4.................................... 136 6.4.1 TCP......................... 136 6.4.2 echo........................... 138 6.4.3 finger......................... 140 6.5.................................... 144 6.5.1 muduo Boost.Asio libevent2.......... 145 6.5.2 muduo libevent2...... 148 6.5.3 muduo Nginx................... 153 6.5.4 muduo ZeroMQ................... 156 6.6 muduo........................... 157 6.6.1............................ 157 6.6.2................. 160

x 7 muduo 177 7.1 TCP.............................. 178 7.2.................................... 185 7.3 Boost.Asio........................... 194 7.3.1 TCP................................ 194 7.3.2................................ 195 7.3.3 LengthHeaderCodec.................... 197 7.3.4............................. 198 7.3.5............................. 200 7.4 muduo Buffer....................... 204 7.4.1 muduo IO.......................... 204 7.4.2 non-blocking buffer.... 205 7.4.3 Buffer.......................... 207 7.4.4 Buffer.......................... 209 7.4.5 Buffer............................. 211 7.4.6............................. 217 7.4.7............................ 218 7.5 Google Protobuf........ 220 7.5.1 Protobuf............. 220 7.5.2 type name Message............ 221 7.5.3 Protobuf.......................... 226 7.6 muduo Protobuf............ 228 7.6.1 codec...................... 229 7.6.2 ProtobufCodec......................... 232 7.6.3 dispatcher................ 232 7.6.4 ProtobufCodec ProtobufDispatcher....... 233 7.6.5 ProtobufDispatcher.................. 234 7.6.6 ProtobufCodec ProtobufDispatcher......... 236 7.7........................ 237 7.7.1...................... 237 7.7.2 muduo.................... 238

xi 7.8..................................... 240 7.8.1............................. 240 7.8.2 Linux............................ 241 7.8.3 muduo......................... 242 7.8.4 Boost.Asio Timer........................ 243 7.8.5 Java Netty............................ 245 7.9..................... 248 7.10 timing wheel........................ 250 7.10.1 timing wheel.......................... 251 7.10.2............................ 254 7.11............................. 257 7.12.................. 260 7.13 socks4a.............................. 264 7.13.1 TCP.............................. 264 7.13.2 socks4a.......................... 267 7.13.3 N : 1 1 : N........................ 267 7.14.................................... 267 7.15................................. 268 7.15.1 UDNS................................. 270 7.15.2 c-ares DNS............................... 272 7.15.3 curl................................... 273 7.15.4.................................. 275 8 muduo 277 8.0 EventLoop........................... 277 8.1 Reactor.............................. 280 8.1.1 Channel class............................. 280 8.1.2 Poller class.............................. 283 8.1.3 EventLoop........................... 287 8.2 TimerQueue.............................. 290 8.2.1 TimerQueue class........................... 290 8.2.2 EventLoop........................... 292

xii 8.3 EventLoop::runInLoop()........................ 293 8.3.1 TimerQueue................... 296 8.3.2 EventLoopThread class........................ 297 8.4 TCP................................ 299 8.5 TcpServer............................ 303 8.5.1 TcpServer class............................ 304 8.5.2 TcpConnection class......................... 305 8.6 TcpConnection........................... 308 8.7 Buffer................................ 313 8.7.1 TcpConnection Buffer............. 314 8.7.2 Buffer::readFd().......................... 315 8.8 TcpConnection........................... 316 8.9 TcpConnection.............................. 320 8.9.1 SIGPIPE................................ 321 8.9.2 TCP No Delay TCP keepalive.................. 321 8.9.3 WriteCompleteCallback HighWaterMarkCallback....... 322 8.10 TcpServer............................... 324 8.11 Connector................................... 327 8.12 TcpClient................................... 332 8.13 epoll...................................... 333 8.14................................. 336 3 337 9 339 9.1........................... 341 9.1.1........................ 343 9.1.2..................... 344 9.2........................... 349 9.2.1 7 24................ 352 9.2.2............... 354 9.3........................ 356

xiii 9.4........................... 360 9.4.1................................ 361 9.4.2................................ 362 9.4.3 TCP............................ 363 9.5......................... 364 9.6............................... 367 9.6.1.......................... 368 9.6.2 ICE................... 369 9.7........................ 370 9.7.1......................... 370 9.7.2........................ 373 9.7.3........................ 374 9.7.4..................... 375 9.7.5................................ 379 9.8............... 380 9.8.1 1......................... 382 9.8.2 2........... 383 9.8.3 3............. 386 9.8.4 4 naming service............. 389 10 C++ 391 10.1 C......................... 394 10.1.1 C...................... 395 10.1.2 C........................... 398 10.2 C++................................ 399 10.2.1................................ 399 10.2.2................................ 402 10.3 C++ linking............................. 404 10.3.1................................ 406 10.3.2 inline............................... 407 10.3.3.................................. 409 10.3.4................................. 414

xiv 10.4........................ 415 10.4.1............................. 416 10.4.2.......................... 417 10.5........................ 418 10.5.1............................ 423 10.5.2........................ 424 10.5.3............................ 428 11 C++ 429 11.1 C++................................ 429 11.2............................ 431 11.2.1......................... 432 11.2.2 ABI..................... 433 11.2.3........................ 435 11.2.4 COM........................... 435 11.2.5................................ 436 11.3........................ 436 11.3.1 C++.................... 437 11.3.2.................. 438 11.3.3........................ 439 11.3.4 Linux COM............ 442 11.3.5 Java.......................... 443 11.4............................ 443 11.5 boost::function boost::bind............. 447 11.5.1................................ 450 11.5.2............................ 451 11.5.3..................... 453 11.6 iostream............................ 457 11.6.1 stdio..................... 457 11.6.2 iostream......................... 461 11.6.3 iostream................. 463

xv 11.6.4 iostream..................... 464 11.6.5 iostream..................... 468 11.6.6 300 memory buffer output stream........... 476 11.6.7 C++ IO................... 480 11.7............................... 482 11.7.1............................. 482 11.7.2............................ 483 11.7.3............................ 488 11.7.4 C++.......................... 488 11.7.5............................ 490 11.7.6...................... 493 11.7.7............................ 495 12 C++ 501 12.1......................... 501 12.1.1..................... 503 12.1.2...................... 505 12.2 ::operator new()...................... 507 12.2.1......................... 507 12.2.2 ::operator new()................... 508 12.2.3 ::operator new()................. 508 12.2.4............................ 509 12.2.5 ::operator new()................... 510 12.2.6 malloc()...................... 512 12.2.7 class ::operator new()......... 513 12.2.8.................... 513 12.3........................... 514 12.3.1............................ 515 12.3.2 C/C++......................... 516 12.3.3............................ 516 12.3.4......................... 517 12.3.5................................ 521

xvi 12.4 mock........................ 522 12.4.1......................... 522 12.4.2 link seam...................... 524 12.5 namespace............................. 526 12.5.1 C static................. 526 12.5.2 C++ static................ 526 12.5.3 namespace..................... 527 12.5.4................................ 529 12.6....................... 529 12.6.1 diff........................ 530 12.6.2 grep....................... 537 12.6.3............................. 538 12.7 std::string............................... 539 12.7.1 eager copy....................... 540 12.7.2 copy-on-write..................... 542 12.7.3 SSO........................ 543 12.8 STL algorithm................ 546 12.8.1 next_permutation().............. 546 12.8.2 unique()................... 548 12.8.3 {make,push,pop}_heap()............ 549 12.8.4 partition().... 553 12.8.5 lower_bound() IP............. 554 4 559 A 561 B C++ Primer 4 C++ 579 C Boost 591 D TCP 593 599

1 C++

1 synchronization primitives mutex race condition C++ Boost shared_ptr weak_ptr 1 Observer 2009 12 C++ C++ Observer 1.1 C++ race condition race condition C++ shared_ptr C++ 1 class TR1 std::tr1 C++11 3

4 1 1.1.1 [JCP] class interleaving C++ class std:: string std::vector std::map class 1.1.2 MutexLock MutexLockGuard C++ 2.4 MutexLock critical section RAII [CCS 13] Windows struct CRITI- CAL_SECTION Linux pthread_mutex_t 2 MutexLock class MutexLockGuard MutexLockGuard class 2.1 1.1.3 Counter class Counter 1 // A thread-safe counter 2 class Counter : boost::noncopyable 3 { 4 // copy-ctor and assignment should be private by default for a class. 5 public: 6 Counter() : value_(0) {} 2 2.1.1

1.2 5 7 int64_t value() const; 8 int64_t getandincrease(); 9 10 private: 11 int64_t value_; 12 mutable MutexLock mutex_; 13 }; 14 15 int64_t Counter::value() const 16 { 17 MutexLockGuard lock(mutex_); // lock 18 return value_; // 19 } 20 21 int64_t Counter::getAndIncrease() 22 { 23 MutexLockGuard lock(mutex_); 24 int64_t ret = value_++; 25 return ret; 26 } 27 // In a real world, atomic operations are preferred. 28 // class class Counter mutex_ lock contention L24 Counter mutex_ mutable const Counter::value() non-const mutex_ mutex_ static / Counter Counter race condition 1.2 this this this escape

6 1 // Don't do this. class Foo : public Observer // Observer 10 { public: Foo(Observable* s) { s->register_(this); // } virtual void update(); }; // Do this. class Foo : public Observer { public: Foo(); virtual void update(); // void observe(observable* s) { s->register_(this); } }; Foo* pfoo = new Foo; Observable* s = getsubject(); pfoo->observe(s); // s->register_(pfoo); +initialize() C++ initialize() this Foo Foo::Foo() most-derived class

1.3 7 1.3 3 mutex 1.3.1 mutex mutex (1) (2) Foo::~Foo() { MutexLockGuard lock(mutex_); // free internal state (1) } void Foo::update() { MutexLockGuard lock(mutex_); // (2) // make use of internal state } A B Foo x A x B x->update() extern Foo* x; // visible by all threads // thread A delete x; x = NULL; // helpless // thread B if (x) { x->update(); } A NULL B x x race condition 1. A (1) 2. B if (x) (2) mutex_ (2) core dump delete NULL 3 dangling pointer wild pointer http://en.wikipedia.org/wiki/dangling_pointer

8 1 1.3.2 mutex class MutexLock class MutexLock MutexLock 1.1 class swap() void swap(counter& a, Counter& b) { MutexLockGuard alock(a.mutex_); MutexLockGuard block(b.mutex_); int64_t value = a.value_; a.value_ = b.value_; b.value_ = value; } // potential dead lock A swap(a, b); B swap(b, a); operator=() Counter& Counter::operator=(const Counter& rhs) { if (this == &rhs) return *this; } MutexLockGuard mylock(mutex_); // potential dead lock MutexLockGuard itslock(rhs.mutex_); value_ = rhs.value_; // value_ = rhs.value() return *this; mutex mutex 1.4 Observer

1.4 Observer 9 [CCS 99] free(3) C/C++ 4 composition aggregation association composition / x owner owner x x owner scoped_ptr owner C++ association / a b a b b a aggregation association a b b 1.1 A x B x lock contention x x race condition 4 Java reference null

10 1 Observer recipes/thread/test/observer.cc 1 class Observer // : boost::noncopyable 2 { 3 public: 4 virtual ~Observer(); 5 virtual void update() = 0; 6 //... 7 }; 8 9 class Observable // : boost::noncopyable 10 { 11 public: 12 void register_(observer* x); 13 void unregister(observer* x); 14 15 void notifyobservers() { 16 for (Observer* x : observers_) { // C++11 17 x->update(); // (3) 18 } 19 } 20 private: 21 std::vector<observer*> observers_; 22 }; Observable Observer (L17) Observer x Observer unregister() 23 class Observer 24 { 25 // 26 void observe(observable* s) { 27 s->register_(this); 28 subject_ = s; 29 } 30 31 virtual ~Observer() { 32 subject_->unregister(this); 33 } 34 35 Observable* subject_; 36 }; Observer unregister(this) race conditions L32 subject_ subject_ 1. A L32 unregister 2. B L17 x L32

1.5 11 x Observer 5 Observer L32 core dump race condition isalive() 1.5 raw pointer Observable Observer* Observer Observer race condition subject_ Observable* C++ shared_ptr p1 p2 Object p1 p2 1-1 A p1 p1 NULL p2 1-1 C/C++ p1 p2 Object p1=0 p2 Object 1-1 5 C++

12 1 p1 p2 1-2 proxy Object C p1 p2 p1 proxy Object p2 1-2 Object proxy 0 1-3 p2 proxy Object p1=0 proxy=0 Object p2 1-3 Object race condition p2 proxy Object p1 proxy proxy reference counting p1 p2 sp1 sp2 proxy 1. 2 1-4 sp1 sp2 pointer count = 2 1-4 Object

1.6 shared_ptr/weak_ptr 13 2. sp1 1 1-5 sp1 sp2 pointer count = 1 1-5 Object 3. sp2 0 proxy Object 1-6 sp1 sp2 pointer count = 0 1-6 Object another layer of indirection 6 Object handle/body idiom handle 7 C++ TR1 1.6 shared_ptr/weak_ptr shared_ptr Boost std::tr1 C++11 C++ shared_ptr<t> class template 6 http://en.wikipedia.org/wiki/abstraction_layer 7 Edsger W. Dijkstra The Humble Programmer http://www.cs.utexas.edu/~ewd/transcriptions/ewd03xx/ewd340.html

14 1 0 weak_ptr weak shared_ptr shared_ptr shared_ptr x shared_ptr x x shared_ptr reset() x weak_ptr promote shared_ptr shared_ptr /lock() shared_ptr/weak_ptr shared_ptr/weak_ptr std::string STL 8 C++ clean up 1.7 9 C Nginx C 10 C++ 2004 C++ 8 http://blog.csdn.net/myan/article/details/1906 9 Java C http://blog.csdn.net/myan/article/details/1482614 10 http://trac.nginx.org/nginx/ticket/{134,135,162}

1.7 15 C++ 1. buffer overrun 2. / 3. double delete 4. memory leak 5. new[]/delete 6. memory fragmentation 5 6 9.2.1 A.1.8 1. std::vector<char>/std::string Buffer class 2. / shared_ptr/weak_ptr 3. scoped_ptr 4. scoped_ptr 5. new[]/delete new[] std::vector/scoped_array std:: vector/std::list std::string C++ delete security data safety scoped_ptr/shared_ptr/weak_ptr shared_ptr<foo>* pfoo = new shared_ptr<foo>(new Foo); // WRONG semantic x T incomplete x.cpp 10.3.2

16 1 1.8 Observer weak_ptr Observer Observable weak_ptr<observer> recipes/thread/test/observer_safe.cc 39 class Observable // not 100% thread safe! 40 { 41 public: 42 void register_(weak_ptr<observer> x); // const weak_ptr<observer>& 43 // void unregister(weak_ptr<observer> x); // 44 void notifyobservers(); 45 46 private: 47 mutable MutexLock mutex_; 48 std::vector<weak_ptr<observer> > observers_; 49 typedef std::vector<weak_ptr<observer> >::iterator Iterator; 50 }; 51 52 void Observable::notifyObservers() 53 { 54 MutexLockGuard lock(mutex_); 55 Iterator it = observers_.begin(); // Iterator 49 56 while (it!= observers_.end()) 57 { 58 shared_ptr<observer> obj(it->lock()); // 59 if (obj) 60 { 61 // 2 62 obj->update(); // obj 63 ++it; 64 } 65 else 66 { 67 // weak_ptr 68 it = observers_.erase(it); 69 } 70 } 71 } recipes/thread/test/observer_safe.cc (3) p. 10 L17 L48 vector<shared_ptr<observer> > observers_; Observer* weak_ptr<observer> Observer 1.14

1.9 shared_ptr 17 Observer shared_ptr Observer subject_->unregister(this) subject_ Observable shared_ptr subject_ weak_ptr<observable> lock contention Observable register_() unregister() notifyobservers() update() register_() unregister() L62 update() (un)register mutex_ mutex_ core dump vector observers_ mutex_ std::list ++it mutex Pthreads mutex 2.1.1 Java intrinsic lock synchronized synchronized 1.9 shared_ptr shared_ptr shared_ptr 100% shared_ptr 11 shared_ptr std::string shared_ptr 1 shared_ptr 2 shared_ptr 3~5 shared_ptr 11 http://www.boost.org/doc/libs/release/libs/smart_ptr/shared_ptr.htm#threadsafety

18 1 shared_ptr mutex MutexLock mutex; // ReadersWriterLock SpinLock shared_ptr<foo> globalptr; // globalptr doit() void doit(const shared_ptr<foo>& pfoo); globalptr race condition http://blog.csdn.net/solstice/article/details/8547547 globalptr void read() { shared_ptr<foo> localptr; { MutexLockGuard lock(mutex); localptr = globalptr; // read globalptr } // use localptr since here localptr doit(localptr); } void write() { shared_ptr<foo> newptr(new Foo); // { MutexLockGuard lock(mutex); globalptr = newptr; // write to globalptr } // use newptr since here newptr doit(newptr); } read() write() globalptr Foo shared_ptr local copy local copy shared_ptr reference to const new Foo globalptr.reset(new Foo) globalptr.reset() localptr globalptr swap() write() globalptr = newptr; globalptr Foo

1.10 shared_ptr 19 1.10 shared_ptr shared_ptr x shared_ptr shared_ptr p. 16 L48 observers_ vector<shared_ ptr<observer> > unregister() Observer unregister() unregister() Observer Java boost::bind boost::bind shared_ptr boost::function class Foo { void doit(); }; shared_ptr<foo> pfoo(new Foo); boost::function<void()> func = boost::bind(&foo::doit, pfoo); // long life foo func shared_ptr<foo> Foo shared_ptr const reference const reference shared_ptr Foo void save(const shared_ptr<foo>& pfoo); void validateaccount(const Foo& foo); bool validate(const shared_ptr<foo>& pfoo) { validateaccount(*pfoo); //... } // pass by const reference // pass by const reference pass by const reference void onmessage(const string& msg) { shared_ptr<foo> pfoo(new Foo(msg)); // if (validate(pfoo)) { // pfoo save(pfoo); // pfoo } }

20 1 shared_ptr pfoo shared_ptr<void> shared_ptr DLL A B Foo Foo inline Foo Factory shared_ptr shared_ptr<t> functor Scott Meyers 12 x shared_ptr x shared_ptr BlockingQueue<shared_ptr<void> > RAII handle RAII C++ RAII C++ C++ C++ new delete new delete RAII [CCS 13] new handle shared_ptr delete shared_ptr owner child shared_ptr child owner weak_ptr 12 http://www.artima.com/cppsource/top_cpp_aha_moments.html

1.11 21 1.11 Stock Google key "NASDAQ:GOOG" IBM "NYSE:IBM" Stock Stock Stock Stock StockFactory 13 key Stock shared_ptr // version 1: questionable code class StockFactory : boost::noncopyable { public: shared_ptr<stock> get(const string& key); private: mutable MutexLock mutex_; std::map<string, shared_ptr<stock> > stocks_; }; get() stocks_ key stocks_[key] Stock stocks_[key] Stock map shared_ptr Observable weak_ptr // version 2: std::map<string, weak_ptr<stock> > stocks_; shared_ptr<stock> StockFactory::get(const string& key) { shared_ptr<stock> pstock; MutexLockGuard lock(mutex_); weak_ptr<stock>& wkstock = stocks_[key]; // key pstock = wkstock.lock(); // if (!pstock) { pstock.reset(new Stock(key)); wkstock = pstock; // stocks_[key] wkstock } return pstock; } 13 recipes/thread/test/factory.cc

22 1 Stock stocks_ stocks_.size() Stock Stock 0 key shared_ptr shared_ptr d d(ptr) ptr shared_ptr shared_ptr bridge template<class Y, class D> shared_ptr::shared_ptr(y* p, D d); template<class Y, class D> void shared_ptr::reset(y* p, D d); // Y T Y* T* Stock stocks_ // version 3 class StockFactory : boost::noncopyable { // get() pstock.reset(new Stock(key)); // pstock.reset(new Stock(key), // boost::bind(&stockfactory::deletestock, this, _1)); // *** private: void deletestock(stock* stock) { if (stock) { MutexLockGuard lock(mutex_); stocks_.erase(stock->key()); } delete stock; // sorry, I lied } // assuming StockFactory lives longer than all Stock's... //... pstock.reset() boost::function Stock* p StockFactory deletestock StockFactory this boost::function *** Stock- Factory Stock core dump Observer Observable::unregister() Observable

1.11 23 1.11.2 1.11.1 enable_shared_from_this StockFactory::get() this boost::function *** StockFactory Stock Stock StockFactory::deleteStock core dump shared_ptr StockFactory::get() shared_ptr<stockfactory> enable_shared_from_this 14 this shared_ptr class StockFactory : public boost::enable_shared_from_this<stockfactory>, boost::noncopyable { /*... */ }; shared_from_this() StockFactory stack object heap object shared_ptr shared_ptr<stockfactory> stockfactory(new StockFactory); this shared_ptr<stockfactory> // version 4 shared_ptr<stock> StockFactory::get(const string& key) { // change // pstock.reset(new Stock(key), // boost::bind(&stockfactory::deletestock, this, _1)); // to pstock.reset(new Stock(key), boost::bind(&stockfactory::deletestock, shared_from_this(), _1)); //... boost::function shared_ptr<stockfactory> StockFactory::deleteStock StockFactory shared_from_this() StockFactory shared_ptr StockFactory 14 http://en.wikipedia.org/wiki/curiously_recurring_template_pattern

24 1 1.11.2 shared_ptr boost::bind boost:function Stock- Factory boost:function Observable::notifyObservers() weak_ptr weak_ptr boost::function shared_ptr StockFactory class StockFactory : public boost::enable_shared_from_this<stockfactory>, boost::noncopyable { public: shared_ptr<stock> get(const string& key) { shared_ptr<stock> pstock; MutexLockGuard lock(mutex_); weak_ptr<stock>& wkstock = stocks_[key]; // wkstock pstock = wkstock.lock(); if (!pstock) { pstock.reset(new Stock(key), boost::bind(&stockfactory::weakdeletecallback, boost::weak_ptr<stockfactory>(shared_from_this()), _1)); // shared_from_this() weak_ptr // boost::bind wkstock = pstock; } return pstock; } private: static void weakdeletecallback(const boost::weak_ptr<stockfactory>& wkfactory, Stock* stock) { shared_ptr<stockfactory> factory(wkfactory.lock()); // if (factory) // factory stocks_ { factory->removestock(stock); } delete stock; // sorry, I lied }

1.11 25 void removestock(stock* stock) { if (stock) { MutexLockGuard lock(mutex_); stocks_.erase(stock->key()); } } private: mutable MutexLock mutex_; std::map<string, weak_ptr<stock> > stocks_; }; void testlonglifefactory() { shared_ptr<stockfactory> factory(new StockFactory); { shared_ptr<stock> stock = factory->get("nyse:ibm"); shared_ptr<stock> stock2 = factory->get("nyse:ibm"); assert(stock == stock2); // stock destructs here } // factory destructs here } void testshortlifefactory() { shared_ptr<stock> stock; { shared_ptr<stockfactory> factory(new StockFactory); stock = factory->get("nyse:ibm"); shared_ptr<stock> stock2 = factory->get("nyse:ibm"); assert(stock == stock2); // factory destructs here } // stock destructs here } Stock StockFactory shared_ptr weak_ptr Factory singleton 15 StockFactory Stock stocks_ 2.1 2.8 15 recipes/thread/weakcallback.h C++11 variadic template rvalue reference

26 1 1.12 shared_ptr/weak_ptr C++ 1. façade Foo Foo façade objid/handle check-out check-in 16 race condition façade Foo Java ConcurrentHashMap buckets bucket contention 2. 1.4 3. 17 shared_ptr shared_ptr 4. C++11 unique_ptr shared_ptr Google Go Concurrency is hard without garbage collection C/C++ Java util.concurrent C++11 1.13 API 16 Jeff Grossman A technique for safe deletion with object locking [Gr00] 17 http://blog.csdn.net/solstice/article/details/5238671#comments

1.13 27 18 1.9 shared_ptr C++ C Java Java Concurrency in Practice [JCP] C++ Java IPC race condition CPU CPU CPU CPU ilovecpp race condition shared_ptr/scoped_ptr 18 50% http://blog.csdn.net/myan/article/details/3144661 Google

28 1 shared_ptr boost::bind shared_ptr weak_ptr shared_ptr boost::shared_ptr http://www.boost.org/doc/libs/release/libs/smart_ptr/shared_ptr.htm C++11 unique_ptr auto_ptr shared_ptr TR1 C++ 19 C++ STL shared_ptr std::vector shared_ptr / 1.14 Observer 1.8 shared_ptr/weak_ptr Observer Observer Observer OO Observer Observer friend Observer Observer class Foo 1 30 work around Base class 19 C++ new delete memory leak C++ smart pointer STL Boost ( ) 4 smart pointers C++ memory leak!

1.14 Observer 29 Observer Java Java 8 Closure C# delegate C++ boost::function/ boost::bind 20 C++ Observer Signal/Slots QT thread safe race condition free thread contention free Signal/Slots shared_ptr 1.8 Observer 2.8 shared_ptr copy-on-write C++11 variadic template trivial template<typename Signature> class SignalTrivial; recipes/thread/signalslottrivial.h // NOT thread safe!!! template <typename RET, typename... ARGS> class SignalTrivial<RET(ARGS...)> { public: typedef std::function<void (ARGS...)> Functor; void connect(functor&& func) { functors_.push_back(std::forward<functor>(func)); } void call(args&&... args) { for (const Functor& f: functors_) { f(args...); } } private: std::vector<functor> functors_; }; recipes/thread/signalslottrivial.h Signal/Slots Slot unregister recipes/thread/signalslot.h 20 11.5 boost::function boost::bind function/bind http://blog.csdn.net/myan/article/details/5928531

30 1 C++ Ruminations on C++ Koenig Moo 2003 C++ 6 C++ handle/body idiom C++ shared_ptr C++ auto_ptr C++ shared_ptr C++ 21 2001 STL 10 STL vector 21 http://en.wikipedia.org/wiki/reinventing_the_wheel

2 muduo

6 muduo 6.1 2010 3 ACE 1 Linux Windows x86-64 IA32 muduo ARM UDP TCP IPv6 IPv4 muduo IO + one event loop per thread IO API API non-trivial templates 90% app lib library framework 5000 FreeBSD/Darwin Mac IO multiplexing poll(2) epoll(4) Google Protocol Buffers RPC 1 http://blog.csdn.net/solstice/archive/2010/03/10/5364096.aspx 125

126 6 muduo C++ muduo 2 Google code http://code.google.com/p/muduo/ muduo git https://github.com/chenshuo/muduo muduo 2010 5 8 0.1.0 2012 11 0.8.2 Sockets API Python Hello hello-server.py 1 #!/usr/bin/python 2 3 import socket, time 4 5 serversocket = socket.socket(socket.af_inet, socket.sock_stream) 6 serversocket.bind(('', 8888)) 7 serversocket.listen(5) 8 9 while True: 10 (clientsocket, address) = serversocket.accept() # 11 data = clientsocket.recv(4096) # 12 datetime = time.asctime()+'\n' 13 clientsocket.send('hello ' + data) # 14 clientsocket.send('my time is ' + datetime) # 15 clientsocket.close() # hello-server.py 20 # import 21 sock = socket.socket(socket.af_inet, socket.sock_stream) 22 sock.connect((sys.argv[1], 8888)) # 23 sock.send(os.getlogin() + '\n') # 24 message = sock.recv(4096) # 25 print message # 26 sock.close() # hello-client.py hello-client.py Sockets API socket(2) bind(2) listen(2) accept(2) connect(2) recv(2) send(2) close(2) gethostbyname(3) 3 2 http://www.oschina.net/question/28_61182 3 L22

6.2 127 $./hello-client.py localhost Hello schen My time is Sun May 13 12:56:44 2012 $./hello-client.py atom Hello schen Java Python Sockets Sockets API 6.6 6.2 tar http://code.google.com/p/muduo/downloads/list muduo- 0.8.2-beta.tar.gz muduo Linux timerfd eventfd Linux 2.6.28 Debian 6.0 Squeeze / Ubuntu 10.04 LTS 2.6.32 g++ 4.4 32-bit 64-bit x86 muduo Fedora 13 CentOS 6 Arch Linux AUR 4 Linux 2.6 5 muduo backport.diff muduo Samsung S3C2440 ARM9 Raspberry Pi ARM11 muduo armlinux.diff muduo CMake 6 build system $ sudo apt-get install cmake muduo Boost 7 $ sudo apt-get install libboost-dev libboost-test-dev 4 http://aur.archlinux.org/packages.php?id=49251 5 Debian 5.0 Lenny Ubuntu 8.04 CentOS 5 6 2.8 CentOS 6 2.6 Protobuf 7 TR1 Boost

128 6 muduo muduo curl c-ares DNS Google Protobuf cmake $ sudo apt-get install libcurl4-openssl-dev libc-ares-dev $ sudo apt-get install protobuf-compiler libprotobuf-dev muduo $ tar zxf muduo-0.8.2-beta.tar.gz $ cd muduo/ $./build.sh -j2 muduo../build/debug/{bin,lib} $./build.sh install muduo../build/debug-install/{include,lib} muduo-protorpc muduo-udns release -O2 $ BUILD_TYPE=release./build.sh -j2 muduo../build/release/{bin,lib} $ BUILD_TYPE=release./build.sh install muduo../build/release-install/{include,lib} muduo-protorpc muduo-udns muduo 1.0 BUILD_TYPE release bin/inspector_test http://10.0.0.10:12345/ http://10.0.0.10:12345/proc/status 10.0.0.10 Linux box IP muduo muduo 8 C++ muduo../build/debug-install/include../build/debug-install/lib -lmuduo_net -lmuduo_base CMake makefile muduo https://github.com/chenshuo/ muduo-tutorial 8 11

6.3 129 6.3 muduo muduo -- build.sh -- ChangeLog -- CMakeLists.txt -- License -- README -- muduo muduo -- base ::muduo namespace \-- net ::muduo::net namespace -- poller poll(2) epoll(4) IO multiplexing -- http Web -- inspect Web \-- protorpc Google Protobuf RPC -- examples \-- TODO muduo class ThreadPool class muduo/ base/threadpool.h muduo/base/threadpool.cc muduo/base muduo \-- base -- AsyncLogging.{h,cc} backend -- Atomic.h -- BlockingQueue.h -- BoundedBlockingQueue.h -- Condition.h Mutex.h -- copyable.h tag -- CountDownLatch.{h,cc} -- Date.{h,cc} Julian -- Exception.{h,cc} stack trace -- Logging.{h,cc} AsyncLogging -- Mutex.h -- ProcessInfo.{h,cc} -- Singleton.h singleton -- StringPiece.h Google -- tests -- Thread.{h,cc} -- ThreadLocal.h -- ThreadLocalSingleton.h singleton -- ThreadPool.{h,cc} -- Timestamp.{h,cc} UTC -- TimeZone.{h,cc} \-- Types.h muduo::string

130 6 muduo muduo Reactor EventLoop IO muduo object-based objectoriented boost::function + boost::bind muduo class muduo/net muduo/net/poller 4300 muduo \-- net -- Acceptor.{h,cc} -- Buffer.{h,cc} IO -- Callbacks.h -- Channel.{h,cc} Socket -- CMakeLists.txt -- Connector.{h,cc} -- Endian.h -- EventLoop.{h,cc} -- EventLoopThread.{h,cc} EventLoop -- EventLoopThreadPool.{h,cc} muduo IO -- InetAddress.{h,cc} IP -- Poller.{h,cc} IO multiplexing -- poller IO multiplexing -- DefaultPoller.cc MUDUO_USE_POLL -- EPollPoller.{h,cc} epoll(4) IO multiplexing \-- PollPoller.{h,cc} poll(2) IO multiplexing -- Socket.{h,cc} Sockets -- SocketsOps.{h,cc} Sockets API -- TcpClient.{h,cc} TCP -- TcpConnection.{h,cc} muduo 300 -- TcpServer.{h,cc} TCP -- tests -- Timer.{h,cc} -- TimerId.h \-- TimerQueue.{h,cc} -lmuduo_http -lmuduo_inspect HttpServer Inspector http Java JMX 9.5 muduo/net/{http,inspect,protorpc}

6.3 131 muduo \-- net -- http HTTP HTTP -- CMakeLists.txt -- HttpContext.h -- HttpRequest.h -- HttpResponse.{h,cc} -- HttpServer.{h,cc} \-- tests/httpserver_test.cc HTTP -- inspect HTTP -- CMakeLists.txt -- Inspector.{h,cc} -- ProcessInspector.{h,cc} \-- tests/inspector_test.cc \-- protorpc Google Protobuf RPC -- CMakeLists.txt -- google-inl.h -- RpcChannel.{h,cc} -- RpcCodec.{h,cc} -- rpc.proto \-- RpcServer.{h,cc} 6.3.1 muduo muduo 5 Buffer EventLoop TcpConnection TcpClient TcpServer -- include \-- muduo -- base \-- net -- Buffer.h -- Callbacks.h -- Channel.h -- Endian.h -- EventLoop.h -- EventLoopThread.h -- InetAddress.h -- TcpClient.h -- TcpConnection.h -- TcpServer.h -- TimerId.h -- http -- HttpRequest.h -- HttpResponse.h \-- HttpServer.h -- inspect -- Inspector.h \-- ProcessInspector.h

132 6 muduo \-- protorpc -- RpcChannel.h -- RpcCodec.h \-- RpcServer.h \-- lib -- libmuduo_base.a, libmuduo_net.a -- libmuduo_http.a, libmuduo_inspect.a \-- libmuduo_protorpc.a 6-1 muduo poller/epollpoller.h SocketsOps.h poller/pollpoller.h Poller.h Acceptor.h Socket.h EventLoopThreadPool.h EventLoopThread.h TcpClient.h TcpServer.h 6-1 TimerQueue.h Timer.h EventLoop.h TcpConnection.h Connector.h Channel.h Callbacks.h TimerId.h Buffer.h InetAddress.h muduo forward declaration Acceptor.h Channel.h Connector.h TcpConnection.h EventLoop class EventLoop.h TcpClient.h Connector class TcpServer.h Acceptor EventLoopThreadPool EventLoop.h Poller TimerQueue TcpConnection.h Channel Socket class Buffer Netty ChannelBuffer buffer class buffer read(2)/write(2) 7.4

6.3 133 InetAddress IPv4 end point IP gethostbyname(3) IO EventLoop Reactor EventLoop IO eventfd(2) pipe(2) TimerQueue Poller IO multiplexing EventLoopThread EventLoop::loop() TcpConnection TCP TcpClient TcpServer TcpConnection shared_ptr Buffer TcpConnection Buffer InetAddress class Channel selectable IO channel IO file descriptor Acceptor Connector EventLoop TimerQueue TcpConnection Socket RAII handle file descriptor fd Acceptor TcpConnection EventLoop TimerQueue fd Socket class SocketsOps Sockets Poller PollPoller EPollPoller EventLoop PollPoller EPollPoller poll(2) epoll(4) IO multiplexing poll poll(2) strace(1) Connector TCP TcpClient Acceptor TCP TcpServer TimerQueue timerfd poll/epoll_wait TimerQueue std::map Timer O(log N) N EventLoop EventLoopThreadPool IO TcpConnection EventLoop TcpServer

134 6 muduo 6-2 muduo Buffer TcpConnection EventLoop Channel +loop() +runafter() +runevery() +runat() +runinloop() +queueinloop() File Descriptor Socket owns -fd_ : File Desc. +handleevent() TcpConnection Acceptor Connector +poll() Poller -handleread() -handlewrite() -handleclose() -handleerror() -handleread() -handlewrite() -handleerror() TcpServer TcpClient PollPoller EPollPoller +poll() +poll() 6-2 6.3.2 muduo examples Boost.Asio Java Netty Python Twisted examples -- asio Boost.Asio -- chat codec \-- tutorial timers -- cdns c-ares DNS -- curl curl HTTP -- filetransfer TCP -- hub pub/sub/hub -- idleconnection -- maxconnection -- multiplexer 1:n -- netty JBoss Netty -- discard -- echo \-- uptime TCP -- pingpong pingpong

6.3 135 -- protobuf Google Protobuf -- codec -- rpc RPC Sudoku \-- rpcbench RPC -- roundtrip -- shorturl -- simple 5 -- allinone 5 -- chargen RFC 864 -- chargenclient chargen -- daytime RFC 867 -- discard RFC 863 -- echo RFC 862 -- time RFC 868 \-- timeclient time -- socks4a Socks4a TcpClient -- sudoku muduo -- twisted Python Twisted \-- finger finger01 ~ 07 \-- zeromq ZeroMQ muduo License muduo http://github.com/chenshuo/muduo-udns UDNS DNS http://github.com/chenshuo/muduo-protorpc RPC 9 6.3.3 muduo one loop per thread + thread pool EventLoop TcpConnection EventLoop IO file descriptor TcpConnection EventLoop TCP TCP TcpConnection EventLoop TcpServer accept(2) TcpConnection IO accept(2) EventLoop EventLoop- ThreadPool round-robin 6.6 Sudoku muduo 9 muduo-protorpc Ubuntu Linux 12.04 apt-get Protobuf Protobuf 2.4.1

136 6 muduo muduo TCP muduo bug signal 10 6.4 muduo 8 muduo Linux 2.6.x TCP IO IO muduo TCP Sockets API Brooks 11 muduo accidental complexity 6.4.1 TCP recv(2) accept(2) send(2) Win32 TCP 1. accept connect TCP 2. close shutdown read(2) 0 10 Signal signalfd(2) EventLoop muduo-protorpc zurg slave 11 http://www.cs.nott.ac.uk/~cah/g51iss/documents/nosilverbullet.html

6.4 137 3. 3.5 TCP close(2) back-off edge trigger level trigger 12 EPOLLOUT busy-loop epoll(4) poll(2) 40kB TCP 25kB 15kB OS 15kB TCP socket 50kB kb 50kB write() lighttpd \r\n\r\n bug 13 10ms readable lighttpd 14 12 13 http://redmine.lighttpd.net/issues/show/2105 14 http://download.lighttpd.net/lighttpd/security/lighttpd_sa_2010_01.txt

138 6 muduo 10 000 50kB (s) 1GB muduo readv(2) IO muduo 6.4.2 echo muduo echo 1. EchoServer class examples/simple/echo/echo.h 4 #include <muduo/net/tcpserver.h> 5 6 // RFC 862 7 class EchoServer 8 { 9 public: 10 EchoServer(muduo::net::EventLoop* loop, 11 const muduo::net::inetaddress& listenaddr); 12 13 void start(); // calls server_.start(); 14 15 private: 16 void onconnection(const muduo::net::tcpconnectionptr& conn); 17 18 void onmessage(const muduo::net::tcpconnectionptr& conn, 19 muduo::net::buffer* buf, 20 muduo::timestamp time); 21 22 muduo::net::eventloop* loop_; 23 muduo::net::tcpserver server_; 24 }; examples/simple/echo/echo.h

6.4 139 examples/simple/echo/echo.cc 10 EchoServer::EchoServer(muduo::net::EventLoop* loop, 11 const muduo::net::inetaddress& listenaddr) 12 : loop_(loop), 13 server_(loop, listenaddr, "EchoServer") 14 { 15 server_.setconnectioncallback( 16 boost::bind(&echoserver::onconnection, this, _1)); 17 server_.setmessagecallback( 18 boost::bind(&echoserver::onmessage, this, _1, _2, _3)); 19 } examples/simple/echo/echo.cc 2. EchoServer::onConnection() EchoServer::onMessage() examples/simple/echo/echo.cc 26 void EchoServer::onConnection(const muduo::net::tcpconnectionptr& conn) 27 { 28 LOG_INFO << "EchoServer - " << conn->peeraddress().toipport() << " -> " 29 << conn->localaddress().toipport() << " is " 30 << (conn->connected()? "UP" : "DOWN"); 31 } 32 33 void EchoServer::onMessage(const muduo::net::tcpconnectionptr& conn, 34 muduo::net::buffer* buf, 35 muduo::timestamp time) 36 { 37 muduo::string msg(buf->retrieveallasstring()); 38 LOG_INFO << conn->name() << " echo " << msg.size() << " bytes, " 39 << "data received at " << time.tostring(); 40 conn->send(msg); 41 } examples/simple/echo/echo.cc L37 L40 echo L40 send(msg) muduo event handler onconnection() conn TcpConnection shared_ptr TcpConnection::connected() bool TcpConnection peeraddress() localaddress() InetAddress IP port

140 6 muduo onmessage() conn TCP buf buf retrieve buf buffer time epoll_wait(2) read(2) Timestamp pass-by-value pass-by-(const)reference x86-64 3. main() EventLoop 1 #include "echo.h" 2 3 #include <muduo/base/logging.h> 4 #include <muduo/net/eventloop.h> 5 6 // using namespace muduo; 7 // using namespace muduo::net; 8 9 int main() 10 { 11 LOG_INFO << "pid = " << getpid(); 12 muduo::net::eventloop loop; 13 muduo::net::inetaddress listenaddr(2007); 14 EchoServer server(&loop, listenaddr); 15 server.start(); 16 loop.loop(); 17 } examples/simple/echo/main.cc examples/simple/echo/main.cc muduo/examples/simple/echo echo TcpServer EventLoop TcpConnection Buffer class class class namespace 6.4.3 finger Python Twisted Reactor muduo muduo deferreds finger Twisted muduo finger finger01 finger07 examples/twisted/ finger

6.4 141 1. 1 #include <muduo/net/eventloop.h> 2 3 using namespace muduo; 4 using namespace muduo::net; 5 6 int main() 7 { 8 EventLoop loop; 9 loop.loop(); 10 } examples/twisted/finger/finger01.cc examples/twisted/finger/finger01.cc 2. 1079 muduo examples/twisted/finger/finger02.cc 1 #include <muduo/net/eventloop.h> 2 #include <muduo/net/tcpserver.h> 3 4 using namespace muduo; 5 using namespace muduo::net; 6 7 int main() 8 { 9 EventLoop loop; 10 TcpServer server(&loop, InetAddress(1079), "Finger"); 11 server.start(); 12 loop.loop(); 13 } examples/twisted/finger/finger02.cc 3. namespace 7 void onconnection(const TcpConnectionPtr& conn) 8 { 9 if (conn->connected()) 10 { 11 conn->shutdown(); 12 } 13 } 14 15 int main() 16 { 17 EventLoop loop; 18 TcpServer server(&loop, InetAddress(1079), "Finger"); 19 server.setconnectioncallback(onconnection); examples/twisted/finger/finger03.cc

142 6 muduo 20 server.start(); 21 loop.loop(); 22 } examples/twisted/finger/finger03.cc 4. \r\n Buffer::findCRLF() O(N 2 ) CPU examples/twisted/finger/finger04.cc 7 void onmessage(const TcpConnectionPtr& conn, 8 Buffer* buf, 9 Timestamp receivetime) 10 { 11 if (buf->findcrlf()) 12 { 13 conn->shutdown(); 14 } 15 } 16 17 int main() 18 { 19 EventLoop loop; 20 TcpServer server(&loop, InetAddress(1079), "Finger"); 21 server.setmessagecallback(onmessage); 22 server.start(); 23 loop.loop(); 24 } examples/twisted/finger/finger04.cc 5. \r\n --- examples/twisted/finger/finger04.cc 2010-08-29 00:03:14 +0800 +++ examples/twisted/finger/finger05.cc 2010-08-29 00:06:05 +0800 @@ -7,12 +7,13 @@ void onmessage(const TcpConnectionPtr& conn, Buffer* buf, Timestamp receivetime) { if (buf->findcrlf()) { + conn->send("no such user\r\n"); conn->shutdown(); } }

6.4 143 6. UserMap L30 UserMap examples/twisted/finger/finger06.cc 9 typedef std::map<string, string> UserMap; 10 UserMap users; 11 12 string getuser(const string& user) 13 { 14 string result = "No such user"; 15 UserMap::iterator it = users.find(user); 16 if (it!= users.end()) 17 { 18 result = it->second; 19 } 20 return result; 21 } 22 23 void onmessage(const TcpConnectionPtr& conn, 24 Buffer* buf, 25 Timestamp receivetime) 26 { 27 const char* crlf = buf->findcrlf(); 28 if (crlf) 29 { 30 string user(buf->peek(), crlf); 31 conn->send(getuser(user) + "\r\n"); 32 buf->retrieveuntil(crlf + 2); 33 conn->shutdown(); 34 } 35 } 36 37 int main() 38 { 39 EventLoop loop; 40 TcpServer server(&loop, InetAddress(1079), "Finger"); 41 server.setmessagecallback(onmessage); 42 server.start(); 43 loop.loop(); 44 } examples/twisted/finger/finger06.cc 7. UserMap L39 --- examples/twisted/finger/finger06.cc 2010-08-29 00:14:33 +0800 +++ examples/twisted/finger/finger07.cc 2010-08-29 00:15:22 +0800 @@ -36,6 +36,7 @@ int main() { + users["schen"] = "Happy and well"; EventLoop loop; TcpServer server(&loop, InetAddress(1079), "Finger");

144 6 muduo server.setmessagecallback(onmessage); server.start(); loop.loop(); } telnet(1) finger Telnet $./bin/twisted_finger07 $ telnet localhost 1079 Trying ::1... Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. muduo No such user Connection closed by foreign host. $ telnet localhost 1079 Trying ::1... Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. schen Happy and well Connection closed by foreign host. 6.5 muduo 2010 8 muduo TCP

6.5 145 muduo percentile 15 16 6.5.1 muduo Boost.Asio libevent2 muduo ping pong muduo Boost.Asio 15% libevent2 18% 70%. boost 1.40 asio 1.4.3 asio 1.4.5 http://think-async.com/asio/download libevent 2.0.6-rc http://monkey.org/~provos/libevent-2.0.6-rc.tar.gz muduo 0.1.1 asio http://asio.cvs.sourceforge.net/viewvc/asio/asio/src/tests/performance/ libevent2 ping pong recipes/pingpong/libevent/ muduo libevent2 muduo examples/pingpong/ gist 17 muduo asio -O2 -finline-limit=1000 $ BUILD_TYPE=release./build.sh # muduo DELL 490 Intel Xeon E5320 CPU 8 1.86GHz 16GiB Ubuntu Linux Server 10.04.1 LTS x86_64 g++ 4.4.3 15 http://zedshaw.com/essays/programmer_stats.html 16 http://www.percona.com/files/presentations/velocity2012-beyond-the-numbers.pdf 17 http://gist.github.com/564985

146 6 muduo asio 18 ping pong muduo asio libevent2 ping pong echo TCP echo echo ZeroMQ 1/10/100/1000/10 000 100 1000 1/2/3/4 8 4 ping pong 16KiB shell http://gist.github.com/564985 CPU TCP 110MiB/s Python CPU CPU 1 CPU 6-3 18 http://think-async.com/asio/linuxperformanceimprovements

吞吐量 (MiB/s) 6.5 147 500 450 400 350 300 250 200 150 100 50 0 单线程 连接数 1 10 100 1000 10000 muduo 191.0 449.6 406.3 379.2 383.7 asio 1.4.5 192.3 420.8 343.7 324.1 329.9 asio 1.4.3 166.2 342.8 289.7 271.6 276.6 libevent2.0.6 174.2 240.2 240.2 210.5 184.7 6-3 muduo libevent2 70% libevent2 socket 4096 buffer.c evbuffer_ read() muduo muduo 16 384 4096 6-4 6-4

第6章 148 muduo 网络库简介 测试结果表明 muduo 的吞吐量平均比 libevent2 高 18% 以上 多线程测试的结果 见图 6-5 数字越大越好 多线程 (100 连接) 多线程 (1000 连接) 1000 700 900 600 700 500 吞吐量 (MiB/s) 吞吐量 (MiB/s) 800 600 500 400 400 300 300 200 200 100 100 1 2 3 线程数 4 1 2 3 4 muduo 405.0 648.1 801.0 876.8 muduo 375.7 510.1 594.0 620.7 asio 1.4.3 291.0 353.0 345.5 334.9 asio 1.4.3 273.0 339.3 312.2 297.8 asio 1.4.5 363.5 398.4 421.7 419.2 asio 1.4.5 340.8 425.0 389.6 369.4 0 线程数 0 图 6-5 测试结果表明 muduo 的吞吐量平均比 asio 高 15% 以上 讨论 muduo 出乎意料地比 asio 性能优越 我想主要得益于其简单的设计和简洁的 代码 asio 在多线程测试中表现不佳 我猜测其主要原因是测试代码只使用了一 个 io_service 如果改用 io_service per CPU 的话 其性能应该有所提高 我 对 asio 的了解程度仅限于能读懂其代码 希望能有 asio 高手编写 io_service per CPU 的 ping pong 测试 以便与 muduo 做一个公平的比较 由于 libevent2 每次最多从网络读取 4096 字节 这大大限制了它的吞吐量 ping pong 测试很容易实现 欢迎其他网络库 ACE POCO libevent 等 也能 加入到对比中来 期待这些库的高手出马 6.5.2 击鼓传花 对比 muduo 与 libevent2 的事件处理效率 前 面 我 们 比 较 了 muduo 和 libevent2 的 吞 吐 量 得 到 的 结 论 是 muduo 比 libevent2 快 18%. 有人会说 libevent2 并不是为高吞吐量的应用场景而设计的 这样的比较不公平 胜之不武 为了公平起见 这回我们用 libevent2 自带的性能测 试程序 击鼓传花 来对比 muduo 和 libevent2 在高并发情况下的 IO 事件处理效率 Linux 多线程服务端编程 使用 muduo C++ 网络库 (excerpt) http://www.chenshuo.com/book/

6.5 149 DELL E6400 1000 1 100 1000 socketpair(2) pipe(2) 1 1 1 2 3 100 100 1000 100 1000 100 ~ 100 000 10 256 000 libevent2 test/bench.c 2.0.6-rc bug libevent2 muduo examples/pingpong/bench.cc 100 1000 100 event watcher libev 19 timer event IO event 25 24 20 libev Marc Lehmann muduo libevent2 6-6 + libevent2 + muduo 1 2 3 4 5 6 7.5 10 19 http://libev.schmorp.de/bench.html 20 recipes/pingpong/libevent/run_bench.sh

150 6 muduo 140000 120000 muduo libevent2 muduo (ctl_add) total time per iteration 100 active clients Dell WS490 100000 80000 60000 40000 20000 0 100 1000 10000 100000 file descriptors 160000 140000 muduo libevent2 muduo (ctl_add) 1000 active clients 120000 100000 80000 60000 40000 20000 0 1000 10000 100000 file descriptors 700 time spent in event processing 100 active clients 650 600 550 500 450 400 350 300 250 100 1000 10000 100000 file descriptors muduo libevent2 muduo (ctl_add) 5000 4500 muduo libevent2 muduo (ctl_add) 1000 active clients 4000 3500 3000 2500 2000 1500 1000 10000 100000 file descriptors time (in us) (lower is better) time (in us) (lower is better) time (in us) (lower is better) time (in us) (lower is better) 6-6

6.5 151 1. libevent2 event watcher muduo 20% 2. a. 100 1000 30 000 1000 30 000 libevent2 b. 1000 10 000 libevent2 muduo 10 000 muduo 1. muduo 2. libevent2 libev Marc Lehmann 21 libevent2 muduo epoll_ctl(fd, EPOLL_CTL_ ADD,...) event watcher 24 muduo epoll_ctl(fd, EPOLL_CTL_MOD,...) event watcher libevent2 epoll_ctl(fd, EPOLL_CTL_ADD,...) fd EEXIST File exists EPOLL_CTL_ADD EPOLL_CTL_MOD libevent2 muduo EPOLL_CTL_ADD event watcher 6-6 muduo libevent2 kernel muduo muduo/net/poller/epollpoller.cc 138 173 6-7 CPU 2.4 GHz 1.86 GHz 21 http://lists.schmorp.de/pipermail/libev/2010q2/001041.html

152 6 muduo 80000 70000 muduo libevent2 muduo (ctl_add) total time per iteration 100 active clients Dell E6400 60000 50000 40000 30000 20000 10000 0 100 1000 10000 100000 file descriptors 90000 80000 70000 muduo libevent2 muduo (ctl_add) 1000 active clients 60000 50000 40000 30000 20000 10000 0 1000 10000 100000 file descriptors 400 time spent in event processing 100 active clients 380 360 340 320 300 280 260 240 220 200 180 100 1000 10000 100000 file descriptors muduo libevent2 muduo (ctl_add) 3000 2800 muduo libevent2 muduo (ctl_add) 1000 active clients 2600 2400 2200 2000 1800 1600 1400 1000 10000 100000 file descriptors time (in us) (lower is better) time (in us) (lower is better) time (in us) (lower is better) time (in us) (lower is better) 6-7

6.5 153 muduo libevent2 10 000 muduo 6.5.3 muduo Nginx Nginx 1.0.12 muduo 0.3.1 HTTP muduo HTTP muduo/net/http/ HTTP server 8 DELL 490 Xeon E5320 CPU ab 22 weighttp 23 4 i5-2500 CPU Nginx muduo HTTP server HTTP client muduo HTTP muduo/net/http/tests/httpserver_test.cc void onrequest(const HttpRequest& req, HttpResponse* resp) { if (req.path() == "/") { //... } else if (req.path() == "/hello") { resp->setstatuscode(httpresponse::k200ok); resp->setstatusmessage("ok"); resp->setcontenttype("text/plain"); resp->addheader("server", "Muduo"); resp->setbody("hello, world!\n"); } else { resp->setstatuscode(httpresponse::k404notfound); resp->setstatusmessage("not Found"); resp->setcloseconnection(true); } } int main(int argc, char* argv[]) { int numthreads = 0; 22 http://httpd.apache.org/docs/2.4/programs/ab.html 23 http://redmine.lighttpd.net/projects/weighttp/wiki

154 6 muduo if (argc > 1) { benchmark = true; Logger::setLogLevel(Logger::WARN); numthreads = atoi(argv[1]); } EventLoop loop; HttpServer server(&loop, InetAddress(8000), "dummy"); server.sethttpcallback(onrequest); server.setthreadnum(numthreads); server.start(); loop.loop(); } muduo/net/http/tests/httpserver_test.cc Nginx HTTP echo 24 #user nobody; worker_processes 4; events { worker_connections 10240; } http { include default_type mime.types; application/octet-stream; access_log sendfile tcp_nopush off; on; on; keepalive_timeout 65; server { listen 8080; server_name localhost; location / { root html; index index.html index.htm; } } } location /hello { default_type text/plain; echo "hello, world!"; } 24 http://wiki.nginx.org/httpechomodule https://gist.github.com/1967026

Requests per second Requests per second 6.5 155 /hello "hello, world!"./ab -n 100000 -k -r -c 1000 10.0.0.9:8080/hello 6-8 HTTP ab CPU 70% muduo vs. Nginx 1 worker/thread 50000 45000 40000 35000 30000 25000 20000 15000 10000 5000 0 1 2 5 10 20 50 100 200 500 1000 muduo 3512 7387 27759 44552 46521 45599 45595 44957 40429 37845 Nginx 3502 6639 26319 39404 40672 40192 40278 39859 38825 34741 6-8 muduo 4 Nginx 4 6-9 20 top(1) ab CPU 85% weighttp muduo vs. Nginx 4 workers/threads 120000 100000 80000 60000 40000 20000 0 1 2 5 10 20 50 100 200 500 1000 2000 5000 10000 muduo 3507 7440 27512 51422 92050 109178 108706 107216 105266 101840 100185 94956 88094 Nginx 3504 6625 26591 48134 99259 108125 106288 108671 97347 93237 88303 79873 76502 6-9

Average latency [us] lower is better 156 6 muduo CPU top(1) 10 000 4 workers/threads muduo 4 83% Nginx 4 75% 1000 4 workers/threads muduo 4 85% Nginx 4 78% Nginx CPU CPU CPU benchmark IO benchmark CPU 100% IO muduo Nginx muduo Nginx muduo qps 10 muduo HTTP HTTP muduo Nginx httpd muduo 6.5.4 muduo ZeroMQ ZeroMQ 25 muduo muduo examples/zeromq/ 6.5.1 ping pong echo 6-10 16KiB muduo ZeroMQ 350.0 300.0 muduo vs. ZeroMQ latency on GbE 250.0 200.0 150.0 100.0 50.0 0.0 1 2 4 8 16 32 64 128 256 512 1k 2k 4k 8k 16k muduo 43.2 43.1 43.1 43.6 44.3 44.7 118 125 138 133 149 165 181 154 230 ZeroMQ 62.2 61.8 61.9 62.4 62.8 63.8 138 149 155 154 172 193 206 182 289 6-10 25 http://www.zeromq.org/results:perf-howto

6.6 muduo 157 6.6 muduo Sudoku Solver muduo muduo examples/sudoku/ 6.6.1 Sudoku Solver Sudoku Solver 9.8 muduo Buffer 7.4 3.6 echo A echo TCP echo Sudoku \r\n TCP [id:]<81digits>\r\n [id:]<81digits>\r\n [id:]nosolution\r\n [id:] id Parallel Pipelining id Parallel Pipelining Protobuf RPC 2 26 27 54 out-of-order RPC 26 http://blog.csdn.net/lanphaday/archive/2011/04/11/6316099.aspx 27 http://blog.csdn.net/solstice/article/details/5950190

158 6 muduo <81digits> Sudoku 9 9 0 Sudoku NoSolution 1 000000010400000000020000000000050407008000300001090000300400200050100000000806000\r\n 693784512487512936125963874932651487568247391741398625319475268856129743274836159\r\n 2 a:000000010400000000020000000000050407008000300001090000300400200050100000000806000\r\n a:693784512487512936125963874932651487568247391741398625319475268856129743274836159\r\n 3 b:000000010400000000020000000000050407008000300001090000300400200050100000000806005\r\n b:nosolution\r\n telnet Sudoku Solver Sudoku Client Sudoku Solver 9981 9 9 = 81 Sudoku Sudoku 28 Sudoku string solvesudoku(const string& puzzle); <81digits> <81digits> NoSolution pure function 6.4.2 echo EchoServer SudokuServer onmessage() examples/sudoku/server_basic.cc onmessage() solvesudoku() TCP 28 http://blog.csdn.net/solstice/archive/2008/02/15/2096209.aspx

6.6 muduo 159 const int kcells = 81; // 81 examples/sudoku/server_basic.cc void onmessage(const TcpConnectionPtr& conn, Buffer* buf, Timestamp) { LOG_DEBUG << conn->name(); size_t len = buf->readablebytes(); while (len >= kcells + 2) // 2 { const char* crlf = buf->findcrlf(); if (crlf) // { string request(buf->peek(), crlf); // string id; buf->retrieveuntil(crlf + 2); // retrieve string::iterator colon = find(request.begin(), request.end(), ':'); if (colon!= request.end()) // id { id.assign(request.begin(), colon); request.erase(request.begin(), colon+1); } if (request.size() == implicit_cast<size_t>(kcells)) // { string result = solvesudoku(request); // if (id.empty()) { conn->send(result+"\r\n"); } else { conn->send(id+":"+result+"\r\n"); } } else // { conn->send("bad Request!\r\n"); conn->shutdown(); } } else // { break; } } } examples/sudoku/server_basic.cc server_basic.cc Sudoku 7.4 CPU server_basic CPU

160 6 muduo server_basic 8 8 server_basic 9981 9982 9988 (s) 8 server_basic load balancer 6.6.2 W. Richard Stevens UNIX 2 27 Client-Server Design Alternatives 20 90 [UNP] 3 30 UNP CSDA [UNP] non-blocking IO Sockets API 2000 HTTP httpd Reactor IBM Lotus TCP Lotus Linux IBM Linux Lotus Server Linux POSA2 http://bulk.fefe.de/scalable-networking.pdf http://www.kegel.com/c10k.html http://gee.cs.oswego.edu/dl/cpjslides/nio.pdf 6-1 12 chat chat A TCP echo/httpd/sudoku chat httpd/sudoku

6.6 muduo 161 6-1 方案 并发模型 [UNP] 对应 多进程 多线程 阻塞 IO IO 复用 长连接 0 accept+read/write 0 否 否 是 否 否 无 否 低 否 是 常 一次服务一个客户 1 accept+fork 1 是 否 是 否 是 低 是 高 否 是 变 process-per-connection 2 accept+thread 6 否 是 是 否 是 中 是 中 是 是 变 thread-per-connection 3 prefork 2/3/4/5 是 否 是 否 是 低 是 高 否 是 变 见 [UNP] 4 pre threaded 7/8 否 是 是 否 是 中 是 中 是 是 变 见 [UNP] 5 poll (reactor) 6.8 节 否 否 否 是 是 高 否 低 是 是 常 单线程 reactor 6 reactor + thread-per-task 无 否 是 否 是 是 中 是 中 是 否 变 thread-per-request 7 reactor + worker thread 无 否 是 否 是 是 中 是 中 是 是 变 worker-thread-per-connection 8 reactor + thread pool 无 否 是 否 是 是 高 是 低 是 否 常 主线程 IO, 工作线程计算 9 reactors in threads 无 否 是 否 是 是 高 是 低 是 是 常 one loop per thread (muduo) 10 reactors in processes 无 是 否 否 是 是 高 是 低 否 是 常 one loop per process (Nginx) 11 reactors + thread pool 无 否 是 否 是 是 高 是 低 是 否 常 最灵活的 IO 与 CPU 配置 并发性 多核 开销 互通 顺序性 线程数 特点

162 6 muduo UNP CSDA 0 5 5 Reactor muduo 6 7 8 9 3.3 micro benchmark Thread_ bench.cc BlockingQueue_bench.cc E5320 Linux 2.6.32 fork()+exit(): 534.7µs pthread_create()+pthread_join(): 42.5µs 26.1µs push/pop a blocking queue : 11.5µs Sudoku resolve: 100us 20 200µs 0 iterative [UNP] Figure 1.9 [UNP] daytime write-only Python 0 echo server Python recipes/python/echo-iterative.py 3 import socket 4 5 def handle(client_socket, client_address): 6 while True: 7 data = client_socket.recv(4096) 8 if data: 9 sent = client_socket.send(data) # sendall? 10 else: 11 print "disconnect", client_address 12 client_socket.close() 13 break 14 15 if name == " main ": 16 listen_address = ("0.0.0.0", 2007) 17 server_socket = socket.socket(socket.af_inet, socket.sock_stream) 18 server_socket.bind(listen_address) 19 server_socket.listen(5) 20 21 while True: 22 (client_socket, client_address) = server_socket.accept() 23 print "got connection from", client_address 24 handle(client_socket, client_address) recipes/python/echo-iterative.py L6~L13 echo L21~L24

6.6 muduo 163 L9 sendall() 1 Unix [UNP] child-per-client fork()-per-client process-per-connection PostgreSQL Perforce fork() fork() Sudoku Python L9~L16 self.request client_socket ForkingTCPServer EchoHandler.handle() IO recipes/python/echo-fork.py 1 #!/usr/bin/python 2 3 from SocketServer import BaseRequestHandler, TCPServer 4 from SocketServer import ForkingTCPServer, ThreadingTCPServer 5 6 class EchoHandler(BaseRequestHandler): 7 def handle(self): 8 print "got connection from", self.client_address 9 while True: 10 data = self.request.recv(4096) 11 if data: 12 sent = self.request.send(data) # sendall? 13 else: 14 print "disconnect", self.client_address 15 self.request.close() 16 break 17 18 if name == " main ": 19 listen_address = ("0.0.0.0", 2007) 20 server = ForkingTCPServer(listen_address, EchoHandler) 21 server.serve_forever() recipes/python/echo-fork.py 2 Java thread-per-connection Java 1.4 NIO Java 1 Sudoku scheduler Python ThreadingTCPServer EchoHandler.handle()

164 6 muduo $ diff -U2 echo-fork.py echo-thread.py if name == " main ": listen_address = ("0.0.0.0", 2007) - server = ForkingTCPServer(listen_address, EchoHandler) + server = ThreadingTCPServer(listen_address, EchoHandler) server.serve_forever() EchoHandler.handle() 0 $ diff -U2 echo-fork.py echo-single.py if name == " main ": listen_address = ("0.0.0.0", 2007) - server = ForkingTCPServer(listen_address, EchoHandler) + server = TCPServer(listen_address, EchoHandler) server.serve_forever() 3 1 [UNP] accept(2) thundering herd 4 2 [UNP] 3 4 Apache httpd thread of control read() TCP read() write() / read() TCP echo client stdin proxy a b b a proxy A TCP / [UNP] echo client 7.13 Python TCP relay Python Pinhole http://code.activestate.com/recipes/114642/ IO multiplexing select/poll/epoll/kqueue thread of control IO IO select/poll non-blocking IO non-blocking IO buffer 7.4 IO multiplexing event-driven Doug Schmidt

6.6 muduo 165 Reactor event-driven Reactor / libevent muduo Netty twisted POE proxy Python IO multiplexing echo server 29 non-blocking L28 socket L14 L15~L32 IO fileno L16, L17 listening fd accept IO watch list connections L18~L23 L24~L32 echo L28 recipes/python/echo-poll.py 6 server_socket = socket.socket(socket.af_inet, socket.sock_stream) 7 server_socket.setsockopt(socket.sol_socket, socket.so_reuseaddr, 1) 8 server_socket.bind(('', 2007)) 9 server_socket.listen(5) 10 # server_socket.setblocking(0) 11 poll = select.poll() # epoll() should work the same 12 poll.register(server_socket.fileno(), select.pollin) 13 14 connections = {} 15 while True: 16 events = poll.poll(10000) # 10 seconds 17 for fileno, event in events: 18 if fileno == server_socket.fileno(): 19 (client_socket, client_address) = server_socket.accept() 20 print "got connection from", client_address 21 # client_socket.setblocking(0) 22 poll.register(client_socket.fileno(), select.pollin) 23 connections[client_socket.fileno()] = client_socket 24 elif event & select.pollin: 25 client_socket = connections[fileno] 26 data = client_socket.recv(4096) 27 if data: 28 client_socket.send(data) # sendall() partial? 29 else: 30 poll.unregister(fileno) 31 client_socket.close() 32 del connections[fileno] recipes/python/echo-poll.py IO multiplexing 29 http://scotdoyle.com/python-epoll-howto.html#async-examples

166 6 muduo listen TCP 3 L28~L30 $ diff echo-poll.py chat-poll.py -U4 --- echo-poll.py 2012-08-20 08:50:49.000000000 +0800 +++ chat-poll.py 2012-08-20 08:50:49.000000000 +0800 23 elif event & select.pollin: 24 clientsocket = connections[fileno] 25 data = clientsocket.recv(4096) 26 if data: 27 - clientsocket.send(data) # sendall() partial? 28 + for (fd, othersocket) in connections.iteritems(): 29 + if othersocket!= clientsocket: 30 + othersocket.send(data) # sendall() partial? 31 else: 32 poll.unregister(fileno) 33 clientsocket.close() 34 del connections[fileno] Doug Schmidt routine Reactor Windows GUI IO multiplexing WndProc switch-case GUI 1 LRESULT CALLBACK WndProc(HWND hwnd, UINT message, WPARAM wparam, LPARAM lparam) 2 { 3 switch (message) 4 { 5 case WM_DESTROY: 6 PostQuitMessage(0); 7 return 0; 8 // many more cases 9 } 10 return DefWindowProc (hwnd, message, wparam, lparam) ; 11 } Reactor IO

6.6 muduo 167 Reactor 6-11 select/poll/epoll_wait IO Reactor IO socket poll poll 6-11 compute() 100ms sleep() IO IO Thread IO Thread Conn 1 readable poll Conn 1 readable poll read decode Conn 2 readable read 1 decode compute compute encode encode Conn 2 readable write poll read decode compute encode write poll write 1 poll read 2 decode compute encode write 2 poll 6-11 5 Reactor 6-11 server_basic.cc IO CPU 2 5 2 read(2) 5 poll(2) read(2)

168 6 muduo Python Reactor L42~L46 handlers listening fd handle_accept handler handle_request handle_input handle_input recipes/python/echo-reactor.py 6 server_socket = socket.socket(socket.af_inet, socket.sock_stream) 7 server_socket.setsockopt(socket.sol_socket, socket.so_reuseaddr, 1) 8 server_socket.bind(('', 2007)) 9 server_socket.listen(5) 10 # serversocket.setblocking(0) 11 12 poll = select.poll() # epoll() should work the same 13 connections = {} 14 handlers = {} 15 16 def handle_input(socket, data): 17 socket.send(data) # sendall() partial? 18 19 def handle_request(fileno, event): 20 if event & select.pollin: 21 client_socket = connections[fileno] 22 data = client_socket.recv(4096) 23 if data: 24 handle_input(client_socket, data) 25 else: 26 poll.unregister(fileno) 27 client_socket.close() 28 del connections[fileno] 29 del handlers[fileno] 30 31 def handle_accept(fileno, event): 32 (client_socket, client_address) = server_socket.accept() 33 print "got connection from", client_address 34 # client_socket.setblocking(0) 35 poll.register(client_socket.fileno(), select.pollin) 36 connections[client_socket.fileno()] = client_socket 37 handlers[client_socket.fileno()] = handle_request 38 39 poll.register(server_socket.fileno(), select.pollin) 40 handlers[server_socket.fileno()] = handle_accept 41 42 while True: 43 events = poll.poll(10000) # 10 seconds 44 for fileno, event in events: 45 handler = handlers[fileno] 46 handler(fileno, event) recipes/python/echo-reactor.py

6.6 muduo 169 handle_input $ diff echo-reactor.py chat-reactor.py -U1 def handle_input(socket, data): - socket.send(data) # sendall() partial? + for (fd, other_socket) in connections.iteritems(): + if other_socket!= socket: + other_socket.send(data) # sendall() partial? IO IO IO Windows GUI 6 Sudoku Reactor CPU 8 out-of-order 2 Sudoku 1 id response request 7 IO thread-per-connection 2 7 6 client CPU 6 TCP burst requests 8 core 7 12.5% CPU 8 9 8 6 6-12 IO Reactor thread pool IO Sudoku Solver examples/sudoku/server_threadpool.cc

170 6 muduo IO Thread Threads in pool Conn 1 readable poll wait wait Conn 2 readable read poll read put in pool put in pool decode poll write 2 write 1 poll compute encode wait decode compute encode wait 6-12 8 Reactor 5 onmessage() ThreadPool 8 id $ diff server_basic.cc server_threadpool.cc -u --- server_basic.cc 2012-04-20 20:19:56.000000000 +0800 +++ server_threadpool.cc 2012-06-10 22:15:02.000000000 +0800 @@ -96,16 +100,7 @@ void onmessage(const TcpConnectionPtr& conn,... if (puzzle.size() == implicit_cast<size_t>(kcells)) { - string result = solvesudoku(puzzle); - if (id.empty()) - { - conn->send(result+"\r\n"); - } - else - { - conn->send(id+":"+result+"\r\n"); - } + threadpool_.run(boost::bind(&solve, conn, puzzle, id)); } @@ -114,17 +109,40 @@ + static void solve(const TcpConnectionPtr& conn, + const string& puzzle, + const string& id) + {

6.6 muduo 171 + string result = solvesudoku(puzzle); + if (id.empty()) + { + conn->send(result+"\r\n"); + } + else + { + conn->send(id+":"+result+"\r\n"); + } + } + EventLoop* loop_; TcpServer server_; + ThreadPool threadpool_; Timestamp starttime_; }; IO Java Servlet 2.x IO fsync(2)/fdatasync(2) 30 IO Reactor 9 Reactor 9 muduo Netty one loop per thread main Reactor accept(2) sub Reactor muduo round-robin sub Reactor sub Reactor CPU muduo Reactor pool CPU 8 11 IO Reactor 8 9 thread pool Reactor IO IO muduo 6-13 30 Linux IO http://antirez.com/post/fsync-different-threaduseless.html

172 6 muduo Base IO Thread IO Thread 1 IO Thread 2 Conn 1 established poll poll poll Conn 2 established accept poll assign poll accept assign Conn 3 established poll accept assign poll Conn 1 readable Conn 2 readable poll poll read decode compute read encode decode write poll 6-13 9 examples/sudoku/server_multiloop.cc server_basic.cc server_.setthreadnum(numthreads); $ diff server_basic.cc server_multiloop.cc -up --- server_basic.cc 2011-06-15 13:40:59.000000000 +0800 +++ server_multiloop.cc 2011-06-15 13:39:53.000000000 +0800 @@ -21,19 +21,22 @@ class SudokuServer - SudokuServer(EventLoop* loop, const InetAddress& listenaddr) + SudokuServer(EventLoop* loop, const InetAddress& listenaddr, int numthreads) : loop_(loop), server_(loop, listenaddr, "SudokuServer"), starttime_(timestamp::now()) { server_.setconnectioncallback( boost::bind(&sudokuserver::onconnection, this, _1)); server_.setmessagecallback( boost::bind(&sudokuserver::onmessage, this, _1, _2, _3)); + server_.setthreadnum(numthreads); } 10 Nginx 11 8 9 Reactor IO IO IO

6.6 muduo 173 6-14 Base IO Thread IO Thread 1 IO Thread 2 Threads in pool Conn 1 established poll poll poll wait wait Conn 2 established accept poll assign poll accept assign poll Conn 2 readable poll Conn 1 readable read read decode decode put in pool compute poll put in pool poll encode compute write wait encode write poll wait 6-14 8 server_.setthreadnum(numthreads); event loop event loops ZeroMQ 31 event loop event loop muduo::tcpserver::setthreadnum() event loop IO event loop IO thread pool event loop TCP TCP event loop event loop muduo event loop 10 31 http://www.zeromq.org/area:faq#toc3

174 6 muduo 10 event loop 1ms event loop event handler epoll_wait() event loop event loop 3.3 C++ one loop per thread + thread pool event loop non-blocking IO thread pool 2010 2 : Reactor in-house C++ muduo 32 5 muduo 4 6-2 6-2 IO 2 thread-per-connection 1 N 5 Reactor 1 8 Reactor + 1 C 2 9 one loop per thread 1 C 1 11 one loop per thread + 1 C 1 C 2 6-2 N C 1 C 2 CPU 32 Characteristics of multithreading models for high-performance IO driven network applications http://arxiv.org/ftp/arxiv/papers/0909/0909.4934.pdf

6.6 muduo 175 IO TCP read()/write() 5 IO concurrent parallel 8 Reactor 5 9 5 VIP VIP 5 VIP 6-11 muduo IO 11 IO 4.6 9 9

3

11.5 boost::function boost::bind 447 pimpl explicit forward pimpl pimpl C++ C libevent2 struct event_base opaque pointer libevent non-virtual virtual virtual function bind-byvtable-offset non-virtual function bind-by-name loader resolution mangled name Internet IP C Java JNI C Python/Perl/Ruby C C C Linux class free function muduo/base/ Timestamp.h class Timestamp muduo::timedifference() free function C++ Java 11.5 boost::function boost::bind boost::function boost::bind boost::function boost::bind std::tr1 C++11 C++ Scott Meyers [EC3 35] boost::function boost:bind function/bind 11 12 class hierarchy class class 11 http://blog.csdn.net/myan/archive/2010/10/09/5928531.aspx 12 http://blog.csdn.net/myan/archive/2010/09/14/5884695.aspx

448 11 C++ class class class C++ wrapper adapter 13 C++ : boost::function boost::bind x A x is-a A B x is more like a B C++ interfaces 14 Ruby duck typing Google Go 15 tag object operations 13 Linus 2007 C++ C++ http://blog.csdn.net/turingbook/article/details/1775488 14 http://www.cnblogs.com/solstice/archive/2011/04/22/2024791.html 15 http://golang.org/doc/go_lang_faq.html#inheritance

11.5 boost::function boost::bind 449 C++ better C data abstraction OO GP object based global function C++ Handler class onconnect() ondisconnect() onmessage() ontimer() override C++ MyHandler Handler MyHandler delete override OO framework Java 16 C++ GC MyHandler delete this; MyHandler TCP MyHandler HandlerFactory MyHandlerFactory C++ 2005 TR1 C++11 boost::function + boost::bind std::tr1::function + std::tr1::bind C++11 std::function + std::bind muduo boost::function TCP IO boost::bind class boost::function muduo muduo 16 Java 8 Closure C# delegate

第 11 章 450 反思 C++ 面向对象与虚函数 言归正传 说说 boost::function 和 boost::bind 取代虚函数的具体做法 11.5.1 基本用途 boost::function 就像 C# 里的 delegate 可以指向任何函数 包括成员函数 当 用 bind 把某个成员函数绑到某个对象上时 我们得到了一个 closure 闭包 例如 class Foo { public: void methoda(); void methodint(int a); void methodstring(const string& str); }; class Bar { public: void methodb(); }; boost::function<void()> f1; // 无参数 无返回值 Foo foo; f1 = boost::bind(&foo::methoda, &foo); f1(); // 调用 foo.methoda(); Bar bar; f1 = boost::bind(&bar::methodb, &bar); f1(); // 调用 bar.methodb(); f1 = boost::bind(&foo::methodint, &foo, 42); f1(); // 调用 foo.methodint(42); f1 = boost::bind(&foo::methodstring, &foo, "hello"); f1(); // 调用 foo.methodstring("hello") // 注意 bind 拷贝的是实参类型 (const char*) 不是形参类型 (string) // 这里形参中的 string 对象的构造发生在调用 f1 的时候 而非 bind 的时候 // 因此要留意 bind 的实参 (cosnt char*) 的生命期 它应该不短于 f1 的生命期 // 必要时可通过 bind(&foo::methodstring, &foo, string(atempbuf)) 来保证安全 boost::function<void(int)> f2; // int 参数 无返回值 f2 = boost::bind(&foo::methodint, &foo, _1); f2(53); // 调用 foo.methodint(53); 如果没有 boost::bind 那么 boost::function 就什么都不是 而有了 bind 同 一个类的不同对象可以 delegate 给不同的实现 从而实现不同的行为 孟岩 简直 就无敌了 Linux 多线程服务端编程 使用 muduo C++ 网络库 (excerpt) http://www.chenshuo.com/book/

11.5 boost::function boost::bind 451 11.5.2 class C++ 1 OO Thread base class Thread::run() derived class run() Thread Java Thread class class method helper class(es) OO boost::function Thread ThreadCallback ThreadCallback Thread Thread::start() Java Thread Runnable C# Thread delegate ThreadStart boost::thread // boost::function Thread class class Thread { public: typedef boost::function<void()> ThreadCallback; Thread(ThreadCallback cb) : cb_(cb) { } void start() { /* some magic to call run() in new created thread */ } private: void run() { cb_(); } ThreadCallback cb_; //... };

452 11 C++ class Foo // { public: void runinthread(); void runinanotherthread(int) }; Foo foo; Thread thread1(boost::bind(&foo::runinthread, &foo)); Thread thread2(boost::bind(&foo::runinanotherthread, &foo, 43)); thread1.start(); // thread2.start(); 2 boost::function NetServer class EchoService NetServer main() 17 network library class Connection; class NetServer : boost::noncopyable { public: typedef boost::function<void (Connection*)> ConnectionCallback; typedef boost::function<void (Connection*, const void*, int len)> MessageCallback; NetServer(uint16_t port); ~NetServer(); void registerconnectioncallback(const ConnectionCallback&); void registermessagecallback(const MessageCallback&); void sendmessage(connection*, const void* buf, int len); private: //... }; network library user code class EchoService { public: // NetServer::sendMessage typedef boost::function<void(connection*, const void*, int)> SendMessageCallback; EchoService(const SendMessageCallback& sendmsgcb) : sendmessagecb_(sendmsgcb) // boost::function { } 17 muduo

11.5 boost::function boost::bind 453 // NetServer::MessageCallback void onmessage(connection* conn, const void* buf, int size) { printf("received Msg from Connection %d: %.*s\n", conn->id(), size, (const char*)buf); sendmessagecb_(conn, buf, size); // echo back } // NetServer::ConnectionCallback void onconnection(connection* conn) { printf("connection from %s:%d is %s\n", conn->ipaddr(), conn->port(), conn->connected()? "UP" : "DOWN"); } private: SendMessageCallback sendmessagecb_; }; // int main() { NetServer server(7); EchoService echo(bind(&netserver::sendmessage, &server, _1, _2, _3)); server.registermessagecallback( bind(&echoservice::onmessage, &echo, _1, _2, _3)); server.registerconnectioncallback( bind(&echoservice::onconnection, &echo, _1)); server.run(); } user code 11.5.3 class concept OO attribute/annotation JUnit 3.x void test*() JUnit 4.x NUnit 2.x annotation/attribute test case

454 11 C++ closure closure boost::function+boost::bind class object-based closure OO Factory Method boost::function<base* ()> Strategy Strategy ConcreteStrategyA ConcreteStrategyB boost::function Command boost::function Command Template Method boost::function boost::function 23 C++ 18 19 closure paradigm EchoService EchoService SendMessageCallback DataSink 18 19 http://norvig.com/design-patterns/

11.5 boost::function boost::bind 455 AbstractDataSink interface sendmessage() class NetDataSink MockDataSink EchoService AbstractDataSink* SendMessageCallback boost::bind() MockServer OO public boost::noncopyable boost::enable_ shared_from_this 1.11 enable_shared_from_this IO multiplexing select() POSIX poll() Linux epoll() FreeBSD kqueue() NetLoop base class classes switch-case Penguin Bird Bird virtual function fly() interface Flyable Runnable Runnable Flyable Runnable interface method interface tag boost::function

456 11 C++ class Penguin // { public: void run(); void swim(); }; class Sparrow // { public: void fly(); void run(); }; // boost::function typedef boost::function<void()> FlyCallback; typedef boost::function<void()> RunCallback; typedef boost::function<void()> SwimCallback; // run fly class class Foo { public: Foo(FlyCallback flycb, RunCallback runcb) : flycb_(flycb), runcb_(runcb) { } private: FlyCallback flycb_; RunCallback runcb_; }; // run swim class class Bar { public: Bar(SwimCallback swimcb, RunCallback runcb) : swimcb_(swimcb), runcb_(runcb) { } private: SwimCallback swimcb_; RunCallback runcb_; }; int main() { Sparrow s; Penguin p; // Foo Bar Foo foo(bind(&sparrow::fly, &s), bind(&sparrow::run, &s)); Bar bar(bind(&penguin::swim, &p), bind(&penguin::run, &p)); }

4

A Sockets API TCP/IP A.1.5 x86-64 Linux Sockets API PC PC Windows 1. 2. W. Richard Stevents TCP TCP TCP A.1 561

562 A A.1.1 Sockets API Sockets API 2005 C# PC C# PDA PDA TCP PDA PC 42 PC PDA PC 7 PC FTP PC PDA single point of failure TCP read/write blocking 2010 muduo Sockets API Reactor C++ Sockets API muduo Sockets API TCP Sockets API TCP 6.4.1 TCP Sockets API A.1.5

A.1 563 A.1.2 TCP UDP client library server libmemcached memcached libpq PostgreSQL Servlet HTTP RPC TCP/IP troubleshooting library framework Sockets API Sockets API strace Sockets API TCP/IP tcpdump A.1.3 Linux 10 FreeBSD FreeBSD 2000 2000 Linux epoll FreeBSD 2001 4.1 kqueue C10k 10 2011 6 Linux 1 Linux bug work around FreeBSD Linux Linux A.1.4 Windows Linux FreeBSD Solaris 1 http://en.wikipedia.org/wiki/usage_share_of_operating_systems

564 A AIX HP-UX Linux POSIX Linux SO_NOSIGPIPE Linux pipe(2) FreeBSD muduo Windows libevent libuv Java Netty A.1.5 big topic ping ping bug 1. firmware 2. 3. TCP/IP 4. HTTP FTP DNS SMTP POP3 NFS 5. HAProxy squid varnish Web load balancer 6. ZooKeeper memcached 7. 8. POP3 SMTP

A.1 565 7 TCP/IP Sockets API muduo libevent muduo Netty gevent muduo A.1.6 HTTP HTTP 1.1 bug spec HTTP 1.1 9.2.2 TCP accept 2 accept HAProxy lighttpd CPU one loop per thread 3.3 6.6 httpd Nginx lighttpd 2 Linux http://linux.dell.com/files/presentations/linux_plumbers_conf_2010/scaling_techniques_ for_servers_with_high_connection%20rates.pdf

566 A IO bound CPU Disk IO trade-off echo qps 1ms 8 8 000 qps IO A.1.7 3 master NAT TCP HTTP proxy HTTP server Web crawler HTTP HTTP proxy 3 http://blog.csdn.net/solstice/article/details/5334243 3

A.1 567 Web browser HTTP server 4 wget A.1.8 7 24 7 24 7 24 9.2 7 24 7 24 7 24 5 6 64-bit malloc memcached malloc memory pool 12.2.8 4 http://blog.codingnow.com/2006/04/iocp_kqueue_epoll.html 5 http://stackoverflow.com/questions/3770457/what-is-memory-fragmentation 6 http://stackoverflow.com/questions/60871/how-to-solve-memory-fragmentation

568 A Linux Kernel PC 3% 5% STL new/delete premature optimization A.1.9 RFC daytime HTTP 1.0 echo chargen TIME_WAIT hold TIME_WAIT TIME_WAIT (s) TIME_WAIT buggy TIME_WAIT XML JSON Protobuf

A.1 569 race condition p. 348 snapshot delta 30 end-to-end principle happens-before relationship A.1.10 7 STL STL STL STL STL 1. 2. TCP/IP 3. TCP/IP stack UNIX TCP/IP TCP/IP manpage manpage TCP/IP Linux TCP/IP 1. TCP self-connection 8 2. Linux bug TCP TCP window clamping bug work around manpage 7 http://jjhou.boolan.com/programmer-5-talk.htm 8 8.11 ACE http://blog.csdn.net/solstice/article/details/5364096

570 A error code ephemeral port backlog connect manpage TCP/IP TCP/IP TCP TCP/IP send_packet() on_receive_packet() on_timer() libnet libpcap TCP/IP TCP/IP lwip Mini/Tiny/Toy/Trivial/Yet-Another TCP/IP TUN/TAP TCP/IP D Sockets API FTDI USB-SPI ENC28J60 TCP/IP stack IP ICMP Echo TCP 3000 UDP DNS A.1.11 TCP echo chat proxy echo echo HTTP chat a b fork()-per-connection b c b a c

A.1 571 proxy 7.13 TCP A.1.12 Sockets API IPython muduo 9 Sockets API IPython IO C TCP epoll $ ipython In [1]: import socket, select In [2]: s = socket.socket(socket.af_inet, socket.sock_stream) In [3]: s.setsockopt(socket.sol_socket, socket.so_reuseaddr, 1) In [4]: s.bind(('', 5000)) In [5]: s.listen(5) In [6]: client, address = s.accept() # client.fileno() == 4 In [7]: client.recv(1024) Out[7]: 'Hello\n' # In [8]: epoll = select.epoll() In [9]: epoll.register(client.fileno(), select.epollin) # In [10]: epoll.poll(60) Out[10]: [(4, 1)] # # 4 select.epollin == 1 In [11]: client.recv(1024) # Out[11]: 'World\n' In [12]: client.setblocking(0) # In [13]: client.recv(1024) # EAGAIN == 11 error: [Errno 11] Resource temporarily unavailable In [14]: epoll.poll(60) Out[14]: [(4, 1)] In [15]: client.recv(1024) Out[15]: 'Bye!\n' # epoll_wait() # In [16]: client.close() nc 9 http://blog.csdn.net/solstice/article/details/5497814

572 A $ nc localhost 5000 Hello <enter> World <enter> Bye! <enter> muduo log strace netcat/tempest/ipython tcpdump muduo Reactor Linux C++ IO 4000 A.1.13 TCP TCP Effective TCP/IP Programming 9 Realize That TCP Is a Reliable Protocol, Not an Infallible Protocol TCP 7.5 Google Protobuf check sum IP header check sum TCP header check sum CRC32 TCP IP header TCP header checksum 16-bit check sum 16-bit integers checksum sum 16-bit checksum CRC32 CRC A-1 client server TCP segment segment IP packet ethernet frame A-1 a router ethernet frame b c server d CRC a b c d TCP header checksum payload router NAT NAT c a d payload TCP header checksum

A.2 573 client a server d switch 1 switch 2 b c router A-1 IP bit payload check sum check sum ethernet CRC When the CRC and TCP checksum disagree The Limitations of the Ethernet CRC and TCP/IP checksums for error detection 10 Amazon S3 2008 7 11 bit check sum Google 12 MD5 end-to-end principle A.2 Unix W. Richard Stevens 6 [APUE] UNIX TCP/IP 10 http://noahdavids.org/self_published/crc_and_checksum.html 11 http://status.aws.amazon.com/s3-20080720.html 12 http://www.ukuug.org/events/spring2007/programme/thatcouldnthappentous.pdf 14

574 A [UNPv2] [APUE] IPC TCP/IP TCP/IP TCP/IP Illustrated, Vol. 1: The Protocols TCP/IP TCPv1 TCPv1 13 TCPv1 TCP/IP W. Richard Stevens TCP/IP tcpdump TCP 17 24 TCP packet TCP 1. Positive acknowledgement with retransmission 2. Flow control using sliding window Nagle 3. Congestion control slow start congestion avoidance fast retransmit TCP TCP flow control TCP connection TCPv1 1993 10Mbit 100Mbit switch hub 1Gbit 13 http://portal.acm.org/citation.cfm?id=161724

A.2 575 10Gbit TCP flow control TCP IPv6 TCP window scale option TCP timestamps option TCP selective ack option Linux tcpdump TCP TCPv1 2011 10 2 Unix Network Programming, Vol. 1: Networking API 2 3 3 XTI UNP W. Richard Stevens UNP 2 UNP 3 UNP Sockets API Sockets API W. Richard Stevens UNP 2 14 I have found when teaching network programming that about 80% of all network programming problems have nothing to do with network programming, per se. That is, the problems are not with the API functions such as accept and select, but the problems arise from a lack of understanding of the underlying network protocols. For example, I have found that once a student understands TCP s three-way handshake and four-packet connection termination, many network programming problems are immediately understood. TCP/IP Nagle UNP 3 UNP UNIX UNP UDP TCP IPv4 IPv6 15 14 http://www.kohala.com/start/preface.unpv12e.html 15 http://blog.csdn.net/myan/archive/2010/09/11/5877305.aspx

576 A MFC Scott Meyers Effective C++ W. Richard Stevens Jeffrey Richter CRC32 check sum check sum CRC CRC CRC CRC zlib UNP Sockets Out-of-Band Data Signal-Driven IO UNP UNP daytime echo TCP

A.2 577 W. Richard Stevens UNP UNP Unix accept() + fork() C10k TCPv1 UNP TCPv1 UNP APUE Effective TCP/IP Programming W. Richard Stevens W. Richard Stevens UNP Effective TCP/IP Programming 6 TCP TCP 2001 2011 TCP/IP Illustrated, Vol. 2: The Implementation TCPv2 1200 4.4BSD TCP/IP 15 000 C Gary Wright mbuf IP ICMP IP IGMP IP Sockets ARP 3/4 TCP TCP TCPv2 IGMP IP host-to-host IP packet

578 A TCP TCP/IP stack inode BSD 20 80 4MiB VAX TCP/IP mbuf buffer TCP/IP 4.4BSD timer Linux TCP/IP Linux TCP/IP 3 Pattern-Oriented Software Architecture Volume 2: Patterns for Concurrent and Networked Objects POSA2 UNP UNP Sockets API POSA2 library/framework event-driven libevent Java Netty Java Mina Perl POE Python Twisted POSA2 C++ RAII

B C++ Primer 4 C++ C++ Primer 4 34 B.1 C++ 2009 Stanley Lippman C++ C++ Java C# Python C++ C++ 1 C++ 2 3 C++ 10% 10% 4 1 http://shootout.alioth.debian.org/ Google https://days2011.scala-lang.org/sites/days2011/files/ws3-1-hundt.pdf 2 C++ Bjarne Stroustrup C++ http://www2.research.att.com/~bs/applications.html 3 C++ http://aristeia.com/talknotes/misra_day_2010.pdf 4 Milo Yip C++ Unreal/Source Havok/FMOD C++ http://www.cnblogs.com/miloyip/archive/2010/09/17/behind_cplusplus.html 579

580 B C++ Primer 4 C++ GC C++ 5 C++ Java C# C++ Bjarne Stroustrup C++ system programming 6 infrastructure C++ 7 Herb Sutter 8 C++ efficiency flexibility 9 abstraction productivity 10 C++ is about efficient programming with abstractions C++ 11 C++ 12 13 & inline int find_longest(const std::vector<std::string>& words) { // std::max_element(words.begin(), words.end(), LengthCompare()); } CPU C++ memory layout locality of reference 5 C++ clean up http://blog.csdn.net/myan/article/details/1906 6 CPU 7 Software Development for Infrastructure http://www2.research.att.com/~bs/computer-jan12.pdf 8 Herb Sutter C++ and Beyond 2011 Why C++? http://channel9.msdn.com/posts/c-and-beyond-2011-herb-sutter-why-c 9 just-in-time compilation 10 Stanley Lippman Linux 1970 C++ C++ 11 Ulrich Drepper Stop Underutilizing Your Computer SIMD http://www.redhat.com/f/pdf/summit/udrepper_945_stop_underutilizing.pdf 12 Technical Report on C++ Performance http://www.open-std.org/jtc1/sc22/wg21/docs/18015.html 13 Scott Meyers Effective C++ in an Embedded Environment http://www.artima.com/shop/effective_cpp_in_an_embedded_environment

B.2 C++ 581 memory hierarchy 14 Scott Meyers CPU Caches and Why You Care 15 Herb Sutter Machine Architecture 16 / GC 17 C++ C Java Python TIOBE C++ Java B.2 C++ C++ features Google 18 C++ C++ 19 20 C++ C++ Bjarne Stroustrup The C++ Programming Language Stanley Lippman C++ Primer C++ 21 C++ Primer 10 C++ 14 std::list O(1) std::vector O(N) vector compact vector list http://ecn.channel9.msdn.com/events/goingnative12/gn12cpp11style.pdf vector 15 http://aristeia.com/talknotes/accu2011_cpucaches.pdf 16 http://www.nwcpp.org/downloads/2007/machine_architecture_-_nwcpp.pdf 17 Bjarne Stroustrup Abstraction and the C++ machine model C++ Java http://www2.research.att.com/ bs/abstraction-and-machine.pdf 18 50% http://blog.csdn.net/myan/article/details/3144661 19 C++ 20 Java C# Python 21 C++ Primer 3/e http://jjhou.boolan.com/cpp-primer-foreword.pdf

582 B C++ Primer 4 C++ C++ Primer 3 C++ Primer 4 4 3 22 3 4 Barbara Moo C++ Primer 4 C++ Huffman C++ C++ syntax semantics C++ STL C++ 23 C++ C++ 24 1 C++ C++ Primer 4 C C++ C++ C C++ Primer 4 C++ C++ 25 26 C++ 900 22 Bjarne Stroustrup Programming Principles and Practice Using C++ use only the 4th edition 23 C++ Primer 4/e 24 22 Accelerated C++ C : C 25 iostream locale/facet 26 Stanley Lippman Virtual base class support wanders off into the Byzantine... The material is simply too esoteric to warrant discussion...

B.2 C++ 583 C++ Primer 4 C++ 3 string vector class string C++ Primer 4 10.4.2 10.6 15.9 Liskov C++ Primer 4 2005 2003 C++ 27 C++ 2011 C++11 TR1 28 C++11 29 C++ C++ Primer 4 C++ GNU G++ Visual C++ 30 C++ C++ C++ C++ 31 C++ C++ C++ C C++ C++ C++ 32 27 1998 C++ 28 TR1 2005 C++ bind/function 29 C++ Primer 5 C++11 30 G++ Linux Unix Visual C++ Windows C++ Intel C++ Linux G++ Windows Visual C++ 31 Cfront http://www.softwarepreservation.org/projects/c_plus_plus 32 C++ http://stackoverflow.com/questions/3931312

584 B C++ Primer 4 C++ C++ 33 C++ 34 C++ module package C C++ 35 vector<t> vector int std::string B-1 3 + 2 4 B-1 36 calculate() + Node 3 * NumberNode BinaryNode 2 4 AddNode MultiplyNode B-1 33 G++ 4.x 32-bit 64-bit 64-bit C++ Visual C++ 2010 Express Visual C++ 6.0 34 C++ C++ 10 35 std::vector<int> 36 Packrat http://www.relisoft.com/book/lang/poly/3tree.html

B.3 585 B-2 BinaryNode<std::plus<double> > BinaryNode<std:: multiplies<double> > BinaryNode<T> Node NumberNode BinaryNode<T> B-2 15.8 Handle C++ C++ C++ 37 C++ C++ 38 Google Protobuf leveldb PCRE C++ muduo Chromium Google C++ STL Boost C++ C++ C Java C++ B.3 C++ C++ Primer C++ Effective C++ 3 39 [EC3] 37 http://blog.csdn.net/myan/article/details/3247071 38 39 Scott Meyers

586 B C++ Primer 4 C++ C++ C++ Primer C++ Effective C++ 3 Effective C++ 3 C++ 2 3 C++ C++ override 2 40 3 class 41 C++ idiom RAII 42 TCP RAII C++ 5 43 C++ delete C++ Effective C++ 3 3 C++ C 44 TR1 bind/function 45 Stephan T. Lavavej PPT 40 Andrew Koenig Teaching C++ Badly: Introduce Constructors and Destructors at the Same Time http://drdobbs.com/blogs/cpp/229500116 41 std::string std::vector boost::shared_ptr class 42 heap 43 TR1 shared_ptr weak_ptr boost::scoped_ptr 44 Java 7 try-with-resources Python with C# using 45 function/bind http://blog.csdn.net/myan/article/details/5928531

B.3 587 TR1 46 47 STL 48 C++ Primer C++ C++ 49 [CCS] concept model refinement C++ STL STL 50 C++ type traits C++ 51 C 52 C++ C++ C++ Effective C++ 3 1 31 C++ C/C++ 53 46 http://blogs.msdn.com/b/vcblog/archive/2008/02/22/tr1-slide-decks.aspx 47 48 Matthew Austern 49 Herb Sutter 50 Iterator http://jjhou.boolan.com/programmer-3-traits.pdf 51 C++ http://blog.csdn.net/myan/article/details/1920 52 C++ 53 http://www.math.pku.edu.cn/teachers/qiuzy/technotes/expression2009.pdf

588 B C++ Primer 4 C++ 35 class 43 pimpl C++ 56 swap() swap() 59 #include using 73 by value by reference 76 vector 79 value smart pointer 5 entity 6 8 9 22 32 class value class base class trait class policy class exception class 33 class monolithic class 37 public 57 class namespace C++ Google C++ 54 LLVM 55 B.4 16 2010 11 54 http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#exceptions 55 http://llvm.org/docs/codingstandards.html#ci_rtti_exceptions

B.4 589 PDF C++ 15% 56 5 C++11 57 TR1 TR1 URL C++ [CCS] soxxxxxx http://stackoverflow.com/questions/xxxxxx http://www.informit.com/store/product.aspx?isbn=0201721481 http://book.douban.com/subject/10944985/ PDF http://chenshuo.com/cp4/ giantchen@gmail.com http://weibo.com/giantchen 2012 5 56 10 11 lower_bound upper_bound 57 Scott Meyers C++11 http://www.artima.com/shop/overview_of_the_new_cpp

C Boost Boost C++ Boost 1 C++ STL Boost vector map shared_ptr vector map shared_ptr weak_ptr C++ Boost Boost Boost noncopyable scoped_ptr static_assert Boost date_time 2 circular_buffer function/bind shared_ptr C++ delete lexical_cast Boost Boost 1 2010 8 2 boost::date_time muduo::timezone 591

592 C Boost regex RegEx class class RegEx regex immutable mutable basic_regex match_results match_regex regex Donald Knuth Coders at Work C/C++ C strlen strcpy strcmp C++ complex string vector class STL STL Boost Boost Boost Boost generic programming concept model refinement Boost.Threads STL Boost Boost.Preprocessor C++ Lua Boost.Proto C++ ANTLR parser Boost.Spirit Boost C++

D TCP TCP 1 IP TCP TCP/IP IPv6 TCP 65536 TCP 1.2.3.4:8765 A B B frame A 10 TCP B 100 TCP Linux socket(2) accept(2) TCP file descriptor dup() fork() TCP TCP/IP TCP 1. TCP TCP accept(2) 1 http://weibo.com/1701018393/ecuxdrta0nn 593

594 D TCP 2. TCP TCP connect(2) Sockets API TCP/IP TCP/IP TCP TCP TCP IP packet TCP 1 TCP faketcp 1a. TCP IP packet SYN TCP segment 1b. SYN ACK TCP segment 1c. ACK segment faketcp TCP faketcp TCP/IP IP packet Ethernet frame faketcp IP:PORT TCP faketcp 2 TCP faketcp 2a. SYN TCP segment 2b. SYN ACK TCP segment 2c. ACK segment faketcp SYN SYN+ACK TCP faketcp faketcp TCP faketcp TCP

595 faketcp faketcp TCP recipes/faketcp make Ubuntu Linux 10.04 PC hostname atom D-1 router atom 10.0.0.0/24.1.2 Ethernet D-1 A TUN/TAP TCP/IP D-2 192.168.0.0/24 faketcp.x TUN router atom.1 10.0.0.0/24.1.2 Ethernet D-2 atom /dev/net/tun tun0 192.168.0.1/24 faketcp 192.168.0.0/24 atom 192.168.0.2~192.168.0.254 IP packet faketcp faketcp IP atom IP packet

596 D TCP ICMP echo ping faketcp recipes/faketcp/ icmpecho.cc ICMP echo request icmp_input() recipes/faketcp/faketcp.cc 3 1. 1 sudo./icmpecho allocted tunnel interface tun0 2. 2 $ sudo ifconfig tun0 192.168.0.1/24 $ sudo tcpdump -i tun0 3. 3 $ ping 192.168.0.2 $ ping 192.168.0.3 $ ping 192.168.0.234 192.168.0.X IP ping TCP SYN TCP segment RST segment recipes/faketcp/rejectall.cc 3 faketcp./rejectall 3 $ nc 192.168.0.2 2000 $ nc 192.168.0.2 3333 $ nc 192.168.0.7 5555 IP TCP TCP SYN TCP segment SYN+ACK FIN segment FIN+ACK recipes/faketcp/acceptall.cc 3 faketcp./acceptall nc 192.168.0.X IP port 4 netstat -tpn nc netstat Send-Q

597 TCP payload TCP segment ACK recipes/faketcp/discardall.cc 3 faketcp./discardall nc 192.168.0.X IP port 4 netstat -tpn 0 2 TCP 1 atom recipes/faketcp/connectmany.cc 4 1. 1 sudo./connectmany 192.168.0.1 2007 1000 192.168.0.1:2007 1000 allocted tunnel interface tun0 press enter key to start connecting 192.168.0.1:2007 2. 2 $ sudo ifconfig tun0 192.168.0.1/24 $ sudo tcpdump -i tun0 3. 3 TCP httpd muduo echo discard listen 2007 4. 1 4 netstat -tpn TCP TCP/IP A TCP mini tcp stack IPv4 2 48 IP TCP end points end point {ip, port} end point end point 2 48 IP 2 32 IP 2 16

598 D TCP NAT Linux http://urbanairship.com/blog/2010/09/29/linux-kernel-tuning-for-c500k/ http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3 http://www.erlang-factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf

[JCP] [RWC] [APUE] [UNP] [UNPv2] [TCPv1] Brian Goetz. Java Concurrency in Practice. Addison-Wesley 2006 Bryan Cantrill and Jeff Bonwick. Real-World Concurrency. ACM Queue 2008 9. http://queue.acm.org/detail.cfm?id=1454462 W. Richard Stevens and Stephen A. Rago. Advanced Programming in the UNIX Environment 2nd ed. Addison-Wesley 2005 UNIX 2. 2006 W. Richard Stevens. UNIX 1 API 3.. 2006 Unix Network Programming vol. 1 The Sockets Networking API 3rd ed UNIX 1. 2004 W. Richard Stevens. Unix Network Programming vol. 2 Interprocess Communications 2nd ed. Prentice Hall 1999 UNIX 2 2. 2002 W. Richard Stevens. TCP/IP Illustrated vol. 1: The Protocols. Addison- Wesley 1994 TCP/IP 1. 2010 [TCPv2] W. Richard Stevens. TCP/IP Illustrated vol. 2: The Implementation. Addison-Wesley 1995 TCP/IP 2. 2010 [CC2e] [EC3] [ESTL] Steve McConnell. 2.. 2006 Code Complete 2nd ed Scott Meyers. Effective C++ 3.. 2006 Scott Meyers. Effective STL. Addison-Wesley 2001 599

600 [CCS] [LLL] [WELC] [TPoP] [K&R] [ExpC] Herb Sutter and Andrei Alexandrescu. C++.. 2008 C++ Coding Standards: 101 Rules Guidelines and Best Practices.. 2009 Michael Feathers... 2007 Working Effectively with Legacy Code Brian W. Kernihgan and Rob Pike... 2000 The Practice of Programming Brian W. Kernighan and Dennis M. Ritchie. The C Programming Language 2nd ed. Prentice Hall 1988 C 2. 2000 Peter van der Linden. Expert C Programming: Deep C Secrets. Prentice Hall 1994 [CS:APP] Randal E. Bryant and David R. O Hallaron. 2.. 2011 Computer Systems: A Programmer s Perspective [D&E] [ERL] [DCC] [Gr00] [jjhou02] Bjarne Stroustrup. C++.. 2002 The Design and Evolution of C++ Joe Armstrong. Erlang.. 2008 Programming Erlang Luiz A. Barroso and Urs Hölzle. The Datacenter as a Computer. Morgan and Claypool Publishers 2009 http://www.morganclaypool.com/doi/abs/10.2200/s00193ed1v01y200905cac006 Jeff Grossman. A Technique for Safe Deletion with Object Locking. More C++ Gems. Robert C. Martin (ed.). Cambridge University Press 2000. Memory Pool. 2002 9. http://jjhou.boolan.com/programmer-13-memory-pool.pdf [Alex10] Andrei Alexandrescu. Scalable Use of the STL. C++ and Beyond 2010. http://www.artima.com/shop/cpp_and_beyond_2010