W. Richard Stevens UNIX Sockets API echo Sockets TCP OOB IO C struct C/C++ UNIX fork() select(2)/poll(2)/epoll(4) IO IO CPU 100% libevent IO UNIX CPU

Transcription

1 Linux muduo C++ C++ TCP C++ x86-64 Linux TCP one loop per thread Linux native muduo C++ IT 5 C++ muduo 2 C++ C++ Primer 4

2 W. Richard Stevens UNIX Sockets API echo Sockets TCP OOB IO C struct C/C++ UNIX fork() select(2)/poll(2)/epoll(4) IO IO CPU 100% libevent IO UNIX CPU IO UNIX 2 IPC 7 24 C++ STL C++ Linux muduo C++

3 C++ x86-64 Linux TCP 5 IO one loop per thread Linux native muduo muduo IO C++ one loop per thread IO muduo Linux A C++ C TR1 C++ C++ C C++ UNIX UNIX C++ Primer Pthreads Sockets API C Linux g C++11 x86-64 CPU GB 5 IO 1 C++ C++ 2 muduo muduo 3 C++ 4 C++ UDP iii

4 iv Observer Reactor Singleton class heap event loop STL algorithm C++ instance resolution instantiation override dereference class 6 syscall (s) fd file descriptor CPU core kb MB GB KiB MiB GiB L42 42 [JCP] [CC2e] mutex socket C++ class this mutable class fork(2) muduo::eventloop fork(2) fork() manpage 2 man 2 fork - EventLoop- ThreadPool URL muduo/base/date.h GitHub chenshuo/recipes/ recipes/thread GitHub URL muduo/base/types.h 1 muduo::string typedef 15 namespace muduo 16 { #ifdef MUDUO_STD_STRING 19 using std::string; 20 #else //!MUDUO_STD_STRING 21 typedef gnu_cxx:: sso_string string; 22 #endif muduo/base/types.h muduo/base/types.h muduo muduo/examples/xxx examples/xxx 8 recipes/reactor/xxx reactor/xxx

5 v diff -u URL

6 1 C C muduo muduo muduo muduo C C C A B C++ Primer 4 C C Boost D TCP vi

7 1 C MutexLock MutexLockGuard Counter mutex mutex Observer shared_ptr/weak_ptr Observer shared_ptr shared_ptr enable_shared_from_this Observer mutex vii

8 viii mutex condition variable MutexLock MutexLockGuard Condition Singleton sleep(3) shared_ptr copy-on-write one loop per thread TCP C C/C Linux pthread_cancel C exit(3) C thread IO

9 ix 4.7 RAII RAII fork() fork() signal Linux muduo muduo TCP echo finger muduo Boost.Asio libevent muduo libevent muduo Nginx muduo ZeroMQ muduo

10 x 7 muduo TCP Boost.Asio TCP LengthHeaderCodec muduo Buffer muduo IO non-blocking buffer Buffer Buffer Buffer Google Protobuf Protobuf type name Message Protobuf muduo Protobuf codec ProtobufCodec dispatcher ProtobufCodec ProtobufDispatcher ProtobufDispatcher ProtobufCodec ProtobufDispatcher muduo

11 xi Linux muduo Boost.Asio Timer Java Netty timing wheel timing wheel socks4a TCP socks4a N : 1 1 : N UDNS c-ares DNS curl muduo EventLoop Reactor Channel class Poller class EventLoop TimerQueue TimerQueue class EventLoop

12 xii 8.3 EventLoop::runInLoop() TimerQueue EventLoopThread class TCP TcpServer TcpServer class TcpConnection class TcpConnection Buffer TcpConnection Buffer Buffer::readFd() TcpConnection TcpConnection SIGPIPE TCP No Delay TCP keepalive WriteCompleteCallback HighWaterMarkCallback TcpServer Connector TcpClient epoll

13 xiii TCP ICE naming service C C C C C C++ linking inline

14 xiv C C ABI COM C Linux COM Java boost::function boost::bind iostream stdio iostream iostream

15 xv iostream iostream memory buffer output stream C++ IO C C ::operator new() ::operator new() ::operator new() ::operator new() malloc() class ::operator new() C/C

16 xvi 12.4 mock link seam namespace C static C++ static namespace diff grep std::string eager copy copy-on-write SSO STL algorithm next_permutation() unique() {make,push,pop}_heap() partition() lower_bound() IP A 561 B C++ Primer 4 C C Boost 591 D TCP

17 1 C++

18

19 1 synchronization primitives mutex race condition C++ Boost shared_ptr weak_ptr 1 Observer C++ C++ Observer 1.1 C++ race condition race condition C++ shared_ptr C++ 1 class TR1 std::tr1 C++11 3

20 [JCP] class interleaving C++ class std:: string std::vector std::map class MutexLock MutexLockGuard C MutexLock critical section RAII [CCS 13] Windows struct CRITI- CAL_SECTION Linux pthread_mutex_t 2 MutexLock class MutexLockGuard MutexLockGuard class Counter class Counter 1 // A thread-safe counter 2 class Counter : boost::noncopyable 3 { 4 // copy-ctor and assignment should be private by default for a class. 5 public: 6 Counter() : value_(0) {}

21 int64_t value() const; 8 int64_t getandincrease(); 9 10 private: 11 int64_t value_; 12 mutable MutexLock mutex_; 13 }; int64_t Counter::value() const 16 { 17 MutexLockGuard lock(mutex_); // lock 18 return value_; // 19 } int64_t Counter::getAndIncrease() 22 { 23 MutexLockGuard lock(mutex_); 24 int64_t ret = value_++; 25 return ret; 26 } 27 // In a real world, atomic operations are preferred. 28 // class class Counter mutex_ lock contention L24 Counter mutex_ mutable const Counter::value() non-const mutex_ mutex_ static / Counter Counter race condition 1.2 this this this escape

22 6 1 // Don't do this. class Foo : public Observer // Observer 10 { public: Foo(Observable* s) { s->register_(this); // } virtual void update(); }; // Do this. class Foo : public Observer { public: Foo(); virtual void update(); // void observe(observable* s) { s->register_(this); } }; Foo* pfoo = new Foo; Observable* s = getsubject(); pfoo->observe(s); // s->register_(pfoo); +initialize() C++ initialize() this Foo Foo::Foo() most-derived class

23 mutex mutex mutex (1) (2) Foo::~Foo() { MutexLockGuard lock(mutex_); // free internal state (1) } void Foo::update() { MutexLockGuard lock(mutex_); // (2) // make use of internal state } A B Foo x A x B x->update() extern Foo* x; // visible by all threads // thread A delete x; x = NULL; // helpless // thread B if (x) { x->update(); } A NULL B x x race condition 1. A (1) 2. B if (x) (2) mutex_ (2) core dump delete NULL 3 dangling pointer wild pointer

24 mutex class MutexLock class MutexLock MutexLock 1.1 class swap() void swap(counter& a, Counter& b) { MutexLockGuard alock(a.mutex_); MutexLockGuard block(b.mutex_); int64_t value = a.value_; a.value_ = b.value_; b.value_ = value; } // potential dead lock A swap(a, b); B swap(b, a); operator=() Counter& Counter::operator=(const Counter& rhs) { if (this == &rhs) return *this; } MutexLockGuard mylock(mutex_); // potential dead lock MutexLockGuard itslock(rhs.mutex_); value_ = rhs.value_; // value_ = rhs.value() return *this; mutex mutex 1.4 Observer

25 1.4 Observer 9 [CCS 99] free(3) C/C++ 4 composition aggregation association composition / x owner owner x x owner scoped_ptr owner C++ association / a b a b b a aggregation association a b b 1.1 A x B x lock contention x x race condition 4 Java reference null

26 10 1 Observer recipes/thread/test/observer.cc 1 class Observer // : boost::noncopyable 2 { 3 public: 4 virtual ~Observer(); 5 virtual void update() = 0; 6 //... 7 }; 8 9 class Observable // : boost::noncopyable 10 { 11 public: 12 void register_(observer* x); 13 void unregister(observer* x); void notifyobservers() { 16 for (Observer* x : observers_) { // C x->update(); // (3) 18 } 19 } 20 private: 21 std::vector<observer*> observers_; 22 }; Observable Observer (L17) Observer x Observer unregister() 23 class Observer 24 { 25 // 26 void observe(observable* s) { 27 s->register_(this); 28 subject_ = s; 29 } virtual ~Observer() { 32 subject_->unregister(this); 33 } Observable* subject_; 36 }; Observer unregister(this) race conditions L32 subject_ subject_ 1. A L32 unregister 2. B L17 x L32

27 x Observer 5 Observer L32 core dump race condition isalive() 1.5 raw pointer Observable Observer* Observer Observer race condition subject_ Observable* C++ shared_ptr p1 p2 Object p1 p2 1-1 A p1 p1 NULL p2 1-1 C/C++ p1 p2 Object p1=0 p2 Object C++

28 12 1 p1 p2 1-2 proxy Object C p1 p2 p1 proxy Object p2 1-2 Object proxy p2 proxy Object p1=0 proxy=0 Object p2 1-3 Object race condition p2 proxy Object p1 proxy proxy reference counting p1 p2 sp1 sp2 proxy sp1 sp2 pointer count = Object

29 1.6 shared_ptr/weak_ptr sp sp1 sp2 pointer count = Object 3. sp2 0 proxy Object 1-6 sp1 sp2 pointer count = Object another layer of indirection 6 Object handle/body idiom handle 7 C++ TR1 1.6 shared_ptr/weak_ptr shared_ptr Boost std::tr1 C++11 C++ shared_ptr<t> class template Edsger W. Dijkstra The Humble Programmer

30 weak_ptr weak shared_ptr shared_ptr shared_ptr x shared_ptr x x shared_ptr reset() x weak_ptr promote shared_ptr shared_ptr /lock() shared_ptr/weak_ptr shared_ptr/weak_ptr std::string STL 8 C++ clean up C Nginx C 10 C C Java C

31 C++ 1. buffer overrun 2. / 3. double delete 4. memory leak 5. new[]/delete 6. memory fragmentation A std::vector<char>/std::string Buffer class 2. / shared_ptr/weak_ptr 3. scoped_ptr 4. scoped_ptr 5. new[]/delete new[] std::vector/scoped_array std:: vector/std::list std::string C++ delete security data safety scoped_ptr/shared_ptr/weak_ptr shared_ptr<foo>* pfoo = new shared_ptr<foo>(new Foo); // WRONG semantic x T incomplete x.cpp

32 Observer weak_ptr Observer Observable weak_ptr<observer> recipes/thread/test/observer_safe.cc 39 class Observable // not 100% thread safe! 40 { 41 public: 42 void register_(weak_ptr<observer> x); // const weak_ptr<observer>& 43 // void unregister(weak_ptr<observer> x); // 44 void notifyobservers(); private: 47 mutable MutexLock mutex_; 48 std::vector<weak_ptr<observer> > observers_; 49 typedef std::vector<weak_ptr<observer> >::iterator Iterator; 50 }; void Observable::notifyObservers() 53 { 54 MutexLockGuard lock(mutex_); 55 Iterator it = observers_.begin(); // Iterator while (it!= observers_.end()) 57 { 58 shared_ptr<observer> obj(it->lock()); // 59 if (obj) 60 { 61 // 2 62 obj->update(); // obj 63 ++it; 64 } 65 else 66 { 67 // weak_ptr 68 it = observers_.erase(it); 69 } 70 } 71 } recipes/thread/test/observer_safe.cc (3) p. 10 L17 L48 vector<shared_ptr<observer> > observers_; Observer* weak_ptr<observer> Observer 1.14

33 1.9 shared_ptr 17 Observer shared_ptr Observer subject_->unregister(this) subject_ Observable shared_ptr subject_ weak_ptr<observable> lock contention Observable register_() unregister() notifyobservers() update() register_() unregister() L62 update() (un)register mutex_ mutex_ core dump vector observers_ mutex_ std::list ++it mutex Pthreads mutex Java intrinsic lock synchronized synchronized 1.9 shared_ptr shared_ptr shared_ptr 100% shared_ptr 11 shared_ptr std::string shared_ptr 1 shared_ptr 2 shared_ptr 3~5 shared_ptr 11

34 18 1 shared_ptr mutex MutexLock mutex; // ReadersWriterLock SpinLock shared_ptr<foo> globalptr; // globalptr doit() void doit(const shared_ptr<foo>& pfoo); globalptr race condition globalptr void read() { shared_ptr<foo> localptr; { MutexLockGuard lock(mutex); localptr = globalptr; // read globalptr } // use localptr since here localptr doit(localptr); } void write() { shared_ptr<foo> newptr(new Foo); // { MutexLockGuard lock(mutex); globalptr = newptr; // write to globalptr } // use newptr since here newptr doit(newptr); } read() write() globalptr Foo shared_ptr local copy local copy shared_ptr reference to const new Foo globalptr.reset(new Foo) globalptr.reset() localptr globalptr swap() write() globalptr = newptr; globalptr Foo

35 1.10 shared_ptr shared_ptr shared_ptr x shared_ptr shared_ptr p. 16 L48 observers_ vector<shared_ ptr<observer> > unregister() Observer unregister() unregister() Observer Java boost::bind boost::bind shared_ptr boost::function class Foo { void doit(); }; shared_ptr<foo> pfoo(new Foo); boost::function<void()> func = boost::bind(&foo::doit, pfoo); // long life foo func shared_ptr<foo> Foo shared_ptr const reference const reference shared_ptr Foo void save(const shared_ptr<foo>& pfoo); void validateaccount(const Foo& foo); bool validate(const shared_ptr<foo>& pfoo) { validateaccount(*pfoo); //... } // pass by const reference // pass by const reference pass by const reference void onmessage(const string& msg) { shared_ptr<foo> pfoo(new Foo(msg)); // if (validate(pfoo)) { // pfoo save(pfoo); // pfoo } }

36 20 1 shared_ptr pfoo shared_ptr<void> shared_ptr DLL A B Foo Foo inline Foo Factory shared_ptr shared_ptr<t> functor Scott Meyers 12 x shared_ptr x shared_ptr BlockingQueue<shared_ptr<void> > RAII handle RAII C++ RAII C++ C++ C++ new delete new delete RAII [CCS 13] new handle shared_ptr delete shared_ptr owner child shared_ptr child owner weak_ptr 12

37 Stock Google key "NASDAQ:GOOG" IBM "NYSE:IBM" Stock Stock Stock Stock StockFactory 13 key Stock shared_ptr // version 1: questionable code class StockFactory : boost::noncopyable { public: shared_ptr<stock> get(const string& key); private: mutable MutexLock mutex_; std::map<string, shared_ptr<stock> > stocks_; }; get() stocks_ key stocks_[key] Stock stocks_[key] Stock map shared_ptr Observable weak_ptr // version 2: std::map<string, weak_ptr<stock> > stocks_; shared_ptr<stock> StockFactory::get(const string& key) { shared_ptr<stock> pstock; MutexLockGuard lock(mutex_); weak_ptr<stock>& wkstock = stocks_[key]; // key pstock = wkstock.lock(); // if (!pstock) { pstock.reset(new Stock(key)); wkstock = pstock; // stocks_[key] wkstock } return pstock; } 13 recipes/thread/test/factory.cc

38 22 1 Stock stocks_ stocks_.size() Stock Stock 0 key shared_ptr shared_ptr d d(ptr) ptr shared_ptr shared_ptr bridge template<class Y, class D> shared_ptr::shared_ptr(y* p, D d); template<class Y, class D> void shared_ptr::reset(y* p, D d); // Y T Y* T* Stock stocks_ // version 3 class StockFactory : boost::noncopyable { // get() pstock.reset(new Stock(key)); // pstock.reset(new Stock(key), // boost::bind(&stockfactory::deletestock, this, _1)); // *** private: void deletestock(stock* stock) { if (stock) { MutexLockGuard lock(mutex_); stocks_.erase(stock->key()); } delete stock; // sorry, I lied } // assuming StockFactory lives longer than all Stock's... //... pstock.reset() boost::function Stock* p StockFactory deletestock StockFactory this boost::function *** Stock- Factory Stock core dump Observer Observable::unregister() Observable

39 enable_shared_from_this StockFactory::get() this boost::function *** StockFactory Stock Stock StockFactory::deleteStock core dump shared_ptr StockFactory::get() shared_ptr<stockfactory> enable_shared_from_this 14 this shared_ptr class StockFactory : public boost::enable_shared_from_this<stockfactory>, boost::noncopyable { /*... */ }; shared_from_this() StockFactory stack object heap object shared_ptr shared_ptr<stockfactory> stockfactory(new StockFactory); this shared_ptr<stockfactory> // version 4 shared_ptr<stock> StockFactory::get(const string& key) { // change // pstock.reset(new Stock(key), // boost::bind(&stockfactory::deletestock, this, _1)); // to pstock.reset(new Stock(key), boost::bind(&stockfactory::deletestock, shared_from_this(), _1)); //... boost::function shared_ptr<stockfactory> StockFactory::deleteStock StockFactory shared_from_this() StockFactory shared_ptr StockFactory 14

40 shared_ptr boost::bind boost:function Stock- Factory boost:function Observable::notifyObservers() weak_ptr weak_ptr boost::function shared_ptr StockFactory class StockFactory : public boost::enable_shared_from_this<stockfactory>, boost::noncopyable { public: shared_ptr<stock> get(const string& key) { shared_ptr<stock> pstock; MutexLockGuard lock(mutex_); weak_ptr<stock>& wkstock = stocks_[key]; // wkstock pstock = wkstock.lock(); if (!pstock) { pstock.reset(new Stock(key), boost::bind(&stockfactory::weakdeletecallback, boost::weak_ptr<stockfactory>(shared_from_this()), _1)); // shared_from_this() weak_ptr // boost::bind wkstock = pstock; } return pstock; } private: static void weakdeletecallback(const boost::weak_ptr<stockfactory>& wkfactory, Stock* stock) { shared_ptr<stockfactory> factory(wkfactory.lock()); // if (factory) // factory stocks_ { factory->removestock(stock); } delete stock; // sorry, I lied }

41 void removestock(stock* stock) { if (stock) { MutexLockGuard lock(mutex_); stocks_.erase(stock->key()); } } private: mutable MutexLock mutex_; std::map<string, weak_ptr<stock> > stocks_; }; void testlonglifefactory() { shared_ptr<stockfactory> factory(new StockFactory); { shared_ptr<stock> stock = factory->get("nyse:ibm"); shared_ptr<stock> stock2 = factory->get("nyse:ibm"); assert(stock == stock2); // stock destructs here } // factory destructs here } void testshortlifefactory() { shared_ptr<stock> stock; { shared_ptr<stockfactory> factory(new StockFactory); stock = factory->get("nyse:ibm"); shared_ptr<stock> stock2 = factory->get("nyse:ibm"); assert(stock == stock2); // factory destructs here } // stock destructs here } Stock StockFactory shared_ptr weak_ptr Factory singleton 15 StockFactory Stock stocks_ recipes/thread/weakcallback.h C++11 variadic template rvalue reference

42 shared_ptr/weak_ptr C++ 1. façade Foo Foo façade objid/handle check-out check-in 16 race condition façade Foo Java ConcurrentHashMap buckets bucket contention shared_ptr shared_ptr 4. C++11 unique_ptr shared_ptr Google Go Concurrency is hard without garbage collection C/C++ Java util.concurrent C API 16 Jeff Grossman A technique for safe deletion with object locking [Gr00] 17

43 shared_ptr C++ C Java Java Concurrency in Practice [JCP] C++ Java IPC race condition CPU CPU CPU CPU ilovecpp race condition shared_ptr/scoped_ptr 18 50% Google

44 28 1 shared_ptr boost::bind shared_ptr weak_ptr shared_ptr boost::shared_ptr C++11 unique_ptr auto_ptr shared_ptr TR1 C++ 19 C++ STL shared_ptr std::vector shared_ptr / 1.14 Observer 1.8 shared_ptr/weak_ptr Observer Observer Observer OO Observer Observer friend Observer Observer class Foo 1 30 work around Base class 19 C++ new delete memory leak C++ smart pointer STL Boost ( ) 4 smart pointers C++ memory leak!

45 1.14 Observer 29 Observer Java Java 8 Closure C# delegate C++ boost::function/ boost::bind 20 C++ Observer Signal/Slots QT thread safe race condition free thread contention free Signal/Slots shared_ptr 1.8 Observer 2.8 shared_ptr copy-on-write C++11 variadic template trivial template<typename Signature> class SignalTrivial; recipes/thread/signalslottrivial.h // NOT thread safe!!! template <typename RET, typename... ARGS> class SignalTrivial<RET(ARGS...)> { public: typedef std::function<void (ARGS...)> Functor; void connect(functor&& func) { functors_.push_back(std::forward<functor>(func)); } void call(args&&... args) { for (const Functor& f: functors_) { f(args...); } } private: std::vector<functor> functors_; }; recipes/thread/signalslottrivial.h Signal/Slots Slot unregister recipes/thread/signalslot.h boost::function boost::bind function/bind

46 30 1 C++ Ruminations on C++ Koenig Moo 2003 C++ 6 C++ handle/body idiom C++ shared_ptr C++ auto_ptr C++ shared_ptr C STL 10 STL vector 21

47 2 muduo

48

49 6 muduo ACE 1 Linux Windows x86-64 IA32 muduo ARM UDP TCP IPv6 IPv4 muduo IO + one event loop per thread IO API API non-trivial templates 90% app lib library framework 5000 FreeBSD/Darwin Mac IO multiplexing poll(2) epoll(4) Google Protocol Buffers RPC

50 126 6 muduo C++ muduo 2 Google code muduo git muduo Sockets API Python Hello hello-server.py 1 #!/usr/bin/python 2 3 import socket, time 4 5 serversocket = socket.socket(socket.af_inet, socket.sock_stream) 6 serversocket.bind(('', 8888)) 7 serversocket.listen(5) 8 9 while True: 10 (clientsocket, address) = serversocket.accept() # 11 data = clientsocket.recv(4096) # 12 datetime = time.asctime()+'\n' 13 clientsocket.send('hello ' + data) # 14 clientsocket.send('my time is ' + datetime) # 15 clientsocket.close() # hello-server.py 20 # import 21 sock = socket.socket(socket.af_inet, socket.sock_stream) 22 sock.connect((sys.argv[1], 8888)) # 23 sock.send(os.getlogin() + '\n') # 24 message = sock.recv(4096) # 25 print message # 26 sock.close() # hello-client.py hello-client.py Sockets API socket(2) bind(2) listen(2) accept(2) connect(2) recv(2) send(2) close(2) gethostbyname(3) L22

51 $./hello-client.py localhost Hello schen My time is Sun May 13 12:56: $./hello-client.py atom Hello schen Java Python Sockets Sockets API tar muduo beta.tar.gz muduo Linux timerfd eventfd Linux Debian 6.0 Squeeze / Ubuntu LTS g bit 64-bit x86 muduo Fedora 13 CentOS 6 Arch Linux AUR 4 Linux muduo backport.diff muduo Samsung S3C2440 ARM9 Raspberry Pi ARM11 muduo armlinux.diff muduo CMake 6 build system $ sudo apt-get install cmake muduo Boost 7 $ sudo apt-get install libboost-dev libboost-test-dev Debian 5.0 Lenny Ubuntu 8.04 CentOS CentOS Protobuf 7 TR1 Boost

52 128 6 muduo muduo curl c-ares DNS Google Protobuf cmake $ sudo apt-get install libcurl4-openssl-dev libc-ares-dev $ sudo apt-get install protobuf-compiler libprotobuf-dev muduo $ tar zxf muduo beta.tar.gz $ cd muduo/ $./build.sh -j2 muduo../build/debug/{bin,lib} $./build.sh install muduo../build/debug-install/{include,lib} muduo-protorpc muduo-udns release -O2 $ BUILD_TYPE=release./build.sh -j2 muduo../build/release/{bin,lib} $ BUILD_TYPE=release./build.sh install muduo../build/release-install/{include,lib} muduo-protorpc muduo-udns muduo 1.0 BUILD_TYPE release bin/inspector_test Linux box IP muduo muduo 8 C++ muduo../build/debug-install/include../build/debug-install/lib -lmuduo_net -lmuduo_base CMake makefile muduo muduo-tutorial 8 11

53 muduo muduo -- build.sh -- ChangeLog -- CMakeLists.txt -- License -- README -- muduo muduo -- base ::muduo namespace \-- net ::muduo::net namespace -- poller poll(2) epoll(4) IO multiplexing -- http Web -- inspect Web \-- protorpc Google Protobuf RPC -- examples \-- TODO muduo class ThreadPool class muduo/ base/threadpool.h muduo/base/threadpool.cc muduo/base muduo \-- base -- AsyncLogging.{h,cc} backend -- Atomic.h -- BlockingQueue.h -- BoundedBlockingQueue.h -- Condition.h Mutex.h -- copyable.h tag -- CountDownLatch.{h,cc} -- Date.{h,cc} Julian -- Exception.{h,cc} stack trace -- Logging.{h,cc} AsyncLogging -- Mutex.h -- ProcessInfo.{h,cc} -- Singleton.h singleton -- StringPiece.h Google -- tests -- Thread.{h,cc} -- ThreadLocal.h -- ThreadLocalSingleton.h singleton -- ThreadPool.{h,cc} -- Timestamp.{h,cc} UTC -- TimeZone.{h,cc} \-- Types.h muduo::string

54 130 6 muduo muduo Reactor EventLoop IO muduo object-based objectoriented boost::function + boost::bind muduo class muduo/net muduo/net/poller 4300 muduo \-- net -- Acceptor.{h,cc} -- Buffer.{h,cc} IO -- Callbacks.h -- Channel.{h,cc} Socket -- CMakeLists.txt -- Connector.{h,cc} -- Endian.h -- EventLoop.{h,cc} -- EventLoopThread.{h,cc} EventLoop -- EventLoopThreadPool.{h,cc} muduo IO -- InetAddress.{h,cc} IP -- Poller.{h,cc} IO multiplexing -- poller IO multiplexing -- DefaultPoller.cc MUDUO_USE_POLL -- EPollPoller.{h,cc} epoll(4) IO multiplexing \-- PollPoller.{h,cc} poll(2) IO multiplexing -- Socket.{h,cc} Sockets -- SocketsOps.{h,cc} Sockets API -- TcpClient.{h,cc} TCP -- TcpConnection.{h,cc} muduo TcpServer.{h,cc} TCP -- tests -- Timer.{h,cc} -- TimerId.h \-- TimerQueue.{h,cc} -lmuduo_http -lmuduo_inspect HttpServer Inspector http Java JMX 9.5 muduo/net/{http,inspect,protorpc}

55 muduo \-- net -- http HTTP HTTP -- CMakeLists.txt -- HttpContext.h -- HttpRequest.h -- HttpResponse.{h,cc} -- HttpServer.{h,cc} \-- tests/httpserver_test.cc HTTP -- inspect HTTP -- CMakeLists.txt -- Inspector.{h,cc} -- ProcessInspector.{h,cc} \-- tests/inspector_test.cc \-- protorpc Google Protobuf RPC -- CMakeLists.txt -- google-inl.h -- RpcChannel.{h,cc} -- RpcCodec.{h,cc} -- rpc.proto \-- RpcServer.{h,cc} muduo muduo 5 Buffer EventLoop TcpConnection TcpClient TcpServer -- include \-- muduo -- base \-- net -- Buffer.h -- Callbacks.h -- Channel.h -- Endian.h -- EventLoop.h -- EventLoopThread.h -- InetAddress.h -- TcpClient.h -- TcpConnection.h -- TcpServer.h -- TimerId.h -- http -- HttpRequest.h -- HttpResponse.h \-- HttpServer.h -- inspect -- Inspector.h \-- ProcessInspector.h

56 132 6 muduo \-- protorpc -- RpcChannel.h -- RpcCodec.h \-- RpcServer.h \-- lib -- libmuduo_base.a, libmuduo_net.a -- libmuduo_http.a, libmuduo_inspect.a \-- libmuduo_protorpc.a 6-1 muduo poller/epollpoller.h SocketsOps.h poller/pollpoller.h Poller.h Acceptor.h Socket.h EventLoopThreadPool.h EventLoopThread.h TcpClient.h TcpServer.h 6-1 TimerQueue.h Timer.h EventLoop.h TcpConnection.h Connector.h Channel.h Callbacks.h TimerId.h Buffer.h InetAddress.h muduo forward declaration Acceptor.h Channel.h Connector.h TcpConnection.h EventLoop class EventLoop.h TcpClient.h Connector class TcpServer.h Acceptor EventLoopThreadPool EventLoop.h Poller TimerQueue TcpConnection.h Channel Socket class Buffer Netty ChannelBuffer buffer class buffer read(2)/write(2) 7.4

57 InetAddress IPv4 end point IP gethostbyname(3) IO EventLoop Reactor EventLoop IO eventfd(2) pipe(2) TimerQueue Poller IO multiplexing EventLoopThread EventLoop::loop() TcpConnection TCP TcpClient TcpServer TcpConnection shared_ptr Buffer TcpConnection Buffer InetAddress class Channel selectable IO channel IO file descriptor Acceptor Connector EventLoop TimerQueue TcpConnection Socket RAII handle file descriptor fd Acceptor TcpConnection EventLoop TimerQueue fd Socket class SocketsOps Sockets Poller PollPoller EPollPoller EventLoop PollPoller EPollPoller poll(2) epoll(4) IO multiplexing poll poll(2) strace(1) Connector TCP TcpClient Acceptor TCP TcpServer TimerQueue timerfd poll/epoll_wait TimerQueue std::map Timer O(log N) N EventLoop EventLoopThreadPool IO TcpConnection EventLoop TcpServer

58 134 6 muduo 6-2 muduo Buffer TcpConnection EventLoop Channel +loop() +runafter() +runevery() +runat() +runinloop() +queueinloop() File Descriptor Socket owns -fd_ : File Desc. +handleevent() TcpConnection Acceptor Connector +poll() Poller -handleread() -handlewrite() -handleclose() -handleerror() -handleread() -handlewrite() -handleerror() TcpServer TcpClient PollPoller EPollPoller +poll() +poll() muduo examples Boost.Asio Java Netty Python Twisted examples -- asio Boost.Asio -- chat codec \-- tutorial timers -- cdns c-ares DNS -- curl curl HTTP -- filetransfer TCP -- hub pub/sub/hub -- idleconnection -- maxconnection -- multiplexer 1:n -- netty JBoss Netty -- discard -- echo \-- uptime TCP -- pingpong pingpong

59 protobuf Google Protobuf -- codec -- rpc RPC Sudoku \-- rpcbench RPC -- roundtrip -- shorturl -- simple 5 -- allinone 5 -- chargen RFC chargenclient chargen -- daytime RFC discard RFC echo RFC time RFC 868 \-- timeclient time -- socks4a Socks4a TcpClient -- sudoku muduo -- twisted Python Twisted \-- finger finger01 ~ 07 \-- zeromq ZeroMQ muduo License muduo UDNS DNS RPC muduo one loop per thread + thread pool EventLoop TcpConnection EventLoop IO file descriptor TcpConnection EventLoop TCP TCP TcpConnection EventLoop TcpServer accept(2) TcpConnection IO accept(2) EventLoop EventLoop- ThreadPool round-robin 6.6 Sudoku muduo 9 muduo-protorpc Ubuntu Linux apt-get Protobuf Protobuf 2.4.1

60 136 6 muduo muduo TCP muduo bug signal muduo 8 muduo Linux 2.6.x TCP IO IO muduo TCP Sockets API Brooks 11 muduo accidental complexity TCP recv(2) accept(2) send(2) Win32 TCP 1. accept connect TCP 2. close shutdown read(2) 0 10 Signal signalfd(2) EventLoop muduo-protorpc zurg slave 11

61 TCP close(2) back-off edge trigger level trigger 12 EPOLLOUT busy-loop epoll(4) poll(2) 40kB TCP 25kB 15kB OS 15kB TCP socket 50kB kb 50kB write() lighttpd \r\n\r\n bug 13 10ms readable lighttpd

62 138 6 muduo kB (s) 1GB muduo readv(2) IO muduo echo muduo echo 1. EchoServer class examples/simple/echo/echo.h 4 #include <muduo/net/tcpserver.h> 5 6 // RFC class EchoServer 8 { 9 public: 10 EchoServer(muduo::net::EventLoop* loop, 11 const muduo::net::inetaddress& listenaddr); void start(); // calls server_.start(); private: 16 void onconnection(const muduo::net::tcpconnectionptr& conn); void onmessage(const muduo::net::tcpconnectionptr& conn, 19 muduo::net::buffer* buf, 20 muduo::timestamp time); muduo::net::eventloop* loop_; 23 muduo::net::tcpserver server_; 24 }; examples/simple/echo/echo.h

63 examples/simple/echo/echo.cc 10 EchoServer::EchoServer(muduo::net::EventLoop* loop, 11 const muduo::net::inetaddress& listenaddr) 12 : loop_(loop), 13 server_(loop, listenaddr, "EchoServer") 14 { 15 server_.setconnectioncallback( 16 boost::bind(&echoserver::onconnection, this, _1)); 17 server_.setmessagecallback( 18 boost::bind(&echoserver::onmessage, this, _1, _2, _3)); 19 } examples/simple/echo/echo.cc 2. EchoServer::onConnection() EchoServer::onMessage() examples/simple/echo/echo.cc 26 void EchoServer::onConnection(const muduo::net::tcpconnectionptr& conn) 27 { 28 LOG_INFO << "EchoServer - " << conn->peeraddress().toipport() << " -> " 29 << conn->localaddress().toipport() << " is " 30 << (conn->connected()? "UP" : "DOWN"); 31 } void EchoServer::onMessage(const muduo::net::tcpconnectionptr& conn, 34 muduo::net::buffer* buf, 35 muduo::timestamp time) 36 { 37 muduo::string msg(buf->retrieveallasstring()); 38 LOG_INFO << conn->name() << " echo " << msg.size() << " bytes, " 39 << "data received at " << time.tostring(); 40 conn->send(msg); 41 } examples/simple/echo/echo.cc L37 L40 echo L40 send(msg) muduo event handler onconnection() conn TcpConnection shared_ptr TcpConnection::connected() bool TcpConnection peeraddress() localaddress() InetAddress IP port

64 140 6 muduo onmessage() conn TCP buf buf retrieve buf buffer time epoll_wait(2) read(2) Timestamp pass-by-value pass-by-(const)reference x main() EventLoop 1 #include "echo.h" 2 3 #include <muduo/base/logging.h> 4 #include <muduo/net/eventloop.h> 5 6 // using namespace muduo; 7 // using namespace muduo::net; 8 9 int main() 10 { 11 LOG_INFO << "pid = " << getpid(); 12 muduo::net::eventloop loop; 13 muduo::net::inetaddress listenaddr(2007); 14 EchoServer server(&loop, listenaddr); 15 server.start(); 16 loop.loop(); 17 } examples/simple/echo/main.cc examples/simple/echo/main.cc muduo/examples/simple/echo echo TcpServer EventLoop TcpConnection Buffer class class class namespace finger Python Twisted Reactor muduo muduo deferreds finger Twisted muduo finger finger01 finger07 examples/twisted/ finger

65 #include <muduo/net/eventloop.h> 2 3 using namespace muduo; 4 using namespace muduo::net; 5 6 int main() 7 { 8 EventLoop loop; 9 loop.loop(); 10 } examples/twisted/finger/finger01.cc examples/twisted/finger/finger01.cc muduo examples/twisted/finger/finger02.cc 1 #include <muduo/net/eventloop.h> 2 #include <muduo/net/tcpserver.h> 3 4 using namespace muduo; 5 using namespace muduo::net; 6 7 int main() 8 { 9 EventLoop loop; 10 TcpServer server(&loop, InetAddress(1079), "Finger"); 11 server.start(); 12 loop.loop(); 13 } examples/twisted/finger/finger02.cc 3. namespace 7 void onconnection(const TcpConnectionPtr& conn) 8 { 9 if (conn->connected()) 10 { 11 conn->shutdown(); 12 } 13 } int main() 16 { 17 EventLoop loop; 18 TcpServer server(&loop, InetAddress(1079), "Finger"); 19 server.setconnectioncallback(onconnection); examples/twisted/finger/finger03.cc

66 142 6 muduo 20 server.start(); 21 loop.loop(); 22 } examples/twisted/finger/finger03.cc 4. \r\n Buffer::findCRLF() O(N 2 ) CPU examples/twisted/finger/finger04.cc 7 void onmessage(const TcpConnectionPtr& conn, 8 Buffer* buf, 9 Timestamp receivetime) 10 { 11 if (buf->findcrlf()) 12 { 13 conn->shutdown(); 14 } 15 } int main() 18 { 19 EventLoop loop; 20 TcpServer server(&loop, InetAddress(1079), "Finger"); 21 server.setmessagecallback(onmessage); 22 server.start(); 23 loop.loop(); 24 } examples/twisted/finger/finger04.cc 5. \r\n --- examples/twisted/finger/finger04.cc :03: examples/twisted/finger/finger05.cc :06:05 -7,12 void onmessage(const TcpConnectionPtr& conn, Buffer* buf, Timestamp receivetime) { if (buf->findcrlf()) { + conn->send("no such user\r\n"); conn->shutdown(); } }

67 UserMap L30 UserMap examples/twisted/finger/finger06.cc 9 typedef std::map<string, string> UserMap; 10 UserMap users; string getuser(const string& user) 13 { 14 string result = "No such user"; 15 UserMap::iterator it = users.find(user); 16 if (it!= users.end()) 17 { 18 result = it->second; 19 } 20 return result; 21 } void onmessage(const TcpConnectionPtr& conn, 24 Buffer* buf, 25 Timestamp receivetime) 26 { 27 const char* crlf = buf->findcrlf(); 28 if (crlf) 29 { 30 string user(buf->peek(), crlf); 31 conn->send(getuser(user) + "\r\n"); 32 buf->retrieveuntil(crlf + 2); 33 conn->shutdown(); 34 } 35 } int main() 38 { 39 EventLoop loop; 40 TcpServer server(&loop, InetAddress(1079), "Finger"); 41 server.setmessagecallback(onmessage); 42 server.start(); 43 loop.loop(); 44 } examples/twisted/finger/finger06.cc 7. UserMap L examples/twisted/finger/finger06.cc :14: examples/twisted/finger/finger07.cc :15:22 -36,6 int main() { + users["schen"] = "Happy and well"; EventLoop loop; TcpServer server(&loop, InetAddress(1079), "Finger");

68 144 6 muduo server.setmessagecallback(onmessage); server.start(); loop.loop(); } telnet(1) finger Telnet $./bin/twisted_finger07 $ telnet localhost 1079 Trying ::1... Trying Connected to localhost. Escape character is '^]'. muduo No such user Connection closed by foreign host. $ telnet localhost 1079 Trying ::1... Trying Connected to localhost. Escape character is '^]'. schen Happy and well Connection closed by foreign host. 6.5 muduo muduo TCP

69 muduo percentile muduo Boost.Asio libevent2 muduo ping pong muduo Boost.Asio 15% libevent2 18% 70%. boost 1.40 asio asio libevent rc muduo asio libevent2 ping pong recipes/pingpong/libevent/ muduo libevent2 muduo examples/pingpong/ gist 17 muduo asio -O2 -finline-limit=1000 $ BUILD_TYPE=release./build.sh # muduo DELL 490 Intel Xeon E5320 CPU GHz 16GiB Ubuntu Linux Server LTS x86_64 g

70 146 6 muduo asio 18 ping pong muduo asio libevent2 ping pong echo TCP echo echo ZeroMQ 1/10/100/1000/ /2/3/4 8 4 ping pong 16KiB shell CPU TCP 110MiB/s Python CPU CPU 1 CPU

71 吞吐量 (MiB/s) 单线程 连接数 muduo asio asio libevent muduo libevent2 70% libevent2 socket 4096 buffer.c evbuffer_ read() muduo muduo

72 第6章 148 muduo 网络库简介 测试结果表明 muduo 的吞吐量平均比 libevent2 高 18% 以上 多线程测试的结果 见图 6-5 数字越大越好 多线程 (100 连接) 多线程 (1000 连接) 吞吐量 (MiB/s) 吞吐量 (MiB/s) 线程数 muduo muduo asio asio asio asio 线程数 0 图 6-5 测试结果表明 muduo 的吞吐量平均比 asio 高 15% 以上 讨论 muduo 出乎意料地比 asio 性能优越 我想主要得益于其简单的设计和简洁的 代码 asio 在多线程测试中表现不佳 我猜测其主要原因是测试代码只使用了一 个 io_service 如果改用 io_service per CPU 的话 其性能应该有所提高 我 对 asio 的了解程度仅限于能读懂其代码 希望能有 asio 高手编写 io_service per CPU 的 ping pong 测试 以便与 muduo 做一个公平的比较 由于 libevent2 每次最多从网络读取 4096 字节 这大大限制了它的吞吐量 ping pong 测试很容易实现 欢迎其他网络库 ACE POCO libevent 等 也能 加入到对比中来 期待这些库的高手出马 击鼓传花 对比 muduo 与 libevent2 的事件处理效率 前 面 我 们 比 较 了 muduo 和 libevent2 的 吞 吐 量 得 到 的 结 论 是 muduo 比 libevent2 快 18%. 有人会说 libevent2 并不是为高吞吐量的应用场景而设计的 这样的比较不公平 胜之不武 为了公平起见 这回我们用 libevent2 自带的性能测 试程序 击鼓传花 来对比 muduo 和 libevent2 在高并发情况下的 IO 事件处理效率 Linux 多线程服务端编程 使用 muduo C++ 网络库 (excerpt)

73 DELL E socketpair(2) pipe(2) ~ libevent2 test/bench.c rc bug libevent2 muduo examples/pingpong/bench.cc event watcher libev 19 timer event IO event libev Marc Lehmann muduo libevent libevent2 + muduo recipes/pingpong/libevent/run_bench.sh

74 150 6 muduo muduo libevent2 muduo (ctl_add) total time per iteration 100 active clients Dell WS file descriptors muduo libevent2 muduo (ctl_add) 1000 active clients file descriptors 700 time spent in event processing 100 active clients file descriptors muduo libevent2 muduo (ctl_add) muduo libevent2 muduo (ctl_add) 1000 active clients file descriptors time (in us) (lower is better) time (in us) (lower is better) time (in us) (lower is better) time (in us) (lower is better) 6-6

75 libevent2 event watcher muduo 20% 2. a libevent2 b libevent2 muduo muduo 1. muduo 2. libevent2 libev Marc Lehmann 21 libevent2 muduo epoll_ctl(fd, EPOLL_CTL_ ADD,...) event watcher 24 muduo epoll_ctl(fd, EPOLL_CTL_MOD,...) event watcher libevent2 epoll_ctl(fd, EPOLL_CTL_ADD,...) fd EEXIST File exists EPOLL_CTL_ADD EPOLL_CTL_MOD libevent2 muduo EPOLL_CTL_ADD event watcher 6-6 muduo libevent2 kernel muduo muduo/net/poller/epollpoller.cc CPU 2.4 GHz 1.86 GHz 21

76 152 6 muduo muduo libevent2 muduo (ctl_add) total time per iteration 100 active clients Dell E file descriptors muduo libevent2 muduo (ctl_add) 1000 active clients file descriptors 400 time spent in event processing 100 active clients file descriptors muduo libevent2 muduo (ctl_add) muduo libevent2 muduo (ctl_add) 1000 active clients file descriptors time (in us) (lower is better) time (in us) (lower is better) time (in us) (lower is better) time (in us) (lower is better) 6-7

77 muduo libevent muduo muduo Nginx Nginx muduo HTTP muduo HTTP muduo/net/http/ HTTP server 8 DELL 490 Xeon E5320 CPU ab 22 weighttp 23 4 i CPU Nginx muduo HTTP server HTTP client muduo HTTP muduo/net/http/tests/httpserver_test.cc void onrequest(const HttpRequest& req, HttpResponse* resp) { if (req.path() == "/") { //... } else if (req.path() == "/hello") { resp->setstatuscode(httpresponse::k200ok); resp->setstatusmessage("ok"); resp->setcontenttype("text/plain"); resp->addheader("server", "Muduo"); resp->setbody("hello, world!\n"); } else { resp->setstatuscode(httpresponse::k404notfound); resp->setstatusmessage("not Found"); resp->setcloseconnection(true); } } int main(int argc, char* argv[]) { int numthreads = 0;

78 154 6 muduo if (argc > 1) { benchmark = true; Logger::setLogLevel(Logger::WARN); numthreads = atoi(argv[1]); } EventLoop loop; HttpServer server(&loop, InetAddress(8000), "dummy"); server.sethttpcallback(onrequest); server.setthreadnum(numthreads); server.start(); loop.loop(); } muduo/net/http/tests/httpserver_test.cc Nginx HTTP echo 24 #user nobody; worker_processes 4; events { worker_connections 10240; } http { include default_type mime.types; application/octet-stream; access_log sendfile tcp_nopush off; on; on; keepalive_timeout 65; server { listen 8080; server_name localhost; location / { root html; index index.html index.htm; } } } location /hello { default_type text/plain; echo "hello, world!"; }

79 Requests per second Requests per second /hello "hello, world!"./ab -n k -r -c :8080/hello 6-8 HTTP ab CPU 70% muduo vs. Nginx 1 worker/thread muduo Nginx muduo 4 Nginx top(1) ab CPU 85% weighttp muduo vs. Nginx 4 workers/threads muduo Nginx

80 Average latency [us] lower is better muduo CPU top(1) workers/threads muduo 4 83% Nginx 4 75% workers/threads muduo 4 85% Nginx 4 78% Nginx CPU CPU CPU benchmark IO benchmark CPU 100% IO muduo Nginx muduo Nginx muduo qps 10 muduo HTTP HTTP muduo Nginx httpd muduo muduo ZeroMQ ZeroMQ 25 muduo muduo examples/zeromq/ ping pong echo KiB muduo ZeroMQ muduo vs. ZeroMQ latency on GbE k 2k 4k 8k 16k muduo ZeroMQ

81 6.6 muduo muduo Sudoku Solver muduo muduo examples/sudoku/ Sudoku Solver Sudoku Solver 9.8 muduo Buffer echo A echo TCP echo Sudoku \r\n TCP [id:]<81digits>\r\n [id:]<81digits>\r\n [id:]nosolution\r\n [id:] id Parallel Pipelining id Parallel Pipelining Protobuf RPC out-of-order RPC

82 158 6 muduo <81digits> Sudoku Sudoku NoSolution \r\n \r\n 2 a: \r\n a: \r\n 3 b: \r\n b:nosolution\r\n telnet Sudoku Solver Sudoku Client Sudoku Solver = 81 Sudoku Sudoku 28 Sudoku string solvesudoku(const string& puzzle); <81digits> <81digits> NoSolution pure function echo EchoServer SudokuServer onmessage() examples/sudoku/server_basic.cc onmessage() solvesudoku() TCP 28

83 6.6 muduo 159 const int kcells = 81; // 81 examples/sudoku/server_basic.cc void onmessage(const TcpConnectionPtr& conn, Buffer* buf, Timestamp) { LOG_DEBUG << conn->name(); size_t len = buf->readablebytes(); while (len >= kcells + 2) // 2 { const char* crlf = buf->findcrlf(); if (crlf) // { string request(buf->peek(), crlf); // string id; buf->retrieveuntil(crlf + 2); // retrieve string::iterator colon = find(request.begin(), request.end(), ':'); if (colon!= request.end()) // id { id.assign(request.begin(), colon); request.erase(request.begin(), colon+1); } if (request.size() == implicit_cast<size_t>(kcells)) // { string result = solvesudoku(request); // if (id.empty()) { conn->send(result+"\r\n"); } else { conn->send(id+":"+result+"\r\n"); } } else // { conn->send("bad Request!\r\n"); conn->shutdown(); } } else // { break; } } } examples/sudoku/server_basic.cc server_basic.cc Sudoku 7.4 CPU server_basic CPU

84 160 6 muduo server_basic 8 8 server_basic (s) 8 server_basic load balancer W. Richard Stevens UNIX 2 27 Client-Server Design Alternatives [UNP] 3 30 UNP CSDA [UNP] non-blocking IO Sockets API 2000 HTTP httpd Reactor IBM Lotus TCP Lotus Linux IBM Linux Lotus Server Linux POSA chat chat A TCP echo/httpd/sudoku chat httpd/sudoku

85 6.6 muduo 方案 并发模型 [UNP] 对应 多进程 多线程 阻塞 IO IO 复用 长连接 0 accept+read/write 0 否 否 是 否 否 无 否 低 否 是 常 一次服务一个客户 1 accept+fork 1 是 否 是 否 是 低 是 高 否 是 变 process-per-connection 2 accept+thread 6 否 是 是 否 是 中 是 中 是 是 变 thread-per-connection 3 prefork 2/3/4/5 是 否 是 否 是 低 是 高 否 是 变 见 [UNP] 4 pre threaded 7/8 否 是 是 否 是 中 是 中 是 是 变 见 [UNP] 5 poll (reactor) 6.8 节 否 否 否 是 是 高 否 低 是 是 常 单线程 reactor 6 reactor + thread-per-task 无 否 是 否 是 是 中 是 中 是 否 变 thread-per-request 7 reactor + worker thread 无 否 是 否 是 是 中 是 中 是 是 变 worker-thread-per-connection 8 reactor + thread pool 无 否 是 否 是 是 高 是 低 是 否 常 主线程 IO, 工作线程计算 9 reactors in threads 无 否 是 否 是 是 高 是 低 是 是 常 one loop per thread (muduo) 10 reactors in processes 无 是 否 否 是 是 高 是 低 否 是 常 one loop per process (Nginx) 11 reactors + thread pool 无 否 是 否 是 是 高 是 低 是 否 常 最灵活的 IO 与 CPU 配置 并发性 多核 开销 互通 顺序性 线程数 特点

86 162 6 muduo UNP CSDA Reactor muduo micro benchmark Thread_ bench.cc BlockingQueue_bench.cc E5320 Linux fork()+exit(): 534.7µs pthread_create()+pthread_join(): 42.5µs 26.1µs push/pop a blocking queue : 11.5µs Sudoku resolve: 100us µs 0 iterative [UNP] Figure 1.9 [UNP] daytime write-only Python 0 echo server Python recipes/python/echo-iterative.py 3 import socket 4 5 def handle(client_socket, client_address): 6 while True: 7 data = client_socket.recv(4096) 8 if data: 9 sent = client_socket.send(data) # sendall? 10 else: 11 print "disconnect", client_address 12 client_socket.close() 13 break if name == " main ": 16 listen_address = (" ", 2007) 17 server_socket = socket.socket(socket.af_inet, socket.sock_stream) 18 server_socket.bind(listen_address) 19 server_socket.listen(5) while True: 22 (client_socket, client_address) = server_socket.accept() 23 print "got connection from", client_address 24 handle(client_socket, client_address) recipes/python/echo-iterative.py L6~L13 echo L21~L24

87 6.6 muduo 163 L9 sendall() 1 Unix [UNP] child-per-client fork()-per-client process-per-connection PostgreSQL Perforce fork() fork() Sudoku Python L9~L16 self.request client_socket ForkingTCPServer EchoHandler.handle() IO recipes/python/echo-fork.py 1 #!/usr/bin/python 2 3 from SocketServer import BaseRequestHandler, TCPServer 4 from SocketServer import ForkingTCPServer, ThreadingTCPServer 5 6 class EchoHandler(BaseRequestHandler): 7 def handle(self): 8 print "got connection from", self.client_address 9 while True: 10 data = self.request.recv(4096) 11 if data: 12 sent = self.request.send(data) # sendall? 13 else: 14 print "disconnect", self.client_address 15 self.request.close() 16 break if name == " main ": 19 listen_address = (" ", 2007) 20 server = ForkingTCPServer(listen_address, EchoHandler) 21 server.serve_forever() recipes/python/echo-fork.py 2 Java thread-per-connection Java 1.4 NIO Java 1 Sudoku scheduler Python ThreadingTCPServer EchoHandler.handle()

88 164 6 muduo $ diff -U2 echo-fork.py echo-thread.py if name == " main ": listen_address = (" ", 2007) - server = ForkingTCPServer(listen_address, EchoHandler) + server = ThreadingTCPServer(listen_address, EchoHandler) server.serve_forever() EchoHandler.handle() 0 $ diff -U2 echo-fork.py echo-single.py if name == " main ": listen_address = (" ", 2007) - server = ForkingTCPServer(listen_address, EchoHandler) + server = TCPServer(listen_address, EchoHandler) server.serve_forever() 3 1 [UNP] accept(2) thundering herd 4 2 [UNP] 3 4 Apache httpd thread of control read() TCP read() write() / read() TCP echo client stdin proxy a b b a proxy A TCP / [UNP] echo client 7.13 Python TCP relay Python Pinhole IO multiplexing select/poll/epoll/kqueue thread of control IO IO select/poll non-blocking IO non-blocking IO buffer 7.4 IO multiplexing event-driven Doug Schmidt

89 6.6 muduo 165 Reactor event-driven Reactor / libevent muduo Netty twisted POE proxy Python IO multiplexing echo server 29 non-blocking L28 socket L14 L15~L32 IO fileno L16, L17 listening fd accept IO watch list connections L18~L23 L24~L32 echo L28 recipes/python/echo-poll.py 6 server_socket = socket.socket(socket.af_inet, socket.sock_stream) 7 server_socket.setsockopt(socket.sol_socket, socket.so_reuseaddr, 1) 8 server_socket.bind(('', 2007)) 9 server_socket.listen(5) 10 # server_socket.setblocking(0) 11 poll = select.poll() # epoll() should work the same 12 poll.register(server_socket.fileno(), select.pollin) connections = {} 15 while True: 16 events = poll.poll(10000) # 10 seconds 17 for fileno, event in events: 18 if fileno == server_socket.fileno(): 19 (client_socket, client_address) = server_socket.accept() 20 print "got connection from", client_address 21 # client_socket.setblocking(0) 22 poll.register(client_socket.fileno(), select.pollin) 23 connections[client_socket.fileno()] = client_socket 24 elif event & select.pollin: 25 client_socket = connections[fileno] 26 data = client_socket.recv(4096) 27 if data: 28 client_socket.send(data) # sendall() partial? 29 else: 30 poll.unregister(fileno) 31 client_socket.close() 32 del connections[fileno] recipes/python/echo-poll.py IO multiplexing 29

90 166 6 muduo listen TCP 3 L28~L30 $ diff echo-poll.py chat-poll.py -U4 --- echo-poll.py :50: chat-poll.py :50: elif event & select.pollin: 24 clientsocket = connections[fileno] 25 data = clientsocket.recv(4096) 26 if data: 27 - clientsocket.send(data) # sendall() partial? 28 + for (fd, othersocket) in connections.iteritems(): 29 + if othersocket!= clientsocket: 30 + othersocket.send(data) # sendall() partial? 31 else: 32 poll.unregister(fileno) 33 clientsocket.close() 34 del connections[fileno] Doug Schmidt routine Reactor Windows GUI IO multiplexing WndProc switch-case GUI 1 LRESULT CALLBACK WndProc(HWND hwnd, UINT message, WPARAM wparam, LPARAM lparam) 2 { 3 switch (message) 4 { 5 case WM_DESTROY: 6 PostQuitMessage(0); 7 return 0; 8 // many more cases 9 } 10 return DefWindowProc (hwnd, message, wparam, lparam) ; 11 } Reactor IO

91 6.6 muduo 167 Reactor 6-11 select/poll/epoll_wait IO Reactor IO socket poll poll 6-11 compute() 100ms sleep() IO IO Thread IO Thread Conn 1 readable poll Conn 1 readable poll read decode Conn 2 readable read 1 decode compute compute encode encode Conn 2 readable write poll read decode compute encode write poll write 1 poll read 2 decode compute encode write 2 poll Reactor 6-11 server_basic.cc IO CPU read(2) 5 poll(2) read(2)

92 168 6 muduo Python Reactor L42~L46 handlers listening fd handle_accept handler handle_request handle_input handle_input recipes/python/echo-reactor.py 6 server_socket = socket.socket(socket.af_inet, socket.sock_stream) 7 server_socket.setsockopt(socket.sol_socket, socket.so_reuseaddr, 1) 8 server_socket.bind(('', 2007)) 9 server_socket.listen(5) 10 # serversocket.setblocking(0) poll = select.poll() # epoll() should work the same 13 connections = {} 14 handlers = {} def handle_input(socket, data): 17 socket.send(data) # sendall() partial? def handle_request(fileno, event): 20 if event & select.pollin: 21 client_socket = connections[fileno] 22 data = client_socket.recv(4096) 23 if data: 24 handle_input(client_socket, data) 25 else: 26 poll.unregister(fileno) 27 client_socket.close() 28 del connections[fileno] 29 del handlers[fileno] def handle_accept(fileno, event): 32 (client_socket, client_address) = server_socket.accept() 33 print "got connection from", client_address 34 # client_socket.setblocking(0) 35 poll.register(client_socket.fileno(), select.pollin) 36 connections[client_socket.fileno()] = client_socket 37 handlers[client_socket.fileno()] = handle_request poll.register(server_socket.fileno(), select.pollin) 40 handlers[server_socket.fileno()] = handle_accept while True: 43 events = poll.poll(10000) # 10 seconds 44 for fileno, event in events: 45 handler = handlers[fileno] 46 handler(fileno, event) recipes/python/echo-reactor.py

93 6.6 muduo 169 handle_input $ diff echo-reactor.py chat-reactor.py -U1 def handle_input(socket, data): - socket.send(data) # sendall() partial? + for (fd, other_socket) in connections.iteritems(): + if other_socket!= socket: + other_socket.send(data) # sendall() partial? IO IO IO Windows GUI 6 Sudoku Reactor CPU 8 out-of-order 2 Sudoku 1 id response request 7 IO thread-per-connection client CPU 6 TCP burst requests 8 core % CPU IO Reactor thread pool IO Sudoku Solver examples/sudoku/server_threadpool.cc

94 170 6 muduo IO Thread Threads in pool Conn 1 readable poll wait wait Conn 2 readable read poll read put in pool put in pool decode poll write 2 write 1 poll compute encode wait decode compute encode wait Reactor 5 onmessage() ThreadPool 8 id $ diff server_basic.cc server_threadpool.cc -u --- server_basic.cc :19: server_threadpool.cc :15: ,16 void onmessage(const TcpConnectionPtr& conn,... if (puzzle.size() == implicit_cast<size_t>(kcells)) { - string result = solvesudoku(puzzle); - if (id.empty()) - { - conn->send(result+"\r\n"); - } - else - { - conn->send(id+":"+result+"\r\n"); - } + threadpool_.run(boost::bind(&solve, conn, puzzle, id)); -114,17 + static void solve(const TcpConnectionPtr& conn, + const string& puzzle, + const string& id) + {

95 6.6 muduo string result = solvesudoku(puzzle); + if (id.empty()) + { + conn->send(result+"\r\n"); + } + else + { + conn->send(id+":"+result+"\r\n"); + } + } + EventLoop* loop_; TcpServer server_; + ThreadPool threadpool_; Timestamp starttime_; }; IO Java Servlet 2.x IO fsync(2)/fdatasync(2) 30 IO Reactor 9 Reactor 9 muduo Netty one loop per thread main Reactor accept(2) sub Reactor muduo round-robin sub Reactor sub Reactor CPU muduo Reactor pool CPU 8 11 IO Reactor 8 9 thread pool Reactor IO IO muduo Linux IO

96 172 6 muduo Base IO Thread IO Thread 1 IO Thread 2 Conn 1 established poll poll poll Conn 2 established accept poll assign poll accept assign Conn 3 established poll accept assign poll Conn 1 readable Conn 2 readable poll poll read decode compute read encode decode write poll examples/sudoku/server_multiloop.cc server_basic.cc server_.setthreadnum(numthreads); $ diff server_basic.cc server_multiloop.cc -up --- server_basic.cc :40: server_multiloop.cc :39: ,19 class SudokuServer - SudokuServer(EventLoop* loop, const InetAddress& listenaddr) + SudokuServer(EventLoop* loop, const InetAddress& listenaddr, int numthreads) : loop_(loop), server_(loop, listenaddr, "SudokuServer"), starttime_(timestamp::now()) { server_.setconnectioncallback( boost::bind(&sudokuserver::onconnection, this, _1)); server_.setmessagecallback( boost::bind(&sudokuserver::onmessage, this, _1, _2, _3)); + server_.setthreadnum(numthreads); } 10 Nginx Reactor IO IO IO

97 6.6 muduo Base IO Thread IO Thread 1 IO Thread 2 Threads in pool Conn 1 established poll poll poll wait wait Conn 2 established accept poll assign poll accept assign poll Conn 2 readable poll Conn 1 readable read read decode decode put in pool compute poll put in pool poll encode compute write wait encode write poll wait server_.setthreadnum(numthreads); event loop event loops ZeroMQ 31 event loop event loop muduo::tcpserver::setthreadnum() event loop IO event loop IO thread pool event loop TCP TCP event loop event loop muduo event loop

98 174 6 muduo 10 event loop 1ms event loop event handler epoll_wait() event loop event loop 3.3 C++ one loop per thread + thread pool event loop non-blocking IO thread pool : Reactor in-house C++ muduo 32 5 muduo IO 2 thread-per-connection 1 N 5 Reactor 1 8 Reactor + 1 C 2 9 one loop per thread 1 C 1 11 one loop per thread + 1 C 1 C N C 1 C 2 CPU 32 Characteristics of multithreading models for high-performance IO driven network applications

99 6.6 muduo 175 IO TCP read()/write() 5 IO concurrent parallel 8 Reactor VIP VIP 5 VIP 6-11 muduo IO 11 IO

100 3

101 11.5 boost::function boost::bind 447 pimpl explicit forward pimpl pimpl C++ C libevent2 struct event_base opaque pointer libevent non-virtual virtual virtual function bind-byvtable-offset non-virtual function bind-by-name loader resolution mangled name Internet IP C Java JNI C Python/Perl/Ruby C C C Linux class free function muduo/base/ Timestamp.h class Timestamp muduo::timedifference() free function C++ Java 11.5 boost::function boost::bind boost::function boost::bind boost::function boost::bind std::tr1 C++11 C++ Scott Meyers [EC3 35] boost::function boost:bind function/bind class hierarchy class class

102 C++ class class class C++ wrapper adapter 13 C++ : boost::function boost::bind x A x is-a A B x is more like a B C++ interfaces 14 Ruby duck typing Google Go 15 tag object operations 13 Linus 2007 C++ C

103 11.5 boost::function boost::bind 449 C++ better C data abstraction OO GP object based global function C++ Handler class onconnect() ondisconnect() onmessage() ontimer() override C++ MyHandler Handler MyHandler delete override OO framework Java 16 C++ GC MyHandler delete this; MyHandler TCP MyHandler HandlerFactory MyHandlerFactory C TR1 C++11 boost::function + boost::bind std::tr1::function + std::tr1::bind C++11 std::function + std::bind muduo boost::function TCP IO boost::bind class boost::function muduo muduo 16 Java 8 Closure C# delegate

104 第 11 章 450 反思 C++ 面向对象与虚函数 言归正传 说说 boost::function 和 boost::bind 取代虚函数的具体做法 基本用途 boost::function 就像 C# 里的 delegate 可以指向任何函数 包括成员函数 当 用 bind 把某个成员函数绑到某个对象上时 我们得到了一个 closure 闭包 例如 class Foo { public: void methoda(); void methodint(int a); void methodstring(const string& str); }; class Bar { public: void methodb(); }; boost::function<void()> f1; // 无参数 无返回值 Foo foo; f1 = boost::bind(&foo::methoda, &foo); f1(); // 调用 foo.methoda(); Bar bar; f1 = boost::bind(&bar::methodb, &bar); f1(); // 调用 bar.methodb(); f1 = boost::bind(&foo::methodint, &foo, 42); f1(); // 调用 foo.methodint(42); f1 = boost::bind(&foo::methodstring, &foo, "hello"); f1(); // 调用 foo.methodstring("hello") // 注意 bind 拷贝的是实参类型 (const char*) 不是形参类型 (string) // 这里形参中的 string 对象的构造发生在调用 f1 的时候 而非 bind 的时候 // 因此要留意 bind 的实参 (cosnt char*) 的生命期 它应该不短于 f1 的生命期 // 必要时可通过 bind(&foo::methodstring, &foo, string(atempbuf)) 来保证安全 boost::function<void(int)> f2; // int 参数 无返回值 f2 = boost::bind(&foo::methodint, &foo, _1); f2(53); // 调用 foo.methodint(53); 如果没有 boost::bind 那么 boost::function 就什么都不是 而有了 bind 同 一个类的不同对象可以 delegate 给不同的实现 从而实现不同的行为 孟岩 简直 就无敌了 Linux 多线程服务端编程 使用 muduo C++ 网络库 (excerpt)

105 11.5 boost::function boost::bind class C++ 1 OO Thread base class Thread::run() derived class run() Thread Java Thread class class method helper class(es) OO boost::function Thread ThreadCallback ThreadCallback Thread Thread::start() Java Thread Runnable C# Thread delegate ThreadStart boost::thread // boost::function Thread class class Thread { public: typedef boost::function<void()> ThreadCallback; Thread(ThreadCallback cb) : cb_(cb) { } void start() { /* some magic to call run() in new created thread */ } private: void run() { cb_(); } ThreadCallback cb_; //... };

106 C++ class Foo // { public: void runinthread(); void runinanotherthread(int) }; Foo foo; Thread thread1(boost::bind(&foo::runinthread, &foo)); Thread thread2(boost::bind(&foo::runinanotherthread, &foo, 43)); thread1.start(); // thread2.start(); 2 boost::function NetServer class EchoService NetServer main() 17 network library class Connection; class NetServer : boost::noncopyable { public: typedef boost::function<void (Connection*)> ConnectionCallback; typedef boost::function<void (Connection*, const void*, int len)> MessageCallback; NetServer(uint16_t port); ~NetServer(); void registerconnectioncallback(const ConnectionCallback&); void registermessagecallback(const MessageCallback&); void sendmessage(connection*, const void* buf, int len); private: //... }; network library user code class EchoService { public: // NetServer::sendMessage typedef boost::function<void(connection*, const void*, int)> SendMessageCallback; EchoService(const SendMessageCallback& sendmsgcb) : sendmessagecb_(sendmsgcb) // boost::function { } 17 muduo

107 11.5 boost::function boost::bind 453 // NetServer::MessageCallback void onmessage(connection* conn, const void* buf, int size) { printf("received Msg from Connection %d: %.*s\n", conn->id(), size, (const char*)buf); sendmessagecb_(conn, buf, size); // echo back } // NetServer::ConnectionCallback void onconnection(connection* conn) { printf("connection from %s:%d is %s\n", conn->ipaddr(), conn->port(), conn->connected()? "UP" : "DOWN"); } private: SendMessageCallback sendmessagecb_; }; // int main() { NetServer server(7); EchoService echo(bind(&netserver::sendmessage, &server, _1, _2, _3)); server.registermessagecallback( bind(&echoservice::onmessage, &echo, _1, _2, _3)); server.registerconnectioncallback( bind(&echoservice::onconnection, &echo, _1)); server.run(); } user code class concept OO attribute/annotation JUnit 3.x void test*() JUnit 4.x NUnit 2.x annotation/attribute test case

108 C++ closure closure boost::function+boost::bind class object-based closure OO Factory Method boost::function<base* ()> Strategy Strategy ConcreteStrategyA ConcreteStrategyB boost::function Command boost::function Command Template Method boost::function boost::function 23 C closure paradigm EchoService EchoService SendMessageCallback DataSink

109 11.5 boost::function boost::bind 455 AbstractDataSink interface sendmessage() class NetDataSink MockDataSink EchoService AbstractDataSink* SendMessageCallback boost::bind() MockServer OO public boost::noncopyable boost::enable_ shared_from_this 1.11 enable_shared_from_this IO multiplexing select() POSIX poll() Linux epoll() FreeBSD kqueue() NetLoop base class classes switch-case Penguin Bird Bird virtual function fly() interface Flyable Runnable Runnable Flyable Runnable interface method interface tag boost::function

110 C++ class Penguin // { public: void run(); void swim(); }; class Sparrow // { public: void fly(); void run(); }; // boost::function typedef boost::function<void()> FlyCallback; typedef boost::function<void()> RunCallback; typedef boost::function<void()> SwimCallback; // run fly class class Foo { public: Foo(FlyCallback flycb, RunCallback runcb) : flycb_(flycb), runcb_(runcb) { } private: FlyCallback flycb_; RunCallback runcb_; }; // run swim class class Bar { public: Bar(SwimCallback swimcb, RunCallback runcb) : swimcb_(swimcb), runcb_(runcb) { } private: SwimCallback swimcb_; RunCallback runcb_; }; int main() { Sparrow s; Penguin p; // Foo Bar Foo foo(bind(&sparrow::fly, &s), bind(&sparrow::run, &s)); Bar bar(bind(&penguin::swim, &p), bind(&penguin::run, &p)); }

111 4

112

113 A Sockets API TCP/IP A.1.5 x86-64 Linux Sockets API PC PC Windows W. Richard Stevents TCP TCP TCP A.1 561

114 562 A A.1.1 Sockets API Sockets API 2005 C# PC C# PDA PDA TCP PDA PC 42 PC PDA PC 7 PC FTP PC PDA single point of failure TCP read/write blocking 2010 muduo Sockets API Reactor C++ Sockets API muduo Sockets API TCP Sockets API TCP TCP Sockets API A.1.5

115 A A.1.2 TCP UDP client library server libmemcached memcached libpq PostgreSQL Servlet HTTP RPC TCP/IP troubleshooting library framework Sockets API Sockets API strace Sockets API TCP/IP tcpdump A.1.3 Linux 10 FreeBSD FreeBSD Linux epoll FreeBSD kqueue C10k Linux 1 Linux bug work around FreeBSD Linux Linux A.1.4 Windows Linux FreeBSD Solaris 1

116 564 A AIX HP-UX Linux POSIX Linux SO_NOSIGPIPE Linux pipe(2) FreeBSD muduo Windows libevent libuv Java Netty A.1.5 big topic ping ping bug 1. firmware TCP/IP 4. HTTP FTP DNS SMTP POP3 NFS 5. HAProxy squid varnish Web load balancer 6. ZooKeeper memcached POP3 SMTP

117 A TCP/IP Sockets API muduo libevent muduo Netty gevent muduo A.1.6 HTTP HTTP 1.1 bug spec HTTP TCP accept 2 accept HAProxy lighttpd CPU one loop per thread httpd Nginx lighttpd 2 Linux for_servers_with_high_connection%20rates.pdf

118 566 A IO bound CPU Disk IO trade-off echo qps 1ms qps IO A master NAT TCP HTTP proxy HTTP server Web crawler HTTP HTTP proxy 3 3

119 A Web browser HTTP server 4 wget A bit malloc memcached malloc memory pool

120 568 A Linux Kernel PC 3% 5% STL new/delete premature optimization A.1.9 RFC daytime HTTP 1.0 echo chargen TIME_WAIT hold TIME_WAIT TIME_WAIT (s) TIME_WAIT buggy TIME_WAIT XML JSON Protobuf

121 A race condition p. 348 snapshot delta 30 end-to-end principle happens-before relationship A STL STL STL STL STL TCP/IP 3. TCP/IP stack UNIX TCP/IP TCP/IP manpage manpage TCP/IP Linux TCP/IP 1. TCP self-connection 8 2. Linux bug TCP TCP window clamping bug work around manpage ACE

122 570 A error code ephemeral port backlog connect manpage TCP/IP TCP/IP TCP TCP/IP send_packet() on_receive_packet() on_timer() libnet libpcap TCP/IP TCP/IP lwip Mini/Tiny/Toy/Trivial/Yet-Another TCP/IP TUN/TAP TCP/IP D Sockets API FTDI USB-SPI ENC28J60 TCP/IP stack IP ICMP Echo TCP 3000 UDP DNS A.1.11 TCP echo chat proxy echo echo HTTP chat a b fork()-per-connection b c b a c

123 A proxy 7.13 TCP A.1.12 Sockets API IPython muduo 9 Sockets API IPython IO C TCP epoll $ ipython In [1]: import socket, select In [2]: s = socket.socket(socket.af_inet, socket.sock_stream) In [3]: s.setsockopt(socket.sol_socket, socket.so_reuseaddr, 1) In [4]: s.bind(('', 5000)) In [5]: s.listen(5) In [6]: client, address = s.accept() # client.fileno() == 4 In [7]: client.recv(1024) Out[7]: 'Hello\n' # In [8]: epoll = select.epoll() In [9]: epoll.register(client.fileno(), select.epollin) # In [10]: epoll.poll(60) Out[10]: [(4, 1)] # # 4 select.epollin == 1 In [11]: client.recv(1024) # Out[11]: 'World\n' In [12]: client.setblocking(0) # In [13]: client.recv(1024) # EAGAIN == 11 error: [Errno 11] Resource temporarily unavailable In [14]: epoll.poll(60) Out[14]: [(4, 1)] In [15]: client.recv(1024) Out[15]: 'Bye!\n' # epoll_wait() # In [16]: client.close() nc 9

124 572 A $ nc localhost 5000 Hello <enter> World <enter> Bye! <enter> muduo log strace netcat/tempest/ipython tcpdump muduo Reactor Linux C++ IO 4000 A.1.13 TCP TCP Effective TCP/IP Programming 9 Realize That TCP Is a Reliable Protocol, Not an Infallible Protocol TCP 7.5 Google Protobuf check sum IP header check sum TCP header check sum CRC32 TCP IP header TCP header checksum 16-bit check sum 16-bit integers checksum sum 16-bit checksum CRC32 CRC A-1 client server TCP segment segment IP packet ethernet frame A-1 a router ethernet frame b c server d CRC a b c d TCP header checksum payload router NAT NAT c a d payload TCP header checksum

125 A client a server d switch 1 switch 2 b c router A-1 IP bit payload check sum check sum ethernet CRC When the CRC and TCP checksum disagree The Limitations of the Ethernet CRC and TCP/IP checksums for error detection 10 Amazon S bit check sum Google 12 MD5 end-to-end principle A.2 Unix W. Richard Stevens 6 [APUE] UNIX TCP/IP

126 574 A [UNPv2] [APUE] IPC TCP/IP TCP/IP TCP/IP Illustrated, Vol. 1: The Protocols TCP/IP TCPv1 TCPv1 13 TCPv1 TCP/IP W. Richard Stevens TCP/IP tcpdump TCP TCP packet TCP 1. Positive acknowledgement with retransmission 2. Flow control using sliding window Nagle 3. Congestion control slow start congestion avoidance fast retransmit TCP TCP flow control TCP connection TCPv Mbit 100Mbit switch hub 1Gbit 13

127 A Gbit TCP flow control TCP IPv6 TCP window scale option TCP timestamps option TCP selective ack option Linux tcpdump TCP TCPv Unix Network Programming, Vol. 1: Networking API XTI UNP W. Richard Stevens UNP 2 UNP 3 UNP Sockets API Sockets API W. Richard Stevens UNP 2 14 I have found when teaching network programming that about 80% of all network programming problems have nothing to do with network programming, per se. That is, the problems are not with the API functions such as accept and select, but the problems arise from a lack of understanding of the underlying network protocols. For example, I have found that once a student understands TCP s three-way handshake and four-packet connection termination, many network programming problems are immediately understood. TCP/IP Nagle UNP 3 UNP UNIX UNP UDP TCP IPv4 IPv

128 576 A MFC Scott Meyers Effective C++ W. Richard Stevens Jeffrey Richter CRC32 check sum check sum CRC CRC CRC CRC zlib UNP Sockets Out-of-Band Data Signal-Driven IO UNP UNP daytime echo TCP

129 A W. Richard Stevens UNP UNP Unix accept() + fork() C10k TCPv1 UNP TCPv1 UNP APUE Effective TCP/IP Programming W. Richard Stevens W. Richard Stevens UNP Effective TCP/IP Programming 6 TCP TCP TCP/IP Illustrated, Vol. 2: The Implementation TCPv BSD TCP/IP C Gary Wright mbuf IP ICMP IP IGMP IP Sockets ARP 3/4 TCP TCP TCPv2 IGMP IP host-to-host IP packet

130 578 A TCP TCP/IP stack inode BSD MiB VAX TCP/IP mbuf buffer TCP/IP 4.4BSD timer Linux TCP/IP Linux TCP/IP 3 Pattern-Oriented Software Architecture Volume 2: Patterns for Concurrent and Networked Objects POSA2 UNP UNP Sockets API POSA2 library/framework event-driven libevent Java Netty Java Mina Perl POE Python Twisted POSA2 C++ RAII

131 B C++ Primer 4 C++ C++ Primer 4 34 B.1 C Stanley Lippman C++ C++ Java C# Python C++ C++ 1 C C++ 10% 10% Google 2 C++ Bjarne Stroustrup C C Milo Yip C++ Unreal/Source Havok/FMOD C

132 580 B C++ Primer 4 C++ GC C++ 5 C++ Java C# C++ Bjarne Stroustrup C++ system programming 6 infrastructure C++ 7 Herb Sutter 8 C++ efficiency flexibility 9 abstraction productivity 10 C++ is about efficient programming with abstractions C++ 11 C & inline int find_longest(const std::vector<std::string>& words) { // std::max_element(words.begin(), words.end(), LengthCompare()); } CPU C++ memory layout locality of reference 5 C++ clean up 6 CPU 7 Software Development for Infrastructure 8 Herb Sutter C++ and Beyond 2011 Why C++? 9 just-in-time compilation 10 Stanley Lippman Linux 1970 C++ C++ 11 Ulrich Drepper Stop Underutilizing Your Computer SIMD 12 Technical Report on C++ Performance 13 Scott Meyers Effective C++ in an Embedded Environment

133 B.2 C memory hierarchy 14 Scott Meyers CPU Caches and Why You Care 15 Herb Sutter Machine Architecture 16 / GC 17 C++ C Java Python TIOBE C++ Java B.2 C++ C++ features Google 18 C++ C C++ C++ Bjarne Stroustrup The C++ Programming Language Stanley Lippman C++ Primer C++ 21 C++ Primer 10 C++ 14 std::list O(1) std::vector O(N) vector compact vector list vector Bjarne Stroustrup Abstraction and the C++ machine model C++ Java bs/abstraction-and-machine.pdf 18 50% 19 C++ 20 Java C# Python 21 C++ Primer 3/e

134 582 B C++ Primer 4 C++ C++ Primer 3 C++ Primer Barbara Moo C++ Primer 4 C++ Huffman C++ C++ syntax semantics C++ STL C++ 23 C++ C C++ C++ Primer 4 C C++ C++ C C++ Primer 4 C++ C C Bjarne Stroustrup Programming Principles and Practice Using C++ use only the 4th edition 23 C++ Primer 4/e Accelerated C++ C : C 25 iostream locale/facet 26 Stanley Lippman Virtual base class support wanders off into the Byzantine... The material is simply too esoteric to warrant discussion...

135 B.2 C C++ Primer 4 C++ 3 string vector class string C++ Primer Liskov C++ Primer C++ 27 C C++11 TR1 28 C C++ C++ Primer 4 C++ GNU G++ Visual C++ 30 C++ C++ C++ C++ 31 C++ C++ C++ C C++ C++ C C++ 28 TR C++ bind/function 29 C++ Primer 5 C G++ Linux Unix Visual C++ Windows C++ Intel C++ Linux G++ Windows Visual C++ 31 Cfront 32 C++

136 584 B C++ Primer 4 C++ C++ 33 C++ 34 C++ module package C C++ 35 vector<t> vector int std::string B B-1 36 calculate() + Node 3 * NumberNode BinaryNode 2 4 AddNode MultiplyNode B-1 33 G++ 4.x 32-bit 64-bit 64-bit C++ Visual C Express Visual C C++ C std::vector<int> 36 Packrat

137 B B-2 BinaryNode<std::plus<double> > BinaryNode<std:: multiplies<double> > BinaryNode<T> Node NumberNode BinaryNode<T> B Handle C++ C++ C++ 37 C++ C++ 38 Google Protobuf leveldb PCRE C++ muduo Chromium Google C++ STL Boost C++ C++ C Java C++ B.3 C++ C++ Primer C++ Effective C [EC3] Scott Meyers

138 586 B C++ Primer 4 C++ C++ C++ Primer C++ Effective C++ 3 Effective C++ 3 C C++ C++ override class 41 C++ idiom RAII 42 TCP RAII C C++ delete C++ Effective C C++ C 44 TR1 bind/function 45 Stephan T. Lavavej PPT 40 Andrew Koenig Teaching C++ Badly: Introduce Constructors and Destructors at the Same Time 41 std::string std::vector boost::shared_ptr class 42 heap 43 TR1 shared_ptr weak_ptr boost::scoped_ptr 44 Java 7 try-with-resources Python with C# using 45 function/bind

139 B TR STL 48 C++ Primer C++ C++ 49 [CCS] concept model refinement C++ STL STL 50 C++ type traits C++ 51 C 52 C++ C++ C++ Effective C C++ C/C Matthew Austern 49 Herb Sutter 50 Iterator 51 C C

140 588 B C++ Primer 4 C++ 35 class 43 pimpl C++ 56 swap() swap() 59 #include using 73 by value by reference 76 vector 79 value smart pointer 5 entity class value class base class trait class policy class exception class 33 class monolithic class 37 public 57 class namespace C++ Google C++ 54 LLVM 55 B

141 B PDF C++ 15% 56 5 C TR1 TR1 URL C++ [CCS] soxxxxxx PDF [email protected] lower_bound upper_bound 57 Scott Meyers C++11

142

143 C Boost Boost C++ Boost 1 C++ STL Boost vector map shared_ptr vector map shared_ptr weak_ptr C++ Boost Boost Boost noncopyable scoped_ptr static_assert Boost date_time 2 circular_buffer function/bind shared_ptr C++ delete lexical_cast Boost Boost boost::date_time muduo::timezone 591

144 592 C Boost regex RegEx class class RegEx regex immutable mutable basic_regex match_results match_regex regex Donald Knuth Coders at Work C/C++ C strlen strcpy strcmp C++ complex string vector class STL STL Boost Boost Boost Boost generic programming concept model refinement Boost.Threads STL Boost Boost.Preprocessor C++ Lua Boost.Proto C++ ANTLR parser Boost.Spirit Boost C++

145 D TCP TCP 1 IP TCP TCP/IP IPv6 TCP TCP :8765 A B B frame A 10 TCP B 100 TCP Linux socket(2) accept(2) TCP file descriptor dup() fork() TCP TCP/IP TCP 1. TCP TCP accept(2)

146 594 D TCP 2. TCP TCP connect(2) Sockets API TCP/IP TCP/IP TCP TCP TCP IP packet TCP 1 TCP faketcp 1a. TCP IP packet SYN TCP segment 1b. SYN ACK TCP segment 1c. ACK segment faketcp TCP faketcp TCP/IP IP packet Ethernet frame faketcp IP:PORT TCP faketcp 2 TCP faketcp 2a. SYN TCP segment 2b. SYN ACK TCP segment 2c. ACK segment faketcp SYN SYN+ACK TCP faketcp faketcp TCP faketcp TCP

147 595 faketcp faketcp TCP recipes/faketcp make Ubuntu Linux PC hostname atom D-1 router atom / Ethernet D-1 A TUN/TAP TCP/IP D /24 faketcp.x TUN router atom / Ethernet D-2 atom /dev/net/tun tun /24 faketcp /24 atom ~ IP packet faketcp faketcp IP atom IP packet

148 596 D TCP ICMP echo ping faketcp recipes/faketcp/ icmpecho.cc ICMP echo request icmp_input() recipes/faketcp/faketcp.cc sudo./icmpecho allocted tunnel interface tun $ sudo ifconfig tun /24 $ sudo tcpdump -i tun $ ping $ ping $ ping X IP ping TCP SYN TCP segment RST segment recipes/faketcp/rejectall.cc 3 faketcp./rejectall 3 $ nc $ nc $ nc IP TCP TCP SYN TCP segment SYN+ACK FIN segment FIN+ACK recipes/faketcp/acceptall.cc 3 faketcp./acceptall nc X IP port 4 netstat -tpn nc netstat Send-Q

149 597 TCP payload TCP segment ACK recipes/faketcp/discardall.cc 3 faketcp./discardall nc X IP port 4 netstat -tpn 0 2 TCP 1 atom recipes/faketcp/connectmany.cc sudo./connectmany : allocted tunnel interface tun0 press enter key to start connecting : $ sudo ifconfig tun /24 $ sudo tcpdump -i tun TCP httpd muduo echo discard listen netstat -tpn TCP TCP/IP A TCP mini tcp stack IPv IP TCP end points end point {ip, port} end point end point 2 48 IP 2 32 IP 2 16

150 598 D TCP NAT Linux

151 [JCP] [RWC] [APUE] [UNP] [UNPv2] [TCPv1] Brian Goetz. Java Concurrency in Practice. Addison-Wesley 2006 Bryan Cantrill and Jeff Bonwick. Real-World Concurrency. ACM Queue W. Richard Stevens and Stephen A. Rago. Advanced Programming in the UNIX Environment 2nd ed. Addison-Wesley 2005 UNIX W. Richard Stevens. UNIX 1 API Unix Network Programming vol. 1 The Sockets Networking API 3rd ed UNIX W. Richard Stevens. Unix Network Programming vol. 2 Interprocess Communications 2nd ed. Prentice Hall 1999 UNIX W. Richard Stevens. TCP/IP Illustrated vol. 1: The Protocols. Addison- Wesley 1994 TCP/IP [TCPv2] W. Richard Stevens. TCP/IP Illustrated vol. 2: The Implementation. Addison-Wesley 1995 TCP/IP [CC2e] [EC3] [ESTL] Steve McConnell Code Complete 2nd ed Scott Meyers. Effective C Scott Meyers. Effective STL. Addison-Wesley

152 600 [CCS] [LLL] [WELC] [TPoP] [K&R] [ExpC] Herb Sutter and Andrei Alexandrescu. C C++ Coding Standards: 101 Rules Guidelines and Best Practices Michael Feathers Working Effectively with Legacy Code Brian W. Kernihgan and Rob Pike The Practice of Programming Brian W. Kernighan and Dennis M. Ritchie. The C Programming Language 2nd ed. Prentice Hall 1988 C Peter van der Linden. Expert C Programming: Deep C Secrets. Prentice Hall 1994 [CS:APP] Randal E. Bryant and David R. O Hallaron Computer Systems: A Programmer s Perspective [D&E] [ERL] [DCC] [Gr00] [jjhou02] Bjarne Stroustrup. C The Design and Evolution of C++ Joe Armstrong. Erlang Programming Erlang Luiz A. Barroso and Urs Hölzle. The Datacenter as a Computer. Morgan and Claypool Publishers Jeff Grossman. A Technique for Safe Deletion with Object Locking. More C++ Gems. Robert C. Martin (ed.). Cambridge University Press Memory Pool [Alex10] Andrei Alexandrescu. Scalable Use of the STL. C++ and Beyond