W. Richard Stevens UNIX Sockets API echo Sockets TCP OOB IO C struct C/C++ UNIX fork() select(2)/poll(2)/epoll(4) IO IO CPU 100% libevent IO UNIX CPU

Transcription

1 Linux muduo C++ C++ TCP C++ x86-64 Linux TCP one loop per thread Linux native muduo C++ IT 5 C++ muduo 2 C++ C++ Primer 4

2 W. Richard Stevens UNIX Sockets API echo Sockets TCP OOB IO C struct C/C++ UNIX fork() select(2)/poll(2)/epoll(4) IO IO CPU 100% libevent IO UNIX CPU IO UNIX 2 IPC 7 24 C++ STL C++ Linux muduo C++

3 C++ x86-64 Linux TCP 5 IO one loop per thread Linux native muduo muduo IO C++ one loop per thread IO muduo Linux A C++ C TR1 C++ C++ C C++ UNIX UNIX C++ Primer Pthreads Sockets API C Linux g C++11 x86-64 CPU GB 5 IO 1 C++ C++ 2 muduo muduo 3 C++ 4 C++ UDP iii

4 iv Observer Reactor Singleton class heap event loop STL algorithm C++ instance resolution instantiation override dereference class 6 syscall (s) fd file descriptor CPU core kb MB GB KiB MiB GiB L42 42 [JCP] [CC2e] mutex socket C++ class this mutable class fork(2) muduo::eventloop fork(2) fork() manpage 2 man 2 fork - EventLoop- ThreadPool URL muduo/base/date.h GitHub chenshuo/recipes/ recipes/thread GitHub URL muduo/base/types.h 1 muduo::string typedef 15 namespace muduo 16 { #ifdef MUDUO_STD_STRING 19 using std::string; 20 #else //!MUDUO_STD_STRING 21 typedef gnu_cxx:: sso_string string; 22 #endif muduo/base/types.h muduo/base/types.h muduo muduo/examples/xxx examples/xxx 8 recipes/reactor/xxx reactor/xxx

5 v diff -u URL

6 1 C C muduo muduo muduo muduo C C C A B C++ Primer 4 C C Boost D TCP vi

7 1 C MutexLock MutexLockGuard Counter mutex mutex Observer shared_ptr/weak_ptr Observer shared_ptr shared_ptr enable_shared_from_this Observer mutex vii

8 viii mutex condition variable MutexLock MutexLockGuard Condition Singleton sleep(3) shared_ptr copy-on-write one loop per thread TCP C C/C Linux pthread_cancel C exit(3) C thread IO

9 ix 4.7 RAII RAII fork() fork() signal Linux muduo muduo TCP echo finger muduo Boost.Asio libevent muduo libevent muduo Nginx muduo ZeroMQ muduo

10 x 7 muduo TCP Boost.Asio TCP LengthHeaderCodec muduo Buffer muduo IO non-blocking buffer Buffer Buffer Buffer Google Protobuf Protobuf type name Message Protobuf muduo Protobuf codec ProtobufCodec dispatcher ProtobufCodec ProtobufDispatcher ProtobufDispatcher ProtobufCodec ProtobufDispatcher muduo

11 xi Linux muduo Boost.Asio Timer Java Netty timing wheel timing wheel socks4a TCP socks4a N : 1 1 : N UDNS c-ares DNS curl muduo EventLoop Reactor Channel class Poller class EventLoop TimerQueue TimerQueue class EventLoop

12 xii 8.3 EventLoop::runInLoop() TimerQueue EventLoopThread class TCP TcpServer TcpServer class TcpConnection class TcpConnection Buffer TcpConnection Buffer Buffer::readFd() TcpConnection TcpConnection SIGPIPE TCP No Delay TCP keepalive WriteCompleteCallback HighWaterMarkCallback TcpServer Connector TcpClient epoll

13 xiii TCP ICE naming service C C C C C C++ linking inline

14 xiv C C ABI COM C Linux COM Java boost::function boost::bind iostream stdio iostream iostream

15 xv iostream iostream memory buffer output stream C++ IO C C ::operator new() ::operator new() ::operator new() ::operator new() malloc() class ::operator new() C/C

16 xvi 12.4 mock link seam namespace C static C++ static namespace diff grep std::string eager copy copy-on-write SSO STL algorithm next_permutation() unique() {make,push,pop}_heap() partition() lower_bound() IP A 561 B C++ Primer 4 C C Boost 591 D TCP

17 1 C++

18

19 1 synchronization primitives mutex race condition C++ Boost shared_ptr weak_ptr 1 Observer C++ C++ Observer 1.1 C++ race condition race condition C++ shared_ptr C++ 1 class TR1 std::tr1 C++11 3

20 [JCP] class interleaving C++ class std:: string std::vector std::map class MutexLock MutexLockGuard C MutexLock critical section RAII [CCS 13] Windows struct CRITI- CAL_SECTION Linux pthread_mutex_t 2 MutexLock class MutexLockGuard MutexLockGuard class Counter class Counter 1 // A thread-safe counter 2 class Counter : boost::noncopyable 3 { 4 // copy-ctor and assignment should be private by default for a class. 5 public: 6 Counter() : value_(0) {}

21 int64_t value() const; 8 int64_t getandincrease(); 9 10 private: 11 int64_t value_; 12 mutable MutexLock mutex_; 13 }; int64_t Counter::value() const 16 { 17 MutexLockGuard lock(mutex_); // lock 18 return value_; // 19 } int64_t Counter::getAndIncrease() 22 { 23 MutexLockGuard lock(mutex_); 24 int64_t ret = value_++; 25 return ret; 26 } 27 // In a real world, atomic operations are preferred. 28 // class class Counter mutex_ lock contention L24 Counter mutex_ mutable const Counter::value() non-const mutex_ mutex_ static / Counter Counter race condition 1.2 this this this escape

22 6 1 // Don't do this. class Foo : public Observer // Observer 10 { public: Foo(Observable* s) { s->register_(this); // } virtual void update(); }; // Do this. class Foo : public Observer { public: Foo(); virtual void update(); // void observe(observable* s) { s->register_(this); } }; Foo* pfoo = new Foo; Observable* s = getsubject(); pfoo->observe(s); // s->register_(pfoo); +initialize() C++ initialize() this Foo Foo::Foo() most-derived class

23 mutex mutex mutex (1) (2) Foo::~Foo() { MutexLockGuard lock(mutex_); // free internal state (1) } void Foo::update() { MutexLockGuard lock(mutex_); // (2) // make use of internal state } A B Foo x A x B x->update() extern Foo* x; // visible by all threads // thread A delete x; x = NULL; // helpless // thread B if (x) { x->update(); } A NULL B x x race condition 1. A (1) 2. B if (x) (2) mutex_ (2) core dump delete NULL 3 dangling pointer wild pointer

24 mutex class MutexLock class MutexLock MutexLock 1.1 class swap() void swap(counter& a, Counter& b) { MutexLockGuard alock(a.mutex_); MutexLockGuard block(b.mutex_); int64_t value = a.value_; a.value_ = b.value_; b.value_ = value; } // potential dead lock A swap(a, b); B swap(b, a); operator=() Counter& Counter::operator=(const Counter& rhs) { if (this == &rhs) return *this; } MutexLockGuard mylock(mutex_); // potential dead lock MutexLockGuard itslock(rhs.mutex_); value_ = rhs.value_; // value_ = rhs.value() return *this; mutex mutex 1.4 Observer

25 1.4 Observer 9 [CCS 99] free(3) C/C++ 4 composition aggregation association composition / x owner owner x x owner scoped_ptr owner C++ association / a b a b b a aggregation association a b b 1.1 A x B x lock contention x x race condition 4 Java reference null

26 10 1 Observer recipes/thread/test/observer.cc 1 class Observer // : boost::noncopyable 2 { 3 public: 4 virtual ~Observer(); 5 virtual void update() = 0; 6 //... 7 }; 8 9 class Observable // : boost::noncopyable 10 { 11 public: 12 void register_(observer* x); 13 void unregister(observer* x); void notifyobservers() { 16 for (Observer* x : observers_) { // C x->update(); // (3) 18 } 19 } 20 private: 21 std::vector<observer*> observers_; 22 }; Observable Observer (L17) Observer x Observer unregister() 23 class Observer 24 { 25 // 26 void observe(observable* s) { 27 s->register_(this); 28 subject_ = s; 29 } virtual ~Observer() { 32 subject_->unregister(this); 33 } Observable* subject_; 36 }; Observer unregister(this) race conditions L32 subject_ subject_ 1. A L32 unregister 2. B L17 x L32

27 x Observer 5 Observer L32 core dump race condition isalive() 1.5 raw pointer Observable Observer* Observer Observer race condition subject_ Observable* C++ shared_ptr p1 p2 Object p1 p2 1-1 A p1 p1 NULL p2 1-1 C/C++ p1 p2 Object p1=0 p2 Object C++

28 12 1 p1 p2 1-2 proxy Object C p1 p2 p1 proxy Object p2 1-2 Object proxy p2 proxy Object p1=0 proxy=0 Object p2 1-3 Object race condition p2 proxy Object p1 proxy proxy reference counting p1 p2 sp1 sp2 proxy sp1 sp2 pointer count = Object

29 1.6 shared_ptr/weak_ptr sp sp1 sp2 pointer count = Object 3. sp2 0 proxy Object 1-6 sp1 sp2 pointer count = Object another layer of indirection 6 Object handle/body idiom handle 7 C++ TR1 1.6 shared_ptr/weak_ptr shared_ptr Boost std::tr1 C++11 C++ shared_ptr<t> class template Edsger W. Dijkstra The Humble Programmer

30 weak_ptr weak shared_ptr shared_ptr shared_ptr x shared_ptr x x shared_ptr reset() x weak_ptr promote shared_ptr shared_ptr /lock() shared_ptr/weak_ptr shared_ptr/weak_ptr std::string STL 8 C++ clean up C Nginx C 10 C C Java C

31 C++ 1. buffer overrun 2. / 3. double delete 4. memory leak 5. new[]/delete 6. memory fragmentation A std::vector<char>/std::string Buffer class 2. / shared_ptr/weak_ptr 3. scoped_ptr 4. scoped_ptr 5. new[]/delete new[] std::vector/scoped_array std:: vector/std::list std::string C++ delete security data safety scoped_ptr/shared_ptr/weak_ptr shared_ptr<foo>* pfoo = new shared_ptr<foo>(new Foo); // WRONG semantic x T incomplete x.cpp

32 Observer weak_ptr Observer Observable weak_ptr<observer> recipes/thread/test/observer_safe.cc 39 class Observable // not 100% thread safe! 40 { 41 public: 42 void register_(weak_ptr<observer> x); // const weak_ptr<observer>& 43 // void unregister(weak_ptr<observer> x); // 44 void notifyobservers(); private: 47 mutable MutexLock mutex_; 48 std::vector<weak_ptr<observer> > observers_; 49 typedef std::vector<weak_ptr<observer> >::iterator Iterator; 50 }; void Observable::notifyObservers() 53 { 54 MutexLockGuard lock(mutex_); 55 Iterator it = observers_.begin(); // Iterator while (it!= observers_.end()) 57 { 58 shared_ptr<observer> obj(it->lock()); // 59 if (obj) 60 { 61 // 2 62 obj->update(); // obj 63 ++it; 64 } 65 else 66 { 67 // weak_ptr 68 it = observers_.erase(it); 69 } 70 } 71 } recipes/thread/test/observer_safe.cc (3) p. 10 L17 L48 vector<shared_ptr<observer> > observers_; Observer* weak_ptr<observer> Observer 1.14

33 1.9 shared_ptr 17 Observer shared_ptr Observer subject_->unregister(this) subject_ Observable shared_ptr subject_ weak_ptr<observable> lock contention Observable register_() unregister() notifyobservers() update() register_() unregister() L62 update() (un)register mutex_ mutex_ core dump vector observers_ mutex_ std::list ++it mutex Pthreads mutex Java intrinsic lock synchronized synchronized 1.9 shared_ptr shared_ptr shared_ptr 100% shared_ptr 11 shared_ptr std::string shared_ptr 1 shared_ptr 2 shared_ptr 3~5 shared_ptr 11

34 18 1 shared_ptr mutex MutexLock mutex; // ReadersWriterLock SpinLock shared_ptr<foo> globalptr; // globalptr doit() void doit(const shared_ptr<foo>& pfoo); globalptr race condition globalptr void read() { shared_ptr<foo> localptr; { MutexLockGuard lock(mutex); localptr = globalptr; // read globalptr } // use localptr since here localptr doit(localptr); } void write() { shared_ptr<foo> newptr(new Foo); // { MutexLockGuard lock(mutex); globalptr = newptr; // write to globalptr } // use newptr since here newptr doit(newptr); } read() write() globalptr Foo shared_ptr local copy local copy shared_ptr reference to const new Foo globalptr.reset(new Foo) globalptr.reset() localptr globalptr swap() write() globalptr = newptr; globalptr Foo

35 1.10 shared_ptr shared_ptr shared_ptr x shared_ptr shared_ptr p. 16 L48 observers_ vector<shared_ ptr<observer> > unregister() Observer unregister() unregister() Observer Java boost::bind boost::bind shared_ptr boost::function class Foo { void doit(); }; shared_ptr<foo> pfoo(new Foo); boost::function<void()> func = boost::bind(&foo::doit, pfoo); // long life foo func shared_ptr<foo> Foo shared_ptr const reference const reference shared_ptr Foo void save(const shared_ptr<foo>& pfoo); void validateaccount(const Foo& foo); bool validate(const shared_ptr<foo>& pfoo) { validateaccount(*pfoo); //... } // pass by const reference // pass by const reference pass by const reference void onmessage(const string& msg) { shared_ptr<foo> pfoo(new Foo(msg)); // if (validate(pfoo)) { // pfoo save(pfoo); // pfoo } }

36 20 1 shared_ptr pfoo shared_ptr<void> shared_ptr DLL A B Foo Foo inline Foo Factory shared_ptr shared_ptr<t> functor Scott Meyers 12 x shared_ptr x shared_ptr BlockingQueue<shared_ptr<void> > RAII handle RAII C++ RAII C++ C++ C++ new delete new delete RAII [CCS 13] new handle shared_ptr delete shared_ptr owner child shared_ptr child owner weak_ptr 12

37 Stock Google key "NASDAQ:GOOG" IBM "NYSE:IBM" Stock Stock Stock Stock StockFactory 13 key Stock shared_ptr // version 1: questionable code class StockFactory : boost::noncopyable { public: shared_ptr<stock> get(const string& key); private: mutable MutexLock mutex_; std::map<string, shared_ptr<stock> > stocks_; }; get() stocks_ key stocks_[key] Stock stocks_[key] Stock map shared_ptr Observable weak_ptr // version 2: std::map<string, weak_ptr<stock> > stocks_; shared_ptr<stock> StockFactory::get(const string& key) { shared_ptr<stock> pstock; MutexLockGuard lock(mutex_); weak_ptr<stock>& wkstock = stocks_[key]; // key pstock = wkstock.lock(); // if (!pstock) { pstock.reset(new Stock(key)); wkstock = pstock; // stocks_[key] wkstock } return pstock; } 13 recipes/thread/test/factory.cc

38 22 1 Stock stocks_ stocks_.size() Stock Stock 0 key shared_ptr shared_ptr d d(ptr) ptr shared_ptr shared_ptr bridge template<class Y, class D> shared_ptr::shared_ptr(y* p, D d); template<class Y, class D> void shared_ptr::reset(y* p, D d); // Y T Y* T* Stock stocks_ // version 3 class StockFactory : boost::noncopyable { // get() pstock.reset(new Stock(key)); // pstock.reset(new Stock(key), // boost::bind(&stockfactory::deletestock, this, _1)); // *** private: void deletestock(stock* stock) { if (stock) { MutexLockGuard lock(mutex_); stocks_.erase(stock->key()); } delete stock; // sorry, I lied } // assuming StockFactory lives longer than all Stock's... //... pstock.reset() boost::function Stock* p StockFactory deletestock StockFactory this boost::function *** Stock- Factory Stock core dump Observer Observable::unregister() Observable

39 enable_shared_from_this StockFactory::get() this boost::function *** StockFactory Stock Stock StockFactory::deleteStock core dump shared_ptr StockFactory::get() shared_ptr<stockfactory> enable_shared_from_this 14 this shared_ptr class StockFactory : public boost::enable_shared_from_this<stockfactory>, boost::noncopyable { /*... */ }; shared_from_this() StockFactory stack object heap object shared_ptr shared_ptr<stockfactory> stockfactory(new StockFactory); this shared_ptr<stockfactory> // version 4 shared_ptr<stock> StockFactory::get(const string& key) { // change // pstock.reset(new Stock(key), // boost::bind(&stockfactory::deletestock, this, _1)); // to pstock.reset(new Stock(key), boost::bind(&stockfactory::deletestock, shared_from_this(), _1)); //... boost::function shared_ptr<stockfactory> StockFactory::deleteStock StockFactory shared_from_this() StockFactory shared_ptr StockFactory 14

40 shared_ptr boost::bind boost:function Stock- Factory boost:function Observable::notifyObservers() weak_ptr weak_ptr boost::function shared_ptr StockFactory class StockFactory : public boost::enable_shared_from_this<stockfactory>, boost::noncopyable { public: shared_ptr<stock> get(const string& key) { shared_ptr<stock> pstock; MutexLockGuard lock(mutex_); weak_ptr<stock>& wkstock = stocks_[key]; // wkstock pstock = wkstock.lock(); if (!pstock) { pstock.reset(new Stock(key), boost::bind(&stockfactory::weakdeletecallback, boost::weak_ptr<stockfactory>(shared_from_this()), _1)); // shared_from_this() weak_ptr // boost::bind wkstock = pstock; } return pstock; } private: static void weakdeletecallback(const boost::weak_ptr<stockfactory>& wkfactory, Stock* stock) { shared_ptr<stockfactory> factory(wkfactory.lock()); // if (factory) // factory stocks_ { factory->removestock(stock); } delete stock; // sorry, I lied }

41 void removestock(stock* stock) { if (stock) { MutexLockGuard lock(mutex_); stocks_.erase(stock->key()); } } private: mutable MutexLock mutex_; std::map<string, weak_ptr<stock> > stocks_; }; void testlonglifefactory() { shared_ptr<stockfactory> factory(new StockFactory); { shared_ptr<stock> stock = factory->get("nyse:ibm"); shared_ptr<stock> stock2 = factory->get("nyse:ibm"); assert(stock == stock2); // stock destructs here } // factory destructs here } void testshortlifefactory() { shared_ptr<stock> stock; { shared_ptr<stockfactory> factory(new StockFactory); stock = factory->get("nyse:ibm"); shared_ptr<stock> stock2 = factory->get("nyse:ibm"); assert(stock == stock2); // factory destructs here } // stock destructs here } Stock StockFactory shared_ptr weak_ptr Factory singleton 15 StockFactory Stock stocks_ recipes/thread/weakcallback.h C++11 variadic template rvalue reference

42 shared_ptr/weak_ptr C++ 1. façade Foo Foo façade objid/handle check-out check-in 16 race condition façade Foo Java ConcurrentHashMap buckets bucket contention shared_ptr shared_ptr 4. C++11 unique_ptr shared_ptr Google Go Concurrency is hard without garbage collection C/C++ Java util.concurrent C API 16 Jeff Grossman A technique for safe deletion with object locking [Gr00] 17

43 shared_ptr C++ C Java Java Concurrency in Practice [JCP] C++ Java IPC race condition CPU CPU CPU CPU ilovecpp race condition shared_ptr/scoped_ptr 18 50% Google

44 28 1 shared_ptr boost::bind shared_ptr weak_ptr shared_ptr boost::shared_ptr C++11 unique_ptr auto_ptr shared_ptr TR1 C++ 19 C++ STL shared_ptr std::vector shared_ptr / 1.14 Observer 1.8 shared_ptr/weak_ptr Observer Observer Observer OO Observer Observer friend Observer Observer class Foo 1 30 work around Base class 19 C++ new delete memory leak C++ smart pointer STL Boost ( ) 4 smart pointers C++ memory leak!

45 1.14 Observer 29 Observer Java Java 8 Closure C# delegate C++ boost::function/ boost::bind 20 C++ Observer Signal/Slots QT thread safe race condition free thread contention free Signal/Slots shared_ptr 1.8 Observer 2.8 shared_ptr copy-on-write C++11 variadic template trivial template<typename Signature> class SignalTrivial; recipes/thread/signalslottrivial.h // NOT thread safe!!! template <typename RET, typename... ARGS> class SignalTrivial<RET(ARGS...)> { public: typedef std::function<void (ARGS...)> Functor; void connect(functor&& func) { functors_.push_back(std::forward<functor>(func)); } void call(args&&... args) { for (const Functor& f: functors_) { f(args...); } } private: std::vector<functor> functors_; }; recipes/thread/signalslottrivial.h Signal/Slots Slot unregister recipes/thread/signalslot.h boost::function boost::bind function/bind

46 30 1 C++ Ruminations on C++ Koenig Moo 2003 C++ 6 C++ handle/body idiom C++ shared_ptr C++ auto_ptr C++ shared_ptr C STL 10 STL vector 21

47 2 muduo

48

49 6 muduo ACE 1 Linux Windows x86-64 IA32 muduo ARM UDP TCP IPv6 IPv4 muduo IO + one event loop per thread IO API API non-trivial templates 90% app lib library framework 5000 FreeBSD/Darwin Mac IO multiplexing poll(2) epoll(4) Google Protocol Buffers RPC

50 126 6 muduo C++ muduo 2 Google code muduo git muduo Sockets API Python Hello hello-server.py 1 #!/usr/bin/python 2 3 import socket, time 4 5 serversocket = socket.socket(socket.af_inet, socket.sock_stream) 6 serversocket.bind(('', 8888)) 7 serversocket.listen(5) 8 9 while True: 10 (clientsocket, address) = serversocket.accept() # 11 data = clientsocket.recv(4096) # 12 datetime = time.asctime()+'\n' 13 clientsocket.send('hello ' + data) # 14 clientsocket.send('my time is ' + datetime) # 15 clientsocket.close() # hello-server.py 20 # import 21 sock = socket.socket(socket.af_inet, socket.sock_stream) 22 sock.connect((sys.argv[1], 8888)) # 23 sock.send(os.getlogin() + '\n') # 24 message = sock.recv(4096) # 25 print message # 26 sock.close() # hello-client.py hello-client.py Sockets API socket(2) bind(2) listen(2) accept(2) connect(2) recv(2) send(2) close(2) gethostbyname(3) L22

51 $./hello-client.py localhost Hello schen My time is Sun May 13 12:56: $./hello-client.py atom Hello schen Java Python Sockets Sockets API tar muduo beta.tar.gz muduo Linux timerfd eventfd Linux Debian 6.0 Squeeze / Ubuntu LTS g bit 64-bit x86 muduo Fedora 13 CentOS 6 Arch Linux AUR 4 Linux muduo backport.diff muduo Samsung S3C2440 ARM9 Raspberry Pi ARM11 muduo armlinux.diff muduo CMake 6 build system $ sudo apt-get install cmake muduo Boost 7 $ sudo apt-get install libboost-dev libboost-test-dev Debian 5.0 Lenny Ubuntu 8.04 CentOS CentOS Protobuf 7 TR1 Boost

52 128 6 muduo muduo curl c-ares DNS Google Protobuf cmake $ sudo apt-get install libcurl4-openssl-dev libc-ares-dev $ sudo apt-get install protobuf-compiler libprotobuf-dev muduo $ tar zxf muduo beta.tar.gz $ cd muduo/ $./build.sh -j2 muduo../build/debug/{bin,lib} $./build.sh install muduo../build/debug-install/{include,lib} muduo-protorpc muduo-udns release -O2 $ BUILD_TYPE=release./build.sh -j2 muduo../build/release/{bin,lib} $ BUILD_TYPE=release./build.sh install muduo../build/release-install/{include,lib} muduo-protorpc muduo-udns muduo 1.0 BUILD_TYPE release bin/inspector_test Linux box IP muduo muduo 8 C++ muduo../build/debug-install/include../build/debug-install/lib -lmuduo_net -lmuduo_base CMake makefile muduo muduo-tutorial 8 11

53 muduo muduo -- build.sh -- ChangeLog -- CMakeLists.txt -- License -- README -- muduo muduo -- base ::muduo namespace \-- net ::muduo::net namespace -- poller poll(2) epoll(4) IO multiplexing -- http Web -- inspect Web \-- protorpc Google Protobuf RPC -- examples \-- TODO muduo class ThreadPool class muduo/ base/threadpool.h muduo/base/threadpool.cc muduo/base muduo \-- base -- AsyncLogging.{h,cc} backend -- Atomic.h -- BlockingQueue.h -- BoundedBlockingQueue.h -- Condition.h Mutex.h -- copyable.h tag -- CountDownLatch.{h,cc} -- Date.{h,cc} Julian -- Exception.{h,cc} stack trace -- Logging.{h,cc} AsyncLogging -- Mutex.h -- ProcessInfo.{h,cc} -- Singleton.h singleton -- StringPiece.h Google -- tests -- Thread.{h,cc} -- ThreadLocal.h -- ThreadLocalSingleton.h singleton -- ThreadPool.{h,cc} -- Timestamp.{h,cc} UTC -- TimeZone.{h,cc} \-- Types.h muduo::string

54 130 6 muduo muduo Reactor EventLoop IO muduo object-based objectoriented boost::function + boost::bind muduo class muduo/net muduo/net/poller 4300 muduo \-- net -- Acceptor.{h,cc} -- Buffer.{h,cc} IO -- Callbacks.h -- Channel.{h,cc} Socket -- CMakeLists.txt -- Connector.{h,cc} -- Endian.h -- EventLoop.{h,cc} -- EventLoopThread.{h,cc} EventLoop -- EventLoopThreadPool.{h,cc} muduo IO -- InetAddress.{h,cc} IP -- Poller.{h,cc} IO multiplexing -- poller IO multiplexing -- DefaultPoller.cc MUDUO_USE_POLL -- EPollPoller.{h,cc} epoll(4) IO multiplexing \-- PollPoller.{h,cc} poll(2) IO multiplexing -- Socket.{h,cc} Sockets -- SocketsOps.{h,cc} Sockets API -- TcpClient.{h,cc} TCP -- TcpConnection.{h,cc} muduo TcpServer.{h,cc} TCP -- tests -- Timer.{h,cc} -- TimerId.h \-- TimerQueue.{h,cc} -lmuduo_http -lmuduo_inspect HttpServer Inspector http Java JMX 9.5 muduo/net/{http,inspect,protorpc}

55 muduo \-- net -- http HTTP HTTP -- CMakeLists.txt -- HttpContext.h -- HttpRequest.h -- HttpResponse.{h,cc} -- HttpServer.{h,cc} \-- tests/httpserver_test.cc HTTP -- inspect HTTP -- CMakeLists.txt -- Inspector.{h,cc} -- ProcessInspector.{h,cc} \-- tests/inspector_test.cc \-- protorpc Google Protobuf RPC -- CMakeLists.txt -- google-inl.h -- RpcChannel.{h,cc} -- RpcCodec.{h,cc} -- rpc.proto \-- RpcServer.{h,cc} muduo muduo 5 Buffer EventLoop TcpConnection TcpClient TcpServer -- include \-- muduo -- base \-- net -- Buffer.h -- Callbacks.h -- Channel.h -- Endian.h -- EventLoop.h -- EventLoopThread.h -- InetAddress.h -- TcpClient.h -- TcpConnection.h -- TcpServer.h -- TimerId.h -- http -- HttpRequest.h -- HttpResponse.h \-- HttpServer.h -- inspect -- Inspector.h \-- ProcessInspector.h

56 132 6 muduo \-- protorpc -- RpcChannel.h -- RpcCodec.h \-- RpcServer.h \-- lib -- libmuduo_base.a, libmuduo_net.a -- libmuduo_http.a, libmuduo_inspect.a \-- libmuduo_protorpc.a 6-1 muduo poller/epollpoller.h SocketsOps.h poller/pollpoller.h Poller.h Acceptor.h Socket.h EventLoopThreadPool.h EventLoopThread.h TcpClient.h TcpServer.h 6-1 TimerQueue.h Timer.h EventLoop.h TcpConnection.h Connector.h Channel.h Callbacks.h TimerId.h Buffer.h InetAddress.h muduo forward declaration Acceptor.h Channel.h Connector.h TcpConnection.h EventLoop class EventLoop.h TcpClient.h Connector class TcpServer.h Acceptor EventLoopThreadPool EventLoop.h Poller TimerQueue TcpConnection.h Channel Socket class Buffer Netty ChannelBuffer buffer class buffer read(2)/write(2) 7.4

57 InetAddress IPv4 end point IP gethostbyname(3) IO EventLoop Reactor EventLoop IO eventfd(2) pipe(2) TimerQueue Poller IO multiplexing EventLoopThread EventLoop::loop() TcpConnection TCP TcpClient TcpServer TcpConnection shared_ptr Buffer TcpConnection Buffer InetAddress class Channel selectable IO channel IO file descriptor Acceptor Connector EventLoop TimerQueue TcpConnection Socket RAII handle file descriptor fd Acceptor TcpConnection EventLoop TimerQueue fd Socket class SocketsOps Sockets Poller PollPoller EPollPoller EventLoop PollPoller EPollPoller poll(2) epoll(4) IO multiplexing poll poll(2) strace(1) Connector TCP TcpClient Acceptor TCP TcpServer TimerQueue timerfd poll/epoll_wait TimerQueue std::map Timer O(log N) N EventLoop EventLoopThreadPool IO TcpConnection EventLoop TcpServer

58 134 6 muduo 6-2 muduo Buffer TcpConnection EventLoop Channel +loop() +runafter() +runevery() +runat() +runinloop() +queueinloop() File Descriptor Socket owns -fd_ : File Desc. +handleevent() TcpConnection Acceptor Connector +poll() Poller -handleread() -handlewrite() -handleclose() -handleerror() -handleread() -handlewrite() -handleerror() TcpServer TcpClient PollPoller EPollPoller +poll() +poll() muduo examples Boost.Asio Java Netty Python Twisted examples -- asio Boost.Asio -- chat codec \-- tutorial timers -- cdns c-ares DNS -- curl curl HTTP -- filetransfer TCP -- hub pub/sub/hub -- idleconnection -- maxconnection -- multiplexer 1:n -- netty JBoss Netty -- discard -- echo \-- uptime TCP -- pingpong pingpong

59 protobuf Google Protobuf -- codec -- rpc RPC Sudoku \-- rpcbench RPC -- roundtrip -- shorturl -- simple 5 -- allinone 5 -- chargen RFC chargenclient chargen -- daytime RFC discard RFC echo RFC time RFC 868 \-- timeclient time -- socks4a Socks4a TcpClient -- sudoku muduo -- twisted Python Twisted \-- finger finger01 ~ 07 \-- zeromq ZeroMQ muduo License muduo UDNS DNS RPC muduo one loop per thread + thread pool EventLoop TcpConnection EventLoop IO file descriptor TcpConnection EventLoop TCP TCP TcpConnection EventLoop TcpServer accept(2) TcpConnection IO accept(2) EventLoop EventLoop- ThreadPool round-robin 6.6 Sudoku muduo 9 muduo-protorpc Ubuntu Linux apt-get Protobuf Protobuf 2.4.1

60 136 6 muduo muduo TCP muduo bug signal muduo 8 muduo Linux 2.6.x TCP IO IO muduo TCP Sockets API Brooks 11 muduo accidental complexity TCP recv(2) accept(2) send(2) Win32 TCP 1. accept connect TCP 2. close shutdown read(2) 0 10 Signal signalfd(2) EventLoop muduo-protorpc zurg slave 11

61 TCP close(2) back-off edge trigger level trigger 12 EPOLLOUT busy-loop epoll(4) poll(2) 40kB TCP 25kB 15kB OS 15kB TCP socket 50kB kb 50kB write() lighttpd \r\n\r\n bug 13 10ms readable lighttpd

62 138 6 muduo kB (s) 1GB muduo readv(2) IO muduo echo muduo echo 1. EchoServer class examples/simple/echo/echo.h 4 #include <muduo/net/tcpserver.h> 5 6 // RFC class EchoServer 8 { 9 public: 10 EchoServer(muduo::net::EventLoop* loop, 11 const muduo::net::inetaddress& listenaddr); void start(); // calls server_.start(); private: 16 void onconnection(const muduo::net::tcpconnectionptr& conn); void onmessage(const muduo::net::tcpconnectionptr& conn, 19 muduo::net::buffer* buf, 20 muduo::timestamp time); muduo::net::eventloop* loop_; 23 muduo::net::tcpserver server_; 24 }; examples/simple/echo/echo.h

63 examples/simple/echo/echo.cc 10 EchoServer::EchoServer(muduo::net::EventLoop* loop, 11 const muduo::net::inetaddress& listenaddr) 12 : loop_(loop), 13 server_(loop, listenaddr, "EchoServer") 14 { 15 server_.setconnectioncallback( 16 boost::bind(&echoserver::onconnection, this, _1)); 17 server_.setmessagecallback( 18 boost::bind(&echoserver::onmessage, this, _1, _2, _3)); 19 } examples/simple/echo/echo.cc 2. EchoServer::onConnection() EchoServer::onMessage() examples/simple/echo/echo.cc 26 void EchoServer::onConnection(const muduo::net::tcpconnectionptr& conn) 27 { 28 LOG_INFO << "EchoServer - " << conn->peeraddress().toipport() << " -> " 29 << conn->localaddress().toipport() << " is " 30 << (conn->connected()? "UP" : "DOWN"); 31 } void EchoServer::onMessage(const muduo::net::tcpconnectionptr& conn, 34 muduo::net::buffer* buf, 35 muduo::timestamp time) 36 { 37 muduo::string msg(buf->retrieveallasstring()); 38 LOG_INFO << conn->name() << " echo " << msg.size() << " bytes, " 39 << "data received at " << time.tostring(); 40 conn->send(msg); 41 } examples/simple/echo/echo.cc L37 L40 echo L40 send(msg) muduo event handler onconnection() conn TcpConnection shared_ptr TcpConnection::connected() bool TcpConnection peeraddress() localaddress() InetAddress IP port

64 140 6 muduo onmessage() conn TCP buf buf retrieve buf buffer time epoll_wait(2) read(2) Timestamp pass-by-value pass-by-(const)reference x main() EventLoop 1 #include "echo.h" 2 3 #include <muduo/base/logging.h> 4 #include <muduo/net/eventloop.h> 5 6 // using namespace muduo; 7 // using namespace muduo::net; 8 9 int main() 10 { 11 LOG_INFO << "pid = " << getpid(); 12 muduo::net::eventloop loop; 13 muduo::net::inetaddress listenaddr(2007); 14 EchoServer server(&loop, listenaddr); 15 server.start(); 16 loop.loop(); 17 } examples/simple/echo/main.cc examples/simple/echo/main.cc muduo/examples/simple/echo echo TcpServer EventLoop TcpConnection Buffer class class class namespace finger Python Twisted Reactor muduo muduo deferreds finger Twisted muduo finger finger01 finger07 examples/twisted/ finger

65 #include <muduo/net/eventloop.h> 2 3 using namespace muduo; 4 using namespace muduo::net; 5 6 int main() 7 { 8 EventLoop loop; 9 loop.loop(); 10 } examples/twisted/finger/finger01.cc examples/twisted/finger/finger01.cc muduo examples/twisted/finger/finger02.cc 1 #include <muduo/net/eventloop.h> 2 #include <muduo/net/tcpserver.h> 3 4 using namespace muduo; 5 using namespace muduo::net; 6 7 int main() 8 { 9 EventLoop loop; 10 TcpServer server(&loop, InetAddress(1079), "Finger"); 11 server.start(); 12 loop.loop(); 13 } examples/twisted/finger/finger02.cc 3. namespace 7 void onconnection(const TcpConnectionPtr& conn) 8 { 9 if (conn->connected()) 10 { 11 conn->shutdown(); 12 } 13 } int main() 16 { 17 EventLoop loop; 18 TcpServer server(&loop, InetAddress(1079), "Finger"); 19 server.setconnectioncallback(onconnection); examples/twisted/finger/finger03.cc

66 142 6 muduo 20 server.start(); 21 loop.loop(); 22 } examples/twisted/finger/finger03.cc 4. \r\n Buffer::findCRLF() O(N 2 ) CPU examples/twisted/finger/finger04.cc 7 void onmessage(const TcpConnectionPtr& conn, 8 Buffer* buf, 9 Timestamp receivetime) 10 { 11 if (buf->findcrlf()) 12 { 13 conn->shutdown(); 14 } 15 } int main() 18 { 19 EventLoop loop; 20 TcpServer server(&loop, InetAddress(1079), "Finger"); 21 server.setmessagecallback(onmessage); 22 server.start(); 23 loop.loop(); 24 } examples/twisted/finger/finger04.cc 5. \r\n --- examples/twisted/finger/finger04.cc :03: examples/twisted/finger/finger05.cc :06:05 -7,12 void onmessage(const TcpConnectionPtr& conn, Buffer* buf, Timestamp receivetime) { if (buf->findcrlf()) { + conn->send("no such user\r\n"); conn->shutdown(); } }

67 UserMap L30 UserMap examples/twisted/finger/finger06.cc 9 typedef std::map<string, string> UserMap; 10 UserMap users; string getuser(const string& user) 13 { 14 string result = "No such user"; 15 UserMap::iterator it = users.find(user); 16 if (it!= users.end()) 17 { 18 result = it->second; 19 } 20 return result; 21 } void onmessage(const TcpConnectionPtr& conn, 24 Buffer* buf, 25 Timestamp receivetime) 26 { 27 const char* crlf = buf->findcrlf(); 28 if (crlf) 29 { 30 string user(buf->peek(), crlf); 31 conn->send(getuser(user) + "\r\n"); 32 buf->retrieveuntil(crlf + 2); 33 conn->shutdown(); 34 } 35 } int main() 38 { 39 EventLoop loop; 40 TcpServer server(&loop, InetAddress(1079), "Finger"); 41 server.setmessagecallback(onmessage); 42 server.start(); 43 loop.loop(); 44 } examples/twisted/finger/finger06.cc 7. UserMap L examples/twisted/finger/finger06.cc :14: examples/twisted/finger/finger07.cc :15:22 -36,6 int main() { + users["schen"] = "Happy and well"; EventLoop loop; TcpServer server(&loop, InetAddress(1079), "Finger");

68 144 6 muduo server.setmessagecallback(onmessage); server.start(); loop.loop(); } telnet(1) finger Telnet $./bin/twisted_finger07 $ telnet localhost 1079 Trying ::1... Trying Connected to localhost. Escape character is '^]'. muduo No such user Connection closed by foreign host. $ telnet localhost 1079 Trying ::1... Trying Connected to localhost. Escape character is '^]'. schen Happy and well Connection closed by foreign host. 6.5 muduo muduo TCP

69 muduo percentile muduo Boost.Asio libevent2 muduo ping pong muduo Boost.Asio 15% libevent2 18% 70%. boost 1.40 asio asio libevent rc muduo asio libevent2 ping pong recipes/pingpong/libevent/ muduo libevent2 muduo examples/pingpong/ gist 17 muduo asio -O2 -finline-limit=1000 $ BUILD_TYPE=release./build.sh # muduo DELL 490 Intel Xeon E5320 CPU GHz 16GiB Ubuntu Linux Server LTS x86_64 g

70 146 6 muduo asio 18 ping pong muduo asio libevent2 ping pong echo TCP echo echo ZeroMQ 1/10/100/1000/ /2/3/4 8 4 ping pong 16KiB shell CPU TCP 110MiB/s Python CPU CPU 1 CPU

71 吞吐量 (MiB/s) 单线程 连接数 muduo asio asio libevent muduo libevent2 70% libevent2 socket 4096 buffer.c evbuffer_ read() muduo muduo

72 第6章 148 muduo 网络库简介 测试结果表明 muduo 的吞吐量平均比 libevent2 高 18% 以上 多线程测试的结果 见图 6-5 数字越大越好 多线程 (100 连接) 多线程 (1000 连接) 吞吐量 (MiB/s) 吞吐量 (MiB/s) 线程数 muduo muduo asio asio asio asio 线程数 0 图 6-5 测试结果表明 muduo 的吞吐量平均比 asio 高 15% 以上 讨论 muduo 出乎意料地比 asio 性能优越 我想主要得益于其简单的设计和简洁的 代码 asio 在多线程测试中表现不佳 我猜测其主要原因是测试代码只使用了一 个 io_service 如果改用 io_service per CPU 的话 其性能应该有所提高 我 对 asio 的了解程度仅限于能读懂其代码 希望能有 asio 高手编写 io_service per CPU 的 ping pong 测试 以便与 muduo 做一个公平的比较 由于 libevent2 每次最多从网络读取 4096 字节 这大大限制了它的吞吐量 ping pong 测试很容易实现 欢迎其他网络库 ACE POCO libevent 等 也能 加入到对比中来 期待这些库的高手出马 击鼓传花 对比 muduo 与 libevent2 的事件处理效率 前 面 我 们 比 较 了 muduo 和 libevent2 的 吞 吐 量 得 到 的 结 论 是 muduo 比 libevent2 快 18%. 有人会说 libevent2 并不是为高吞吐量的应用场景而设计的 这样的比较不公平 胜之不武 为了公平起见 这回我们用 libevent2 自带的性能测 试程序 击鼓传花 来对比 muduo 和 libevent2 在高并发情况下的 IO 事件处理效率 Linux 多线程服务端编程 使用 muduo C++ 网络库 (excerpt)

73 DELL E socketpair(2) pipe(2) ~ libevent2 test/bench.c rc bug libevent2 muduo examples/pingpong/bench.cc event watcher libev 19 timer event IO event libev Marc Lehmann muduo libevent libevent2 + muduo recipes/pingpong/libevent/run_bench.sh

74 150 6 muduo muduo libevent2 muduo (ctl_add) total time per iteration 100 active clients Dell WS file descriptors muduo libevent2 muduo (ctl_add) 1000 active clients file descriptors 700 time spent in event processing 100 active clients file descriptors muduo libevent2 muduo (ctl_add) muduo libevent2 muduo (ctl_add) 1000 active clients file descriptors time (in us) (lower is better) time (in us) (lower is better) time (in us) (lower is better) time (in us) (lower is better) 6-6

75 libevent2 event watcher muduo 20% 2. a libevent2 b libevent2 muduo muduo 1. muduo 2. libevent2 libev Marc Lehmann 21 libevent2 muduo epoll_ctl(fd, EPOLL_CTL_ ADD,...) event watcher 24 muduo epoll_ctl(fd, EPOLL_CTL_MOD,...) event watcher libevent2 epoll_ctl(fd, EPOLL_CTL_ADD,...) fd EEXIST File exists EPOLL_CTL_ADD EPOLL_CTL_MOD libevent2 muduo EPOLL_CTL_ADD event watcher 6-6 muduo libevent2 kernel muduo muduo/net/poller/epollpoller.cc CPU 2.4 GHz 1.86 GHz 21

76 152 6 muduo muduo libevent2 muduo (ctl_add) total time per iteration 100 active clients Dell E file descriptors muduo libevent2 muduo (ctl_add) 1000 active clients file descriptors 400 time spent in event processing 100 active clients file descriptors muduo libevent2 muduo (ctl_add) muduo libevent2 muduo (ctl_add) 1000 active clients file descriptors time (in us) (lower is better) time (in us) (lower is better) time (in us) (lower is better) time (in us) (lower is better) 6-7