A Sockets API TCP/IP A.1.5 x86-64 Linux Sockets API PC PC Windows 1. 2. W. Richard Stevents TCP TCP TCP A.1 561
562 A A.1.1 Sockets API Sockets API 2005 C# PC C# PDA PDA TCP PDA PC 42 PC PDA PC 7 PC FTP PC PDA single point of failure TCP read/write blocking 2010 muduo Sockets API Reactor C++ Sockets API muduo Sockets API TCP Sockets API TCP 6.4.1 TCP Sockets API A.1.5
A.1 563 A.1.2 TCP UDP client library server libmemcached memcached libpq PostgreSQL Servlet HTTP RPC TCP/IP troubleshooting library framework Sockets API Sockets API strace Sockets API TCP/IP tcpdump A.1.3 Linux 10 FreeBSD FreeBSD 2000 2000 Linux epoll FreeBSD 2001 4.1 kqueue C10k 10 Linux 1 Linux bug work around FreeBSD Linux Linux A.1.4 Windows Linux FreeBSD Solaris 1 http://en.wikipedia.org/wiki/usage_share_of_operating_systems
564 A AIX HP-UX Linux POSIX Linux SO_NOSIGPIPE Linux pipe(2) FreeBSD muduo Windows libevent libuv Java Netty A.1.5 big topic ping ping bug 1. firmware 2. 3. TCP/IP 4. HTTP FTP DNS SMTP POP3 NFS 5. HAProxy squid varnish Web load balancer 6. ZooKeeper memcached 7. 8. POP3 SMTP
A.1 565 7 TCP/IP Sockets API muduo libevent muduo Netty gevent muduo A.1.6 HTTP HTTP 1.1 bug spec HTTP 1.1 9.2.2 TCP accept 2 accept HAProxy lighttpd CPU one loop per thread 3.3 6.6 httpd Nginx lighttpd 2 Linux http://linux.dell.com/files/presentations/linux_plumbers_conf_2010/scaling_techniques_ for_servers_with_high_connection%20rates.pdf
566 A IO bound CPU Disk IO trade-off echo qps 1ms 8 8 000 qps IO A.1.7 3 master NAT TCP HTTP proxy HTTP server Web crawler HTTP HTTP proxy 3 http://blog.csdn.net/solstice/article/details/5334243 3
A.1 567 Web browser HTTP server 4 wget A.1.8 7 24 7 24 7 24 9.2 7 24 7 24 7 24 5 6 64-bit malloc memcached malloc memory pool 12.2.8 4 http://blog.codingnow.com/2006/04/iocp_kqueue_epoll.html 5 http://stackoverflow.com/questions/3770457/what-is-memory-fragmentation 6 http://stackoverflow.com/questions/60871/how-to-solve-memory-fragmentation
568 A Linux Kernel PC 3% 5% STL new/delete premature optimization A.1.9 RFC daytime HTTP 1.0 echo chargen TIME_WAIT hold TIME_WAIT TIME_WAIT (s) TIME_WAIT buggy TIME_WAIT XML JSON Protobuf
A.1 569 race condition p. 348 snapshot delta 30 end-to-end principle happens-before relationship A.1.10 7 STL STL STL STL STL 1. 2. TCP/IP 3. TCP/IP stack UNIX TCP/IP TCP/IP manpage manpage TCP/IP Linux TCP/IP 1. TCP self-connection 8 2. Linux bug TCP TCP window clamping bug work around manpage 7 http://jjhou.boolan.com/programmer-5-talk.htm 8 8.11 ACE http://blog.csdn.net/solstice/article/details/5364096
570 A error code ephemeral port backlog connect manpage TCP/IP TCP/IP TCP TCP/IP send_packet() on_receive_packet() on_timer() libnet libpcap TCP/IP TCP/IP lwip Mini/Tiny/Toy/Trivial/Yet-Another TCP/IP TUN/TAP TCP/IP D Sockets API FTDI USB-SPI ENC28J60 TCP/IP stack IP ICMP Echo TCP 3000 UDP DNS A.1.11 TCP echo chat proxy echo echo HTTP chat a b fork()-per-connection b c b a c
A.1 571 proxy 7.13 TCP A.1.12 Sockets API IPython muduo 9 Sockets API IPython IO C TCP epoll $ ipython In [1]: import socket, select In [2]: s = socket.socket(socket.af_inet, socket.sock_stream) In [3]: s.setsockopt(socket.sol_socket, socket.so_reuseaddr, 1) In [4]: s.bind(('', 5000)) In [5]: s.listen(5) In [6]: client, address = s.accept() # client.fileno() == 4 In [7]: client.recv(1024) Out[7]: 'Hello\n' # In [8]: epoll = select.epoll() In [9]: epoll.register(client.fileno(), select.epollin) # In [10]: epoll.poll(60) Out[10]: [(4, 1)] # # 4 select.epollin == 1 In [11]: client.recv(1024) # Out[11]: 'World\n' In [12]: client.setblocking(0) # In [13]: client.recv(1024) # EAGAIN == 11 error: [Errno 11] Resource temporarily unavailable In [14]: epoll.poll(60) Out[14]: [(4, 1)] In [15]: client.recv(1024) Out[15]: 'Bye!\n' # epoll_wait() # In [16]: client.close() nc 9 http://blog.csdn.net/solstice/article/details/5497814
572 A $ nc localhost 5000 Hello <enter> World <enter> Bye! <enter> muduo log strace netcat/tempest/ipython tcpdump muduo Reactor Linux C++ IO 4000 A.1.13 TCP TCP Effective TCP/IP Programming 9 Realize That TCP Is a Reliable Protocol, Not an Infallible Protocol TCP 7.5 Google Protobuf check sum IP header check sum TCP header check sum CRC32 TCP IP header TCP header checksum 16-bit check sum 16-bit integers checksum sum 16-bit checksum CRC32 CRC A-1 client server TCP segment segment IP packet ethernet frame A-1 a router ethernet frame b c server d CRC a b c d TCP header checksum payload router NAT NAT c a d payload TCP header checksum
A.2 573 client a server d switch 1 switch 2 b c router A-1 IP bit payload check sum check sum ethernet CRC When the CRC and TCP checksum disagree The Limitations of the Ethernet CRC and TCP/IP checksums for error detection 10 Amazon S3 2008 7 11 bit check sum Google 12 MD5 end-to-end principle A.2 Unix W. Richard Stevens 6 [APUE] UNIX TCP/IP 10 http://noahdavids.org/self_published/crc_and_checksum.html 11 http://status.aws.amazon.com/s3-20080720.html 12 http://www.ukuug.org/events/spring2007/programme/thatcouldnthappentous.pdf 14
574 A [UNPv2] [APUE] IPC TCP/IP TCP/IP TCP/IP Illustrated, Vol. 1: The Protocols TCP/IP TCPv1 TCPv1 13 TCPv1 TCP/IP W. Richard Stevens TCP/IP tcpdump TCP 17 24 TCP package TCP 1. Positive acknowledgement with retransmission 2. Flow control using sliding window Nagle 3. Congestion control slow start congestion avoidance fast retransmit TCP TCP flow control TCP connection TCPv1 1993 10Mbit 100Mbit switch hub 1Gbit 13 http://portal.acm.org/citation.cfm?id=161724
A.2 575 10Gbit TCP flow control TCP IPv6 TCP window scale option TCP timestamps option TCP selective ack option Linux tcpdump TCP TCPv1 2011 10 2 Unix Network Programming, Vol. 1: Networking API 2 3 3 XTI UNP W. Richard Stevens UNP 2 UNP 3 UNP Sockets API Sockets API W. Richard Stevens UNP 2 14 I have found when teaching network programming that about 80% of all network programming problems have nothing to do with network programming, per se. That is, the problems are not with the API functions such as accept and select, but the problems arise from a lack of understanding of the underlying network protocols. For example, I have found that once a student understands TCP s three-way handshake and four-packet connection termination, many network programming problems are immediately understood. TCP/IP Nagle UNP 3 UNP UNIX UNP UDP TCP IPv4 IPv6 15 14 http://www.kohala.com/start/preface.unpv12e.html 15 http://blog.csdn.net/myan/archive/2010/09/11/5877305.aspx
576 A MFC Scott Meyers Effective C++ W. Richard Stevens Jeffrey Richter CRC32 check sum check sum CRC CRC CRC CRC zlib UNP Sockets Out-of-Band Data Signal-Driven IO UNP UNP daytime echo TCP
A.2 577 W. Richard Stevens UNP UNP Unix accept() + fork() C10k TCPv1 UNP TCPv1 UNP APUE Effective TCP/IP Programming W. Richard Stevens W. Richard Stevens UNP Effective TCP/IP Programming 6 TCP TCP 2001 2011 TCP/IP Illustrated, Vol. 2: The Implementation TCPv2 1200 4.4BSD TCP/IP 15 000 C Gary Wright mbuf IP ICMP IP IGMP IP Sockets ARP 3/4 TCP TCP TCPv2 IGMP IP host-to-host IP packet
578 A TCP TCP/IP stack inode BSD 20 80 4MiB VAX TCP/IP mbuf buffer TCP/IP 4.4BSD timer Linux TCP/IP Linux TCP/IP 3 Pattern-Oriented Software Architecture Volume 2: Patterns for Concurrent and Networked Objects POSA2 UNP UNP Sockets API POSA2 library/framework event-driven libevent Java Netty Java Mina Perl POE Python Twisted POSA2 C++ RAII
B C++ Primer 4 C++ C++ Primer 4 34 B.1 C++ 2009 Stanley Lippman C++ C++ Java C# Python C++ C++ 1 C++ 2 3 C++ 10% 10% 4 1 http://shootout.alioth.debian.org/ Google https://days2011.scala-lang.org/sites/days2011/files/ws3-1-hundt.pdf 2 C++ Bjarne Stroustrup C++ http://www2.research.att.com/~bs/applications.html 3 C++ http://aristeia.com/talknotes/misra_day_2010.pdf 4 Milo Yip C++ Unreal/Source Havok/FMOD C++ http://www.cnblogs.com/miloyip/archive/2010/09/17/behind_cplusplus.html 579
580 B C++ Primer 4 C++ GC C++ 5 C++ Java C# C++ Bjarne Stroustrup C++ system programming 6 infrastructure C++ 7 Herb Sutter 8 C++ efficiency flexibility 9 abstraction productivity 10 C++ is about efficient programming with abstractions C++ 11 C++ 12 13 & inline int find_longest(const std::vector<std::string>& words) { // std::max_element(words.begin(), words.end(), LengthCompare()); } CPU C++ memory layout locality of reference 5 C++ clean up http://blog.csdn.net/myan/article/details/1906 6 CPU 7 Software Development for Infrastructure http://www2.research.att.com/~bs/computer-jan12.pdf 8 Herb Sutter C++ and Beyond 2011 Why C++? http://channel9.msdn.com/posts/c-and-beyond-2011-herb-sutter-why-c 9 just-in-time compilation 10 Stanley Lippman Linux 1970 C++ C++ 11 Ulrich Drepper Stop Underutilizing Your Computer SIMD http://www.redhat.com/f/pdf/summit/udrepper_945_stop_underutilizing.pdf 12 Technical Report on C++ Performance http://www.open-std.org/jtc1/sc22/wg21/docs/18015.html 13 Scott Meyers Effective C++ in an Embedded Environment http://www.artima.com/shop/effective_cpp_in_an_embedded_environment
B.2 C++ 581 memory hierarchy 14 Scott Meyers CPU Caches and Why You Care 15 Herb Sutter Machine Architecture 16 / GC 17 C++ C Java Python TIOBE C++ Java B.2 C++ C++ features Google 18 C++ C++ 19 20 C++ C++ Bjarne Stroustrup The C++ Programming Language Stanley Lippman C++ Primer C++ 21 C++ Primer 10 C++ 14 std::list O(1) std::vector O(N) vector compact vector list http://ecn.channel9.msdn.com/events/goingnative12/gn12cpp11style.pdf vector 15 http://aristeia.com/talknotes/accu2011_cpucaches.pdf 16 http://www.nwcpp.org/downloads/2007/machine_architecture_-_nwcpp.pdf 17 Bjarne Stroustrup Abstraction and the C++ machine model C++ Java http://www2.research.att.com/ bs/abstraction-and-machine.pdf 18 50% http://blog.csdn.net/myan/article/details/3144661 19 C++ 20 Java C# Python 21 C++ Primer 3/e http://jjhou.boolan.com/cpp-primer-foreword.pdf
582 B C++ Primer 4 C++ C++ Primer 3 C++ Primer 4 4 3 22 3 4 Barbara Moo C++ Primer 4 C++ Huffman C++ C++ syntax semantics C++ STL C++ 23 C++ C++ 24 1 C++ C++ Primer 4 C C++ C++ C C++ Primer 4 C++ C++ 25 26 C++ 900 22 Bjarne Stroustrup Programming Principles and Practice Using C++ use only the 4th edition 23 C++ Primer 4/e 24 22 Accelerated C++ C : C 25 iostream locale/facet 26 Stanley Lippman Virtual base class support wanders off into the Byzantine... The material is simply too esoteric to warrant discussion...
B.2 C++ 583 C++ Primer 4 C++ 3 string vector class string C++ Primer 4 10.4.2 10.6 15.9 Liskov C++ Primer 4 2005 2003 C++ 27 C++ 2011 C++11 TR1 28 C++11 29 C++ C++ Primer 4 C++ GNU G++ Visual C++ 30 C++ C++ C++ C++ 31 C++ C++ C++ C C++ C++ C++ 32 27 1998 C++ 28 TR1 2005 C++ bind/function 29 C++ Primer 5 C++11 30 G++ Linux Unix Visual C++ Windows C++ Intel C++ Linux G++ Windows Visual C++ 31 Cfront http://www.softwarepreservation.org/projects/c_plus_plus 32 C++ http://stackoverflow.com/questions/3931312
584 B C++ Primer 4 C++ C++ 33 C++ 34 C++ module package C C++ 35 vector<t> vector int std::string B-1 3 + 2 4 B-1 36 calculate() + Node 3 * NumberNode BinaryNode 2 4 AddNode MultiplyNode B-1 33 G++ 4.x 32-bit 64-bit 64-bit C++ Visual C++ 2010 Express Visual C++ 6.0 34 C++ C++ 10 35 std::vector<int> 36 Packrat http://www.relisoft.com/book/lang/poly/3tree.html
B.3 585 B-2 BinaryNode<std::plus<double> > BinaryNode<std:: multiplies<double> > BinaryNode<T> Node NumberNode BinaryNode<T> B-2 15.8 Handle C++ C++ C++ 37 C++ C++ 38 Google Protobuf leveldb PCRE C++ muduo Chromium Google C++ STL Boost C++ C++ C Java C++ B.3 C++ C++ Primer C++ Effective C++ 3 39 [EC3] 37 http://blog.csdn.net/myan/article/details/3247071 38 39 Scott Meyers
586 B C++ Primer 4 C++ C++ C++ Primer C++ Effective C++ 3 Effective C++ 3 C++ 2 3 C++ C++ override 2 40 3 class 41 C++ idiom RAII 42 TCP RAII C++ 5 43 C++ delete C++ Effective C++ 3 3 C++ C 44 TR1 bind/function 45 Stephan T. Lavavej PPT 40 Andrew Koenig Teaching C++ Badly: Introduce Constructors and Destructors at the Same Time http://drdobbs.com/blogs/cpp/229500116 41 std::string std::vector boost::shared_ptr class 42 heap 43 TR1 shared_ptr weak_ptr boost::scoped_ptr 44 Java 7 try-with-resources Python with C# using 45 function/bind http://blog.csdn.net/myan/article/details/5928531
B.3 587 TR1 46 47 STL 48 C++ Primer C++ C++ 49 [CCS] concept model refinement C++ STL STL 50 C++ type traits C++ 51 C 52 C++ C++ C++ Effective C++ 3 1 31 C++ C/C++ 53 46 http://blogs.msdn.com/b/vcblog/archive/2008/02/22/tr1-slide-decks.aspx 47 48 Matthew Austern 49 Herb Sutter 50 Iterator http://jjhou.boolan.com/programmer-3-traits.pdf 51 C++ http://blog.csdn.net/myan/article/details/1920 52 C++ 53 http://www.math.pku.edu.cn/teachers/qiuzy/technotes/expression2009.pdf
588 B C++ Primer 4 C++ 35 class 43 pimpl C++ 56 swap() swap() 59 #include using 73 by value by reference 76 vector 79 value smart pointer 5 entity 6 8 9 22 32 class value class base class trait class policy class exception class 33 class monolithic class 37 public 57 class namespace C++ Google C++ 54 LLVM 55 B.4 16 2010 11 54 http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#exceptions 55 http://llvm.org/docs/codingstandards.html#ci_rtti_exceptions
B.4 589 PDF C++ 15% 56 5 C++11 57 TR1 TR1 URL C++ [CCS] soxxxxxx http://stackoverflow.com/questions/xxxxxx http://www.informit.com/store/product.aspx?isbn=0201721481 http://book.douban.com/subject/10944985/ PDF http://chenshuo.com/cp4/ giantchen@gmail.com http://weibo.com/giantchen 2012 5 56 10 11 lower_bound upper_bound 57 Scott Meyers C++11 http://www.artima.com/shop/overview_of_the_new_cpp
C Boost Boost C++ Boost 1 C++ STL Boost vector map shared_ptr vector map shared_ptr weak_ptr C++ Boost Boost Boost noncopyable scoped_ptr static_assert Boost date_time 2 circular_buffer function/bind shared_ptr C++ delete lexical_cast Boost Boost 1 2010 8 2 boost::date_time muduo::timezone 591
592 C Boost regex RegEx class class RegEx regex immutable mutable basic_regex match_results match_regex regex Donald Knuth Coders at Work C/C++ C strlen strcpy strcmp C++ complex string vector class STL STL Boost Boost Boost Boost generic programming concept model refinement Boost.Threads STL Boost Boost.Preprocessor C++ Lua Boost.Proto C++ ANTLR parser Boost.Spirit Boost C++
D TCP TCP 1 IP TCP TCP/IP IPv6 TCP 65536 TCP 1.2.3.4:8765 A B B frame A 10 TCP B 100 TCP Linux socket(2) accept(2) TCP file descriptor dup() fork() TCP TCP/IP TCP 1. TCP TCP accept(2) 1 http://weibo.com/1701018393/ecuxdrta0nn 593
594 D TCP 2. TCP TCP connect(2) Sockets API TCP/IP TCP/IP TCP TCP TCP IP packet TCP 1 TCP faketcp 1a. TCP IP packet SYN TCP segment 1b. SYN ACK TCP segment 1c. ACK segment faketcp TCP faketcp TCP/IP IP packet Ethernet frame faketcp IP:PORT TCP faketcp 2 TCP faketcp 2a. SYN TCP segment 2b. SYN ACK TCP segment 2c. ACK segment faketcp SYN SYN+ACK TCP faketcp faketcp TCP faketcp TCP
595 faketcp faketcp TCP recipes/faketcp make Ubuntu Linux 10.04 PC hostname atom D-1 router atom 10.0.0.0/24.1.2 Ethernet D-1 A TUN/TAP TCP/IP D-2 192.168.0.0/24 faketcp.x TUN router atom.1 10.0.0.0/24.1.2 Ethernet D-2 atom /dev/net/tun tun0 192.168.0.1/24 faketcp 192.168.0.0/24 atom 192.168.0.2~192.168.0.254 IP packet faketcp faketcp IP atom IP packet
596 D TCP ICMP echo ping faketcp recipes/faketcp/ icmpecho.cc ICMP echo request icmp_input() recipes/faketcp/faketcp.cc 3 1. 1 sudo./icmpecho allocted tunnel interface tun0 2. 2 $ sudo ifconfig tun0 192.168.0.1/24 $ sudo tcpdump -i tun0 3. 3 $ ping 192.168.0.2 $ ping 192.168.0.3 $ ping 192.168.0.234 192.168.0.X IP ping TCP SYN TCP segment RST segment recipes/faketcp/rejectall.cc 3 faketcp./rejectall 3 $ nc 192.168.0.2 2000 $ nc 192.168.0.2 3333 $ nc 192.168.0.7 5555 IP TCP TCP SYN TCP segment SYN+ACK FIN segment FIN+ACK recipes/faketcp/acceptall.cc 3 faketcp./acceptall nc 192.168.0.X IP port 4 netstat -tpn nc netstat Send-Q
597 TCP payload TCP segment ACK recipes/faketcp/discardall.cc 3 faketcp./discardall nc 192.168.0.X IP port 4 netstat -tpn 0 2 TCP 1 atom recipes/faketcp/connectmany.cc 4 1. 1 sudo./connectmany 192.168.0.1 2007 1000 192.168.0.1:2007 1000 allocted tunnel interface tun0 press enter key to start connecting 192.168.0.1:2007 2. 2 $ sudo ifconfig tun0 192.168.0.1/24 $ sudo tcpdump -i tun0 3. 3 TCP httpd muduo echo discard listen 2007 4. 1 4 netstat -tpn TCP TCP/IP A TCP mini tcp stack IPv4 2 48 IP TCP end points end point {ip, port} end point end point 2 48 IP 2 32 IP 2 16
598 D TCP NAT Linux http://urbanairship.com/blog/2010/09/29/linux-kernel-tuning-for-c500k/ http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3 http://www.erlang-factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf
[JCP] [RWC] [APUE] [UNP] [UNPv2] [TCPv1] Brian Goetz. Java Concurrency in Practice. Addison-Wesley 2006 Bryan Cantrill and Jeff Bonwick. Real-World Concurrency. ACM Queue 2008 9. http://queue.acm.org/detail.cfm?id=1454462 W. Richard Stevens and Stephen A. Rago. Advanced Programming in the UNIX Environment 2nd ed. Addison-Wesley 2005 UNIX 2. 2006 W. Richard Stevens. UNIX 1 API 3.. 2006 Unix Network Programming vol. 1 The Sockets Networking API 3rd ed UNIX 1. 2004 W. Richard Stevens. Unix Network Programming vol. 2 Interprocess Communications 2nd ed. Prentice Hall 1999 UNIX 2 2. 2002 W. Richard Stevens. TCP/IP Illustrated vol. 1: The Protocols. Addison- Wesley 1994 TCP/IP 1. 2010 [TCPv2] W. Richard Stevens. TCP/IP Illustrated vol. 2: The Implementation. Addison-Wesley 1995 TCP/IP 2. 2010 [CC2e] [EC3] [ESTL] Steve McConnell. 2.. 2006 Code Complete 2nd ed Scott Meyers. Effective C++ 3.. 2006 Scott Meyers. Effective STL. Addison-Wesley 2001 599
600 [CCS] [LLL] [WELC] [TPoP] [K&R] [ExpC] Herb Sutter and Andrei Alexandrescu. C++.. 2008 C++ Coding Standards: 101 Rules Guidelines and Best Practices.. 2009 Michael Feathers... 2007 Working Effectively with Legacy Code Brian W. Kernihgan and Rob Pike... 2000 The Practice of Programming Brian W. Kernighan and Dennis M. Ritchie. The C Programming Language 2nd ed. Prentice Hall 1988 C 2. 2000 Peter van der Linden. Expert C Programming: Deep C Secrets. Prentice Hall 1994 [CS:APP] Randal E. Bryant and David R. O Hallaron. 2.. 2011 Computer Systems: A Programmer s Perspective [D&E] [ERL] [DCC] [Gr00] [jjhou02] Bjarne Stroustrup. C++.. 2002 The Design and Evolution of C++ Joe Armstrong. Erlang.. 2008 Programming Erlang Luiz A. Barroso and Urs Hölzle. The Datacenter as a Computer. Morgan and Claypool Publishers 2009 http://www.morganclaypool.com/doi/abs/10.2200/s00193ed1v01y200905cac006 Jeff Grossman. A Technique for Safe Deletion with Object Locking. More C++ Gems. Robert C. Martin (ed.). Cambridge University Press 2000. Memory Pool. 2002 9. http://jjhou.boolan.com/programmer-13-memory-pool.pdf [Alex10] Andrei Alexandrescu. Scalable Use of the STL. C++ and Beyond 2010. http://www.artima.com/shop/cpp_and_beyond_2010