562 A A.1.1 Sockets API Sockets API 2005 C# PC C# PDA PDA TCP PDA PC 42 PC PDA PC 7 PC FTP PC PDA single point of failure TCP read/write blocking 2010

Size: px

Start display at page:

Download "562 A A.1.1 Sockets API Sockets API 2005 C# PC C# PDA PDA TCP PDA PC 42 PC PDA PC 7 PC FTP PC PDA single point of failure TCP read/write blocking 2010"

琼殷
5 years ago
Views:

1 A Sockets API TCP/IP A.1.5 x86-64 Linux Sockets API PC PC Windows W. Richard Stevents TCP TCP TCP A.1 561

2 562 A A.1.1 Sockets API Sockets API 2005 C# PC C# PDA PDA TCP PDA PC 42 PC PDA PC 7 PC FTP PC PDA single point of failure TCP read/write blocking 2010 muduo Sockets API Reactor C++ Sockets API muduo Sockets API TCP Sockets API TCP TCP Sockets API A.1.5

3 A A.1.2 TCP UDP client library server libmemcached memcached libpq PostgreSQL Servlet HTTP RPC TCP/IP troubleshooting library framework Sockets API Sockets API strace Sockets API TCP/IP tcpdump A.1.3 Linux 10 FreeBSD FreeBSD Linux epoll FreeBSD kqueue C10k 10 Linux 1 Linux bug work around FreeBSD Linux Linux A.1.4 Windows Linux FreeBSD Solaris 1

4 564 A AIX HP-UX Linux POSIX Linux SO_NOSIGPIPE Linux pipe(2) FreeBSD muduo Windows libevent libuv Java Netty A.1.5 big topic ping ping bug 1. firmware TCP/IP 4. HTTP FTP DNS SMTP POP3 NFS 5. HAProxy squid varnish Web load balancer 6. ZooKeeper memcached POP3 SMTP

5 A TCP/IP Sockets API muduo libevent muduo Netty gevent muduo A.1.6 HTTP HTTP 1.1 bug spec HTTP TCP accept 2 accept HAProxy lighttpd CPU one loop per thread httpd Nginx lighttpd 2 Linux for_servers_with_high_connection%20rates.pdf

6 566 A IO bound CPU Disk IO trade-off echo qps 1ms qps IO A master NAT TCP HTTP proxy HTTP server Web crawler HTTP HTTP proxy 3 3

7 A Web browser HTTP server 4 wget A bit malloc memcached malloc memory pool

8 568 A Linux Kernel PC 3% 5% STL new/delete premature optimization A.1.9 RFC daytime HTTP 1.0 echo chargen TIME_WAIT hold TIME_WAIT TIME_WAIT (s) TIME_WAIT buggy TIME_WAIT XML JSON Protobuf

9 A race condition p. 348 snapshot delta 30 end-to-end principle happens-before relationship A STL STL STL STL STL TCP/IP 3. TCP/IP stack UNIX TCP/IP TCP/IP manpage manpage TCP/IP Linux TCP/IP 1. TCP self-connection 8 2. Linux bug TCP TCP window clamping bug work around manpage ACE

10 570 A error code ephemeral port backlog connect manpage TCP/IP TCP/IP TCP TCP/IP send_packet() on_receive_packet() on_timer() libnet libpcap TCP/IP TCP/IP lwip Mini/Tiny/Toy/Trivial/Yet-Another TCP/IP TUN/TAP TCP/IP D Sockets API FTDI USB-SPI ENC28J60 TCP/IP stack IP ICMP Echo TCP 3000 UDP DNS A.1.11 TCP echo chat proxy echo echo HTTP chat a b fork()-per-connection b c b a c

11 A proxy 7.13 TCP A.1.12 Sockets API IPython muduo 9 Sockets API IPython IO C TCP epoll $ ipython In [1]: import socket, select In [2]: s = socket.socket(socket.af_inet, socket.sock_stream) In [3]: s.setsockopt(socket.sol_socket, socket.so_reuseaddr, 1) In [4]: s.bind(('', 5000)) In [5]: s.listen(5) In [6]: client, address = s.accept() # client.fileno() == 4 In [7]: client.recv(1024) Out[7]: 'Hello\n' # In [8]: epoll = select.epoll() In [9]: epoll.register(client.fileno(), select.epollin) # In [10]: epoll.poll(60) Out[10]: [(4, 1)] # # 4 select.epollin == 1 In [11]: client.recv(1024) # Out[11]: 'World\n' In [12]: client.setblocking(0) # In [13]: client.recv(1024) # EAGAIN == 11 error: [Errno 11] Resource temporarily unavailable In [14]: epoll.poll(60) Out[14]: [(4, 1)] In [15]: client.recv(1024) Out[15]: 'Bye!\n' # epoll_wait() # In [16]: client.close() nc 9

12 572 A $ nc localhost 5000 Hello <enter> World <enter> Bye! <enter> muduo log strace netcat/tempest/ipython tcpdump muduo Reactor Linux C++ IO 4000 A.1.13 TCP TCP Effective TCP/IP Programming 9 Realize That TCP Is a Reliable Protocol, Not an Infallible Protocol TCP 7.5 Google Protobuf check sum IP header check sum TCP header check sum CRC32 TCP IP header TCP header checksum 16-bit check sum 16-bit integers checksum sum 16-bit checksum CRC32 CRC A-1 client server TCP segment segment IP packet ethernet frame A-1 a router ethernet frame b c server d CRC a b c d TCP header checksum payload router NAT NAT c a d payload TCP header checksum

13 A client a server d switch 1 switch 2 b c router A-1 IP bit payload check sum check sum ethernet CRC When the CRC and TCP checksum disagree The Limitations of the Ethernet CRC and TCP/IP checksums for error detection 10 Amazon S bit check sum Google 12 MD5 end-to-end principle A.2 Unix W. Richard Stevens 6 [APUE] UNIX TCP/IP

14 574 A [UNPv2] [APUE] IPC TCP/IP TCP/IP TCP/IP Illustrated, Vol. 1: The Protocols TCP/IP TCPv1 TCPv1 13 TCPv1 TCP/IP W. Richard Stevens TCP/IP tcpdump TCP TCP package TCP 1. Positive acknowledgement with retransmission 2. Flow control using sliding window Nagle 3. Congestion control slow start congestion avoidance fast retransmit TCP TCP flow control TCP connection TCPv Mbit 100Mbit switch hub 1Gbit 13

15 A Gbit TCP flow control TCP IPv6 TCP window scale option TCP timestamps option TCP selective ack option Linux tcpdump TCP TCPv Unix Network Programming, Vol. 1: Networking API XTI UNP W. Richard Stevens UNP 2 UNP 3 UNP Sockets API Sockets API W. Richard Stevens UNP 2 14 I have found when teaching network programming that about 80% of all network programming problems have nothing to do with network programming, per se. That is, the problems are not with the API functions such as accept and select, but the problems arise from a lack of understanding of the underlying network protocols. For example, I have found that once a student understands TCP s three-way handshake and four-packet connection termination, many network programming problems are immediately understood. TCP/IP Nagle UNP 3 UNP UNIX UNP UDP TCP IPv4 IPv

16 576 A MFC Scott Meyers Effective C++ W. Richard Stevens Jeffrey Richter CRC32 check sum check sum CRC CRC CRC CRC zlib UNP Sockets Out-of-Band Data Signal-Driven IO UNP UNP daytime echo TCP

17 A W. Richard Stevens UNP UNP Unix accept() + fork() C10k TCPv1 UNP TCPv1 UNP APUE Effective TCP/IP Programming W. Richard Stevens W. Richard Stevens UNP Effective TCP/IP Programming 6 TCP TCP TCP/IP Illustrated, Vol. 2: The Implementation TCPv BSD TCP/IP C Gary Wright mbuf IP ICMP IP IGMP IP Sockets ARP 3/4 TCP TCP TCPv2 IGMP IP host-to-host IP packet

18 578 A TCP TCP/IP stack inode BSD MiB VAX TCP/IP mbuf buffer TCP/IP 4.4BSD timer Linux TCP/IP Linux TCP/IP 3 Pattern-Oriented Software Architecture Volume 2: Patterns for Concurrent and Networked Objects POSA2 UNP UNP Sockets API POSA2 library/framework event-driven libevent Java Netty Java Mina Perl POE Python Twisted POSA2 C++ RAII

19 B C++ Primer 4 C++ C++ Primer 4 34 B.1 C Stanley Lippman C++ C++ Java C# Python C++ C++ 1 C C++ 10% 10% Google 2 C++ Bjarne Stroustrup C C Milo Yip C++ Unreal/Source Havok/FMOD C

20 580 B C++ Primer 4 C++ GC C++ 5 C++ Java C# C++ Bjarne Stroustrup C++ system programming 6 infrastructure C++ 7 Herb Sutter 8 C++ efficiency flexibility 9 abstraction productivity 10 C++ is about efficient programming with abstractions C++ 11 C & inline int find_longest(const std::vector<std::string>& words) { // std::max_element(words.begin(), words.end(), LengthCompare()); } CPU C++ memory layout locality of reference 5 C++ clean up 6 CPU 7 Software Development for Infrastructure 8 Herb Sutter C++ and Beyond 2011 Why C++? 9 just-in-time compilation 10 Stanley Lippman Linux 1970 C++ C++ 11 Ulrich Drepper Stop Underutilizing Your Computer SIMD 12 Technical Report on C++ Performance 13 Scott Meyers Effective C++ in an Embedded Environment

21 B.2 C memory hierarchy 14 Scott Meyers CPU Caches and Why You Care 15 Herb Sutter Machine Architecture 16 / GC 17 C++ C Java Python TIOBE C++ Java B.2 C++ C++ features Google 18 C++ C C++ C++ Bjarne Stroustrup The C++ Programming Language Stanley Lippman C++ Primer C++ 21 C++ Primer 10 C++ 14 std::list O(1) std::vector O(N) vector compact vector list vector Bjarne Stroustrup Abstraction and the C++ machine model C++ Java bs/abstraction-and-machine.pdf 18 50% 19 C++ 20 Java C# Python 21 C++ Primer 3/e

22 582 B C++ Primer 4 C++ C++ Primer 3 C++ Primer Barbara Moo C++ Primer 4 C++ Huffman C++ C++ syntax semantics C++ STL C++ 23 C++ C C++ C++ Primer 4 C C++ C++ C C++ Primer 4 C++ C C Bjarne Stroustrup Programming Principles and Practice Using C++ use only the 4th edition 23 C++ Primer 4/e Accelerated C++ C : C 25 iostream locale/facet 26 Stanley Lippman Virtual base class support wanders off into the Byzantine... The material is simply too esoteric to warrant discussion...

23 B.2 C C++ Primer 4 C++ 3 string vector class string C++ Primer Liskov C++ Primer C++ 27 C C++11 TR1 28 C C++ C++ Primer 4 C++ GNU G++ Visual C++ 30 C++ C++ C++ C++ 31 C++ C++ C++ C C++ C++ C C++ 28 TR C++ bind/function 29 C++ Primer 5 C G++ Linux Unix Visual C++ Windows C++ Intel C++ Linux G++ Windows Visual C++ 31 Cfront 32 C++

24 584 B C++ Primer 4 C++ C++ 33 C++ 34 C++ module package C C++ 35 vector<t> vector int std::string B B-1 36 calculate() + Node 3 * NumberNode BinaryNode 2 4 AddNode MultiplyNode B-1 33 G++ 4.x 32-bit 64-bit 64-bit C++ Visual C Express Visual C C++ C std::vector<int> 36 Packrat

25 B B-2 BinaryNode<std::plus<double> > BinaryNode<std:: multiplies<double> > BinaryNode<T> Node NumberNode BinaryNode<T> B Handle C++ C++ C++ 37 C++ C++ 38 Google Protobuf leveldb PCRE C++ muduo Chromium Google C++ STL Boost C++ C++ C Java C++ B.3 C++ C++ Primer C++ Effective C [EC3] Scott Meyers

26 586 B C++ Primer 4 C++ C++ C++ Primer C++ Effective C++ 3 Effective C++ 3 C C++ C++ override class 41 C++ idiom RAII 42 TCP RAII C C++ delete C++ Effective C C++ C 44 TR1 bind/function 45 Stephan T. Lavavej PPT 40 Andrew Koenig Teaching C++ Badly: Introduce Constructors and Destructors at the Same Time 41 std::string std::vector boost::shared_ptr class 42 heap 43 TR1 shared_ptr weak_ptr boost::scoped_ptr 44 Java 7 try-with-resources Python with C# using 45 function/bind

27 B TR STL 48 C++ Primer C++ C++ 49 [CCS] concept model refinement C++ STL STL 50 C++ type traits C++ 51 C 52 C++ C++ C++ Effective C C++ C/C Matthew Austern 49 Herb Sutter 50 Iterator 51 C C

28 588 B C++ Primer 4 C++ 35 class 43 pimpl C++ 56 swap() swap() 59 #include using 73 by value by reference 76 vector 79 value smart pointer 5 entity class value class base class trait class policy class exception class 33 class monolithic class 37 public 57 class namespace C++ Google C++ 54 LLVM 55 B

29 B PDF C++ 15% 56 5 C TR1 TR1 URL C++ [CCS] soxxxxxx PDF giantchen@gmail.com lower_bound upper_bound 57 Scott Meyers C++11

31 C Boost Boost C++ Boost 1 C++ STL Boost vector map shared_ptr vector map shared_ptr weak_ptr C++ Boost Boost Boost noncopyable scoped_ptr static_assert Boost date_time 2 circular_buffer function/bind shared_ptr C++ delete lexical_cast Boost Boost boost::date_time muduo::timezone 591

32 592 C Boost regex RegEx class class RegEx regex immutable mutable basic_regex match_results match_regex regex Donald Knuth Coders at Work C/C++ C strlen strcpy strcmp C++ complex string vector class STL STL Boost Boost Boost Boost generic programming concept model refinement Boost.Threads STL Boost Boost.Preprocessor C++ Lua Boost.Proto C++ ANTLR parser Boost.Spirit Boost C++

33 D TCP TCP 1 IP TCP TCP/IP IPv6 TCP TCP :8765 A B B frame A 10 TCP B 100 TCP Linux socket(2) accept(2) TCP file descriptor dup() fork() TCP TCP/IP TCP 1. TCP TCP accept(2)

34 594 D TCP 2. TCP TCP connect(2) Sockets API TCP/IP TCP/IP TCP TCP TCP IP packet TCP 1 TCP faketcp 1a. TCP IP packet SYN TCP segment 1b. SYN ACK TCP segment 1c. ACK segment faketcp TCP faketcp TCP/IP IP packet Ethernet frame faketcp IP:PORT TCP faketcp 2 TCP faketcp 2a. SYN TCP segment 2b. SYN ACK TCP segment 2c. ACK segment faketcp SYN SYN+ACK TCP faketcp faketcp TCP faketcp TCP

35 595 faketcp faketcp TCP recipes/faketcp make Ubuntu Linux PC hostname atom D-1 router atom / Ethernet D-1 A TUN/TAP TCP/IP D /24 faketcp.x TUN router atom / Ethernet D-2 atom /dev/net/tun tun /24 faketcp /24 atom ~ IP packet faketcp faketcp IP atom IP packet

36 596 D TCP ICMP echo ping faketcp recipes/faketcp/ icmpecho.cc ICMP echo request icmp_input() recipes/faketcp/faketcp.cc sudo./icmpecho allocted tunnel interface tun $ sudo ifconfig tun /24 $ sudo tcpdump -i tun $ ping $ ping $ ping X IP ping TCP SYN TCP segment RST segment recipes/faketcp/rejectall.cc 3 faketcp./rejectall 3 $ nc $ nc $ nc IP TCP TCP SYN TCP segment SYN+ACK FIN segment FIN+ACK recipes/faketcp/acceptall.cc 3 faketcp./acceptall nc X IP port 4 netstat -tpn nc netstat Send-Q

37 597 TCP payload TCP segment ACK recipes/faketcp/discardall.cc 3 faketcp./discardall nc X IP port 4 netstat -tpn 0 2 TCP 1 atom recipes/faketcp/connectmany.cc sudo./connectmany : allocted tunnel interface tun0 press enter key to start connecting : $ sudo ifconfig tun /24 $ sudo tcpdump -i tun TCP httpd muduo echo discard listen netstat -tpn TCP TCP/IP A TCP mini tcp stack IPv IP TCP end points end point {ip, port} end point end point 2 48 IP 2 32 IP 2 16

38 598 D TCP NAT Linux

39 [JCP] [RWC] [APUE] [UNP] [UNPv2] [TCPv1] Brian Goetz. Java Concurrency in Practice. Addison-Wesley 2006 Bryan Cantrill and Jeff Bonwick. Real-World Concurrency. ACM Queue W. Richard Stevens and Stephen A. Rago. Advanced Programming in the UNIX Environment 2nd ed. Addison-Wesley 2005 UNIX W. Richard Stevens. UNIX 1 API Unix Network Programming vol. 1 The Sockets Networking API 3rd ed UNIX W. Richard Stevens. Unix Network Programming vol. 2 Interprocess Communications 2nd ed. Prentice Hall 1999 UNIX W. Richard Stevens. TCP/IP Illustrated vol. 1: The Protocols. Addison- Wesley 1994 TCP/IP [TCPv2] W. Richard Stevens. TCP/IP Illustrated vol. 2: The Implementation. Addison-Wesley 1995 TCP/IP [CC2e] [EC3] [ESTL] Steve McConnell Code Complete 2nd ed Scott Meyers. Effective C Scott Meyers. Effective STL. Addison-Wesley

40 600 [CCS] [LLL] [WELC] [TPoP] [K&R] [ExpC] Herb Sutter and Andrei Alexandrescu. C C++ Coding Standards: 101 Rules Guidelines and Best Practices Michael Feathers Working Effectively with Legacy Code Brian W. Kernihgan and Rob Pike The Practice of Programming Brian W. Kernighan and Dennis M. Ritchie. The C Programming Language 2nd ed. Prentice Hall 1988 C Peter van der Linden. Expert C Programming: Deep C Secrets. Prentice Hall 1994 [CS:APP] Randal E. Bryant and David R. O Hallaron Computer Systems: A Programmer s Perspective [D&E] [ERL] [DCC] [Gr00] [jjhou02] Bjarne Stroustrup. C The Design and Evolution of C++ Joe Armstrong. Erlang Programming Erlang Luiz A. Barroso and Urs Hölzle. The Datacenter as a Computer. Morgan and Claypool Publishers Jeff Grossman. A Technique for Safe Deletion with Object Locking. More C++ Gems. Robert C. Martin (ed.). Cambridge University Press Memory Pool [Alex10] Andrei Alexandrescu. Scalable Use of the STL. C++ and Beyond

1 C++ 2 Bjarne Stroustrup C++ (system programming) 6 (infrastructure) C++ 7 Herb Sutter 8 C++ (efficiency) (flexibility) 9 (abstraction) (productivity

1 C++ 2 Bjarne Stroustrup C++ (system programming) 6 (infrastructure) C++ 7 Herb Sutter 8 C++ (efficiency) (flexibility) 9 (abstraction) (productivity 1 C++ 1 C++ Primer C++ (giantchen@gmail.com) 2012-7-11 Creative Commons - - 3.0 Unported (cc by-nc-nd) http://creativecommons.org/licenses/by-nc-nd/3.0/ 1 C++ 2009 Stanley Lippman C++ C++ Java/C#/Python