A study on quality measurement of Web-based system
i
Little Little -ii-
Abstract As the application of Internet gets more and more popular nowadays, its validation and software quality assurance become more and more important at the same time. The purpose of this paper is to develop a Web site reliability certification mechanism, which is used to verify and evaluate hyperlinks of a Website. Another purpose is to evaluate that how many requests a Web server can service in the short time. After we investigate the issues related to Website quality, we study the methods for hyperlink analysis. To reinforce the present methods for hyperlink analysis, we purpose using statistical usage testing method to evaluate the reliability of the Website hyperlinks. Finally, we demonstrate a real example to explain the Website reliability certification mechanism, including usage model building and analysis. The advantages of this mechanism are the complete and automatic hyperlink evaluation process, rapidly building Website usage modeling, and accurate prediction of hyperlink reliability. In addition, we revise the Little s formula which is used to evaluated queue model and apply it in Internet. We refer the characteristic of Internet and use the revised Little s formula to evaluate performance of Website. We can provide the result of evaluation for network manager, who can base on requirement of company and adjust server connect time to reach the Website performance optimization. Keyword Software Quality Assurance hyperlink reliability Statistics usage testing Website usage model queuing theory Little s formula -iii-
... i...ii Abstract...iii...iv...viii...ix... x...xii... 1 1... 1 1.1.... 1 1.2.... 1 2... 3 3... 4... 5 1... 5 2... 5 3... 6... 8 -iv-
1... 8 1.1.... 8 1.2.... 9 2...10 2.1....10 2.2....12 2.3....14 3...16 3.1....16 3.2....18 3.3....18 3.4....21 3.5....21 3.6....23 3.7....24 3.8....26 3.9....29 4...30 4.1....30 4.2....31 4.3....33 -v-
4.4....33...37 1...37 1.1....37 1.2....38 1.3....39 2...42 2.1....42 2.2....44 3...44 3.1....44 3.2....45 3.3. Little...45 3.4. Little...48 3.5....51 4...51 4.1....52 4.2....52 4.3....53...55 1...55 -vi-
2...56...57 -vii-
1...12 2...13 3...16 4...17 5...19 6...22 7 - DR. HYPERLINK...24 8...25 9...27 10 AUTOFEA...31 11 AUTOFEA...32 12 A...32 13 B...33 14 507...36 15...43 16...45 18 W Q...48 19 3DSPOTLIGHT...52 20 3DSPOTLIGHT...54 -viii-
1...11 2...15 3...26 4...30 5 AUTOFEA...33 6 AUTOFEA...34 7 AUTOFEA...34 8 AUTOFEA...35 8 1 2 3...46 9...49 10...51 11...53 12 (%)...53 -ix-
Arcs Arc coverage Balking Connect time Complexity metric Error seeding model Fault avoidance Fault detection and removal Fault tolerance Failure rate model First come first served Hyperlink Hypertext Hyperlink reliability Homepage Idle waiting time Internet service Jockeying Load test Last come first served log Level Markov process Markov chain Mean First Passage Probability of Occurrence -x-
Queue Queuing theory Response time test Reliability growth model Response status code Response time of Website Reneging Stress test System failure System error System fault Software metric Statistics usage testing States Software Not Invoked Software Terminated State coverage Smoke test Service rate of Websites Transition probability Tunneling Usage state WWW Web page -xi-
a b c h s j Lq(t) Lq Ln n N (t)- (t) p, j k i j ^ P R R in,term R in term Rt S i t1 t2 U s Wq(i) Wq ( ) ( ) i -xii-
Wq(i) (t) (t) i ( ) t t -xiii-
1 1.1. e (Queuing theory) 1.2. 90 1
-2-
2-3-
3 1. 2. Little 3. -4-
1 Computer Industry Almanac [1] 2000 1 3,500 3,400 ( B2B B2C) 1,425 1992 47.8% 2 (Stress test) (Response time test) (Load test) -5-
3 (1) (2) HTML(Hyper Text Markup Language) HTML Java JavaScript (3) -6-
-7-
1 1.1. HTML WWW (World Wide Web) (Browser) (Hyperlink) (Hypertext) HTML Internet HTTP(Hyper Text Transport Protocol) FTP (File Transfer Protocol ) POP3 (Post Office Protocol Version) HTML <a href="url"> </a> -8-
(Web page) 1.2. 1.2.1 (log) Web Site Visitor Log Web Site Visitor Log Win95 / NT 4.0 Web Site Visitor Log CLF (Common Log Format) ECLF (Extended Common Log Format) [1] Web Site Visitor Log Web Site Visitor Log 1.2.2-9-
Linkbot [2] Linkbot Watchfire Linkbot 1000 Linkbot HTML Linkbot 1.2.3 Astra e-tester Astra Mercury Interactive Mercury Interactive [3] e-tester Empirix Empirix [4] Astra e-tester Windows 2 2.1. -10-
(System failure) (System error) (System fault) [5] 1 1 (System failure) (System error) (System fault) [5] A. (Fault avoidance) B. (Fault detection and removal) (system testing) (debugging) -11-
C. (Fault tolerance) 2.2. [5] 1 Mills et al. (1987) [5] 60% 3% 1 1-12-
2 2 1. [6] 2. -13-
3. 4. Halstead (Software metric) McCabe (Complexity metric) (Error seeding model) (Failure rate model) (Reliability growth model) [7] 2.3. -14-
2 (Hyperlink reliability) HTTP HTTP (Response status code) 2 [8] HTTP 2 200 OK 200 301 Moved Permanently 302 Moved Temporarily 303 See Other URL URL CGI script 404 Not Found 500 Server Error script URL URL HTTP 404 URL 3-15-
3 3 3.1. Harlan Mills John Musa IBM AT T [9] -16-
(Statistics usage testing) [10, 11] [12] 4 (1) -17-
(2) (3) 3.2. (Markov process) [13] [14] 3.3. [15] (Usage state) s i (i=0,1,2,n,t) s 0 s t -18-
(Markov chain) (States) (Arcs) (Transition probability)[16] 5 (Software Not Invoked) A B (Software Terminated) 1/3 5 1993 Whittaker [13] U s i y sij -19-
j y s ij = p a s ij + k τ p a s a, P sij j k, s j 1 y ij ik y s kj (1) h s = j (2) s j k Z s jk Z = y p (3) s jk s ij s jk j k h s jk h = s jk 1 Z s jk (4) (2) (4) 1. 2. 3. 1-20-
3.4. s i [15] ^ p i, j s i s j ^ P (State coverage) (Arc coverage) p i, 3.5. (Homepage) Terminated [17] in term j s j -21-
(5) 1 6 5 6 2 4 Terminated 6 Homepage Terminated 1 Homepage Link1 Link2 Terminated 1 1 0.3 1 = 0.3-22-
2 Homepage Link1 Link3 Link4 Terminated 2 1 0.7 0.8 1 = 0.56 3 Homepage Link1 Link3 Link5 Link4 Terminated 3 0.7 0.2 1 1 = 0.14 1 2 3 0.3 + 0.56 + 0.14 = 1 Link5 3 Link5 3 0.3 + 0.56 = 0.86 3.6. Dr. Hyperlink HTTP 7-23-
7 - Dr. Hyperlink 3.7. 3.7.1 HTML 8-24-
8 1 root URL Software Not Invoked 7 9 11 Software Terminated 3.7.2 ToolSET_Certify [18] -25-
-26-3.7.3 HTTP 3.7.4 / 95% 99% 3 3 Script # Status / R ( 0 1 ) C=95% (R )95% R 0.02 R=0.92 95% 0.92-0.02 0.92+0.02 C=99% (R )99% R 0.02 R=0.92 99% 0.92-0.02 0.92+0.02 3.8.
Client-Sever Java AWT Applet 9 3.8.1 Client-Sever Java HTTP (Tunnelin) Servlet Java Servlet ASP CGI RMI Server Servlet Client 3.8.2 Java servlet ToolSET_Certify API -27-
3.8.3 HypersonicSQL database URL 3.8.4 Client Java Applet Server SUN SPARC5 Workstation SUN Soloris7 Apache HTTP sever Resin Java Servlet Engine HypersonicSQL Database ToolSET_Certify 3.8.5 Java Servlet Servelet Server-side-Java applet JavaScript servlet servlet class classes JVM(Java Virtual Machine) servlet servlet servlet load classes servlet Java Java RMI(Remote Method Invocation) RMI TCP/IP Method HTTP request-response servlet -28-
1. HTTP HTTP 2. indicator method 3. HTTP request header 4. 5. servlet HTTP servlet method servlet servlet instance 6. method 7. response 8. HTTP response header 9. response 10. HTTP 3.9. Dr. Hyperlink Web Site Visitor Log, Linkbot, Astra e-tester 4-29-
4 Web Site Visitor Log Linkbot Astra e-tester Dr. Hyperlink Windows 95 / 98 / ME / NT / 2000 Bacus Laboratories Windows 98 / NT / 2000 Watchfire Windows 98 Windows 98 Windows 98/Sun Solaris Mercury Empirix Software 4 4.1. AutoFEA [19] AutoFEA (AutoFEA Engineering Software Technology Inc) -30-
AutoFEA 10 10 AutoFEA 4.2. (Level) 4 11 AutoFEA 4 A B 11-31-
11 AutoFEA 12 A -32-
13 B 4.3. AutoFEA 66 123 5.944 432(431.999) 5 5 AutoFEA 66 123 5.944 431.999 431.999 4.4. 4.4.1 6 root Homepage -33-
-Software Terminated 6 AutoFEA Software Not Invoked ( ) 1. click to Homepage Homepage 2. click to SHOP NOW SHOP NOW 3. click to Version -$2495 Version -$2495 4. click to BUY NOW $4995 BUY NOW $4995 5. click to Web Store $4995 Web Store $4995 Software Terminated 6. click to Software Terminated ( ) 4.4.2 with FEMAP for Windows 507 17 HTTP 7 (Mean First Passage) (Probability of Occurrence) 7 AutoFEA 1 2 3-34-
4 5 6 7 8 9 10 11 12 13 14 15 16 17 4.4.3 Dr. Hyperlink 95% 99% 507 100% 99.187% 100% 8 AutoFEA 8 AutoFEA Script # Result R C=95% C=99% % States % Arcs Certified Certified 496 0.9697580.001215 0.001596 100 99.187 497 0.9678070.001224 0.001609 100 99.187 498 0.9678710.001222 0.001606 100 99.187 499 0.9679360.001219 0.001602 100 99.187 500 0.968 0.001217 0.001599 100 99.187 501 0.9660680.001226 0.001611 100 99.187 502 0.9661350.001224 0.001608 100 99.187 503 0.9662030.001221 0.001605 100 99.187-35-
504 0.96627 0.001219 0.001602 100 99.187 505 0.9663370.001216 0.001599 100 99.187 506 0.9664030.001214 0.001596 100 99.187 507 0.9664690.001212 0.001592 100 99.187 14 507 180 180 420 180 420 14 507-36-
1 1.1. 1.1.1 [20] 1.1.2 90 XML[21] script -37-
1.2. [22] (Internet service) HTTP FTP POP3 IMAP(Internet Message Access Protocol) SMTP(Simple Mail Transfer Protocol)[23] HTML HTTP HTTP HTTP HTTP HTTP HTTP HTML HTTP FTP FTP -38-
1.3. [24] 1.3.1 1.3.2 (Smoke test) [1] -39-
[24] 1000 1001 1001 server not available (Queue) [2] [3] -40-
Newport Group 2000 3 8 2003 7 4 6 [25] 1.3.3 evalid WCAT OpenLoad [1] evalid evalid evalid [26] evalid Test Enabled Web Browser evalid evalid 100 1000 evalid HTML [2] Microsoft WCAT load test tool WCAT [27] -41-
IIS(Internet Information Services) NT WCAT WCAT [3] OpenLoad OpenLoad OpenDemand [28] 2 2.1. 1940 Morse 1958 Queues, Inventories, and Maintenance [29] 30 1000 (Customer) [30] 15-42-
15 2.1.1 (Reneging) (Balking) (Jockeying) 2.1.2 (Configuration) 2.1.3 (First come first served) (Last come first served) -43-
2.2. 3 3.1. e 0.1 1 1 10 1 (Response time of Website) -44-
16 T w Ts T 3.2. T = T w + Ts (Connect time) 2 (Service rate of Websites) 3.3. Little Little [31] Little Little -45-
Little A B C D E A+B+C+D+E Wq(i) i N W = Wq( i) (6) i= 1 a b 2 L q (t) t a b [32] 8 1 2 3 (t) (t) L q (t) L q W q (i) W q a b t t (t)- (t) i W = b a Lq ( t) dt (7) -46-
17 t W = b a Lq ( t) dt (8) Little L q = W q (9) Little (Idle waiting time) Little -47-
3.4. Little 18 Little A B C D E D D E D E A B C 18 W q Little 1. 2. 3. -48-
4. Little 9 R t L n c n N W q (i) t1 t2 i ( ) ( ) ( ) 1 Ln= [ (t 2 -t 1 )] - Rt Rs (10) Little W q (i) W q (i) -49-
Rs = Rt Wq (i) < c n W q (i) L n = N-n W q (i) c = N t1 t2 N=( t 2 -t 1 ) = t 2 -t 1 Little 3 Rt L n = N-n = (t 2 -t 1 ) Rs = [ (t 2 -t 1 )] - Rt Rs 1 t2-t1=1 100 (N) N 100 = =100 t1 t2 1 = 10 (c=10) (R t ) 490 ( Rs ) 7-50-
Little L n = [ (t 2 -t 1 )] - Rt 490 =[100 1]- =30 Rs 7 70 = 70 % 100 100 10 70% 3.5. evalid Microsoft WCAT load test tool OpenLoad 10 10 evalid WCAT OpenLoad V V V V X X X V X X X V V V V V X X X V 4-51-
4.1. Little 3D spotlight 19 3D spotlight 1998 [33] PC 3D spotlight ZDNet Computer Gaming World PCWorld MaximumPC PC Magazine The Register 19 3Dspotlight 4.2. Little 500 1000 1500 R s L N -52-
t 1 t 1 R s L n 11 11 N t 2 -t 1 R s L n N- L n 500 1 500 1358 3 47.334(47) 453 (90.6) 1000 1 1000 15412 17 93.412(93) 907 (90.7) 1500 1 1500 21634 20 118.3(118) 1081 (72) 4.3. Little 11 12 12 (%) C=30 C=40 C=50 (%) (s) (%) (s) (%) (s) 500 90.6 3 95.5 11 97.4 19 1000 90.7 17 94.7 26 92.3 34 1500 72 20 79.8 28 86.1 39 20 3Dspotlight -53-
20 3Dspotlight -54-
1 Little 1.1-55-
1.2 1.3 Little Little 2 Java ASP Active X -56-
[1] Bacus Laboratories INC Web site, http://www.bacuslabs.com/wsvlcore.html, (April 2002). [2] Watchfire Inc Website, http://www.watchfire.com, (April 2002). [3] Mercury Interactive Inc Website, http://www-svca.mercuryinteractive.com/, (April 2002). [4] Empirix, http://www.empirix.com/, (April 2002). [5] Lan. Sommerville, Software Engineering, Sixth Edition, Pearson Education Limited, 359-363, (2001). [6] (2001) [7] Hoan Pham, Software reliability, Springer, (2000). [8] Stephane Gigandet, Ashok Sudarsanam, and Anshu Aggarwal, The Inktomi Climate Lab: An Integrated Environment for Analyzing and Simulating Customer Network Traffic, (2001). [9] Stacy J.Prowell, Carmen J. Trammell, Richard C. Linger and Jesse H. Poore: Cleanroom Software Engineering Technology and Process, Addison Wesley Longman Inc, 91-92, (1999). [10] G. H., Walton, J. H. Poore and C. J. Trammell: "Statistical Testing of Software Based on a Usage Model," Software Practice and Experience, vol. 25(1), 97-108, (January 1995). [11] Wen-Kui Chang, S. Twu and W. Teng: Ensuring Functional Test Coverage For Avionics Control Applications Through Statistical Usage Testing, FESMA'99-2nd European Software Measurement Conference 4-8 Amsterdam, the Netherlands, 261-268, (October 1999). [12] Wen-Kui Chang, Shing-Kai Hon, A practice on performance testing for web-based systems, The Asian Journal on Quality / Vo1.1 No. 1, (December 2000). [13] J.A. Whittaker, K. Rekab, M.G. Thomason, A Markov chain model for predicting the reliability of multi-build software, Information and Software -57-
Technology 42, 889-894, (2000). [14] (2001) [15] James A. Whittaker, Jeffrey Voas, Toward a More Reliable Theory of Software Reliability, IEEE Computer, 36-42, (December 2000). [16] 12-29 (December 2001). [17] K. Sayre, J.H. Poore, Stopping criteria for statistical testing, Information and Software Technology 42 851-857, (2000). [18] Q-Labs, ToolCertify User Guide, Version 4.0 (1999). [19] AutoFEA Website, http://www.autofea.com, (April 2002). [20] Wen-Kui Chang, Shing-Kai Hon. A PRACTICAL ANALYSIS OF STRESS TESTING FOR A WEB-EXAM SERVER, 11 ICSQ International Conference on Software Quality, (2000). [21] Norman Walsh, A Technical Introduction to XML, XML white paper, (1999). [22] Ed Wilson, James Naramore, Network Monitoring and Analysis: A Protocol Approach to Troubleshooting, (2000). [23] (2000). [24] Daniel J. Mosley, Client-Server Testing on the Desktop and the Web, Prentice Hall PTR, (2000). [25] Steven Splaine, Stefan P. Jaskiel, Alberto Savoia, The Web Testing Handbook, Software Quality Engineering Pub, (2001). [26] evalid Web site, http://www.soft.com/evalid/evindex.html, (May 2002). [27] Microsoft WCAT load test tool Web site, http://www.microsoft.com/windows2000/techinfo/reskit/en/iisbook/c05_we b_application_stress_tool_and_wcat.htm, (May 2002). [28] Opendemand Web site, http://www.opendemand.com/openload/features.shtml, (May 2002). -58-
[29] Philip Morse, Queues Inventories and Maintenance, (1958). [30] Donald Gross, Carl M. Harris, "Fundamentals of Queueing Theory ", Wiley-Interscience, (1997). [31] Avi Silberschatz, Peter B. Galvin, Abraham Silberschatz, Greg Gagne, Applied Operating System Concepts, first edition, (1999). [32] Randolph W.Hall, Queueing Methods For Services and Manufacturing, Prentice-Hall International Editions, (1999). [33] 3Dspotlight Web site, "http://www.3dspotlight.com", (May 2002). -59-