1 : 33 trends Key words Statistical learning theo ry, suppo rt vecto r m ach ine, m ach ine learning, pattern recogn ition 1, ( ),,,,,, (Statistical L

Size: px

Start display at page:

Download "1 : 33 trends Key words Statistical learning theo ry, suppo rt vecto r m ach ine, m ach ine learning, pattern recogn ition 1, ( ),,,,,, (Statistical L"

谟甫毋
5 years ago
Views:

1 V o l126,n o11 A CTA AU TOM A T ICA S IN ICA Jan, ) (, ),,, (SL T ) V apnik, SL T, (SVM ),, SL T SVM, SL T SVM,,,, INTROD UCT ION TO STAT IST ICAL L EARN ING THEORY AND SUPPORT VECTOR M ACH INES ZHAN G Xuegong (D ep ṫ of A u tom ation, T sing hua U niversity, B eij ing ) (S tate K ey L aboratory of Intellig ent T echnology and Sy stem s of Ch ina) Abstract D ata2based m ach ine learning covers a w ide range of top ics from pattern recogn ition to function regression and den sity estim ation 1), M o st of the ex isting m ethods are based on traditional statistics, w h ich p rovides conclusion only fo r the situation w here samp le size is tending to infinity So they m ay no t wo rk in p ractical cases of lim ited samp les Statistical L earning T heo ry o r SL T is a sm all2samp le statistics by V apnik et al, w h ich concerns m ainly the statistic p rincip les w hen sam 2 p les are lim ited, especially the p roperties of learning p rocedure in such cases p rovides us a new fram ewo rk fo r the general learning p roblem, and a novel pow erful learning m ethod called Suppo rt V ecto r M ach ine o r SVM, w h ich can so lve sm all2 samp le learning p roblem s better SL T It is believed that the study of SL T and SVM is becom ing a new ho t area in the field of m ach ine learning T h is review introduces the basic ideas of SL T and SVM, their m ajo r characteristics and som e current research Tsinghua Tongfang Optical Disc Co, Ltd All rights reserved

2 1 : 33 trends Key words Statistical learning theo ry, suppo rt vecto r m ach ine, m ach ine learning, pattern recogn ition 1, ( ),,,,,, (Statistical L earn ing T heo ry SL T ) V V apnik [ 1 ],,, [ 2, 3, ],, ( ) ;, (Suppo rt V ecto r M ach ine SVM ),, SL T SVM [ 3, ] [ 4 ],,,, ; SL T ; SVM ; 2 2 1, : y x, F (x, y ), (x y ), n (x 1, y 1), (x 2, y 2),, (x n, y n), (1) {f (x, w ) } f (x, w 0), R (w ) = L (y, f (x, w ) ) df (x, y ) (2) Tsinghua Tongfang Optical Disc Co, Ltd All rights reserved

3 34 26, {f (x, w ) }, w, {f (x, w ) } ; L (y, f (x,w ) ) f (x, w ) y,,, y 1), y = {0, 1} {1, - 1},, L (y, f (x, w ) ) = 0, if y = f (x, w ), 1, if y f (x, w ), Bayes, y ( ), L (y, f (x, w ) ) = (y - f (x, w ) ) 2, (4), x p (x,w ), 2 2 L (p (x, w ) ) = - logp (x, w ) (5),,, (1), (2), (ERM ), R emp (w ) = 1 n 6 n (3) L (y i, f (x i, w ) ), (6) (2), (3), ; (4), ; (5) ERM, ERM,,,, n (6) (2), ERM? 2 3 ERM, R emp (w ),,,,,,,,, {x, y }, y [ 0, 1 ],, f (x, Α) = sin (Αx ) (Α ), Α, 1), Tsinghua Tongfang Optical Disc Co, Ltd All rights reserved

4 1 : 35,,,,,, [ 3 ], y = x 2 10, ERM,,,, 100, 71%,, 1) ; 2), 3 [ 2, ] : 1) ; 2) ; 3) ; 4) ( ),, V C 3 1 VC,, V C (V apnik2chervonenk is D im en sion) V C :, h 2 h, h ; V C h, V C V C V C, V C ( ), V C, V C n V C n+ 1, f (x, Α) = sin (Αx ) V C ( ), V C ( ),,, ( ) V C [ 5 ] 3 2, [ 2, ], : ( ), R emp (w ) R (w ) 1- Γ [ 6 ] : Tsinghua Tongfang Optical Disc Co, Ltd All rights reserved

5 36 26 R (w ) R emp (w ) + h (ln (2ngh) + 1) - n ln (Γg4), (7) h V C, n : ( ),, V C R (w ) R emp (w ) + 5 (hgn) (8),, V C ( ),, V C,,,,, V C ( [ 6 ] hgn> 0137, V C 1) ),,, V apnik [ 2 ], 3 3, ERM,,,, ( hgn ),,,,, 1 V C ( 5 ) ;,,, 1 (Structu ral R isk M in im ization [ 4 ] ) SRM [ 2 SRM ] SRM,,,, ( 0),, 1), V C,,, Tsinghua Tongfang Optical Disc Co, Ltd All rights reserved

6 1 : 37 [ 3 ] 1) SRM 4 SVM,, [ 7, 8, 9, ], 4 1 SVM, 2,, H, H 1, H 2, (m argin) ( 0), xgw + b= 0,, 2 (x i, y i),,, n, x R d, y {+ 1, - 1}, y i[ (w g x i) + b ] - 1 0, i = 1,, n (9) 2 gwg, gwg 2 (9) 1 2 gwg 2, H 1, H 2, SVM [ 2 ], N, R, gwg A f (x, w, b) = sgn{ (wgx) + b} (sgn ( ) ) V C h m in ( [R 2 A 2 ], N ) + 1 (10) gwg 2 V C, SRM [ 6 L agrange ], 6 n y iαi = 0 Αi 0, i = 1,, n Αi (11a) (11b) 1), SRM, V C, Tsinghua Tongfang Optical Disc Co, Ltd All rights reserved

7 38 26 Q (Α) = 6 n Αi n ΑiΑjy iy j (x ig x j), (12) i, j= 1 Αi L agrange,, ( ) Αi, f (x) = sgn{ (w g x) + b} = sgn 6 n Α 3 1 y i (x ig x) + b 3, (13) b 3, ( (9) ),, (9) Νi 0, y i[ (w g x i) + b ] Νi 0, i = 1,, n, (14) (w, Ν) = 1 n 2 gw g 2 + C 6 Νi,,, C > 0,, (11b) Αi C, i = 1,, n (15) N, V C N + 1, (10), gwg A V C, V C ( [2 ] V C 10 3 ),,,,,,,,,, (12) (13) (x igx j),,,,, [ 2 K (x i, x j) M ercer, ], K (x i, x j),, (12) Q (Α) = 6 n 1 Αi n ΑiΑjy iy jk (x i, x j), (16) i, j= 1 f (x) = sgn (6 n Α 3 i y ik (x i, x ) + b 3 ), (17),, ( ) SVM,,, Tsinghua Tongfang Optical Disc Co, Ltd All rights reserved

8 1 : SVM,, K (x, x i) = [ (x g x i) + 1 ] q, (18) q ; (RBF) K (x, x i) = exp - gx - x ig 2 Ρ 2, (19) RBF,, 3 Sigm o id, K (x, x i) = tanh (v (x g x i) + c), (20) SVM,, 4 4 SVM [ SVM ], f (x) = wgx+ b {x i, y i},,, n, x i R d, y i R, Ε, y i - w g x i - b Ε, i = 1,, n (21) w g x i + b - y i Ε,, 1, 2 gw g 2, Νi 0 Ν 3 i 0, (21) y i - w g x i - b Ε+ Νi, w g x i + b - y i Ε+ Ν 3 i = 1,, n (22) i, 1 (Νi+ Ν 3 i 2 gw g 2 + C6 n ), C > 0 Ε 6 n (Αi - Α 3 i ) = 0, (23) 0 Αi, Α 3 i C, i = 1,, n, L agrange Αi, Α 3 i W (Α, Α 3 ) = - Ε6 n (Α 3 i + Αi) + 6 n f (x ) = y i (Α 3 i - Αi) - (w g x ) + b = 6 n n (Α 3 i - Αi) (Α 3 j - Αj) (x ig x j), i, j= 1 (24) (Α 3 i - Αi) (x ig x ) + b 3 (25) Tsinghua Tongfang Optical Disc Co, Ltd All rights reserved

9 40 26 SVM, Αi, Α 3 i 0,, ;, K (x i, x j) (24), (25) 4 5 SVM, [ 13, 14, ], (PCA ) {x 1,, x n}, C = 1 n 6 n x ix T i x <(x), C θ = 1 n 6 n <(x i) <(x i) T, v, Κv = C θ v, Κ<(x k) g v = <(x k) g C θ v, k = 1,, n (26), v v = 6 n Αi<(x i), (26), K = {K ij} = { (<(x i) g <(x j) ) } = {K (x i, x j) }, (27) (K ij i j ), nκa = K a, (28) a= [Α1,, Αn ] T K a C θ v, <(x) x, <(x) v, v g <(x) = 6 n Αi<(x i) g <(x) = 6 n ΑiK (x i, x) (29), SVM, K (x i, x j),, 4 6, SVM,, [ 15,, ] SVM [ 2, 8 ],, 215%, 1612%, 519%, 511% ( ), SVM ( (18) (20) ) 410% 411% 412%, SVM, SVM, SVM ( ), SVM 80%,, (,, SVM ), Tsinghua Tongfang Optical Disc Co, Ltd All rights reserved

10 1 : 41 [ 16, 17 SVM, ] [ 18 ] [ 19 ], SVM [ 20 ],M IT SVM, [ ] [ 23, 24 ] [ 25 ] [ 26 ] [ 27 ],, [ 10 12, 28, 29 ] [ 30, 31, 23 SVM ] SVM [ 34 ] [ 32, 33 ] 5,,,,,,,,, 1 V apnik V N E stim ation of Dependencies Based on Emp irical Data Berlin: Sp ringer2v erlag, V apnik V N The N ature of Statistical L earning Theo ry, N Y: Sp ringer2v erlag, 1995 :, 1999 ( ) 3 Cherkassky V, M ulier F L earning from Data: Concep ts, Theo ry and M ethodṡ N Y: John V iley & Sons, :, V apnik V, L evin E, L e Cun Y M easuring the VC2dim ension of a learning m achine N eu ral Comp u tation, 1994, 6: Burges C J C A tuto rial on suppo rt vecto r m achines fo r pattern recognition D ata M ining and K now led g e D isco2 very, (2) 7 Bo ser B, Guyon I, V apnik V A training algo rithm fo r op tim al m argin classifiers, F ifth A nnual W o rkshop on Computational L earning Theo ry P ittsburgh: ACM P ress, Co rtes C, V apnik V Suppo rt2vecto r netwo rkṡ M ach ine L earning, 1995, 20: Scho β lkopf B, Burges C, V apnik V Extracting suppo rt data fo r a given task In: Fayyad U M, U thurusam y R (eds ) P roc of F irst Intl Conf on Know ledge D iscovery & Data M ining, AAA I P ress, 1995, V apnik V, Go low ich S, Smo la A, Suppo rt vecto r m ethod fo r function app roxim ation, regression estim ation, and signal p rocessing In: M ozer M, Jo rdan M, Petsche T (eds) N eural Info rm ation P rocessing System s, M IT P ress, 1997, 9 11 M ller K2R, Smo la A J, R a β tsch G et al P redicting tim e series w ith suppo rt vecto r m achineṡ In: P roc of ICANN 97, Sp ringer L ecture No tes in Computer Science, 1997, D rucker H, Burges C, Kaufm an L et al Suppo rt vecto r regression m achineṡ (eds) N eural Info rm ation P rocessing System s, M IT P ress, 1997, Tsinghua Tongfang Optical Disc Co, Ltd All rights reserved In: M ozerm, Jo rdan M, Petsche T

11 Scho β lkopf B, Smo la A, M ller K2R 589 Kernel p rincipal component analysiṡ In: P roc of ICANN 97, 1997, Scho β lkopf B, Smo la A, M ller K2R Nonlinear component analysis as kernel eigenvalue p roblem N eural Comp u2 tation, 1998, 10 (5) : Scho β lkopf B, Sung K2K, Burges C et al Comparing suppo rt vecto r m achines w ith Gaussian kernels to radial basis function classifiers IE E E T rans on S ig nal P rocessing, 1997, 45 (11) : Scho β lkopf B, Burges C, V apnik V Inco rpo rating invariances in suppo rt vecto r learning m achines, In: von der M alsburg C, von Seelen W, Vo rbr ggen J C et al (eds) A rtificialn euraln etwo rks2icann 96, Sp ingers L ecture No tes in Computer Science, Berlin, 1996, 1112: Scho β lkopf B, Sim ard P, Smo la A et al P rio r know ledge in suppo rt vecto r kernelṡ N IPS 97, Guyon I, M atic N, V apnik V D iscovering info rm ative patterns and data cleaning In: Fayyad U M, P iatetsky2 Shap iro G, Sm yth P et al (eds) A dvances in Know ledge D iscovery & Data M ining, M IT P ress, 1996, Burges C, Scho β lkopf B Imp roving the accuracy and speed of suppo rt vecto r m achineṡ In: M ozer M, Jo rdan M, Petsche T (eds) N eural Info rm ation P rocessing System s, M IT P ress, 1997, 9 20 M atic N, Guyon I, Denker J et al W riter adap tation fo r on2line handw ritten character recognition Conf on Pattern R ecognition and Docum ent A nalysis, 1993, In: 2nd Intl 21 O ren M, Papageo rgiou C, Sinha P et al Pedestrian detection using w avelet temp lates, In: P roc of CV PR 97, Puerto R ico, Hearst M A, Scho β lkopf B, Dum ais S et al Sy stem s, 1998, 13 (4) : T rends and controversies2suppo rt vecto r m achines, IE E E Intellig ent 23 O suna E, F reund R, Giro si F T raining suppo rt vecto r m achines: an app lication to face detection In: P roc of CV PR 97, Puerto R ico, , SVM, 1999 ( ) 25 L u Chunyu, Yan P ingfan, Zhang Changshui, Zhou J ie of ICNNB 98, Beijing, 1998, B lanz V, Scho β lkopf B, B lthoff H Face recognition using suppo rt vecto r m achine In: P roc et al Comparison of view 2based object recognition algo rithm s using realistic 3D models, In: von der M alsburg C, von Seelen W, Vo rbr ggen J C et al (eds) A rtificial N eural N etwo rks ICANN 96, Sp ingers L ecture No tes in Computer Science, Berlin, 1996, 1112: B row n M, L ew is H G, Gunn S R L inear spectral m ixture models and suppo rt vecto r m achines fo r remo te sens2 ing, (subm itted to) IE E E T ranṡ Geoscience and R em ote S ensing, M ukherjee S, O suna E, Giro si F Nonlinear p rediction of chao tic tim e series using a suppo rt vecto r m achine In: P roc of NN SP 97, Kwok J T 2Y, Suppo rt vecto r m ixture fo r classification and regression p roblem ṡ ICPR 98, Bennett K, M angasarian O Robust linear p rogramm ing discrim ination of two linearly inseparable sets O p tim iz ation M ethod s and S of tw are, 1992, 1: O suna E, F reund R, Giro si F A n imp roved training algo rithm fo r suppo rt vecto r m achineṡ In: P roc of NN SP 97, Bennett K P, Dem iriz A Sem i2supervised suppo rt vecto r m achineṡ In: P roc of N IPS 98, Zhang Xuegong U sing class2center vecto rs to build suppo rt vecto r m achineṡ In: P roc of NN SP : A nguita D, R idella S, Rovetta S C ircuital imp lem entation of suppo rt vecto r m achineṡ E lectronics L etters, 1998, 34 (16) : , 1994,, Tsinghua Tongfang Optical Disc Co, Ltd All rights reserved

7 53 1 [5, 7 ],,,,,,,,,,, 111, (,,, ) : (a,m, b ) [A,M B ]; (a,m, b ), [A,M, B ], (a, m, b, (a, m, b ), [A,M, B ], A, A + ( ) ( ), A - ( ) ( ) ;M,M +,

7 53 1 [5, 7 ],,,,,,,,,,, 111, (,,, ) : (a,m, b ) [A,M B ]; (a,m, b ), [A,M, B ], (a, m, b, (a, m, b ), [A,M, B ], A, A + ( ) ( ), A - ( ) ( ) ;M,M +, 1999 7 7 α (, 200331), ; ; ; A nalysis on the Co re System M odel P rincip le of the Basic T heo ry of Ch inese M edicine GU AN X iao si (D ep ṫ of Basic Science, Shanghai T iedao U n iversity, Shanghai