untitled

Size: px

Start display at page:

Download "untitled"

洁董
7 years ago
Views:

1 串料劉立識若行更量力論串料念行串 C4.5 SVM 類料類料不更料 1. 論 (Information Extraction) 理識料料料

2 理料數力領度行更料量數若行更量力料行更料若更料理來料料不利數類料領行領識領識不了了料料理領料行理料不更料理

3 2. 料理 (data cleansing) 料行識不料料 [4] 不論料來料異料來料都料理來料料理料料料行料便異料料不料 (data profiling) 料 (data mining) 兩料 [3] 料料度料率串料量料料料 Galhardas Raman 兩料理 [1] [5] (missing value) 料 (noisy data) 料料料例理略 (ignore tuple) (unknown) (mean)(linear regression) 略不理料料欄 unknown 數數欄利來

4 料理 (bin) (clustering) 連料離料年 (partition) (hierarchy) 度 (density) 利料度來類料異異料來例 motherboard 都了料料料料論料數論數率料度料料欄料若欄料 (uniqueness detection) [6] 欄料欄 data type attribute length whether a default value is specified whether null is permitted distinctness ratio min/max data length ratio order of distinctness ratio with its relation 練料 C4.5 練便欄料料料

5 更料理串料理數料欄例不料 (data profiling) 念串串串欄異 3. 料領料料料來料不料料識料不料來量識串料立 ( 欄 ) 串料料料料識 (subject instance) 料欄串料識串欄料漏識了不料料異

6 1. (missing entities) 識料了料 2. (missing values) 識識略了串料 3. (duplicates) 料料料 4. 不 (invalid values) 識識串欄欄不立料不不易來料異不不不欄不

7 料料立料不降行 3.1 串串串類料串串類類串串異來串串不領不串不串念六串 1. string cardinality ( S c ) 串數 2. string prefix ( S p ) 串 k k 參數 3. string suffix ( S s ) 串 k k 參數 4. string entity ( S e ) 串列

8 5. string numeral ( S n ) 串數 true or false 6. string format ( S f ) 串料 SF 六串 SF = S, S, S, S, S, S } SF { c p s e n f 來料 (v i ) SF(v i ) = (S c (v i ), S p (v i ), S s (v i ), S e (v i ), S n (v i ), S f (v i )) 異欄例北串串例 SF( 北 ) and k=1 S c S p S s S e S n S f 5 北 false string 3.2 串數不立異串說欄類串串見串見串異不數欄串數料六串數理六串欄都見六數數數都串欄都見論異

9 , j {1,2,3,4,5,6 }, SF 串料 S j v i S v ) P S ( v )) 數 j ( i rob( j i S '( v ) 數 j i 兩串數都 w w 參數 T (w) 數列數 w 數 1 不 w T (10) 便 0% 10% 數 1 10% 20% 數 2 類數 10 串行數 S '( v ) S '( v ) = P ( S ( v )) T ( w) 累 j i j 串列累行 G 不 P S ( v )) 數 S j '( vi ) S j '( vi ) = Prob ( S j ( vi )) T ( w) i G i rob j i rob( j i 數例說料數 {1, 2, 3, 4, 5, 6} {1.5%, 11%, 79%, 5%, 3%, 0.5%} w 參數 10 數 {1, 4, 5, 6} 都 0% 10% 數都 1 數 {2} 都 10% 20% 數 2 數 {3} 都 70% 80% 數 8 累數 {6} 數 1 數 {1} 累數 {6,1} 數 1 來數 {5} 累

10 數 {6,1,5} 數 1 來數 {4} 累數 {6,1,5,4} 數 2 來數 {2} 累數 {6,1,5,4,2} 數 3 數 {3} 累數 {6,1,5,4,2,3} 數 10 累 S c ( v i ) P rob ( Sc ( vi )) S c '( vi ) S c ( v i ) P rob ( Sc ( vi )) S c '( vi ) i G % % % % % % % % % % % % 10 見串率數數異了論數異數異若數數異數異不不易兩切不類不行 3.3 料料類行兩類類了立類類練料串數類類類料行類

11 識料練料串數料類料不立料類 (supervised machine learning) C4.5 理論 (statistical learning theory) 量 (support vector machine) 利串數料兩不串類量 C4.5 量都 ( SVM 參數參立 LIBSVM [2]) 不串數料 C4.5 量兩 4. 令異例

12 [7] 領識理 20 年 10 異料異料料 ( 1995 年 2004 年 ) 料練料料來館館林利行烈行理 4.1 例立類料異料 2 x 2 confusion matrix true positive rate true negative rate false positive rate false negative rate 量度 2 x 2 confusion matrix 2 X 2 confusion matrix 來料來料類料 number of true positives (TP) number of false positives (FP) 類料 number of false negatives (FN) number of true negatives (TN) 量度類料

13 類料 1. True positive rate TP-rate = TP/(TP+FN) 類數率 2. True negative rate TN-rate = TN/(FP+TN) 類數率 3. False positive rate FP-rate = FP/(FP+TN) = 1 TN 類數率 4. False negative rate FN-rate = FN/(TP+FN) = 1 TP 類數率類 TP-rate TN-rate FN-rate FP-rate 理類 TP-rate TN-rate 1 數 TP-rate TN-rate 益類行 TP-rate TN-rate 降 FP-rate 4.2 了串不行了串數兩串數異劣練料練料不年年練料料類串數參數 w 參數數 ) 練料練料例數 N:1 料類異串數

14 練料立料例 100 不練料行 w 參數 100 不練料類兩串數料數 TP-rate/FP-rate 類 TP-rate FP-rate 不論 C4.5 SVM 立類累兩異切了類更類數欄不欄理欄料料精料欄料度料年欄料練料立類理 SVM 類說串數 C4.5 C4.5 累 SVM SVM 累 84.91% / 33.57% 90.06% / 29.06% 87.11% / 60.79% 92.12% / 57.19% 84.58% / 13.56% 88.58% / 7.69% 76.54% / 5.99% 79.61% / 4.00% % / 0.00% % / 0.00% 86.21% / 5.04% 89.47% / 4.86% 87.44% / 11.36% 91.13% / 3.81% 80.58% / 2.01% 81.81% / 1.43% % / 0.00% % / 0.00% % / 4.17% % / 2.58% 年 % / 2.88% % / 0.00% 93.96% / 5.00% 97.25% / 3.41% 練料立兩不練料年年參數 w 參數 100 練例數 100 年

15 練料 2001 年料 2002 年 2003 年 2004 年行 TP-rate FP-rate 年練料料年料 ( 1995 年 2004 年 ) 練料行立料類 TP-rate FP-rate 六練料欄數 TP-rate/ FP-rate 年料立類年料立類年料了料料類度欄料度料類度六練料 C4.5 年 C4.5 年 SVM 年 SVM 年 95.78% / 37.49% 90.06% / 29.06% 76.33% / 57.29% 92.12% / 57.19% 95.83% / 13.24% 88.58% / 7.69% 90.83% / 21.66% 79.61% / 4.00% % / 0.00% % / 0.00% 85.23% / 5.62% 89.47% / 4.86% 89.62% / 17.45% 91.13% / 3.81% 58.86% / 16.11% 81.81% / 1.43% % / 0.00% % / 0.00% % / 3.98% % / 2.58% 年 % / 2.56% % / 0.00% 95.07% / 4.44% 97.25% / 3.41% 串數參數 w 參數數不類串數累練料年練例數 100 立料行立料類

16 TP-rate FP-rate w 參數串數參數欄欄 w TP-rate FP-rate TP-rate FP-rate C4.5 SVM C4.5 SVM w C4.5 SVM C4.5 SVM % 92.12% 29.06% 57.19% % 81.81% 3.81% 1.43% % 84.82% 27.50% 51.56% % 86.11% 3.81% 6.65% % 72.95% 27.19% 28.75% % 89.89% 3.81% 9.33% % 80.17% 27.50% 40.00% % 88.12% 3.81% 8.79% % 87.05% 25.93% 49.56% % 89.19% 3.81% 11.40% 欄欄 w TP-rate FP-rate TP-rate FP-rate C4.5 SVM C4.5 SVM w C4.5 SVM C4.5 SVM % 79.61% 7.69% 4.00% % % 0.00% 2.58% % 85.76% 19.01% 6.75% % % 0.00% 2.58% % 89.32% 9.36% 9.00% % % 0.00% 2.58% % 85.70% 4.50% 9.50% % % 0.00% 2.58% % 78.19% 3.94% 3.75% % % 0.00% 2.58% 欄欄年 w TP-rate FP-rate TP-rate FP-rate C4.5 SVM C4.5 SVM w C4.5 SVM C4.5 SVM % 89.47% 0.00% 4.86% % 97.25% 0.00% 3.41% % 88.51% 0.00% 3.47% % 97.25% 0.00% 3.41% % 88.51% 0.00% 3.47% % 97.25% 0.00% 3.41% % 88.64% 0.00% 3.47% % 97.25% 0.00% 3.41% % 88.51% 0.00% 3.47% % 97.25% 0.00% 3.41% w 參數立料類異欄度異例欄度 w 參數數料類類年欄數料數數 w 參數不論數

17 TP-rate FP-rate 不欄數兩理論類 w 參數不 TP-rate FP-rate 益 TP-rate 若 (TP-rate ) FP-rate (FP-rate ) 兩練料練料例立不料類練料例數 100/ / / / /100 立料行立料類 TP-rate FP-rate w 參數 25 年料料率 90% 例例 1:1 例練例類例論 C4.5 立料類兩欄料類 TP-rate FP-rate 益欄例例 TP-rate 來 FP-rate 欄 9:1 例欄 3 1 例年度欄不欄不欄度異欄數量例數 900 不立料類 C4.5 類例 TP-rate FP-rate 欄料數

18 量 1400 練料數數料類更例數例數 (a) 欄 (b) 欄練料類欄數量 200 3:1 例不例數不年欄料數量 30 不不例數 4.3 論串料欄 TP-rate 85% FP-rate 27% 欄 TP-rate 87% FP-rate 4% 欄 TP-rate 91% FP-rate 3% 年欄類料串例料料類欄串念了欄領識數 C4.5 料類

19 力說欄類 string cardinality (S c ) string prefix (S p ) string suffix (S s ) 數串 k 串 k 料料數都 2 3 串 1 度料欄 string cardinality (S c ) string suffix (S s ) 料不了度異不見料類串年 S c S p S s S c S s S p S e S n S s S e S e S f S p S n S f 欄串串數類料 string numeral (S n ) 串欄串串都 string prefix (S p ) string entity (S e ) 欄串欄料見串數 string suffix (S s ) string entity (S e ) 欄串欄兩都串 string entity (S e ) string fromat (S f ) 欄年欄料數料年 84 年 93 年串 8 9 string prefix (S p ) string numeral (S n ) string fromat (S f ) 串

20 5. 論串料料力理 (post processing) 流更料理料 NSC H NSC H 參 [1] Galhardas, H., Florescu, D., and Shasha, D. An Extensible Framework for Data Cleaning, INRIA Technical Report, [2] LIBSVM, [3] Muller, H., and Freytag, J. C. Problems, Methods, and Challenges in Comprehensive Data Cleansing. Technical Report HUB-IB-164, Humboldt University Berlin, [4] Rahm, E. and Do, H.-H., Data Cleaning: Problems and Current Approaches, IEEE Bulletin of the Technical Committee on Data Engineering, Vol. 23, No. 4, December [5] Raman, V. and Hellerstein, J. M. An Interactive Framework for Data Cleaning, UC Berkeley Computer Science Division Report No. UCB/CSD00/1110, September [6] 李念料料立理論 2002 [7] 識立論 2003

untitled

untitled 領參讀什不論不若不什什不來參不什來立來度讀老度不兩落度行刺讀例讀行力讀兩輪度不浪讀率若度輪行行類不類不論例論留理 1 論論論論論論論論論論論了省省度不例理例念理論度類行聯理論度精利 2 省例行類立省精省益省省利