Keywords: Internet slang, Internet buzzwords, text normalization, decision tree, bi-gram language model 研究背景與動機 Web 2.0 Blog Internet Slang ::

Transcription

1 第十七卷第一期 2015 年 3 月 (pp.25~48) * 81% 90% An Internet Slang Translator Based on Decision Tree and Bi-gram Language Model Heng-Li Yang * Hung-Chang Huang Qing-Feng Lin Department of Management Information Systems, National Cheng-Chi University Abstract While conducting text mining on Chinese content, Internet slang is a problem which results in a lower accuracy of text segmentation. Translating Internet slang into formal Chinese would help segmentation and, in addition, revealing the implicit information of the slang. In order to translate Internet slang, this study first collected meanings of slangs and web texts. Next, Internet slang was categorized, and translating methods, which are mainly based on decision tree and bi-gram language model, were developed for each category. The translator was then implemented. Eighty-one percentages of the Internet slang in web texts were correctly detected and translated, with a precision at ninety percentages. It is concluded that the proposed methods are quite applicable to Internet slang translation. * yanh@nccu.edu.tw NSC H MY3 DOI: /JEB (1).02 March

2 Keywords: Internet slang, Internet buzzwords, text normalization, decision tree, bi-gram language model 研究背景與動機 Web 2.0 Blog Internet Slang :: _ :: : O O Google Noise Wu et al., 研究目的與方法 BBS 26 March 2015

3 網路用語 Initialisms Words Presented in Mandarin Word Games Numeral Expressions Huo Xing Wen 文字正規化 Text Normalization Sproat et al., 2001 Aw et al., 2006 Text-to-Speech Sproat et al., 2001; Schwarm and Ostendorf, 2002; Pennell and Liu, 2010 Aw et al., March

4 2006; Pennell and Liu, 2011; Khan and Karim, 2012 Twitter Kouloumpis et al., 2011; Liu et al., 2011 Edit Distance Yang et al Language Model Mutual Information Leveshtein Leveshtein 1966 Sasu, 2011 Word Error Rate; WER NieBen, 2000 Goutte et al., Probability Measure Manning et al., 2008; Goutte et al., 2009 Yang et al Bigram s w 1, w 2,, w n P(s) w i w i 1 n 1 Ps () Pw ( ) Pw ( w ) = 1 i+ 1 i= 1 i (1) P(w 1 ) w 1 P(w i 1 w i ) w i 28 March 2015

5 w i 1 P( ) P( ) P( ) 表情符號 2006 ( ) 3.2 英數符號文 March

6 2010 TMD 3Q U i 3.3 注音文 諧音字 特殊用語 March 2015

7 2006 BBS 網路用語數目現況 建議的轉譯系統流程 1 (1) (2) (3) 1 March

8 1 (4) (5) (6) (7) (8) (9) 4.2 偵測特殊字串 List DetectSpecialStrings(S[1.. n]) // S n L : list for storing special strings, as the output // L T : string for temporarily storing special strings // T // C for each character C in S if C is a special character // C C T 32 March 2015

9 add C to T else if C is a space character // C // T if T is composed of alphabets or T is composed of numbers // T L T add T to L; T : null else // C T add C to T else // C T L T add T to L; T : null return L \^0^/~ XD \^0^/~ XD (1) (2) 1 (3) telnet:// ftp:// 4.3 表情符號的轉譯 (1) XD ^_^ XDDD ^ ^ (2) 0 o o O O ^0^ ^o^ ^O^ (3) _ a 2 2 Feature (1) Cosine Similarity March

10 (2, 1, 0) (2, 1, 1) A (a 1, a 2,, a n ) B (b 1, b 2,, b n ) (2) A B cos( AB, ) = = A B ab 1 1+ ab ab n n (2) a + a + + a b + b + + b n 1 2 n 1 2 (2) XD XDDD (1, 1) (1, 3) D A (a 1, a 2,, a n ) B (b 1, b 2,, b n ) (3) n A B= ab i i= ab 1 1+ ab ab n n (3) i= 1 (3) (4) NieBen, March 2015

11 ^ ^ ^ ^ (2, 4) ^ _ ^_^ (2, 1) ^ _ ^ ^ ^_^ _ 3 ^ ^ 6 (0.8, 8, 3, 6) ^ ^ ^_^ 1 ^ ^ ^_^ ^_ 0 ^ ^ ^_ 2469 x y 1, y 2,..., y n ID3 CHAID CART C4.5 ID3 CHAID C4.5 CART C4.5 University of Waikato Weka 3.6 C4.5 J fold Cross Validation 3 Accuracy March

12 0 Precision Recall % 88% F F-measure F Accuracy Precision Recall F-measure March 2015

13 2 Leaf Node ^ ^ ^ ^ 2 ^_^ (0.8, 8, 3, 6) ^_ (0.7746, 6, 4, 6) 1 0 ^ ^ ^_^ ^_ 英數符號文的轉譯 OMG OMG OMG OMG 斷詞系統 Liu et al ICTCLAS 2 中文斷詞系統, March

14 叫獸 蘿莉 教授 女孩 (Nh) (Na) (DE) (Na) (SHI) (Nf) (VH) (DE) (VH) (Nb) (PERIODCATEGORY) (Nh) (Na) (Nf) (VH) (Nb) DE SHI PERIODCATEGORY 偶 ㄍ ㄊ 我 哥 他 (D) (Nc) (Nc) (DE) (VH) (VH) (FW) (VE) (FW) (Ng) (VE) (P) (VH) (Na) (PERIODCATEGORY) (D) (Nc) (VH) (FW) (VE) (Ng) (P) (Na) PERIODCATEGORY 4.6 特殊用語的轉譯 March 2015

15 TopDownSplit(S[1..n]) // S n // L L : list for storing split tokens // S M i j If S contains an internet slang M from index i to index j add TopDownSplit (S[1..i]) to L // M L add M into L // M L add TopDownSplit (S[j 1..n]) to L // M L Else // S S L add each single character of S into L Return L 3 Top-down 4.7 注音文的轉譯 2010 T (4) March

16 PT ( = c), c C T (4) C T {c 1,, c n } T P(T c) T c c T c* (5) * c = argmax P( T= c) (5) c C T P(T c) (6) PT ( = c) P( wi 1cwi+ 1 ), where wi = c (6) w i-1 w i 1 T P(w i-1 ) P(T c) P(w i-1 ) (7) Pw ( cw ) = Pw ( ) Pc ( w ) P( w c) Pc ( w ) P( w c) (7) i-1 i+ 1 i-1 i-1 i+ 1 i-1 i+ 1 Maximum Likelihood Estimation; MLE P(c w i-1 ) P(w i 1 c) P( ) 0 Add-one Smoothing; Laplace Smoothing P(c w i-1 ) P(w i 1 c) (8) Fw ( i-1c) + 1 Fcw ( i+ 1) + 1 Pc ( wi- 1) =, Pw ( i+ 1 c) = Fw ( ) + 1 Fc () + 1 i-1 (8) F Frequency F(w i-1 ) 1 P(T c) (7) 40 March 2015

17 * Fwi c F cwi c argmax P( T c) argmax ( ( ) ) ( ( ) ) = = = Fc ( )+1 c CT c CT (9) (9) 4 w i-1 w i 1 4 c F(w i-1 c) F(cw i+1 ) F(c) ( Fw ( i-1c) + 1) ( F( cwi + 1) + 1) Fc () 諧音字的轉譯 P( ) P( ) 系統概述.NET Framework 4.0 C# 4 March

18 4 5.2 測試資料來源 ptt.cc PC_Shopping nb-shopping corpus PC_Shopping 2007/8/30 nb-shopping 2006/4/ 轉譯結果評估 TP True Positive FP False Positive FN False Negative 42 March 2015

19 5 TP FP FN Precision(P) =TP/(TP+FP) Recall(R) =TP/(TP+FN) F-measure =2PR/(P+R) FP?????_? FN ^_^ ^_^ _ March

20 % 90% 44 March 2015

21 (1) (2) (3) (4) Context 1 ptt.cc PC_Shopping nb-shopping 1 C4.5 MM March

22 gov.tw/public/mmo/trad/950804_home.ppt 2009 US-China Foreign Language Aw, A., Zhang, M., Xiao, J., & Su, J. (2006). A phrase-based statistical model for SMS text normalization. Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, Sydney, Australia. Goutte, C., Cancedda, N., Dymetman, M., & Foster, G. (2009). Learning machine translation. Cambridge: The MIT Press. Khan, O. A., & Karim, A. (2012). A rule-based model for normalization of SMS text. Proceedings of the 2012 IEEE 24th International Conference on Tools with Artificial Intelligence (ICTAI), Athens, Greece. Kouloumpis, E., Wilson, T., & Moore, J. (2011). Twitter sentiment analysis: The good the bad and the OMG! Proceedings of the 5th International AAAI Conference on Weblogs and Social Media, Barcelona, Spain. Levenshtein, V. (1966). Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10, Liu, F., Weng, F., Wang, B., & Liu, Y. (2011). Insertion, deletion, or substitution? Normalizing text messages without pre-categorization nor supervision. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, USA. Liu, W., Allison, B., & Guthrie, L. (2008). Professor or screaming beast? Detecting words misuse in Chinese. Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 08), Marrakech, Morocco. Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. New York: Cambridge University Press. NieBen, S., Och, F. J., Leusch, G., & Ney, H. (2000). An evaluation tool for machine translation: Fast evaluation for MT research. Proceedings of the 2 nd Language Resources and Evaluation Conference (LREC), Athens, Greece. 46 March 2015

23 Pennell, D. L., & Liu, Y. (2010). Normalization of text messages for text-to-speech. Proceedings of the 2010 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), Dallas, Texas, USA. Pennell, D. L., & Liu, Y. (2011). Toward text message normalization: Modeling abbreviation generation. Proceedings of the 2011 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), Prague, Czech. Schwarm, S., & Ostendorf, M. (2002). Text normalization with varied data sources for conversational speech language modeling. Proceedings of the 2002 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), Orlando, Florida, USA. Sproat, R., Black, A. W., Chen, S., Kumar, S., Ostendorf, M., & Richards, C. (2001). Normalization of non-standard words. Computer Speech and Language, 15(3), Sasu, L. (2011). A probabilistic model for spelling correction. Bulletin of the Transilvania University of Brasov, 4(2), Wu, W., Zhang, B., & Ostendorf, M. (2010). Automatic generation of personalized annotation tags for Twitter users. Human Language Technologies: the 2010 Annual Conference of the North American Chapter of the ACL, Los Angeles, California, USA. Yang, S., Zhao, H., Wang, X., & Lu, B. (2012). Spell checking for Chinese. Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC 12), Istanbul, Turkey. March

24 %98%9F%E6%96%87 icq &next 4866&l f&fid %B0%E6%B9%BE%E5%9C%B0%E5%8C%BA%E7%BD%91%E7%BB%9C%E8 %AF%AD%E8%A8%80%E5%88%97%E8%A1%A8 - %9F%B3%E6%96% %E 8%A1%A8%E6%83%85%E7%AC%A6%E8%99%9F ViewArticle&articl eid Yahoo! question?qid Chinese Digger 48 March 2015