An Automated Term Definition Extraction System using the Web Corpus in Chinese Language

Transcription

1 私 立 東 海 大 學 資 訊 工 程 與 科 學 研 究 所 碩 士 論 文 指 導 教 授 : 呂 芳 懌 博 士 從 全 球 資 訊 網 語 料 自 動 萃 取 中 文 詞 彙 定 義 之 系 統 的 研 究 An Automated Term Definition Extraction System using the Web Corpus in Chinese Language 研 究 生 : 柯 志 杰 中 華 民 國 九 十 七 年 二 月 i

2 中 文 摘 要 本 研 究 提 出 一 個 系 統 稱 為 DefExplorer, 它 可 以 透 過 全 球 資 訊 網 自 動 收 集 及 萃 取 一 個 中 文 詞 彙 的 定 義, 並 自 動 識 別 詞 彙 的 類 型 及 去 除 雜 訊 DefExplorer 使 用 語 意 方 式 過 濾 不 理 想 的 資 訊 本 研 究 提 出 兩 種 候 選 集 合, 分 別 是 一 般 候 選 集 合 與 領 域 候 選 集 合, 並 依 候 選 句 的 相 似 度 對 眾 多 的 定 義 候 選 句 進 行 分 群, 及 估 算 候 選 句 的 重 要 性, 以 供 最 後 解 答 的 選 取 實 驗 顯 示 DefExplorer 能 夠 有 效 從 全 球 資 訊 網 萃 取 詞 彙 的 定 義, 尤 其 是 一 般 字 典 沒 有 收 錄 的 專 有 名 詞, 效 果 尤 佳 關 鍵 字 : 定 義 全 球 資 訊 網 資 訊 萃 取 中 文 文 字 探 勘 知 識 系 統 i

3 Abstract This paper proposes a system, named DefExplorer, which extracts term definitions from the Web, determines the type of Chinese question terms, and selects answers from noisy Web pages automatically. DefExplorer filters out invalid data with a semantic approach. Two types of candidate sets, common and domain specific, are deployed to cluster similar candidates into groups and determine candidates importance for selecting final answers. Experimental results show that DefExplorer can effectively extract term definitions from the Web, especially for the definitions of out-of-vocabulary terms. Keywords: Definitions, Web corpus, Information Extraction, Chinese Language, Text Mining, Knowledge System ii

4 致 謝 首 先 誠 摯 的 感 謝 指 導 教 授 呂 芳 懌 博 士, 老 師 悉 心 的 教 導 使 我 得 以 廣 泛 接 觸 資 訊 各 種 領 域 的 深 奧, 更 不 時 的 討 論 並 指 點 我 正 確 的 論 文 寫 法, 使 我 獲 益 匪 淺 還 要 感 謝 赴 日 留 學 大 分 大 學 時, 末 田 教 授 人 工 智 慧 自 然 語 言 課 程 詳 盡 的 指 導, 奠 定 我 研 究 的 紮 實 基 礎 在 三 年 半 的 日 子 裡, 實 驗 室 裡 共 同 的 生 活 點 滴, 學 術 上 的 討 論 趕 作 業 吃 宵 夜 的 革 命 情 感, 感 謝 眾 位 學 長 同 學 學 弟 妹 的 共 同 砥 礪 從 大 學 專 題 起, 在 實 驗 室 呆 了 太 多 年, 要 感 謝 義 樺 拉 我 近 來 這 個 領 域, 並 感 謝 泰 翔 小 B 小 新 小 賴 文 晉 郁 婷 小 葛 世 彥, 還 有 多 到 叫 不 出 名 字 的 學 弟 們, 你 / 妳 們 的 陪 伴 讓 這 幾 年 的 研 究 生 活 變 得 絢 麗 多 彩 還 要 感 謝 DZ, 跟 你 打 屁 耍 冷 之 間, 總 是 莫 名 得 到 許 多 新 技 術 新 觀 念, 讓 我 受 益 良 多 最 後, 謹 以 此 文 獻 給 我 的 朋 友 與 家 人 iii

5 List of Contents Chapter 1 Introduction...1 Chapter 2 Related Work...3 Chapter 3 Extracting Term Definitions Question Analysis Document Retrieval Semantics Selection Similarity Scoring Candidate Grouping Answer Generation...11 Chapter 4 Experimental Results Clustering Threshold Performance of Dynamic Way Performance of Static Way Integrating with Existing Lexicon Performance on Two Sorted Bases Comparison with Other Systems...23 Chapter 5 Conclusions and Future Research...26 References...27 Appendix A Definition Sentence Patterns...31 Appendix B Question Term Sets for Experiments...32 Appendix C Examples of an Extracting Process...32 Appendix D Examples of Extracting Results and Its Outputs...38 iv

6 List of Figures Figure 1. The DefExplorer system architecture...4 Figure 2. Definition brief and its tokens...8 Figure 3. Dynamic and static ways to generate answer set for a query term...14 Figure 4. User interface of DefExplorer...15 Figure 5. An example of an unsatisfied clustering results...17 Figure 6. Clustering results given different θ values...17 Figure 7. Top-n inclusion rates (combining Table 4 and 5) Figure 8. Comparison of Top-n inclusion rates Figure 9. Comparison of Top-n precision v

7 List of Tables Table 1 Examples of question terms...18 Table 2 Performance dynamic way on different γ s...19 Table 3 Performance of each candidate set on different γ s...19 Table 4 Top-n inclusion rates by using DefExplorer only, given the three question sets211 Table 5 Top-n inclusion rates when DefExplorer is integrated with an existing lexicon. Those that can be found in a given lexicon are listed in appendix B with * signs Table 6 Top-n inclusion rates by using two sorting bases, group size and TF-IDF score233 Table 7 Top-n precision by using two sorting bases, group size and TF-IDF score vi

8 Chapter 1 Introduction As societies and technologies have been developed quickly nowadays, new vocabularies are continuously created and developed to enrich our everyday life. Building up lexicons that contain new vocabularies covering different fields to provide us with an upto-date word search thus becomes more difficult than before. Facing the incessant advancement of vocabularies, when reading newspapers or documents, we will unavoidably find unknown words and terms that fail to be found in any dictionaries. That means it is no longer sufficient for us to look up related information in dictionaries only before we can realize an article completely. This is so called out-of-vocabulary (OOV) problem [1, 2]. Waiting for update of dictionaries takes time and is infeasible. The Web, as currently one of the largest data resources in the World, has provided users with a great number of data and knowledge. People can freely contribute their own information of different fields to the Web. However, given keywords, what retrieved from the Web is very often a huge amount of related and unrelated data, including descriptions and/or statements. People frequently do not know what to do next. Retrieving useful knowledge, e.g., the meaning and definition of a term/word (team as a representative), from the Web is currently a meaningful research, particularly when we have to or would like to know the semantics of an unknown term. In Chinese, explanatory sentences of a term are frequently expressed by simple declarative sentences, like A 是 B (namely, A is B). However, declarative sentences may be too common in format to be discriminated from non-declarative sentences, e.g. question sentences, such as 瑜 珈 是 什 麼 (What is Yoga?) is also in the form of A 是 B. 1

9 The searching results sometimes confuse users, particularly when the term submitted is an unknown one. Definitional question answering is a new area of question answering [3, 4]. Definitional questions are often in a form of X 是 什 麼? (What is X?) or X 是 誰? (Who is X?), such as 瑜 珈 是 什 麼? (What is Yoga?) and 張 藝 謀 是 誰? (Who is Yimou Zhang?), where X is a question term, and should be a noun. Definitional questions are different from other question answering, such as 瑜 珈 源 自 哪 國? (What county is Yoga from?), in that they are not to request a well-defined answer. We define definitions of X as the conceptual facts that are able to be collected and described in dictionaries or encyclopedias. A definitional question usually has more than one answer. All conceptual facts that define X from different viewpoints are all probable answers. This is an important feature an outstanding definitional QA system should have. In addition, a well-defined definition is often very short and precious. In this paper, we propose a term definition searching system, named Definition Explorer (DefExplorer in short), aiming to find out term definitions collected in the Web corpus [5-7], to mine the Web for definitions of a given term, and to effectively screen accurate definitions for this term from noisy data retrieved. The rest of this paper is organized as follows. Section 2 introduces several related text mining studies that use Web corpus. Section 3 describes how DefExplorer extracts term definitions from the Web automatically. The experimental results are illustrated in section 4. Section 5 concludes this article and discusses our future work. 2

10 Chapter 2 Related Work Various techniques for answering definitional questions have been proposed, mainly oriented by TREC (Text Retrieval Conference) Question Answering track [3, 4]. Prager et al. [8] proposed an approach to divide definitional questions into a number of subquestions. For example, a question concerning a person would be replaced by subquestions regarding his/her name, birthday, experience and so on. The answers of the sub-questions will be finally assembled to form the answer of the original one. Blair- Goldensohn et al. [9] used a technique that ranks sentences based on their similarity to a word s centroid composed of sentences concerning the question term. Han et al. [10] introduced a probabilistic model for definitional QA that reflects the characteristics of a given definitional question. That work considers that an answer of the definitional question should contain contents relevant to the topic of the question term, and follow a representative form of the definition style. However, these approaches focus on how to extract definitions from noiseless sources, such as artificial text corpus or online dictionaries, and they are all experimented in English. Web corpus has been proved useful in natural language processing, especially in crosslanguage information retrieval research [1, 2, 11]. Lu et al. [1] and Zhang et al. [2] used Web corpus for an automated term translation extraction, and obtained desirable results. Lam et al. [11] employed Web corpus to automatically mine new terms translations everyday. In this research, we focus on how to search definitions with the assistance of Web corpus. 3

11 Chapter 3 Extracting Term Definitions DefExplorer extracts definitions or their equivalences for Chinese terms in six phases, including question analysis, document retrieval, semantics selection, similarity scoring, candidate clustering (also called candidate clustering), and answer generation. Figure 1 illustrates its system architecture which will be described step by step in the following. Figure 1 The DefExplorer system architecture 4

12 3.1 Question Analysis Given a question term X, question analysis is to identify its type, aiming to classify the term into one of pre-defined domains, such as person, location or organization. A term that fails to be classified into any pre-defined domains is treated as a normal term. Two methods are deployed to identify type of X. The first is to match postfix of the term with pre-built lexicons. For example, if X is ended by 市 (city), it will be classified into type location. The second is generating several Web queries, like X 是 一 個 (X is a ) and X 是 一 位 (X is one who ) [12], to search the Web, and recognizing the type of X from search results, e.g., the results containing an entry X 是 一 間 知 名 軟 體 公 司 (X is a famous software company) give the question term a very high probability that this term is strongly related to an organization since the word 公 司 (company) appearing in the lexicon concerns organizations. 3.2 Document Retrieval After type of the question term is identified, DefExplorer generates queries to search the Web through the use of search engines, and then retrieves related Web pages. No matter the question term is classified into a specific domain or is a normal term, DefExplorer produces a query set, named common query set, consisting of common definition sentence patterns, which are pre-defined definition sentence patterns adapted to all or most question terms concerned. In fact, most term definitions in Chinese are formatted by X 是 Definition (X is Definition). For example, given a question term 台 北, the common query set DefExplorer generates includes common definition sentence patterns, 台 北 是 (Taipei is ), 台 北 意 為 (Taipei means ) and so on, which will be 5

13 submitted, rather than submitting 台 北 (Taipei) only, to search the Web. After that, DefExplorer compares those sentences retrieved, with common definition sentence patterns collected, which are listed in appendix A of this paper. Those that match at least one pattern will be selected as members of the candidate set, named common candidate set. In addition, some definition sentence patterns only appear together with specific domain terms [8, 10], e.g., 出 生 於 (was born in/on/at) and 曾 任 (served as) always follow a person, e.g., 孫 文 出 生 於 1866 年 (Sun Yat-San was born in 1866.) and 柯 林 頓 曾 任 美 國 總 統 (Clinton served as President of the US). 位 於 (located in/at) follows a location or an organization, e.g., 台 北 101 位 於 信 義 計 畫 區 (Taipei 101 is located in Xinyi-plan district.). DefExplorer on receiving a question term concerning, e.g. 李 鴻 章 (Li Hongzhang), generates a query set containing 李 鴻 章 曾 任 (Li Hongzhang served as ) and 李 鴻 章 出 生 於 (Li Hongzhang was born in ) as domain query sets. In other words, DefExplorer generates a common query set and several domain query sets for a domain specific question term. Queries in a domain query set request information of designate targets, e.g., a person s name, birthday and occupation. We generate a query set for each designate target T, named target-t domain query set, which contains pre-collected definition sentence patterns adapted to this domain. Appendix A lists three example patterns. Queries collected in the same domain query set aim to gather the same target information. For example, 位 於 (located at) and 坐 落 於 (situated at) both request a location. We collect them as target-location domain query set. 成 立 於 (established in) that requests the establishing year will be collected it in target-establishing-year domain query set. We search the Web by separately submitting each domain query set, and collect retrieved sentences as domain candidate sets. 6

14 However, using query sets (common and domain) to filter definition sentences, a lot of sentences containing no term definitions will be also retrieved, e.g., searching MP3 是 (MP3 is ), MP3 是 什 麼 (What is MP3?) will be also retrieved, i.e., sentences may be over-extracted in this phase. Therefore, they need to be further filtered. 3.3 Semantics Selection In this phase, we remove candidates that do not match any definition sentence patterns. In the following, a definition candidate in a common or domain candidate set is selected as a definition by using different selection principles. We have found that most term definitions follow some specific common grammatical patterns. Unsuitable candidates can be then filtered out by using language grammars. First, term definitions should be affirmative sentences. Hence, we delete interrogative and exclamatory sentences which are not hard to be detected since they encounter question marks, exclamation marks or several exclamatory words, e.g., 台 北 是 不 錯 啦! (Taipei is nice!) which is ended by an exclamatory word 啦, is a fragmented conversation sentence, rather than a term definition. Next, we analyze morphologies of these probable term definitions to see whether the structure of a retrieved sentence matches any definition sentence patterns or not. According to our observation, about 75 % (2720/3628) of term definitions carry summary information prior to the first comma (or prior to first period if there is no comma before the period) since people very often attempt to point out the key meanings of the given terms in this portion. Those after the comma describe further details or list other less relevant contents. Figure 2 gives an example. There are two definitions, ADSL 是 一 種 寬 頻 技 術, 用 於 提 高 網 路 速 度 and ADSL 是 一 種 寬 頻 的 技 術, 是 在 現 有 銅 線 上 傳 輸 數 7

15 位 資 料, that meet the patterns. But, their similarity is very low because the portions after commas often describe details from different viewpoints. However, the similarity of the portions before the commas of these two sentences is very high. Therefore, only the portion prior to the first comma, called definition brief, is preserved. Figure 2 Definition brief and its tokens After that, we use a part-of-speech dictionary [13] to partition the definition brief into tokens, each of which is associated with information of its own part-of-speech. Now, a candidate set consists of definition briefs rather than original sentences retrieved from the Web. We also count appearing frequency for each token and accordingly calculate TF- IDF scores for each definition brief [14]. However, among 5832 tokens, 323 tokens (about 5.54%) are not included in the part-of-speech dictionary. Hence, they are failed to be tagged with part-of-speech. We treat these words as nouns, since most of Chinese terms are nouns [12]. Also, according to our observation, most ideal term definition briefs are ended by nouns which can be enhanced or modified by adjectives or other nouns [15]. For example, the definition of the term 台 中 (Taichung) might be 台 灣 中 部 的 城 市 (a city in central part of Taiwan) where 城 市 (city) is a noun, and its prefix 台 灣 中 部 的 (in central part of Taiwan) describes 城 市 (city) because Chinese sentences commonly use early words/phrases to modify words that follow. Therefore, a definition brief terminated by a word other than a noun, e.g., ended by an adjective like beautiful or great, is of- 8

16 ten a subjective description, e.g., 台 中 是 很 漂 亮 的 (Taichung is beautiful.), which should not be a definition. Domain candidate sets as stated above request specific types of target information. We use pre-collected domain dictionaries to examine whether the search results accord with type of the request or not. DefExplorer collects well-known location names in the world and postfixes of location names, like 市 (city) and 城 (town), to determine whether a candidate is a location. We retrieve valid candidates as a part of final definition briefs. 3.4 Similarity Scoring To choose the best candidate from the final definition briefs, we assume that appropriate term definitions will be shown up repeatedly and similarly. That is, if a term has a specific definition, then those definitions concerning the term on different Web pages should be all similar. Hence, we calculate the similarity score between any two definition briefs in the same candidate set and cluster those with high similarity together as a group. The purpose is to merge redundant definition briefs and determine importance of each group. We assume that if a group has more definition briefs, its importance will be higher. Before the calculation of similarity score, the synonym problem, which is defined as different terms having the same meaning, should be solved first. We solve this problem by deploying thesaurus dictionary [16, 17]. Definition briefs in a candidate set at the previous stage were split into tokens and tagged with part-of-speech. DefExplorer looks up thesaurus dictionary for each token, and replace the token with a representative term, e.g., 星 國 (Singapore) is replaced by 新 加 坡 (Singapore). 9

17 Since definition briefs are extracted from Chinese Web pages in the world, different areas have different phraseologies. DefExplorer deploys an additional step to substitute area-specific term with a commonly used one, like 數 碼 ( digital used in mainland China) will be substituted by 數 位 ( digital used in Taiwan) in Taiwan. In most situations, a definition brief is often short. DefExplorer calculates similarity score Sim( A B) for definition briefs A and B in a candidate set S by using characters as its comparison units. Even HowNet [18] provides an interface for users to compare the similarity of two terms. But many terms are yet to be defined in it, and comparison performance is another consideration. Sim( A B) LLCS ( ( AB, )) max( LA ( ), LB ( )) = (1) where LCS is the longest common subsequence identified by using the longest common sequence algorithm [19], Lx ( ) the length of x, and, AB S. Sim( A B) = 1 when A B =. If A is completely different from B, Sim( A B) = 0, Therefore, Sim( A B) ranges between 0 and 1. Also, Sim( A B) = Sim( B A). 3.5 Candidate Grouping The way to cluster the definition briefs in a candidate set into groups is that when Sim( A B) is higher than a given clustering threshold θ, A and B are clustered to the same group. However, a fixed clustering threshold value is inappropriate, since a popular term may generate many definition briefs once the correspondent query sets are submitted. Hence, a higher θ value should be given. A less popular term yields few definition briefs. This time, a lower θ value can generate a better clustering result. That is, a dynamic clustering 10

18 approach is used. DefExplorer multiplies maximum similarity score of definition briefs in a candidate set with a constant ( 0< < 1) to produce a critical value θ critical, { } θ critical = max Sim( A B) AB, S (2) with which definition briefs are clustered, where is 0.65 in this article which will be described later. Nevertheless, this clustering approach sometimes yields unsatisfied results. For example, when Sim( A B) > θcritical and Sim( B C) > θcritical, A, B and C will be clustered to the same group. But, sometimes Sim( A C) is lower than θ critical. However, if Sim( A C) is close to θ critical, DefExplorer still classifies them into the same group. But when Sim( A C) is far lower than θ critical, we define another variable θmin < θcritical to further cluster them into different subgroups. After clustering, we check to see whether the smallest Sim( A B) in each group is smaller than θ min or not. If yes, DefExplorer increases θ critical and re-clusters the definition briefs in this group into subgroups again until scores in each group are all higher than θ min. θ min is 0.3 in this study. Since we assume that term definitions will be shown up repeatedly, groups containing only one definition brief will be deleted. Further, given a group G, we calculate sum of similarity score S D for each definition brief D. S = Sim( D D) D where, i (3) Di G DD G i. The one with the highest S D in G is the one the most similar to all remaining definition briefs, and is then selected as the representative of G. The TF-IDF score [14] of group G is the TF-IDF score of G s representative. 3.6 Answer Generation In the final phase, DefExplorer generates answers by selecting definition groups from each candidate set. 11

19 Since a question term may have more than one definitions, DefExplorer selects λ groups as probable answers, named answer set. Answer of a specific domain candidate set may be unique, e.g., the birthday. We call it single-answer candidate set. A candidate set which has more than one probable answer, e.g., one s working experiences, is called multiple-answer candidate set. We use two ways, static and dynamic, to select answers (see Figure 3). Dynamic By dynamic way, the value of λ varies. DefExplorer sorts all groups of a candidate set with a descending order based on their group sizes. Because the amount of a question term s candidate sentences in the Web is often proportional to popular degree of the term. Therefore, given a question term X, DefExplorer selects all groups that have definition briefs not less than ϕ from X s common candidate set and multiple-answer candidate sets as X s answer set A, where ϕ = γ total amount of definition briefs in the candidate set, and γ is a fixed value, e.g., 3%. Further, DefExplorer selects the first group from each single-answer candidate set and also inserts them into A as a part of the final answer set. Let k be number of multiple-answer domain candidate sets, m i and c be numbers of groups collected from multiple-answer domain candidate set i and common candidate set, respectively, as a part of a final answer set. Let s be number of single-answer domain candidate sets, n j be number of definition briefs in group j where group j is a group collected in the final answer set and is collected from the common candidate set, and q ip be number of definition briefs in group p where group p is a group collected in the final answer set and is collected from multiple-answer domain candidate set i. Then, k c mi s λ = + + (4) i= 1 12

20 Let dy be number of definition briefs collected in the final answer set. Then, c k m i dy = nj+ qip + s (5) j= 1 i= 1 p= 1 Static By static way, the value λ is fixed. DefExplorer merges common and domain candidate sets (single and multiple) together, and sorts all groups with a descending order based on their group sizes or TF-IDF score of the representative of each group. The size of a group is number of definition briefs in the group. After sorting on group size (on TF- IDF score), for a single-answer candidate set, DefExplorer keeps the group with the biggest group size (with the highest score), and removes other groups from the candidate set. Finally, top ranked λ groups are selected as the answer set. In the following experiments, we can see that static way can easily identify which system tested can select more accurate answers given the same answer sizes. Let d s be number of definition briefs collected in the final answer set, h be number of multiple-answer domain candidate sets, f and l i be number of groups selected from the common candidate set and multiple-answer domain candidate sets i, respectively, as a part of the final answer set, s s be number of groups selected from single-answer domain candidate sets. n j be number of definition briefs in selected group j from the common candidate set, and q ip be number of definition briefs in the selected group p from multiple-answer domain candidate set i. Then, f d = n + q + s h s j ip s l i (6) j= 1 i= 1 p= 1 And h f li ss λ = + + (7) i= 1 13

21 A group selected as a member of an answer set is called an answer entry, which is represented by the group s representative. Appendix B lists results for each step of DefExplorer given a query term 馬 英 九. Figure 3 Dynamic and static ways to generate answer set for a query term 14

22 Chapter 4 Experimental Results We implement DefExplorer with the algorithms discussed above as a Web application. Figure 4 shows its user interface. Given a question term, DefExplorer searches the Web and generates answer set in which an answer entry is accompanied by its URLs, with which a user can visit source pages he/she is interested in. Figure 4 User interface of DefExplorer Six experiments were performed in this study. The first evaluated the appropriate value of clustering threshold θ. The second and the third focused on performance of generating answer sets by using dynamic and static ways, respectively. The fourth experiment studied the top-n inclusion rate, which will be defined later, when a lexicon is in- 15

23 volved, besides the Web. In the fifth experiment, two different sorting approaches were deployed for ordering and choosing answers from candidate sets. After that, their performances were then compared. The final experiment compared performances of DefExplorer and several existing state-of-art systems. 4.1 Clustering Threshold In order to assign an appropriate clustering threshold θ to DefExplorer for clustering definition briefs, we generate answer sets by giving θ different values ranging from 0.0 to 1.0 and define two factors, Min Cohesion which is minimum similarity of any two definition briefs in a given group, and Max Coupling which is the maximum similarity between two definition briefs selected from two different groups [20, 21], to evaluate the results. Formal definitions of the two factors are as follows. Given two groups G and H collected in a candidate set, and two definition briefs A and B. Min Cohesion = min( Sim( A B) AB, G) (8) Max Coupling = max( Sim( A B) A GB, HG, H) (9) Both of the two factors range from 1 to 0. A high Min Cohesion score, close to 1, means all definition briefs in a given group are very similar. A high Max Coupling score, on the other hand, represents a high similarity of definition briefs between two groups. Figure 5 shows an example, in which A, B, C and D are definition briefs. Since both Sim(A, B) and Sim(A, C) are less than or equal to θ, A, B and C are clustered into a group. However, even Sim(C, D) is smaller than Sim(C, B), D is not clustered into the group. Figure 6 illustrates the plots of the Min Cohesion and Max Coupling against different θ values. It also shows that Min Cohesion is almost equal to Max Coupling at θ =

24 which is the most appropriate θ value to group definition briefs. Therefore, we use the θ value to do the remaining experiments. Figure 5 An example of an unsatisfied clustering results Score Min cohesion Max coupling θ Figure 6 Clustering results given different θ values 17

25 Table 1 Examples of question terms Question Set China Times Economic Daily Yam queries Examples of question terms 吳 郭 魚 (tilapia) / 新 疆 (Xinjiang) / 信 用 卡 (credit card) 標 準 普 爾 (Standard & Poor's) / EMBA / 統 一 發 票 (Uniform Invoice) 大 樂 透 (lotto) / MP3 / 好 玩 遊 戲 區 (interesting games) 4.2 Performance of Dynamic Way In the second experiment, we chose three specific domains, including person, location and organization, as an example, in which question terms belonging to the three domains were further classified into three classes. One domain is mapped to one class. Outliers as stated above are classified to a normal class as normal terms. We created three question sets containing a total of 544 question terms (see appendix C) which are gathered from three different sources, including 268 terms randomly selected from China Times, 176 terms randomly selected from Economic Daily, and 100 terms from the top 100 popular search queries of Yam.com in A question set came from a source. Table 1 lists a part of the question terms accompanied by their English translations. A total of 80,719 Web pages are extracted. We select answers by dynamic way given different γ values. Table 2 shows the performance, including precision, recall, answer size and value of F-measure where answer size is an average number of answer entries (i.e., number of groups) in an answer set. Since amount of pages in the Web is unknown, recall can not be calculated by its usual way. We defined it in this study as number of definition briefs collected in a final answer set over amount of definition briefs in the candidate sets generated by semantics-selection phase. Experimental results show that the 18

26 best precision occurs at γ = 5%, but the corresponding answer size is too small, that is, answer sizes of many question terms are zeros. While γ decreases, both answer size and recall increase. F-measure values have similar trend. That is, the best case occurs at γ = 2%, generating 4.69 answer entries. Therefore, we use γ = 2% as a default configuration of DefExplorer. Table 2 Performance of dynamic way on different γ s γ Answer size Precision Recall F-measure 6% % 28.62% % % 44.10% % % 48.20% % % 51.10% % % 59.30% % % 67.42% On the other hand, in the experiment, we found that only 34.0% (185/544) of question terms collected were classified into one of the three specific domains/classes (recall, others are common terms). Table 3 shows performance of the candidate sets, in which precisions of domain candidate sets are not significantly different from those of the common candidate set on different γ s. Besides, the answer sizes (each is an average of ten-fold values) of domain candidate sets are stable. The reason is that answers selected from domain candidate sets are usually explicit, and most groups of a final answer set have more than 5% of the amount of definition briefs in their original candidate sets. Therefore, changing γ from 2% to 5% has very little influence on the numbers of selected results. Table 3 Performance of each candidate set on different γ s Common candidate set Domain candidate sets All answers γ Ans. size Precision Ans. size Precision Ans. size Precision 5% % % % 4% % % % 19

27 3% % % % 2% % % % However, we also found that number of answers selected from domain candidate sets various significantly for different question terms. Some may obtain several answer entries, while others retrieve no answer even its average answer sizes are stable on different γ s. For instance, 深 圳 (Shenzhen) obtained 15 answers (10 are valid) from domain candidate sets, and 4 answers (all are valid) from the common candidate set. However, given 楊 振 寧 (Chen Ning Yang), a Nobel laureate, DefExplorer obtained no answer from domain candidate sets, but 16 answers (9 are valid) were retrieved from the common candidate set. The reason is as follows. Domain candidate sets often consist of specific features, e.g., birthday and past occupations. Hence, they have unavoidable limitations. For instance, a writer usually does not publish his/her past occupation. People are more interested in his/her writings, therefore, we should collect 著 有 (ever wrote) as a domain sentence pattern. But, writers are minority in the world. To collect sentence patterns for rarely seen domain is difficult, particularly when some specific domains of a given query target are always absent from the Web, e.g., collecting ever wrote for an person who is not writer or collecting occupation for a writer. On the other hand, a common candidate set can provide answers from various aspects, since people give a concept or an object a definition very often from their own viewpoints. The ADSL descriptions mentioned above are an example. Besides, this experiment spent a total of 446 minutes (about 7.5 hours), including Internet access time. 4.3 Performance of Static Way In the third experiment, we generated answer set by static way which selects n (i.e., λ ) biggest groups from all candidate sets. Since the average answer size at γ = 2% is 4.69, 20

28 we give n = 5 ( 4.69), and measure top-n inclusion rates as a metric [1]. Given a question set, its top-n inclusion rate is defined as percentage of question terms whose accurate definitions could be found in the top-n of extracted answer entries collected in their final answer sets. Table 4 shows experimental results of the three resources involved. The worst case occurred at Yam.com s popular search queries. Further, DefExplorer did not work well on general terms. Several general terms do not obtain their accurate definitions, e.g., 成 人 (adult) (see appendix D which lists examples of search results by deploying DefExplorer only, no lexicons) and 好 玩 遊 戲 區 (interesting games). The latter even generated no answer. Table 4 Top-n inclusion rates by using DefExplorer only, given the three question sets Question Set Top-n inclusion rates Top-1 Top-2 Top-3 Top-4 Top-5 (A) China Times 53.0% 65.3% 71.4% 73.4% 79.6% (B) Economic Daily 58.9% 71.4% 76.8% 76.8% 82.1% (C) Yam queries 32.3% 43.4% 48.5% 53.5% 58.6% Among the three question sets, Economic Daily, which mainly contains proper nouns, such as LCD, 新 台 幣 (New Taiwan Dollars) and 標 準 普 爾 (Standard & Poor s), outperformed the other two. The results imply that our approach can more accurately extract term definitions for non-general terms, including proper nouns and domain specific terminologies. 4.4 Integrating with Existing Lexicon Although DefExplorer did not work well on general terms, if those terms that their definitions can not be found in the Web can be found in existing lexicons or encyclopedias, performance can be then improved. In the fourth experiment, DefExplorer was inte- 21

29 grated with an existing Chinese lexicon [22]. Table 5 and Figure 7 show the experimental results. Before the integration, only 38.6% of question terms in Economic Daily question set could find their definitions in an existing dictionary. But after the integration, 96.5% of question terms could obtain their definitions either in the given dictionary or in extracted top-5 answers (those excluded are shown in appendix B with question marks). Now, we can conclude that our approach is helpful in solving the out-of-vocabulary problem. Table 5 Top-n inclusion rates when DefExplorer is integrated with an existing lexicon. Those that can be found in a given lexicon are listed in appendix B with * signs Question Set Dict Dict+ Top-1 Dict+ Top-2 Dict+ Top-3 Dict+ Top-4 Dict+ Top-5 (A) China Times 69.4% 87.8% 89.8% 91.8% 93.9% 95.9% (B) Economic Daily 38.6% 73.7% 86.0% 91.2% 91.2% 96.5% (C) Yam queries 30.3% 53.5% 59.6% 64.7% 69.7% 74.8% 1 China Times Economic Times Yam queries China Times (Dict) Economic Times (Dict) Yam queries (Dict) Inclusion rate Dict Top-1 Top-2 Top-3 Top-4 Top-5 Top-n Figure 7 Top-n inclusion rates (combining Table 4 and 5) 22

30 4.5 Performance of Sorting on Group size and TF-IDF Score In fifth experiment, we use static way to select answers using two different sorted bases, group sizes and TF-IDF scores. The question set is Economic Daily. Tables 6 and 7 respectively show that sorting on TF-IDF scores can obtain better inclusion rate and precision than those of sorting on group sizes. The reason is that correct answers always have higher TF-IDF scores, which of course are ranked higher. Table 6 Top-n inclusion rates by using two sorting bases, group size and TF-IDF score Sorted base Top-n inclusion rates Top-1 Top-2 Top-3 Top-4 Top-5 Group size 58.9% 71.4% 76.8% 76.8% 82.1% TF-IDF score 64.3% 75.0% 85.7% 89.3% 89.3% Table 7 Top-n precision by using two sorting bases, group size and TF-IDF score Sorted base Top-n precision Top-1 Top-2 Top-3 Top-4 Top-5 Group size 58.9% 52.7% 49.1% 45.1% 43.7% TF-IDF score 64.3% 58.0% 58.0% 55.8% 54.4% 4.6 Comparison with Other Systems In the sixth experiment, we compared DefExplorer with three state-of-art systems, Probabilistic Model [10], DefScriber [9] and Google. Several modifications are applied to the first two systems so as to give a more fair comparison. The first is that the two systems are originally designed to extract definitions for English terms. We modify their grammar patterns and part-of-speech dictionary to make them able to process Chinese data. The second is that the two systems originally extract definitions mainly from some reliable 23

31 sources, such as encyclopedias. However, we are only interested in their performance on extracting term definitions from the Web. Hence, Web corpus is the only data source in the experiment. We compared the four systems by searching question terms for Traditional Chinese pages and checking to see whether definitions are shown in the first five searching results or not. The question set is Economic Daily. Figures 8 and 9 show that DefExplorer performs the best both in inclusion rates and precisions. The answers generated by the Probabilistic Model and DefScriber contains lots of noises which, although all in forms of definition patterns, are not real definitions, e.g., ipod 是 浴 缸 的 弟 弟 (ipod is bathtub s brother), which is not a definition on ipod, but is extracted from a joke [23]. The two systems perform better on their own welldefined knowledge corpora than on the Web. In fact, Han et al. in their research also shows that their model performs the best when encyclopedias are the extracting targets, and performs the worst when the target is the Web [10]. Web corpus contains many noises such as jokes and buzzes. DefExplorer can effectively filter them out. The two figures also show that Google s search results at top-1 inclusion rate (23.9%) are higher than those of the two systems, excluding DefExplorer. We found that most correct answers appearing in first search results are pages in Wikipedia. The reason can be conjectured as that most pages in Wikipedia are cited very often so that they have very high PageRank when Google s ranking algorithm is deployed, rather than Google employing optimized algorithms to rank its extracted term definitions. Figure 8 Comparison of Top-n inclusion rates 24

32 DefExplorer (Group size) Probabilistic model Google DefExplorer (TF-IDF score) DefScriber 1 Inclusion rate Top-1 Top-2 Top-3 Top-4 Top-5 Top-n Figure 8. Comparison of Top-n inclusion rates % 80.00% DefExplorer (Group size) Probabilistic model Google DefExplorer (TF-IDF score) DefScriber Precision 60.00% 40.00% 20.00% 0.00% Top-1 Top-2 Top-3 Top-4 Top-5 Top-n Figure 9 Comparison of Top-n precision 25

33 Chapter 5 Conclusions and Future Research In this article, we proposed a system, DefExplorer, which automatically extracts term definitions from the Web, and explained how the system filters out noisy information and how it selects valid definitions. We designed two types of candidate sets, common and domain, and grouped candidates. Finally, answers from candidate sets are evaluated and selected by using different approaches. The experimental results show that the system can extract definitions for out-of-vocabulary terms effectively. We also found that the two candidate sets have their own forte. Common candidate sets provide overall answers, while domain candidate sets support specific features. In the future, we would like to study how to improve the classification rate of question terms. Since in this study, only 34.0% of terms can be classified into one of the concerned domains. At least two ways can improve this situation. The first is to recover abbreviations, like 傳 媒 傳 播 媒 體 and 事 發 事 情 發 生. However, no abbreviation dictionary is currently available. The second is to increase number of domains. In fact, how to collect many more effective domain definition sentence patterns for more domains is an interesting issue. Semantic primitives collected in and analyzed by [18] provide a excellent direction. Besides, the way to discriminate different meanings and definitions of the same terms is a challenge, e.g., the term 藍 光 has two meanings, blue ray and Bluray disc. Both were collected and selected in an answer set in current version of DefExplorer. Those constitute our future research topics. 26

34 References [1] W.H. Lu, L.F. Chien and H.J. Lee, Anchor Text Mining for Translation of Web Queries: A Transitive Translation Approach, ACM Transactions on Information Systems, vol. 22, no. 2, 2004, pp [2] Y. Zhang and P. Vines, Using the Web for Automated Translation Extraction in Cross- Language Information Retrieval, the International ACM SIGIR Conference on Research and Development in Information Retrieval, 2004, pp [3] E. M. Voorhees, Overview of the TREC 2003 Question Answering Track, the Text Retrieval Conference, 2003, pp [4] E. M. Voorhees, Overview of the TREC 2004 Question Answering Track, the Text Retrieval Conference, [5] A. Renouf, Explorations in Corpus Linguistics, TW&lr=&id=HCJL8FFKQAoC&oi=fnd&pg=PA269&dq=Agglomerative+Nesting+(AGNES)+Algorithm&ots=9CCso6z3_R&sig=bmjLWliX7JeYY9L3T0SKzv 7uWFM#PPA104,M1 [6] L. Ru, Z. Tong, Y. Liu and S. Ma (PRC), Automatic Chinese Name Recognition based on Web Corpus Analysis, ACTA press, 27

35 [7] J.L. Zhang, L. Sun and J.M. Min, Using the Web Corpus to Translate the Queries in Crosslingual Information Retrieval, IEEE International Conference on Natural Language Processing and Knowledge Engineering, 2005, pp [8] J. Prager, J. Chu-Carroll, K. Czuba, C. Welty, A. Ittycheriah and R. Mahindru, IBM s PI- QUANT in TREC2003, the Text Retrieval Conference, 2003, pages [9] S. Blair-Goldensohn, K.R. McKeown and A.H. Schlaikjer, A Hybrid Approach for QA Track Definitional Questions, the Text Retrieval Conference, 2003, pp [10] K.S. Han, Y.I. Song and H.C. Rim., Probabilistic Model for Definitional Question Answering, the International ACM SIGIR Conference on Research and Development on Information Retrieval, 2006, pp [11] W. Lam, P.S. Cheung and R.Z. Huang, Mining Events and New Name Translations from Online Daily News, Joint Conference on Digital Libraries, 2004, pp [12] Y.R. Chao, A Grammar of Spoken Chinese, Chinese edition. The Chinese University of Honk Kong, 1968 (translated by P.H. Ting, 2002). [13] S. Yu, X. Zhu, H. Wang and Y. Zhang, The Grammatical Knowledge-base of Contemporary Chinese a complete specification, Beijing Tsinghua University Press, [14] Wikipeida, TF-IDF, 28

36 [15] Z.G. Zhang, Han yu yu fa chang shi, Joint Publishing, Hong Kong, [16] Y.H. Tseng, Automatic Thesaurus Generation for Chinese Documents, Journal of The American Society for Information Science and Technology, vol. 53, 2002, pp [17] 繁 簡 分 歧 詞 表 [18] Z.D. Dong and Q.A. Dong, HowNet, [19] T.H. Cormen, C.E. Leiserson, R.L. Rivest and C. Stein, Introduction to Algorithms, 2 nd ed., the MIT Press, [20] R.S. Pressman, Software Engineering: A Practitioner's Approach, 6 th e.d., McGraw Hill, [21] I. Sommerville, Software Engineering, 8th Edition, Addison Wesley, [22] Ministry of Education, R.O.C., Chinese Dictionary. [23] [17] K.C. Chow, The Emergence of the Copula Shi from the Perspective of Information Structure, Humanitas Taiwanica, [18] Y.C. Sung, The Research on Improving the Performance of Information Retrieval with the Agglomerative Nesting (AGNES) Algorithm using a Chinese news dataset,

37 30

38 Appendix A Definition Sentence Patterns A.1 Common Definition Sentence Patterns (1) X 是 指 (2) X 意 為 (3) X 為 (4) X 是 excluding X 是 不 是, X 是 否 A.2 Domain Definition Sentence Patterns for person, location and organization (1) Person œ X 生 於 [date / location], X 出 生 於 [date / location] œ X 卒 於 [date / location], X 亡 於 [date / location] œ X 曾 任 [jobs], X 任 職 於 [jobs] (2) Location œ X 位 於 [place], X 坐 落 於 [place] œ X 建 於 [date] (3) Organization œ X 成 立 於 [date] œ X 位 於 [place] 31

39 Appendix B Examples of an Extracting Process The following lists results of each step when DefExplorer is used given a query term 馬 英 九 (Ma, Ying-Jeou). Steps Queries Generated Outputs 馬 英 九 是 一 個 Type of the question term: 馬 英 九 是 一 位 person 馬 英 九 是 一 本 Question Analysis Document Retrieval Semantics Selection Similarity Scoring Candidate Grouping Answer Generation Common Query Set: 馬 英 九 是 馬 英 九 為 馬 英 九 是 指 Domain Query Set: 馬 英 九 出 生 於 馬 英 九 曾 任 Total 1,552 candidates in candidate set: 馬 英 九 是 眼 睛 瞎 了, 看 不 到 擺 在 眼 前 的 事 實? 主 持 人 說 網 路 上 都 形 容 馬 英 九 是 馬 囧, 馬 英 九 說, 囧 的 原 意 並 非 困 窘 而 是 光 明 馬 英 九 出 生 於 1950 年 7 月 13 日, 籍 貫 湖 南 衡 山 鄭 安 國 表 示, 馬 英 九 出 生 于 香 港, 對 台 港 關 系 非 常 重 視, 過 去 曾 三 次 訪 問 香 港... - Total 283 definition briefs: Common candidate set: 馬 囧 最 大 在 野 黨 主 席 最 大 在 野 黨 的 主 席 Target-birthday domain candidate set: 出 生 於 1950 年 7 月 13 日 Target-place-of-born domain candidate set: 出 生 于 香 港 - Similarity Score: 最 大 / 在 野 黨 / 主 席 ; 最 大 / 在 野 黨 / 的 / 主 席 Similarity score = 0.75 最 大 / 在 野 黨 / 的 / 主 席 ; 國 民 黨 / 的 / 希 望 Similarity score = Total 22 groups: 最 大 在 野 黨 主 席 (2) => 最 大 在 野 黨 主 席 / 最 大 在 野 黨 的 主 席 一 個 法 律 人 (2) => 一 個 法 律 人 / 一 個 學 法 的 人 - Answer Set: 最 大 在 野 黨 主 席 一 個 法 律 人 32

40 Steps Queries Generated Outputs 藍 光 是 一 個 Type of the question term: 藍 光 是 一 位 normal 藍 光 是 一 本 Question Analysis Document Retrieval Semantics Selection Similarity Scoring Candidate Grouping Answer Generation Common Query Set: 藍 光 是 藍 光 為 藍 光 是 指 Total 806 candidates in candidate set: 我 們 看 晴 天 天 空 是 藍 色 的, 即 因 為 藍 光 是 陽 光 在 空 氣 中 的 主 要 散 射 成 分 藍 光 是 索 尼 公 司 開 發 的 新 一 代 DVD 光 盤 技 術 標 准, 不 但 可 應 用 於 錄 制 擦 寫 或 播 放 HD 高 清 影 像, 同 時 也 可 應 用 於 存 儲 容 量 更 為 巨 大 的 數 字 內 容 藍 光 是 新 力 所 開 發 的 下 一 代 DVD 光 碟 技 術, 以 藍 光 雷 射 讀 取 光 碟 片 資 料, 每 片 光 碟 儲 存 容 量 可 增 加 到 20GB, 較 現 有 的 DVD 每 片 8.7GB 多 了 一 倍 餘... - Total 234 definition briefs: Common candidate set: 陽 光 在 空 氣 中 的 主 要 散 射 成 分 索 尼 公 司 開 發 的 新 一 代 DVD 光 盤 技 術 標 準 新 力 所 開 發 的 下 一 代 DVD 光 碟 技 術 - Similarity Score: 新 力 / 公 司 / 開 發 / 的 / 新 / 一 代 /DVD/ 光 碟 / 技 術 / 標 準 陽 光 / 在 / 空 氣 / 中 / 的 / 主 要 / 散 射 / 成 分 Similarity score = 0.10 新 力 / 公 司 / 開 發 / 的 / 新 / 一 代 /DVD/ 光 碟 / 技 術 / 標 準 新 力 / 所 / 開 發 / 的 / 下 / 一 代 /DVD/ 光 碟 / 技 術 Similarity score = Total 12 groups: 新 力 公 司 開 發 的 新 一 代 DVD 光 碟 技 術 標 準 (2) => 新 力 公 司 開 發 的 新 一 代 DVD 光 碟 技 術 標 準 / 新 力 所 開 發 的 下 一 代 DVD 光 碟 技 術 陽 光 在 空 氣 中 的 主 要 散 射 成 分 (3) => 陽 光 在 空 氣 中 的 主 要 散 射 成 分 / 陽 光 散 射 在 空 氣 中 的 主 要 成 分 / 陽 光 在 空 氣 中 散 射 的 主 要 成 分 - Answer Set: 新 力 公 司 開 發 的 新 一 代 DVD 光 碟 技 術 標 準 陽 光 在 空 氣 中 的 主 要 散 射 成 分 33

41 Appendix C Question Term Sets for Experiments A total of 544 question terms are involved, in which 268, 176 and 100 question terms come from China Times, Economic Daily and top 100 popular search queries of Yam.com in Terms with * signs are those that can be found in the given lexicon [22], and terms with? signs represent that no correct answer is found within answers ranked top-5 in DefExplorer s answer sets. Top-n inclusion rates of the answers as involving DefExplorer only are shown in Table 4. Those when both the lexicon and DefExplorer are involved are listed in Table 5. C.1 China Times (268 question terms) 3.5G 一 夜 情 一 國 兩 制 * 九 一 一 * 九 份 * 二 氧 化 碳 * 人 大 * 十 六 大 上 網 *? 大 蒜 * 女 婿 *? 小 泉 純 一 郎 中 油 * 中 芯 中 秋 節 * 中 國 大 陸 * 中 常 委 * 中 常 會 * 中 選 會 * 互 聯 網 元 宵 * 公 民 * 公 投 公 視 * 公 訴 * 公 路 *? 化 糞 池 * 反 分 裂 法 反 作 用 力 * 反 恐 反 聖 嬰 現 象 天 皇 * 巴 勒 斯 坦 * 文 宣 *? 方 便 麵 * 火 鍋 * 牛 肉 麵? 主 席 * 主 播 * 冬 瓜 * 包 道 格 台 海 四 川 省 * 外 資 *? 失 禁 *? 奶 茶 * 布 希 * 布 施 * 平 壤 * 民 調 瓦 斯 * 田 徑 * 目 的 *? 石 家 莊 立 法 會 交 換 學 生 共 伴 效 應? 印 度 尼 西 亞 * 地 震 *? 年 底 *? 成 都 市 * 成 龍 * 死 刑 * 江 澤 民 老 鼠 會 * 吳 郭 魚 * 志 工 * 抗 戰 * 投 票 率 李 遠 哲 * 乳 球 事 發 *? 咖 啡 * 和 解 *? 周 杰 倫 周 星 馳 坦 言 *? 季 風 * 季 線 定 位 拖 牌 招 牌 * 果 凍 *? 林 信 義 林 務 局 板 塊 * 泛 民 主 派 泛 綠 泛 藍 的 士 * 直 升 機 * 直 播 * 矽 膠 社 會 局 * 社 團 *? 虱 目 魚 * 阿 里 山 阿 妹 * 信 用 卡 * 保 守 黨 俄 羅 斯 * 姚 明 後 遺 症 * 指 紋 * 星 球 * 查 處 *? 毒 樹 果 理 論? 洪 災 * 流 感 炸 彈 客? 相 聲 * 34

42 紅 包 * 紅 絲 帶 紅 襪 隊 背 包 客 胡 錦 濤 軍 售 *? 迪 士 尼 音 響 *? 飛 彈 * 香 油 錢 * 香 港 * 倒 扁 * 修 憲 * 冥 王 星 * 原 住 民 * 哥 斯 大 黎 加 * 座 談 會 *? 恐 怖 分 子 * 核 武 核 電 廠 * 消 保 官 消 費 者 * 海 王 星 * 海 協 會 * 海 軍 * 海 基 會 * 真 調 會 紙 尿 褲 * 紙 鶴 草 案 *? 記 者 會 記 錄 片? 訊 問 * 貢 丸 * 退 休 金 * 退 輔 會 酒 駕 配 票 * 院 會 * 除 夕 * 達 馬 松? 馬 英 九 高 齡 化 * 停 車 費 *? 假 帳? 偷 渡 * 曼 聯 * 基 本 法 * 基 督 徒? 基 督 教 * 專 機 *? 得 票 率 * 情 人 節 * 捷 運 * 教 宗 * 球 證 * 產 學 合 作 眷 村 * 移 民 * 統 一 戰 線 * 軟 件 * 陳 水 扁 * 凱 達 格 蘭 * 單 車 * 寒 流 * 富 邦 惠 普 替 代 役 殘 奧 會 湯 曜 明 無 盟 菩 提 * 菸 草 * 華 人 * 華 府 * 華 僑 * 著 作 權 * 詐 欺 * 超 貸 *? 跆 拳 道 * 進 行 式? 郵 輪 *? 間 諜 * 黃 金 周 傳 媒 *? 奧 組 委 奧 運 * 新 竹 米 粉 新 華 社 * 新 疆 * 楊 振 寧 * 溫 室 效 應 * 溫 家 寶 準 備 程 序 庭 罪 犯 * 聖 戰 * 聖 嬰 現 象 * 董 事 長 * 解 聘 * 資 本 主 義 * 跳 票 * 農 藥 * 遊 民 * 電 話 門 事 件 電 影 *? 預 算 案? 歌 仔 戲 * 漫 畫 家 碩 士 * 管 委 會 精 神 病 * 綠 卡 * 網 民 網 址 * 網 頁 * 網 站 * 罰 球 * 認 同 * 餃 子 *? 廟 會 * 數 獨 歐 尼 爾 * 歐 盟 緯 來 蔡 依 林 輪 迴 * 養 生 *? 養 殖 業 魯 迅 * 魯 爾 * 學 歷 *? 導 演 * 導 彈 * 融 資 *? 親 民 黨 選 舉 人 票 * 彌 功 檢 察 院 * 總 編 * 聯 合 國 * 輿 論 * 鎂 光 燈 * 隱 私 * 鮮 奶 * 鮮 乳 * 瀉 藥 * 藍 軍 覆 議 案 證 據 *? 蘇 貞 昌 議 會 * 霰 * 鐵 人 三 項 * 鐵 馬 * 鐵 鍊 *? 體 味 * 緋 聞 *? 貔 貅 * 35