(1) :,, (2) :,, logo,,,,,,,,, :,,,,, 2, :,,,,,,,,,,, :, ;,, 2. 1,, [ 1 ], : (1 ) ( Two2Dimensional Logical Structure),,, (2) ( Visual Layout P

Size: px

Start display at page:

Download "(1) :,, (2) :,, logo,,,,,,,,, :,,,,, 2, :,,,,,,,,,,, :, ;,, 2. 1,, [ 1 ], : (1 ) ( Two2Dimensional Logical Structure),,, (2) ( Visual Layout P"

诉石
7 years ago
Views:

1 J OU RNAL OF CH IN ESE IN FORMA TION PROCESSIN G Vol. 22, No. 1 Jan., 2008 : (2008) ,, (, ) :,,,,,90 %,85 %, 5, 15 ; 5, : ; ; ;; ; ; : TP391 : A A WebPage Content Block Detection Method Based on Layout Features and Language Features HAN Xian2pei, L IU Kang, ZHAO J un (National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing , China) Abstract : This paper analyzed the different feature types of webpage blocks, and presented a webpage content block detection method based on layout features and language features, which effectively resolved the seesaw problem between detection accuracy and model generality across different types of webpages. The method used the vision2 block tree to represent webpage, built two individual classifiers respectively for webpage s layout features and language features, and used different strategies to combine these two classifiers. The experimental result s show that, with holding the content block detection recall higher than 90 %, the combined classifiers accuracy can reach 85 percents, 5 percents higher than the classifier using only the layout features, and 15 percents higher than the classifier using only the language features ; and the experimental result s also show that the combined classifiers obtained good detection performance over five selected websites which means that it have good generality. Key words : computer application ; Chinese information processing ; webpage cleaning ; content block detection ; webpage segment ; layout feature ; language feature 1,,,,, H TML,,,, : : : : ( ) ; ( , ) : (1984 ),,, ; (1981 ),,, ; (1966 ),,,

2 (1) :,, (2) :,, logo,,,,,,,,, :,,,,, 2, :,,,,,,,,,,, :, ;,, 2. 1,, [ 1 ], : (1 ) ( Two2Dimensional Logical Structure),,, (2) ( Visual Layout Pre2 sentation), H TML,,,,,,, H TML, DOM Tree, DOM Tree,,,, DOM Tree, DOM Tree V IPs [2 ],,, 2. 2,, : (1) : Google PageRank,,,,,,

3 1 : 17 PageRank, (2) : Sina logo Sina, Sina ;Lan Yi [3 ]Site Style Tree DOM Tree,,, (3) :,, (4) :,,, (1) (2),, ; (3) ; (4),,,,,, 3,,Ziv Bar2Yossef [ 4 ],, pagelet, pagelet, pagelet, Lan Yi [3 ] DOM Tree,, Site Styled Tree DOM, DOM,,,,,Suhit Gupta [5 ],, DOM Tree,, DOM Tree,, Deng Cai [ 2 ], V IPs,,,,,Deepayan CHakrabarti [ 6 ],,,,, : (1) (2), 4,,, 2. 2,,,,,, 4. 1 [2 ]

4 中文信息学报 18 过文献 [ 2 ] 中的网页切分算法即 V IPs 算法, 将网页表示成一个树的结构 WebPage = {VB1, VB2,, VBn},其中 VBi 为网页最上层的视觉块同时每个视觉块 VBi 本身也具有递归的层次结构, VBi = {VBi_1, VBi_2,, VBi_ m}, 其中 VBi_ k 为 VBi 图年的子块,依次递归直到视觉块满足不再切分的条件举例说明,如图 1 ( a ) 表示的网页, 可通过文献 [2 ] 中的方法最终表示为图 1 ( c) 中的树结构 Web Page = { VB1, VB2, VB3, VB4 }, 而其中的 VB1 = {VB1_1, VB1_2},其他的块以此类推网页切分及其视觉块树表示举例联合使用布局特征与语言特征的网页视觉块分类器分类的队列中当分类结束时, 就得到了网页的所有主要内容块其中, 原子块指的是这个块内部的在将网页表示成一个网页视觉块树后, 主要内容块发现就变成了网页视觉块的分类问题分类从网页视觉块树的根部开始, 用一个队列存储需要分类的块,首先将根节点加入队列中,分类直到队列中结构已经足够的一致, 其包含的内容要么完全是主要内容,要么完全是噪音信息复合块指的是块内部结构还不够一致, 不能认定其包含的内容完全是主要内容或完全是噪音信息根据第 3 节中的论述, 要对网页视觉块进行分没有需要分类的块如果一个块被分为噪音块, 即这个块的内容是非主要内容,则这个块被丢弃 ; 如果一个块被分为非噪音块, 则通过一个准则判断这个块是否是原子块,如果是一个原子块,则将这个块当成一个主要内容块取出 ; 如果不是一个原子块,则这类,分类器需要尽可能多的使用网页视觉块的特征以达到高准确率,同时使用的特征不针对特定站点, 使得分类器具有通用性本文的分类器同时使用了布局特征和语言特征,都不针对特定站点对多类特征建模,有两种方法,一种是在一个分个块是一个复合块, 将其所有的下一级子节点加入类器中使用所有的特征, 另一种方式是对每一种特

5 1 : 19,, ;,,, 1 14, 5. 1,TREC blog06, 500,, ,1 836, %, 30 %,: (1) : 4. 2, V IPs PDoc ( ) form p Unigram,,,, : (1) k,, k = 1,, (1) : P( Ci) = arg max i k P k ( Ci) (1) k, (2),, 5,, (2) : 4. 2 Unigram, (3) 1 :,, 0. 7, 0. 3 (4) 2 :,,,, % % % % % % % % 1,, 2, 0. 7, 0. 3,,

6 : 2,,, 90 %;, 5 % 15 %,, : (1), (2) 5. 2, TREC blog06, 5 15, U RL 3,4 1, U RL 1 northfield. org mdcbowen. org gutrumbles. com asktaxmoms. com scotsman. com % % % % % % % % % % % % % % % % % % % %

7 1 : ,,, 4 5,, 4, 4, ;5, 5,,,,, : (1),,, (2),,, 6,,,,,,,,, : (1) : [2 ],,,, (2) :,, :,, ;,, : [1 ] Rupesh R. Mehta, Harish Karnick, and Pabitra Mitra. Semantic Structure Analysis of Web Document s. Digital Document Processing [ M ], Springer [2 ] Deng Cai, Shipeng Yu, Ji2Rong Wen and Wei Ying Ma. V IPS : A Vision based Page Segmentation Algorithm [ R]. MSR2TR [3 ] Lan Yi, Bing Liu, Xiaoli Li. Eliminating Noisy Information in Web Pages for Data Mining [ A ]. The Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining [ C] [4 ] Ziv Bar2Yossef, Sridhar Rajagopalan. Template Detection via Data Mining and it s Applications [ A ]. The eleventh international world wide web conference [ C] [5 ] Suhit Gupta, Gail Kaiser, David Neistadt, Peter Grimm. DOM2based Content Extraction of H TML Document s [ A ]. The Twelfth International World Wide Web Conference [ C] [6 ] Deepayan CHakrabarti, Ravi Kumar, Kunal Punera. Page2level Template Detection via Isotonic Smoothing [ A ]. The 16 th International World Wide Web Conference [ C] [7 ] Sandip Debnath, Prasenjit Mitra, C. Lee Giles. Automatic Extraction of Informative Blocks f rom Webpages [ A ] ACM Symposium on Applied Computing [ C] [8 ] Ruihua Song, Haifeng Liu, Ji2Rong Wen, Wei2Ying Ma. Learning Block Importance Models for Web Pages [ A ]. 13th International WWW Conference [ C] [9 ] Shian2Hua Lin, Jan2Ming Ho. Discovering Informative Content Blocks from Web Document [ A ]. The Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining [ C] [10 ] Rupesh R. Mehta, Pabitra Mitra, Harish Karnick. Extracting Semantic Structure of Web Document s Using Content and Visual Information [ A ]. 13th International WWW Conference [ C] [11 ] David Gibson, Kunal Punera, Andrew Tomkins. The Volume and Evolution of Web Page Templates [ A ]. 13th International WWW Conference [ C]

<4D F736F F D20BAABCFC8C5E02DCBD5D6DDBBE1D2E92E646F63>

<4D F736F F D20BAABCFC8C5E02DCBD5D6DDBBE1D2E92E646F63> 基于布局特征与语言特征的网页主要内容块发现韩先培, 刘康, 赵军 ( 中国科学院自动化研究所模式识别国家重点实验室, 北京 100080) 摘要 : 本文综合分析了网页内容块各方面的特征, 提出了一个联合使用布局特征和语言特征的网页主要内容块发现方法, 有效地解决了以往模型中通用性与高准确率不能共存的缺点该方法使用网页视觉块树表示网页, 对网页内容块的布局特征和语言特征分别建立了独立的分类器,