PowerPoint Presentation

Size: px

Start display at page:

Download "PowerPoint Presentation"

役历
5 years ago
Views:

1 Knowledge-based Information Extraction in Taobao 豪文 ( 刘健 ) 阿里集团 -Etao- 搜索算法

2 Global View of IE Our team s work focus here nonstructural Free Text Image NER Event recognition Relation recognition Information Extraction Structural or Semistructural Video XML Web Term extraction 属性抽取表格抽取正文抽取区域发现评论抽取

3 Knowledge-based IE Information Standardization Information source Translator Tag Network Free Text Image Video Title Article Comment Computerreadable knowledgebase Why? Easy to process for computers Easy to understand for human beings

4 Knowledgebase: some related concepts

5 Semantic Spectrum Some steps in the semantic spectrum include the following: glossary: A simple list of terms and their definitions. A glossary focuses on creating a complete list of the terminology of domain-specific terms and acronyms. It is useful for creating clear and unambiguous definitions for terms and because it can be created with simple word processing tools, few technical tools are necessary. controlled vocabulary: A simple list of terms, definitions and naming conventions. A controlled vocabulary frequently has some type of oversight process associated with adding or removing data element definitions to ensure consistency. Terms are often defined in relationship to each other. data dictionary: Terms, definitions, naming conventions and one or more representations of the data elements in a computer system. Data dictionaries often define data types, validation checks such as enumerated values and the formal definitions of each of the enumerated values. data model: Terms, definitions, naming conventions, representations and one or more representations of the data elements as well as the beginning of specification of the relationships between data elements including abstractions and containers. taxonomy: A complete data model in an inheritance hierarchy where all data elements inherit their behaviors from a single "super data element". The difference between a data model and a formal taxonomy is the arrangement of data elements into a formal tree structure where each element in the tree is a formally defined concept with associated properties. ontology: A complete, machine-readable specification of a conceptualization using URIs (and then IRIs) for all data elements, properties and relationship types. The W3C standard language for representing ontologies is the Web Ontology Language (OWL). Ontologies frequently contain formal business rules formed in discrete logic statements that relate data elements to each another.

6 Knowledgebase: Industry Work Wolfram Alpha Google s knowledge graph Baidu s knowledge graph Taobao s work

7 Wolfram alpha

8 Stephen Wolfram 在 2009 年宣布了 Wolfram Alpha 的计划, 它的第一版是用约五百万行的 Mathematica 代码编写的, 并且在个中央处理器上运行 Siri 背后的知识搜索引擎就是 Wolfram Alpha, 目前 Wolfram Alpha 处理的数据流量中有 25% 都是由 Siri 带来的

9 Google s work metaweb

10 Baidu s work

11 Our work

12 Knowledgebase: from scratch Training corpus Keyword extraction Domainrelated Terms Reduced Terms 属性预测 Tag knowledgebase Definition of entities Relation of entities Indicator of entities

13 Tagging In view of IE: Information Extraction with knowledgebase Information Standardization Information source Translator Tag Network Computerreadable knowledgebase

14 Tagging:example Item id: Title: 2012 正品时尚复古蝙蝠袖毛衣开衫针织披肩春秋女式宽松外套女季节 : 春秋风格 : 复古人气热点 : 蝙蝠衫 Tag: 针织 Tag: 披肩 Tag: 宽松 Tag: 时尚产品 : 外套产品 : 开衫产品 : 毛衣

15 Derivative Work Based on tags, we can develop lots of data product 用 tag 表征 query 用 tag 表征用户偏好 query Tags for query Tags for User Prefer. Tag 化的用户需求商品 Product Fashion Info. 潮流资讯用 tag 描述商品 Tags for Product Tags for Fashion Info. 用 tag 提炼潮流资讯的主题 Tag 化的购物资讯

16 Today s key problems: How to get domain-related terms? How to merge the term with same meaning? How to determine the category of a term? How to generate the tag web (knowledgebase)? How to do IE, with the tag web? 课堂讨论 : 分小组比赛, 谁想的方法多

17 Q1: How to get domain-related terms? Task Description: Given a corpus for a certain domain (sometimes background corpus as well), find out domainrelated terms in the corpus Related Research: Name Entity Recognition Domain-related keyword extraction

18 Name Entity Recognition(NER) Named-entity recognition (NER) (also known as entity identification and entity extraction) is a subtask of information extraction that seeks to locate and classify atomic elements in text into predefined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. From Wikipedia

19 Domain-related keyword extraction Basic idea: Foreground corpus vs. background corpus TF-IDF Problems : New term Term combination

20 Top terms extracted from different level-1 categories

21 Case Study: 给定一篇武侠小说, 如何从中抽取出人名? 有哪些方法? 各方法需要怎样的辅助数据? 如果涉及模型, 特征该怎么设计? 分小组讨论 8min 每个组列举可能的办法看哪组列举的多, 且可行

22 How to merge the term with same meaning? Synonym Recognition Why: Computing Scale ( 减少了重复的 tag 参与后续运算 ) Data concentration( 避免相似的 tag 将数据分流使得统计规律不明显 ) Examples: Chinese name vs. English name 耐克 / Nike Misspelling 牛仔裤 / 牛子裤 T 恤 / T 侐 Near-duplicate terms 棉衣 / 棉服

23 The boundary of the problem: 字面相近的未必就是同义的 : 水晶水晶藻水晶藻泥小西服西服

24 Synonym Reduction: our work Linguistic Clue: Contextual similarity Spelling similarity Brand Reduction( 品牌归一 ) 去年的工作, 将属性值表中实质相同的品牌进行合并数据量, 效果

25 Case Study: 给定某个领域的 term 列表, 如何找出同义词分小组讨论 8min 每个组列举可能的办法, 要用到的语料看哪组列举的多, 且可行

26 Creation of tag: How to predict the genre of a term? Given a term, to tell its genre type term tag Burberry predicting Type: brand T-shirt predicting Type: product Lady Gaga predicting Type: star Apple (in 3C) predicting Type: brand Apple (in food) predicting Type: product

27 Methodology: Pattern Rule XX 裙 -> type: product Topic Classification LDA Bayesian MaxEnt Clustering

28 Our work Global prediction Prediction under level-1 category

29 Case Study: 如何构造预测模型, 使得给定任意一个 term, 都能给出该词属于不同类型 ( 比如 : 品牌风格材质产品类型词 ) 的概率哪些语料可以利用, 该产生怎样的特征, 如何建立模型? 分小组讨论 8min 看哪组列举的多, 且可行

30 How to build the tag web with data-driven? Tag-tag relations: superordinate concept vs. subordinate concept ( 上位概念下位概念 ) 卫衣 -> 上衣 Parallel concepts: 老人手机女士手机儿童手机... 音乐手机电视手机... 苹果手机三星手机小米手机...

31 Linguist Clue: co-occurrence Language pattern: A 包括 x1,x2,..

32 Additional Topic: indicators for tags Hotness 热度选购热点 Favorite from different user group Eg. 不同品牌的人群倾向 Semantic Orientation 语义倾向评论挖掘

33 选购热点

34 How to do IE, with the tag web? Tagging with Explicit Information (Information Extraction) Tagging with Implicit Information 风格 : 甜美场景 : 校园

35 Tagging with Explicit Information Related works: NER Keyphrase extraction Word segmentation HMM CRF

36 Keyphrase extraction KEA: Practical Automatic Keyphrase Extraction To find the important phrases to represent the whole article Bayesian Model with two features: tf-idf, first position of a candidate term

37 Global Optimization Problem: Gram Sequence Find the best sequence(with best score) Tag Sequence Method: HMM CRF

38 Word Segmentation Wrong segmentation, big problem Major Challenges: 分词歧义 : 南京市 / 长江大桥 vs 南京市长 / 江大桥未知词错别字 1 下雨天留客, 天留, 我不留 2 下雨天留客, 天留我不留 3 下雨天, 留客, 天留, 我不留 4 下雨天, 留客, 天留我, 不留 5 下雨天留客, 天留我不? 留! 6 下雨天, 留客天, 留我不留? 7 下雨天, 留客天, 留我? 不留! 8 下雨天留客, 天! 留我不? 留! 9 下雨天, 留客! 天! 留我不留? From 度娘

39 Word Segmentation: Methodology Algorithm used in WS: Rule-based: 简单好解释立竿见影, 不容易扩展易退化 Maximum Match Statistics-based: 数据驱动, 数据容易稀疏 HMM CRF Hybrid:

40 CRF B 词的开始 E 词的结束 I 词的中部 S 单字成词

41 AliWS: Word segmentation in Alibaba * 主要是基于统计的方法 * 统计与规则相结合

42 Basic Architecture 分词技术课程介绍和在淘宝的应用 - 一羽

43 Tagging result

44 Tagging with Implicit Information 通过算法来预测商品的使用场景 Demo: 女装着装场景 ren.php 内网访问

45 算法预测的夜店主题

46 算法预测的海边主题

47 算法预测的约会主题符合当前时令

48 算法预测的宴会年会主题

49 Something about our team

50 淘宝搜索导购 team 的数据产品

51 标签挖掘体系概念级分析 ( 由若干 tag 组成一个概念 ) 关联规则挖掘标签级分析 ( 单个标签 ) 市场热点发现卖家特色发现产出 : * 搭配规则 * 有商业价值的 tag 组合 * 店铺的具体经营特色... 同主题 tag 聚合标签生成 Tag-tag 关系预测人群 -tag 偏好分析 query-tag 关联分析文章主题 -tag 关联分析产出 : * 标签网络 * 标签偏好 * 商品打标 ( 高级标签 )... 新词发现关键词提取同义聚合词性预测产出 : * 商品打标 ( 基础标签 ) * 潮流 tag 分行业语料商品标题商品详情页导购文章用户 query 用户评论

53 Thank you, 亲!

<4D6963726F736F667420506F776572506F696E74202D20C8EDBCFEBCDCB9B9CAA6D1D0D0DEBDB2D7F92E707074>

<4D6963726F736F667420506F776572506F696E74202D20C8EDBCFEBCDCB9B9CAA6D1D0D0DEBDB2D7F92E707074> 软件架构师研修讲座胡协刚软件架构师 UML/RUP 专家 szjinco@public.szptt.net.cn 中国软件架构师网东软培训中心小故事 : 七人分粥当前软件团队的开发现状和面临的问题软件项目的特点解决之道 : 从瀑布模型到迭代模型解决项