基于矩阵分解和矩阵变换的多义词向量研究

Size: px

Start display at page:

Download "基于矩阵分解和矩阵变换的多义词向量研究"

馔昧米
5 years ago
Views:

2 Content 1 PCA VS. RPCA

3 2

4 2

5 3 [1] star s1 star s2 star s3 algorithm s1 algorithm s2 stars, movie, song, MVP stars, award, eagle, two-time supergiant, constellation, aurigae hash, algorithms, quick sort, recursive algorithms, optimization, public-key [1] Neelakantan et al Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space. In Proc. of EMNLP

6 4

7 5

8 5 WordNet [1] / [2] (hypernymy-hyponymy) animal/ country/ cat/ China/ [1] Miller WordNet: a lexical database for English. Communications of the ACM [2]

9 bank financial institution slope net trap ( ) income

10 bank financial institution slope net trap ( ) income score(v w s 1, SynH w i ) = cos(v w s 1, v w s k )ispossiblehypernym(synh w i, v w s 1 ) v w s k NN(v w s 1 )

11 bank financial institution slope net trap ( ) income score(v w s 1, SynH w i ) = cos(v w s 1, v w s k )ispossiblehypernym(synh w i, v w s 1 ) v w s k NN(v w s 1 )

12 7 : P pseudo (v w s 0, v w s 1 ) cos(v s 0, v s 1 ) v s 0 NN(v w s ) v 0 s 1 NN(v w s ) 1 θ θ

13 7 : P pseudo (v w s 0, v w s 1 ) cos(v s 0, v s 1 ) v s 0 NN(v w s ) v 0 s 1 NN(v w s ) 1 θ θ /

14 8 (1) (cat s0 -dog s0, cat s1 -dog s1 ) (2) (> 70%)

15 8 (1) (cat s0 -dog s0, cat s1 -dog s1 ) (2) (> 70%) M = w [ i j (v w i v w j )] v 0 cat v 1 cat v 1 cat v 0 cat..... v 0 cat v 2 cat v 1 cat v 2 cat M

16 9 d L min M L F subject to L + E = M, rank(l) = d

17 9 min M L F subject to L + E = M, rank(l) = d d L

18 9 min M L F subject to L + E = M, rank(l) = d d L (Robust PCA, RPCA) min rank(l) + λ 1 S 0 + λ 2 E F subject to L + E = M, rank(l) = d

19 10 RPCA [1]

20 10 RPCA [1] PCA RPCA M (0) = M, t = 0, S = 0 WHILE PCA L (t) + E (t) = M (t) S (t) = E (t) S = S + S (t) M (t+1) = M (t) S (t) t = t + 1 return L (t), S, E (t) M (t) L (t) [2] [1] Wright et al Robust Principal Component Analysis: Exact Recovery of Corrupted Low-Rank Matrices via Convex Optimization. In Proc. NIPS [2]

21 11 T min L(T) = m i=1 1 2 ( Tvwi s i,0 (vwi s i,0 + v wi 2 s i,1 ) 2 + Tv wi s i,1 (vwi s i,0 + v wi 2 s i,1 ) 2 )

22 11 T min L(T) = m i=1 1 2 ( Tvwi s i,0 (vwi s i,0 + v wi 2 s i,1 ) 2 + Tv wi s i,1 (vwi s i,0 + v wi 2 s i,1 ) 2 ) U R n T { Tα = 0, α U Tα = α α U T

23 11 T min L(T) = m i=1 1 2 ( Tvwi s i,0 (vwi s i,0 + v wi 2 s i,1 ) 2 + Tv wi s i,1 (vwi s i,0 + v wi 2 s i,1 ) 2 ) U R n T { Tα = 0, α U Tα = α α U T T Ṽ = {ṽ w s = Tv w s v w s V}.

24 12 WS-353 SCWS [1] skip-gram (MSSG, 300D) [2] skip-gram (NP-MSSG, 300D) [2] NP-MSSG NP-MSSG NP-MSSG + (PCA) NP-MSSG + (RPCA) [3] MUSE [4] [1] Huang et al Improving word representations via global context and multiple word prototypes. In Proc. of ACL [2] Neelakantan et al Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space. In Proc. of EMNLP [3] Li and Jurafsky Do Multi-Sense Embeddings Improve Natural Language Understanding?. In Proc. of EMNLP [4] Lee and Chen MUSE: Modularizing Unsupervised Sense Embeddings. In Proc. of EMNLP

25 13 SentEval [1] (SUBJ), (TREC) (MSRP) BoW SUBJ TREC MSRP (NP-MSSG) [2] (RPCA) [1] Conneau and Kiela SentEval: An Evaluation Toolkit for Universal Sentence Representations. In Proc. of LREC [2] Neelakantan et al Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space. In Proc. of EMNLP

26 PCA VS. RPCA: WordNet Synset 14 PCA -L RPCA S 2 WordNet rank(l) = 1, 2, 3, RPCA PCA 0.5 RPCA PCA RPCA PCA 0.5 RPCA PCA

27 PCA VS. RPCA: 15 rank(l) = 3 PCA RPCA RPCA PCA # 1 after 1,2, eventually 0,1, whilst 0,1, again 1,2, finally 1,2 2 although 1,2, well 2,3, initially 1,2, more 1,4, both 0,2 3 Brian 1,2, February 0,2, Daniel 1,2, September 2,7, Frank 0,2 1 income 2,4, campaigns 1,5, age 6,7, development 4,5, goals 2,6 2 Berlin 0,6, Martin 3,4, Greek 0,3, Jan 0,4/0,6, name 1,3 3 quarterback 3,9, playoff 3,9 NBA 0,1, Houston 1,3, mayor 0,6 RPCA PCA

28 16

[-0.6cm]On Multi-Sense Word Embeddings via Matrix Factorization and Transformation

[-0.6cm]On Multi-Sense Word Embeddings via Matrix Factorization and Transformation I I N G U N I V E R S P E K T Y 1 8 9 8 On Multi-Sense Word Embeddings via Matrix Factorization and Transformation 1300012756 (1) (2) (3) I Abstract On Multi-Sense Word Embeddings via Matrix Factorization