[-0.6cm]On Multi-Sense Word Embeddings via Matrix Factorization and Transformation

Transcription

1 I I N G U N I V E R S P E K T Y On Multi-Sense Word Embeddings via Matrix Factorization and Transformation

2

3

4

5 (1) (2) (3) I

6 Abstract On Multi-Sense Word Embeddings via Matrix Factorization and Transformation Haoyue Shi (Department of Intelligence Science and Technology) Directed by Prof. Junfeng Hu Abstract Multi-sense word embeddings, which learn a vector for each sense, represent the concept of sense with vectors intuitively. However, such embeddings learned without supervision generate many pseudo multi-senses, as such methods are overly sensitive to contextual variations. In this work, we introduce three different algorithms for pseudo multi-sense detection from different perspectives of view, termed (1) external knowledge based pseudo multi-sense detection, (2) neighborhood similarity based pseudo multi-sense detection, and (3) principal component analysis based pseudo multi-sense detection. We test our algorithms with transformation matrix based elimination algorithms and downstream tasks. We show that eliminating the effect of pseudo multi-sense could lead to better performance to represent senses, as well as better performance of the embeddings on a lot of downstream tasks. Key Words : word embeddings, multi-sense word embeddings, matrix factorization, matrix transformation II

7 III

8 PCA RPCA IV

9 (Word Embeddings, Word Vectors) [2, 6, 20, 27 29, 32, 36, 41] 1.1 (1), (2) (3) (1) (2) (3) (4) bank (1) (2) net (1) (2) foot (1) (2) (3) 1.1 [5, 12, 13, 18, 22, 26, 33] A bank account is considered indispensable by most businesses and individuals. In geography, the word bank generally refers to the land alongside a body of water. bank 1

10 1.2 [33] bear( ) bear(1) bear(2) bear(1) 1 emerald, bears, three-toed, snake, periwinkle, ruffed, hoopoe, distinctive 2 bird, wolf, arrow, pelican, emerald, canyon, diamond, buck 3 pride, lady, hide, king, gift, crane, afflict, promise, reap 1.2 bear [31, 50] 2

11 ( 2, ) 3

12 2.1 (backward propagation) 1986 [38] [2, 6, 10, 39, 40, 42, 44, 45] X Y (Mutual Information)[8] I(X;Y) = p(x, y) log y Y x X ( ) p(x, y) p(x) p(y) (2.1) x, y X,Y p(x), p(y) x y, p(x, y) x y X x Y y, (2.1) x 4

13 y (Pointwise Mutual Information, PMI) i(x; y) = log ( ) p(x, y) p(x) p(y) (2.2) V V V M I M I x y Mx, I y = i(v x, v y ) v x c(v x ) (x, y) c(v x, v y ) p(x) p(v x, v y ). p(v x ) = c(v x) N p(v x, v y ) = c(v x, v y ) N (2.3) N i(x; y) = log ( ) c(x, y) N c(x) c(y) (2.4) (Positive Pointwise Mutual Information, PPMI) 0 [44] [43] [21] skip-gram (Negative Sampling, NS) (SGNS) [29] SVD [4, 9] [28, 30] SGNS [23] SGNS (co-occurence matrix) Pennington [36] 5

14 word2vec [29] [15] / FastText Mikolov [28, 29] 1. (Continuous Bag-of-Words, CBoW) 2. skip-gram [19] Omer Levy Yoav Goldberg skip-gram [19] [13, 37] skip-gram [33] w w 6

15 2k D 1 D V 1 w i k sum D V MLP w i 2 w i 1 w i+1 feature w i+2 w i+k look-up table prediction dot product sigmoid w i maximize P(c i w i ) c i w i minimize P(c i N w i ) c i N maximize 1 P(c i w i ) 2.1 skip-gram 7

16 Target Word 2-gram [29] 5-gram [29] [19] nightwing superman superman aquaman superboy superboy batman catwoman aquaman supergirl superman catwoman catwoman manhunter batgirl aquaman dumbledore evernight sunnydale hallows sunnydale collinwood hogwarts half-blood garderobe calarts malfoy blandings greendale snape collinwood millfield nondeterministic non-deterministic pauling non-deterministic finite-state hotelling turing computability nondeterministic heting deterministic buchi lessing finite-state primality hamming gainesville fla texas fla alabama louisiana florida jacksonville gainesville georgia tampa tallahassee california lauderdale texas carolina aspect-oriented aspect-oriented event-driven smalltalk event-driven domain-specific object-oriented event-driven objective-c rule-based prolog dataflow data-driven domain-specific 4gl human-centered singing singing singing dance dance rapping dancing dances dances breakdancing dancers breakdancing miming tap-dancing clowning busking 2.1 k (k=5) [19] skip-gram 2.2 skip-gram 8

17 w i k sum argmin dist w i 2 w i 1 w i+1 context sense vector w i+2 context center vector w i+k Skip-Gram global vector 2.2 [33] skip-gram [12, 18, 22, 33] (Singular Value Decomposition, SVD) [11] (Eigen Value Decomposition, EVD) 9

18 n m M : M = UΣV (2.5) U n Σ n m min(n, m) V m m V Σ {σ 1, σ 2,..., σmin(n, m)} σ i (1 i min(n, m)) M T M σ i 0 σ i M k n k Σ k UΣ k M V Σ k Σ V k UΣ k M SVD M T M M T M 2.1 λ i λ i 0 M σ i = λ i R SVD 2.1 A, A T A A n m A T m n x R m x T A T Ax = (x T A T )(Ax) = (Ax) T (Ax) = m (Ax) 2 i 0 (2.6) i=1 A T A (Principal Component Analysis, PCA)[14, 46] 10

19 0 Q M 0 QT Q n 1 M M M cov 2.1 M cov 0 M cov M M cov M M 0 M L E E, L L M [47] M L E S L E S (Robust Principal Component Analysis, RPCA) [47] subject to min rank(l) + λ 1 E F + λ 2 S 0 L + E + S = M (2.7) 11

20 , F Frobenius 0 0 λ 1 λ 2 [47] subject to min L + λ 1 E F + λ 2 S 1 L + E + S = M (2.8) 1 1 (2.8) [1, 24, 34] [47] (Robust Singular Value Decomposition, RSVD) [25, 48] 2.3 WordNet [31] WordNet Synset(synonym set) Synset (word) WordNet Synset - Synset Synset Synset WordNet WordNet WordNet 12

21 13

22 (metaphor)[16] (multi-sense) (metaphorical expression) (literal expression)[17, 43] Word- Net[31] (hypernym) [50] Synset WordNet Synset Synset Synset w s vs w 1 14

23 WordNet 1. WordNet w Synset Syn w 1, Synw 2,, Synw m. 2. Synset Synset Synset SynH w 1, SynHw 2,, SynHw n Synset w Synset w Synset 3. v w s 1 Synset Synset score(v w s 1, SynH w i ) = v w s k N N(v w s 1 ) cos(v w s 1, v w s k )ispossiblehypernym(synh w i, v w s 1 ) (3.1) WordNet Synset Synset Synset Synset Synset Synset Synset Synset Synset Synset w vs w 0 vs w 1 vs w 0 vs w 0 P pseudo (vs w 0, vs w 1 ) cos(v s 0, v s 1 ) (3.2) v s 0 N N(vs w 0 ) v s 1 N N(vs w 1 ) 1 Workshop(COLING Workshop CL4LC 2016) 15

24 (intra-word sense-wise difference matrix) V vs w w s d w V n w M = [ i j (vi w v w j )] (3.3) w M d w n w (n w 1) 3.1 v 0 cat v 1 cat v 1 cat v 0 cat..... v 0 cat v 2 cat v 1 cat v 2 cat (LREC2018) 16

25 vi w v w j vi w v w j M M 0 M M M M T M [33] M L E S, L M E 0 S S M (2.8) λ 1 λ 2 L rank(l) (10 4 ) 10 2 rank(l) M = L + E E 17

26 t M (t) L (t) E (t) E (t) S (t) 3 σ A (t) E (t) A (t) i, j = 0, 3σ E (t) E (t) i, j 3σ E (t) 1, otherwise (3.4) A (t) E (t) σ E (t) S (t) S (t) = A (t) E (t) (3.5) S (t) S (t) M (t) M (t+1) 1 1 Require: M, d Ensure: L, S, E 1: M (0) = M, t = 0, S = 0 2: while do 3: PCA L (t) + E (t) = M (t) 4: (3.4) A (t) 5: (3.5) S (t) 6: S = S + S (t) 7: M (t+1) = M (t) S (t) 8: t = t + 1 9: end while 10: return L (t), E (t), S 3.1 M 18

27 ρ L (M) L M t N, ρ L (t)(m (t) ) ρ L (t)(m (t+1) ) ρ L (t+1)(m (t+1) ) (3.6) {ρ L (t)(m (t) )} t 1 L = M [49] lim t ρ L (t)(m (t) ) M 0 M {(v w 0 s 0,0, v w 0 s 0,1 ), (v w 1 s 1,0, v w 1 s 1,1 ), (v w m sm,0, v w m sm,1 )} T min L(T) = m i=1 1 2 ( Tvw i s i,0 (vwi s i,0 + v w i 2 s i,1 ) 2 + Tv w i s i,1 (vwi s i,0 + v w i 2 s i,1 ) 2 ) (3.7) T 3 (EMNLP2018) 19

28 T : T 0 ; T 3.2 T α 1, α 2,, α k Ṽ T V Ṽ = {ṽ w s = Tv w s v w s V} (3.8) 3.2 R n α 1, α 2,, α n k(1 k n), T Tα i = 0, 1 i k Tα i = α i, k i n. α T 1 α 1,1 α 1,2 α 1,n α T 2 α 2,1 α 2,2 α 2,n A = = α T n α n,1 α n,2 α n,n 20 (3.9)

29 Tα 1 = 0 α T 1 TT i = 0 Tα 2 = 0 α T 2 TT i = 0 Tα k = 0 α T k TT i = 0 i, 1 i n (3.10) Tα k+1 = α k+1 α T k+1 TT i = α k+1,i, T T i Tα n = α n α n T T i = α n,i T T i (3.10) c i { }} { AT i = (0, 0,, 0, α k+1,i,, α n,i ) T, i, 1 i n (3.11) } {{ } k 0 A R n rank(a) = n, A 1 (3.11) T i T i = A 1 c i T α 1, α 2,, α n k T α 1, α 2,, α k [3] De-Biased Word Embeddings [3] bias de-bias 3.3 [21] 21

30 (PPMI) P P P i P i w i bank bank bank money check bank river lake w s 1 s 1 t 1 t 2 v s 1 PM I = [PMI(s 1, t 1 ); PMI(s 1, t 2 )] [ c(s1, t 1 ) N = c(s 1 ) c(t 1 ) ; c(s ] 1, t 2 ) N (3.12) c(s 1 ) c(t 2 ) v s 2 PM I = [PMI(s 2, t 1 ); PMI(s 2, t 2 )] [ c(s2, t 1 ) N = c(s 2 ) c(t 1 ) ; c(s ] 2, t 2 ) N (3.13) c(s 2 ) c(t 2 ) v w PM I = [PMI(w, t 1 ); PMI(w, t 2 )] [ c(w, t1 ) N = c(w) c(t 1 ) ; c(w, t ] 2) N c(w) c(t 2 ) [ (c(s1, t 1 ) + c(s 2, t 1 )) N = ; (c(s ] 1, t 2 ) + c(s 2, t 2 )) N (c(s 1 ) + c(s 2 )) c(t 1 ) (c(s 1 ) + c(s 2 )) c(t 2 ) (3.14) c(t 1 ) = c(t 2 ) v s 1 PM I vs 2 PM I vw PM I v s 1 PM I vs 2 PM I 1 s 1 s 2 w w s 1 s 2 s 1 s 2 22

31 c(s 1 ) c(s 2 ) s 1 s 2 23

32 [33] [22] [33] 300 NP-MSSG 6000 PyTorch [35] 4.1 k k [13] cat s0 cats s1, dog s2, mouse s3, dogs s1, chicken s cat s4 dog s4, cats s0, cats s2, bat s3, mouse s [33] cat s0 dog s2, cat s3, mr s9, girl s1, mouse s cat s3 dog s4, bat s8, cat s0, air s4, pan s [22] cat s0 cats2, cat s0, dog s0, dog s2, dog s cat s1 cat s2, cat s0, dog s0, dog s2, dog s [18] cat s0 cat s1, cat s2, dog s0, dog s1, dog s cat s1 dog s1, cat s2, cat s0, dog s0, dog s [18, 22, 33] [18, 22] [33] [12] (duplication) [13] [33] 24

33 4.2 STAR princess, series, cast, serial, midway, sparkle, 1940s, leo, closet, co-star 01 silver, boy, cat, version, adventures, stars, emerald, destroyer, terrace, planet 02 energy, disk, wheel, disadvantage, block, puff, radius, diamond, chord 03 version, bronze, standard, colors, ring, emblem, silver, wear, shoulder, red 01 [13] workshop, shop, paper, merchandise, plain, corporation, stock, likeness 03 guard, baseball, starter, tennis, basketball, brazil, class, world, morocco, ncaa 01 appearance, entertainer, pat, alumnus, freelance, brother, session, receiver 01 fictional, ongoing, manga, super, japanese, silver, interactive, asian, fiction 01 die, express, ride, opera, spanish, musical, hour, disaster, sun, blue 01 galaxy, spiral, variable, guide, magnitude, companion, satellite, crater 02 blue, dragon, acbl, diamond, purple, legion, arrow, mercury, eagle, cross 01 [33] fan, legend, show, moesha, heroes, guest-star, flicka, lassie, tv-movie 01 stars, sun, constellation, galaxy, eridani, pegasi, supergiant, ceti, starburst 02 01: person.n.01 02: celestial_body.n.01 03: whole.n.02 ROCK blur, indulgence, pop, noise, bands, lacuna, reformed, wave, genre, taster 01 energy, silver, cat, song, cd, planet, dawn, hero, video, terrace 02 metal, classic, legendary, dubbed, american, hard, belgian, short-lived, debut, da 01 soft, shifting, disappear, fill, crystalline, false, pitch, expanse, heat, pile 03 [13] vinyl, concert, limited, box, summer, double, dance, enhanced, gold, inch 04 hop, well-known, folk, occasional, jazz, music, concert, array, hard, pop 01 morris, miami, wood, ghost, silver, pearl, chase, corner, oak, thousand 03 hard, pop, cm, jazz, hip, hop, r&b, gutter, wave, subculture 01 hard, hip, short-lived, classic, jazz, raw, metal, ep 01 jazz, rally, star, roll, live, entertainer, appearance, session, pop, cover 01 metal, rippling, dense, swirling, chirping, blues, punk, psychedelia, bands, pop 01 [33] sand, rocks, butte, ash, sandy, little, cedar, rocky, sugarloaf, spring-fed 03 hip, alternative, indie, progressive, hop, reggae, roll, rock/metal, post-hardcore 01 01: popular_music.n.01 02: person.n.01 03: material.n.01 04: whole.n star( / ) rock( / / ) WordNet[31] Synset 10 3 Synset Synset Synset whole.n.02 (Synset) [33] 3 50 (MSSG-50D) [13] 3 MSSG 25

34 WordNet Synset 4.3 (3.2) 1 [33] skip-gram skip-gram NORWAY NP-MSSG [33] Denmark, Troms, Sogn, Hedmark Denmark, Sweden, Finland, Iceland Denmark, Sweden, Finland, Netherlands Denmark, Austria, Germany, Belgium + Tag Denmark, Norwegian, Sweden, Trondheim STAR NP-MSSG [33] stars, movie, song, heart stars, award, eagle, two-time supergiant, constellation, aurigae + Tag stars, movie, superstar, MVP supergiant, stars, g5v, white main ALGORITHM NP-MSSG [33] hash, algorithms, quick sort, recursive algorithms, optimization, public-key + Tag algorithms, computation, iteratively NP-MSSG 26

35 4.4 M : M = L + E + S (4.1) L, E, S L S # 1 after 1,2, eventually 0,1, whilst 0,1, again 1,2, finally 1,2 2 although 1,2, well 2,3, initially 1,2, more 1,4, both 0,2 3 Brian 1,2, February 0,2, Daniel 1,2, September 2,7, Frank 0, S p 2 prime s0 minister, cabinet, parliament prime s1 modulo, space, equivalently, real 3.35 yard s0 touchdown, interception, kickoff yard s1 lawn, backyard, garden, porch 2.75 engine s0 jeep, truck, wheel, vehicle, car engine s1 camshaft, turbine, gearbox 0.61 cat s0 dog, pet, wolf, bird, animal cat s1 dog, pets, puppy, cats, fox 0 Word #0 #1 prime yard engine / cat 4.5 k

36 4.5 k 2 yard prime 2 engine / cat RPCA 4.5 (gold label) Spearman rank correlation(ρ [ 1, 1]) WordSim353[9] Stanford Contextual Word Similarity(SCWS, [13]) WordSim SCWS WordSim-353, WS-353 avgsim[37] maxsimc[33] 28

37 avgsim(w a, w b ) = 1 S(w a ) S(w b ) s a S(w a ) s b S(w b ) cos(v w a s a, v w b s b ) (4.2) w a w b S(w a ) S(w b ) maxsimc(w a, w b ) = cos(v w a ŝ a, v w b ŝ b ) (4.3) ŝ a = argmax sa S(a) P(s a C(w a )) ŝ b = argmax sb S(b) P(s b C(w b )) avgsimc[12, 18, 22, 33] WS-353 SCWS [13] skip-gram (MSSG, 300D)[33] skip-gram (NP-MSSG, 300D) [33] NP-MSSG NP-MSSG NP-MSSG + (PCA) NP-MSSG + (RPCA) [22] MUSE[18] WordSim-353 SCWS 29

38 MUSE[18] SCWS NP-MSSG[33] 4.6 PCA RPCA 4.4 RPCA RPCA PCA PCA WordNet Synset PCA RPCA d RPCA PCA 4.2 Synset Synset n RPCA PCA 4.7 PCA RPCA RPCA 3 PCA 2 RPCA PCA RPCA 30

39 0.5 RPCA PCA 0.5 RPCA PCA RPCA PCA 0.5 RPCA PCA d = 1 d = 2, d = 3, d = 5. RPCA PCA # 1 after 1,2, eventually 0,1, whilst 0,1, again 1,2, finally 1,2 2 although 1,2, well 2,3, initially 1,2, more 1,4, both 0,2 3 Brian 1,2, February 0,2, Daniel 1,2, September 2,7, Frank 0,2 1 income 2,4, campaigns 1,5, age 6,7, development 4,5, goals 2,6 2 Berlin 0,6, Martin 3,4, Greek 0,3, Jan 0,4/0,6, name 1,3 3 quarterback 3,9, playoff 3,9 NBA 0,1, Houston 1,3, mayor 0,6 4.7 RPCA PCA 4.7 maxsimc (BoW) Facebook Research SentEval[7] SUBJ TREC MSRP

40 SUBJ TREC MSRP (RPCA)

41 (Word Sense) (Word Embeddings) 33

42 [1] Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences, 2009, 2(1): [2] Yoshua Bengio, Réjean Ducharme, Pascal Vincent et al. A neural probabilistic language model. Journal of machine learning research, 2003, 3(Feb): [3] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou et al. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In: Advances in Neural Information Processing Systems, 2016: [4] Elia Bruni, Gemma Boleda, Marco Baroni et al. Distributional semantics in technicolor. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, 2012: [5] Jianpeng Cheng and Dimitri Kartsaklis. Syntax-Aware Multi-Sense Word Embeddings for Deep Compositional Models of Meaning. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015: [6] Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In: Proceedings of the 25th international conference on Machine learning, 2008: [7] Alexis Conneau and Douwe Kiela. SentEval: An Evaluation Toolkit for Universal Sentence Representations. In: Proceedings of the 11th Language Resource and Evaluation Conference, [8] TM Cover and JA Thomas. Elements of information theory: Wiley Online Library [9] Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias et al. Placing search in context: The concept revisited. In: Proceedings of the 10th international conference on World Wide Web, 2001: [10] Xavier Glorot, Antoine Bordes and Yoshua Bengio. Domain adaptation for large-scale sentiment classification: A deep learning approach. In: Proceedings of the 28th international conference on machine learning (ICML-11), 2011: [11] Gene H Golub and Christian Reinsch. Singular value decomposition and least squares solutions. Numerische mathematik, 1970, 14(5): [12] Fenfei Guo, Mohit Iyyer and Jordan Boyd-Graber. Inducing and Embedding Senses with Scaled Gumbel Softmax. arxiv preprint arxiv: , [13] Eric H Huang, Richard Socher, Christopher D Manning et al. Improving word representations via global context and multiple word prototypes. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, 2012:

43 [14] Ian T Jolliffe. Principal component analysis and factor analysis. In: Principal component analysis. Springer, 1986: [15] Armand Joulin, Edouard Grave, Piotr Bojanowski et al. Bag of Tricks for Efficient Text Classification. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, 2017: [16] George Lakoff. The contemporary theory of metaphor [17] George Lakoff and Mark Johnson. Metaphors we live by. University of Chicago press, [18] Guang-He Lee and Yun-Nung Chen. MUSE: Modularizing Unsupervised Sense Embeddings. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017: [19] Omer Levy and Yoav Goldberg. Dependency-based word embeddings. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2014: [20] Omer Levy and Yoav Goldberg. Linguistic regularities in sparse and explicit word representations. In: Proceedings of the eighteenth conference on computational natural language learning, 2014: [21] Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In: Advances in neural information processing systems, 2014: [22] Jiwei Li and Dan Jurafsky. Do Multi-Sense Embeddings Improve Natural Language Understanding? In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015: [23] Yitan Li, Linli Xu, Fei Tian et al. Word Embedding Revisited: A New Representation Learning and Explicit Matrix Factorization Perspective. In: IJCAI, 2015: [24] Zhouchen Lin, Risheng Liu and Zhixun Su. Linearized alternating direction method with adaptive penalty for low-rank representation. In: Advances in neural information processing systems, 2011: [25] Li Liu, Douglas M Hawkins, Sujoy Ghosh et al. Robust singular value decomposition analysis of microarray data. Proceedings of the National Academy of Sciences, 2003, 100(23): [26] Yang Liu, Zhiyuan Liu, Tat-Seng Chua et al. Topical Word Embeddings. In: AAAI, 2015: [27] Thang Luong, Richard Socher and Christopher Manning. Better word representations with recursive neural networks for morphology. In: Proceedings of the Seventeenth Conference on Computational Natural Language Learning, 2013: [28] Tomas Mikolov, Kai Chen, Greg Corrado et al. Efficient estimation of word representations in vector space. arxiv preprint arxiv: ,

44 [29] Tomas Mikolov, Ilya Sutskever, Kai Chen et al. Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, 2013: [30] Tomas Mikolov, Wen-tau Yih and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2013: [31] George A Miller. WordNet: a lexical database for English. Communications of the ACM, 1995, 38(11): [32] Andriy Mnih and Geoffrey Hinton. Three new graphical models for statistical language modelling. In: Proceedings of the 24th international conference on Machine learning, 2007: [33] Arvind Neelakantan, Jeevan Shankar, Alexandre Passos et al. Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014: [34] Yu Nesterov. Smooth minimization of non-smooth functions. Mathematical programming, 2005, 103(1): [35] Adam Paszke, Sam Gross, Soumith Chintala et al. Pytorch, [36] Jeffrey Pennington, Richard Socher and Christopher Manning. Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014: [37] Joseph Reisinger and Raymond J Mooney. Multi-prototype vector-space models of word meaning. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2010: [38] David E Rumelhart, Geoffrey E Hinton and Ronald J Williams. Learning representations by backpropagating errors. nature, 1986, 323(6088): 533. [39] Holger Schwenk. Continuous space language models. Computer Speech & Language, 2007, 21(3): [40] Richard Socher, Cliff C Lin, Chris Manning et al. Parsing natural scenes and natural language with recursive neural networks. In: Proceedings of the 28th international conference on machine learning (ICML-11), 2011: [41] Joseph Turian, Lev Ratinov and Yoshua Bengio. Word representations: a simple and general method for semi-supervised learning. In: Proceedings of the 48th annual meeting of the association for computational linguistics, 2010: [42] Peter D Turney. Distributional Semantics Beyond Words: Supervised Learning of Analogy and Paraphrase. Transactions of the Association of Computational Linguistics, 2013, 1: [43] Peter D Turney, Yair Neuman, Dan Assaf et al. Literal and metaphorical sense identification through concrete and abstract context. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2011:

45 [44] Peter D Turney and Patrick Pantel. From frequency to meaning: Vector space models of semantics. Journal of artificial intelligence research, 2010, 37: [45] Jason Weston, Samy Bengio and Nicolas Usunier. Wsabie: Scaling up to large vocabulary image annotation. In: IJCAI, 2011: [46] Svante Wold, Kim Esbensen and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 1987, 2(1-3): [47] John Wright, Arvind Ganesh, Shankar Rao et al. Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization. In: Advances in neural information processing systems, 2009: [48] MK Stephen Yeung, Jesper Tegnér and James J Collins. Reverse engineering gene networks using singular value decomposition and robust regression. Proceedings of the National Academy of Sciences, 2002, 99(9): [49].., [50].. ;,

46 1. Haoyue Shi 1, Jiayuan Mao 1, Tete Xiao 1, Yuning Jiang, Jian Sun. Learning Visually- Grounded Semantics from Contrastive Adversarial Samples International Conference on Computational Linguistics (COLING 2018). Santa Fe, New-Mexico, USA. August (1 Equal Contribution, *: Corresponding Authors) 2. Haoyue Shi, Xihao Wang, Yuqi Sun, Junfeng Hu. Constructing High Quality Sensespecific Corpus and Word Embedding via Unsupervised Elimination of Pseudo Multi-Sense. Language Resources and Evaluation Conference (LREC 2018). Miyazaki, Japan. May (*: Corresponding Author) 3. Haoyue Shi, Jia Chen, Alexander G. Hauptmann. Joint Saliency Estimation and Matching using Image Regions for Geo-Localization of Online Video, ACM International Conference on Multimedia Retrieval (ICMR 2017). Bucharest, Romania. June (*: Corresponding Author) 4. Haoyue Shi, Caihua Li, Junfeng Hu. Real Multi-Sense or Pseudo Multi-Sense: An Approach to Improve Word Representation. Workshop on Computational Linguistics for Linguistic Complexity (associated with COLING 2016). Osaka, Japan. December (*: Corresponding Author) 5. Shan Xu, Haoyue Shi, Xiaohui Duan, Tiangang Zhu, Peihua Wu and Dongyue Liu. Cardiovascular Risk Prediction Method Based on Test Analysis and Data Mining Ensemble System IEEE International Conference on Big Data Analysis. Hangzhou, China. March (*: Corresponding Author) 6. Haoyue Shi, Junfeng Hu, Yuqi Sun, Xihao Wang. Improving Sense Discovery via Robust Principal Component Analysis. In Submission to EMNLP (*: Corresponding Author) 7. Haoyue Shi, Hao Zhou, Jiaze Chen, Lei Li. On Tree-Based Neural Sentence Modeling. In Submission to EMNLP (*: Corresponding Author) 38

47 Prof. Alex Hauptmann Prof. Sam Bowman Prof. Alex Hauptmann Google, host Dr. Omer Levy Dr. Sebastian Riedel 39

48 40

49 / / 41