CRFs4ASR.pptx

Size: px

Start display at page:

Download "CRFs4ASR.pptx"

溆培仰
5 years ago
Views:

1 CRF CSLT, RIIT, Tsinghua University

2 ASR 2

3 (node, vertex, site): (link, arc, edge): :, : p(ω 1, ω 2, ω 3 )=p(ω 1 )p(ω 2 ω 1 )p(ω 3 ω 2 ) 3

4 Bayes p(x, y) =p(y)p(x y) x, p(x) = y p(x y )p(y ) p(y x) = p(x y)p(y) à y p(x y )p(y ) 4

5 : I à K J p(i,k J) =p(i J)p(K J) (clique): ( ) : ( ) 5

6 : C ω C =(x C, y C ), Ψ C ( ω C ) ( ), : p(ω )= 1 Z C Ψ C ( ω C ) = p(x, y) Z = Ψ C ( ω C ) ω C 6

7 Ψ( ω C )=exp{ E( ω C )} > 0, E( ω C ) ; Ψ( ω C ) p(ω ) Hammersley-Clifford UI: UF: p(ω ) UI=UF= (i.e. )? 7

8 : X Ω = x X Ω x, Ω x ; x F Ω P ; (Ω, F ) P (ω) ( ω Ω) (Ω, F,P) (i.e. ) X : : (Markov Networks, Finite Lattice etc.) 8

9 p(ω C )= 1 Z, : p(ω C )= 1 K exp{ λ k f k (ω C )} Z C C Ψ C ( x C, y C )= 1 Z k=1 exp{ E C (ω C )} :, C,, ( : ) C 9

10 (CRF) à x = {x 1,x 2,...,x T }, y = {y 1,y 2,...,y T } : p(x, y) = 1 Ψ C ( x C, y C )= 1 Z Z C : p(y x) = p(x, y) y p(x, y ) = T exp{ t=1 K λ k f k (y t,y t 1, x t )} k=1 exp{ T t=1 K k=1 λ kf k (y t,y t 1, x t )} y exp{ T t=1 K k=1 λ kf k (y t,y t 1, x t)} 10

11 (CRF) (CRF): Z(x) = y p(y x) = 1 T K exp{ λ k f k (y t,y t 1, x t )} Z(x) T t=1 k=1 K exp{ λ k f k (yt,y t 1, x t )} t=1 k=1 11

12 : D = {x (i), y (i) } N i=1, (x(i), y (i) )= : p(y x) θ = {λ k } K k=1 (maximum likelihood): l(θ) = N i=1 T t=1 K k=1 λ k f k (y (i) t,y (i) t 1, x(i) t x (i) = {x (i) 1,x(i) 2,...,x(i) T } ) y (i) = {y (i) 1,y(i) 2,...,y(i) T } N log Z(x (i) ) i=1 ( ) l(θ) 12

13 : or k- : L2 + 1/δ 2 N T K l(θ) = λ k f k (y (i) t,y (i) i=1 t=1 k=1 t 1, x(i) t ) N log Z(x (i) ) i=1 θ 0 δ 2 I MAP θ T C k (y,x) = f k (y t,y t 1, x t ), t=1 K k=1 λ 2 k 2δ 2 13

14 l λ k = N C k (y (i), x (i) ) i=1 = N(E[f k ] E θ [f k ]) N i=1 y p(y x (i) )C k (y, x (i) ) λ k δ 2, E[f k ] E θ [f k ] θ f k L1 f k 14

15 l λ k : Newton s Method: Hessian à quasi-newton s method(bfgs): Hessian à, Limited-memory BFGS(L-BFGS): à, : Precond. CG, Mixed CG, Plain CG, GIS etc. 15

16 (inference): : à ; p(y x) Z(x) : y = arg max p(y,x) y p(y t,y t 1 x) : (forward-backward algorithm) α t (y) = y α t 1(y )exp( K k=1 λ kf k (y,y,x t 1 )) β t (y) = y β t+1(y )exp( K k=1 λ kf k (y,y,x t+1 )) p(y t,y t 1 x) α t 1 (y t 1 )exp{ K k=1 λ kf k (y t,y t 1, x t )}β t (y t ) 16

17 Z(x) : Z(x) = T y t=1 exp{ K k=1 λ kf k (yt,y t 1, x t )} ; ( ) Viterbi : δ t (j) = max i Y exp{ K k=1 λ kf k (i, j, x t )}δ t 1 (i) i, j Y, Y (CRF ) 17

18 CRF : WFA A =(Σ,Q,q s,f,e,ρ) WFA A, L,,, l[π] =l[π A = L A π Π A π Π A ] A A : Intersection, L A A [π] =L A [π] L A [π] w A A [π] =w A [π]+w A [π], l[π] =l[π 1 ]=l[π 2 ] Exponential Family à Kernel Function 18

19 CRF : B, g B, C(g) h : X (B) Y(B), h(g, x) Y(g) : φ(y, f(g, x)) = f c (x, y c ) + log exp( f c (x, y c)) c C(g) CRF: p(y g, x) =Z 1 (g, x, f)exp( c Gibbs y Y((g)) f c (x, y c )) c C(g) 19

20 CRF vs. HMM HMM: à p(y t y t 1 ) à p(x t y t ) p(x, y) = T t=1 p(y t y t 1 )p(x t y t ) HMM: T 1 p(x, y) = Z exp{ λ ij 1 yt =i1 yt 1 =j + t=1 i,j Y i Y = 1 T K exp{ λ k f k (y t,y t 1,x t )} Z t=1 k=1 o O µ oi 1 yt =i1 xt =o} 20

21 CRF vs. HMM CRF HMM: p(y t y t 1 ) vs. f k (y t,y t 1, x t ) à CRF (Label Bias Problem) p(x t y t ) vs. f k (y t,y t 1, x t ) à CRF, CRF x t CRF, scaling factor CRF, HMM CRF HMM: HMM, ( ) CRF? 21

22 VS : p(x, y) ( ) x y p(x y) : p(y x), p(x) CRF (CRF y x p(x) ) à 22

23 / 23

24 / ( ) Ψ 1,2 (ω 1, ω 2 )=p(ω 1 )p(x 2 x 1 ) : Ψ 2,3 (ω 2, ω 3 )=p(ω 3 ω 2 ) 24

25 / 25

26 CRF CRF: p(y x) = 1 Z(x) C K C exp{ λ Ck f Ck (x C, y C )} k=1 à ( / ) 26

27 CRF à C = {C p } P p=1 C p à Belief Propagation( ) Viterbi : NP-hard : MCMC : Loopy Belief Propagation ( ) : TRP, BLP etc. 27

28 CRF w p(y, w x) = 1 Z(x) C p C Ψ C C p Ψ C (x C, w C, y C ; θ p ) l(θ) = log p(y x) = log w p(y, w x) p( w y,x) = 1 Z(x, y) C p C Ψ C C p Ψ C (x C, w C, y C ; θ p ) p(y, w x) Z(x, y) p(y x) = = p( w y,x) Z(x) Z(x, y) = Ψ C (x C, w C, y C ; θ p ), Z(x) = w C p C Ψ C C p y Z(x, y) 28

29 CRF l(θ), (L-BFGS, GIS), Generalized EM etc. : ; 29

30 CRFs CRF CRF: TCRF, Skip-Chain CRF, 2D CRF, Semi-Markov CRF, *SQL CRF CRF: HCRF, GHCRF, LDCRF, FCRF : *Kernel CRF 30

31 Linear-Chain CRF in ASR-(1) Linear-Chain CRF + Phonology Attribute + Phone (Eng.) Monophone label 31

32 Linear-Chain CRF in ASR-(1) (, ) f /b/,voi (y,x, t) = 0, otherwise 1, if y t 1 = /n/, y t = /ah/and nasal(x t )=true f /n/,/ah/nas (y,x, t) = f /b/,voi (y,x, t) = 1, if y t = /b/and voiced(x t )=true 0, otherwise voiced(x t ), 0, otherwise if y t = /b/ 32

33 Linear-Chain CRF in ASR-(1) Feature : MLP CRF reference hypothesis CRF re-alignment Frame-Level, 33

34 Linear-Chain CRF in ASR-(2) Articulatory Features + Finals (Mandarin): Monophone label 34

35 Linear-Chain CRF in ASR-(2) : CRF++ ( ) 1, if y t = /zh/and bilabio(x t )=true f /zh/,bil. (y,x, t) = 0, otherwise 1, if y t 1 = /zh/, y t = /zh/, y t+1 = /ang/ f /zh/,/zh/,/ang/,bil. (y,x, t) = and bilabio(x t )=true 0, otherwise Feature : Right-CD HMM (Bi-phone) 35

36 Linear-Chain CRF in ASR-(3) Linear-Chain CRF : 36

37 CRF TCRF Semantic Annotation Skip-Chain CRF Name Entity Recognition 37

38 CRF 2D CRF Web Information Extraction 38

39 Semi-Markov CRF : S K s=1 k=1 p(y x) = λ kg k (y s,y s 1, x, st s,et s ) Z(x) S, S ; st s / et s s Name Entity Recognition 39

40 Semi-Markov CRF in ASR Word, Syllable, Phone, Articulatory, Duration etc. Word ( Bing ) : n-gram Existence Feature f u (y s,y s 1,o et st) =δ(w(y s 1 )=u)δ(u span(st, et))) n-gram Expectation Feature: Hit, False Alarm, False Reject f u (y s,y s 1,o et st) =δ(u pron(w(y s 1 )))δ(i span(st, et))) f u (y s,y s 1,o et st) =δ(u pron(w(y s 1 )))δ(i / span(st, et))) f u (y s,y s 1,o et st) =δ(u / pron(w(y s 1 )))δ(i span(st, et))) 40

41 Semi-Markov CRF in ASR Levenshtein Feature number of times u is matched f match u f sub u f del u f ins u number of times u (in pronunciation) is substituted number of times u is deleted number of times u is inserted Language Model Feature f(y s,y s 1,o et st) = LM(y s,y s 1 ) Baseline Feature f b (y s,y s 1,o et +1 if C(st, et) = 1 and B(st, et) =w(y s 1 ) st) = 1 otherwise. 41

42 Semi-Markov CRF in ASR Feature : HMM (Bi-phone, Monophone) 42

43 CRF Hidden CRF Object Recognition Hidden: State Factorized CRF Joint Noun Phrase Chunking and Part-of-Speech Tagging Hidden: State(POS) 43

44 CRF Latent-Dynamic CRF Shallow Parsing Gesture Recognition Hidden: State Hidden State Number BLP Inference (Linear) 44

45 (Gaussian) Hidden CRF in ASR HMM (GHMM)? Hidden: GMM, State CRFà GHMM? HCRFà GHMM 45

46 (Gaussian) Hidden CRF in ASR MFCC or PLP etc. à monophone label : f (LM) = δ(y = y ) y T f (Tr) = δ(y = y,s y ss t 1 = s, s t = s) f (Occ) sm = f (M1) sm = f (M2) sm = t=1 T δ(s t = s, m t = m) t=1 T δ(s t = s, m t = m)x t t=1 T δ(s t = s, m t = m)x 2 t t=1 y y,s,s s, m s, m s, m 46

47 (Gaussian) Hidden CRF in ASR Why named Gaussian HCRF? : λ LM y = log u y y λ Tr y ss λ Occ sm = 1 2 = log a y ss d λ M2 sm = 1 2δ 2 s,m,d λ M1 sm = µ s,m,d δ 2 s,m,d (log 2πδ 2 s,m,d + µ2 s,m,d δ 2 s,m,d ) y,s,s s, m s, m s, m 47

48 (Gaussian) Hidden CRF in ASR GHCRF à Log-Linear GHMM exp{λ LM y } = u y exp{λ Tr y ss } = a y ss exp{λ Occ sm + λ M1 smx t + λ M2 smx 2 t } = 1 2πδs,m,d 2 exp{ 1 2 (µ s,m,d x t,s,d ) 2 } δ 2 s,m,d : e.g. : HMM 48

49 (Gaussian) Hidden CRF in ASR Phone Classification Phone Recognition 49

50 (Gaussian) Hidden CRF in ASR : MAP, MLLR 50

51 CRF Efficiently Inducing Features Relief Gaussian-Hidden CRF 51

52 CRF à f pk (y c, x c )=1 yc =y q pk(x c c ) à /Belief Propagation ( ) / / 52

Ρ Τ Π Υ 8 ). /0+ 1, 234) ς Ω! Ω! # Ω Ξ %& Π 8 Δ, + 8 ),. Ψ4) (. / 0+ 1, > + 1, / : ( 2 : / < Α : / %& %& Ζ Θ Π Π 4 Π Τ > [ [ Ζ ] ] %& Τ Τ Ζ Ζ Π

Ρ Τ Π Υ 8 ). /0+ 1, 234) ς Ω! Ω! # Ω Ξ %& Π 8 Δ, + 8 ),. Ψ4) (. / 0+ 1, > + 1, / : ( 2 : / < Α : / %& %& Ζ Θ Π Π 4 Π Τ > [ [ Ζ ] ] %& Τ Τ Ζ Ζ Π ! # % & ( ) + (,. /0 +1, 234) % 5 / 0 6/ 7 7 & % 8 9 : / ; 34 : + 3. & < / = : / 0 5 /: = + % >+ ( 4 : 0, 7 : 0,? & % 5. / 0:? : / : 43 : 2 : Α : / 6 3 : ; Β?? : Α 0+ 1,4. Α? + & % ; 4 ( :. Α 6 4 : & %