Transcription

1

2

3 University of Science and Technology of China Dissertation for Doctor s Degree Discriminative Training of Acoustic Models for Automatic Speech Recognition Zhi-Jie Yan Signal & Information Processing Supervisor: Prof. Ren-Hua Wang May, 2008

4

5 ,

6

7 MWCE MCE MWCE ( MWE MPE) MWCE MMIE Trust Region Trust Region EB MMIE Trust Region EB SME SME SME MCE SME SME Soft Margin SME MCE SME MCE MMIE HMM MMIE HMM I

8 HMM II

9 Abstract Abstract Discriminative training of acoustic models has become one of the most important training methods for state-of-the-art speech recognition systems. This topic attracts more and more attentions of researchers, to develop new training criteria, parameter optimization methods, and application techniques. In this context, this thesis focuses on discriminative training of acoustic models and its application in automatic speech recognition. It provides a systematic and in-depth research in this topic, and introduces our innovations in criterion, optimization method, and application of discriminative training. Firstly, this thesis proposes a novel discriminative training criterion, i.e. Minimum Word Classification Error (MWCE). By localizing conventional string-level MCE loss function to word-level, a more direct measure of word classification error is approximated and minimized. Because the word-level criterion better matches performance evaluation criteria in LVCSR, such as WER, an improved word recognition performance can be achieved. Comparing with other sub-string level criteria (e.g. MWE / MPE), MWCE provides another perspective of word-level classification error, and achieves the best recognition performance in our experiments. This result suggests that it is still meaningful to develop new discriminative training criteria which have explicit physical meaning and more reasonable. Secondly, the thesis proposes a new parameter optimization method for discriminative training, i.e. trust region based optimization for MMIE criterion. By imposing a trust region constraint into the optimization process, we avoid some disadvantages of the unbounded optimization of conventional EB method. The new optimization method is more reasonable in mathematics, and also physically meaningful. Meanwhile, because we can reach a global optimum in each iteration, the proposed method is more efficient in optimizing criterion. Our experimental results suggest that the trust region based approach outperforms conventional EB method both in optimizing criterion and recognition performance. Thirdly, this thesis introduces our research to improve the Soft Margin Estimation (SME) method. By imposing some important technologies of discriminative training in recent years, we successfully implement the SME method in LVCSR for the first time. Meanwhile, we propose to use a reasonable frame-level separation measure to III

10 Abstract select certain frame samples that contain important discriminative information. We compare conventional MCE, string-level SME, and the proposed frame-level SME in our experiments. The results show that by using the concept of soft margin, both SME methods can achieve a better performance than MCE. And by imposing a factor which removes noisy frames, the frame-level SME achieves the best recognition performance which significantly outperforms MCE. Lastly, this thesis proposes an application method of discriminative training, i.e. MMIE based HMM topology optimization. We define a heuristic metric according to MMIE criterion, and use it to guild the topology optimization process. The approach tries to exchange Gaussian ernels among HMM states so as to allocate model parameters non-uniformly. Besides, a post-process is also carried out to refine model topology in time axis. By doing this, we provide a more direct connection between topology optimization and discrimination. As a result, the discriminative model topology optimization outperforms other conventional, lielihood based optimization methods in our experiments. Keywords: Discriminative Training, Acoustic Model, ASR, MWCE, Trust Region, SME IV

11 HMM HMM HMM HMM HMM MMIE MMIE MMIE MCE MCE MCE / MWE / MPE MWE / MPE MWE / MPE V

12 EB MWCE MCE MWCE MWCE MWCE MWCE MMIE Trust Region HMM MMIE Trust Region Trust Region KLD Trust Region Trust Region HMM Trust Region HMM Trust Region Trust Region SME SME SME SME SME SME SME VI

13 7 MMIE HMM BIC / MDL MMIE MMIE VII

14 VIII

15 a HMM b( ) c F M N( ) p( ) r, R s, S t, T w, W x, X, o, O / / γ ( ) λ, Λ µ σ 2, Σ AIC AM ASR BIC CT DFE DP DT DTW EB EM FB FT GD GMM GPD Aaie Information Criterion Aaie Acoustic Model Automatic Speech Recognition Bayesian Information Criterion Corrective Training ( ) Discriminative Feature Extraction Dynamic Programming Discriminative Training Dynamic Time Warping Extended Baum-Welch Algorithm Baum-Welch Expectation-Maximization Algorithm Forward-Bacward Algorithm Falsifying Training ( ) Gradient Descent Gaussian Mixture Model Generalized Probability Descent IX

16 HLDA HMM HTK KLD LDA LME LVCSR MAP MCE MDL MFCC MLE MLLR MMIE MPE MVE MWCE MWE PLP SME SVM TR UI WER WPP Heteroscedastic Linear Discriminant Analysis Hidden Marov Model Hidden Marov Toolit Kullbac-Leibler Divergence KL Linear Discriminant Analysis Large Margin Estimation Large Vocabulary Continuous Speech Recognition Maximum A-Posteriori ( ) Minimum Classification Error ( ) Minimum Description Length Mel-Frequency Cepstral Coefficients Maximum Lielihood Estimation ( ) Maximum Lielihood Linear Regression Maximum Mutual Information Estimation ( ) Minimum Phone Error ( ) Minimum Verification Error ( ) Minimum Word Classification Error ( ) Minimum Word Error ( ) Perceptual Linear Prediction Soft Margin Estimation Support Vector Machine Trust Region User Interface Word Error Rate Word Posterior Probability X

17 (User Interface, UI) (Automatic Speech Recognition, ASR) [1] (Dynamic Time Warping, DTW) (Dynamic Programming, DP) [2] 70 LPC [3] (Defense Advanced Research Projects Agency, DARPA) (Speech Understanding Research, SUR) (Carnegie Mellon University, CMU) Harpy [4] Hearsay [5] BBN HWIM [6] IBM [7] HMM HMM HMM [8 10] HMM HMM (Maximum A-Posteriori Estimation, MAP Estimation) [11] (Maximum Lielihood Linear Regression, MLLR) [12] [13,14] 1

18 1 CMU Sphinx [15] BBN BYBLOS [16] SRI DECIPHER [17] Whisper [18] SAPI IBM ViaVoice DARPA (National Institute of Standards and Technology, NIST) (Resource Management, RM) (North American Business, NAB) (Wall Street Journal, WSJ) Switchborad (Maximum Lielihood Estimation, MLE) (Discriminative Training, DT) [19,20] (Hidden Marov Toolit, HTK) [21] [22 24] HMM [25] DARPA (Global Autonomous Language Exploitation, GALE) [26,27] [28] (Large Vocabulary Continuous Speech Recognition, LVCSR) 1.2 O F ( ) T = F (O) F (Error Rate) 2

19 1 语音输入 声学特征提取 获得声学特征 O 最优词序列 W* 解码器 搜索全局最大的 p(o W)p(W) 声学模型 p(o W) 语言模型 p(w) 1.1 (Statistical Speech Recognition) W = {w 1, w 2,..., w N } O p(w O) = p(o W)p(W) p(o) (1-1) p(o W) (Acoustic Model, AM) W (Hidden Marov Model, HMM) 2 p(w) W W (1.1) [29,30] 3

20 1 [31] (Fourier Transformation) (Linear Prediction) (Cepstral Analysis) (Mel-Frequency Cepstral Coefficients, MFCC) [32] (Perceptual Linear Prediction, PLP) [33] [34] (Linear Discriminant Analysis, LDA) [35] (Heteroscedastic Linear Discriminant Analysis, HLDA) [36] n- (n-gram) n-gram n 1 ( 2 ) p(w) p(w) = p(w K 1 ) = K w 0 1 = =1 p(w w 1 1 ) K =1 p(w w 1 max[ n+1,1] ) (1-2) n 1 n-gram n-gram (Perplexity) [37] PP = [ [ K p(w1 K )] 1 K = =1 1 ] K p(w w 1 max[ n+1,1] ) (1-3) 4

21 1 M n-gram M N (Discounting) [38] (Bacing-Off) [39] (Decoder) W (1-1) W = arg max W p(w O) = arg max W p(o W)p(W) p(o) = arg max W p(o W)p(W) (1-4) ( ) (Viterbi Algorithm) [40] 2 [41] A (A -Search) (Stac-Decoding) [42] Viterbi (Pruning) ( Guassian Selection [43] ) Beam [44] (Lexical Prefix Tree) [45] (Language Model Loo-Ahead) [46] (Re-Scoring) n-best [47,48] (Word Graph, Lattice) [49] 5

22 1 [31,50 52] 1.4 MWCE MCE MWCE Trust Region HMM Trust Region Trust Region EB SME SME SME SME MCE SME MMIE HMM MMIE 2 HMM 3 6

23 1 4 MWCE 5 Trust Region 6 SME SME 7 MMIE HMM 8 7

24 2 HMM 2 HMM 2.1 HMM HMM ( ) ( ) HMM HMM HMM HMM HMM Baum HMM [53] [54] [55] [56] [10] HMM HMM HMM HMM (2.2) HMM (2.3) HMM HMM (2.4) HMM (2.5) 2.2 HMM (Marov Process) 8

25 2 HMM X t 1 < t 2 < < t n p ( ) ( ) x tn < φ x t1, x t2,..., x tn 1 = p xtn < φ x tn 1 (2-1) (Marov Chain) S = {1, 2,..., N} s t p(s t s 0, s 1,..., s t 1 ) = p(s t s t 1 ) (2-2) a i j = p(s t = j s t 1 = i) 1 i, j N π i = p ( s 0 = i ) 1 i N (2-3) a i j π i N a i j = 1 1 i N j=1 (2-4) N π j = 1 j=1 (Observable Marov Model) (2.1) [57] A π A = {a i j } = (2-5) 0.5 π = {π i } =

26 2 HMM 上涨 下跌 持平 p(1, 1, 1, 1, 1) = π 1 a 11 a 11 a 11 a 11 = 0.5 (0.6) 4 = (2-6) HMM HMM HMM HMM HMM (2.1) HMM (2.2) [57] p( ) B = p( ) (2-7) p( )

27 2 HMM [ 0.1] [ 0.6] [ 0.3] HMM 3 HMM 1 ( ) X 2 Ω 3 A 4 B 5 π a i j 0, b i (x) 0, π i 0 N a i j = 1 j=1 b i (x) dx = 1 N π i = 1 i=1 (2-8) b i (x) i x HMM p ( x t x1 t 1, ) ( ) st 1 = p xt s t (2-9) HMM HMM [58,59] 11

28 2 HMM 2.3 HMM HMM 3 HMM Φ X = {x 1, x 2,..., x T } Φ X p(x Φ) Φ X Φ X S = {s 1, s 2,..., s T } X X Φ p(x Φ) HMM p(x Φ) S X p(x Φ) = p(x, S Φ) = p(s Φ)p(X S, Φ) (2-10) S S HMM p(s Φ) = p(s 1 Φ) T p(s t s t 1, Φ) = π s1 a s1 s 2... a st 1 s T (2-11) t=2 p(x S, Φ) = T b s1 (x 1 )b s2 (x 2 )... b st (x T ) (2-12) t=1 p(x Φ) = π s1 b s1 (x 1 )a s1 s 2 b s2 (x 2 )... a st 1 s T b st (x T ) (2-13) S (2-13) HMM π s1 s 1 b s1 (x 1 ) a st 1 s t s t b st (x t ) s T N HMM T N T O(N T ) (Forward Algorithm) 12

29 2 HMM 状态 1 α t-1 (s 1 ) α t (s 1 ) 状态 2 α t-1 (s 2 ) 状态 3 α t-1 (s 3 ) t-1 t 2.3 α t (i) HMM x1 t t i α t (i) = p(x t 1, s t = i Φ) (2-14) 1 α 1 (i) = π i b i (x 1 ) (2.3) t 1 t i t 1 t i [ N α t (i) = α t 1 ( j)a ji ]b i (x t ) (2-15) j=1 T α p(x Φ) = N α T (i) (2-16) i=1 p(x Φ) O(N 2 T) p(x Φ) X Φ S = arg max S p(x, S Φ) (2-17) S X HMM X 13

30 2 HMM 状态 1 V t-1 (s 1 ) B t (s 1 ) V t (s 1 ) MAX 状态 2 V t-1 (s 2 ) 状态 3 V t-1 (s 3 ) t-1 t 2.4 Viterbi Viterbi Viterbi t i V t (i) V t (i) = p(x t 1, st 1 1, s t = i Φ) (2-18) s t 1 1 t 1 x t 1 1 V t (i) α t (i) α t (i) V t (i) V t (i) t i B t (i) Viterbi V 1 (i) = π i b i (x 1 ) B 1 (i) = V B [ ] V t (i) = max Vt 1 ( j)a ji bi (x t ) 1 j N B t (i) = arg max 1 j N [ Vt 1 ( j)a ji ] (2-19) (2.4) T p(x, S Φ) = max 1 i N V T (i) s T = arg max 1 i N V T(i) s T B T(s T ) B t(i) (Bactracing) S Vitebi HMM Viterbi p(x, S Φ) p(x Φ) Viterbi O(N 2 T) 14

31 2 HMM HMM HMM ( ) HMM HMM HMM HMM HMM p(x Φ) (Expectation-Maximization Algorithm, EM) Baum-Welch (Forward-Bacward Algorithm, FB) HMM (Hidden Variable) (Incomplete Data) (Statistics) EM EM HMM EM (Complete Data) X S X (X, S ) p(x Φ) X S EM S HMM p(x, S Φ) = p(s X, Φ)p(X Φ) (2-20) log p(x Φ) = log p(x, S Φ) log p(s X, Φ) (2-21) X Φ (0) S E[log p(x Φ)] S X,Φ (0) = E[log p(x, S Φ)] S X,Φ (0) E[log p(s X, Φ)] S X,Φ (0) (2-22) 15

32 2 HMM Q(Φ Φ (0) ) = E[log p(x, S Φ)] S X,Φ (0) = p(s X, Φ (0) ) log p(x, S Φ) (2-23) S H(Φ Φ (0) ) = E[log p(s X, Φ)] S X,Φ (0) = p(s X, Φ (0) ) log p(s X, Φ) (2-24) E[log p(x Φ)] S X,Φ (0) = log p(x Φ) = Q(Φ Φ (0) ) H(Φ Φ (0) ) (2-25) (Jenson s Inequality) [60] a i > 0 a i = 1 S a i log x i log a i x i (2-26) i i i H(Φ Φ (0) ) H(Φ (0) Φ (0) ) = p(s X, Φ (0) ) log log S S p(s X, Φ) p(s X, Φ (0) ) p(s X, Φ (0) p(s X, Φ) ) p(s X, Φ (0) ) = log p(s X, Φ) = 0 S (2-27) log p(x Φ) log p(x Φ (0) ) = [Q(Φ Φ (0) ) Q(Φ (0) Φ (0) )] [H(Φ Φ (0) ) H(Φ (0) Φ (0) )] (2-28) Q(Φ Φ (0) ) Q(Φ (0) Φ (0) ) (2-28) log p(x Φ) Q Q log p(x Φ) Q Q(Φ Φ (0) ) Q (Auxiliary Function) Q log p(x Φ) log p(x Φ) Q (Expectation) (Maximization) EM 16

33 2 HMM 状态 1 a 1i a i1 状态 2 a 2i a i2... a Ni 状态 i a in... 状态 N α t (i) β t (i) t-1 t t Baum-Welch Baum-Welch EM HMM HMM Baum-Welch β t (i) = p(x T t+1 s t = i, Φ) (2-29) HMM t i xt+1 T α t(i) β t (i) (2.5) β T (i) = 1 T 1 [ N ] β t (i) = a i j b j (x t+1 )β t+1 ( j) j=1 (2-30) γ t (i, j) HMM X t i j γ t (i, j) = p(s t 1 = i, s t = j x T 1, Φ) = p(s t 1 = i, s t = j, x T 1 Φ) p(x T 1 Φ) = α t 1(i)a i j b j (x t )β t ( j) N α T () =1 17 (2-31)

34 2 HMM EM Q Φ = {π, A, B} (2-23) Q(Φ Φ (0) ) = a s0 s 1 = π s1 S p(x, S Φ (0) ) p(x Φ (0) ) log p(x, S Φ) (2-32) T p(x, S Φ) = a st 1 s t b st (x t ) t=1 T log p(x, S Φ) = log a st 1 s t + t=1 T log b st (x t ) t=1 (2-33) (2-32) Q(Φ Φ (0) ) = Q(a Φ (0) ) + Q(b Φ (0) ) Q(a Φ (0) ) = t i j p(x, s t 1 = i, s t = j Φ (0) ) p(x Φ (0) ) log a i j (2-34) Q(b Φ (0) ) = t i p(x, s t = i Φ (0) ) p(x Φ (0) ) log b i (x t ) (2-35) Q(a Φ (0) ) Q(b Φ (0) ) N j=1 a i j = 1 a i j = 1 p(x Φ (0) ) 1 p(x Φ (0) ) t p(x, s t 1 = i, s t = j Φ (0) ) p(x, s t 1 = i Φ (0) ) t = γ t (i, j) t γ t (i, ) t (2-36) b i (x) O = {o 1, o 2,..., o M } Q(b Φ (0) ) = t i p(x, s t = i Φ (0) ) p(x Φ (0) ) log b i () δ(x t, o ) (2-37) M =1 b i () = 1 b i () = 1 p(x, s p(x Φ (0) t = i Φ (0) ) δ(x t, o ) ) t 1 p(x, s p(x Φ (0) t = i Φ (0) ) ) t = γ t ( j, i) δ(x t, o ) t j γ t ( j, i) t j (2-38) b i (x) 1 b i (x) = exp [ (x µ i) 2 ] 2πσ 2 2σ 2 i i (2-39) 18

35 2 HMM b i (x) dx = 1 b i (x) µ i σ 2 i (2-35) Q(b Φ (0) ) µ i = = t t p(x, s t = i Φ (0) ) log b i (x t ) p(x Φ (0) ) µ i p(x, s t = i Φ (0) ) p(x Φ (0) ) x t µ i σ 2 i (2-40) Q(b Φ (0) ) σ 2 i = = t t p(x, s t = i Φ (0) ) log b i (x t ) p(x Φ (0) ) σ 2 i p(x, s t = i Φ (0) ) (x t µ i ) 2 σ 2 i p(x Φ (0) ) 2σ 4 i (2-41) 0 µ i σ 2 i Q(b Φ (0) ) µ i σ 2 i γ t ( j, i)x t µ i = t j γ t ( j, i) t j (2-42) σ 2 i = γ t ( j, i)(x t µ i ) 2 t j γ t ( j, i) t j (2-43) EM p(x Φ) Baum-Welch 4 1 Φ (0) 2 E- (E-Step) Q(Φ Φ (0) ) 3 M- (M-Step) Q(Φ Φ (0) ) Φ 4 Φ (0) = Φ 2-4 HMM 2.4 HMM HMM HMM HMM [61] 19

36 2 HMM 音素 HMM 静音 HMM 短停顿 HMM 2.6 HMM [ ] [ ] mono-phone sil y u y in sil sil y+u u y+i in sil tri-phone sil sil-y+u y-u+y u-y+in y-in+sil sil (phone) (syllable) (word) (Coarticulation) (Context-Dependent) (Tri-Phone) (Mono-Phone) (2.1) HMM HMM sil sp 3 HMM (2.6) 20

37 2 HMM HMM (Gaussian Mixture Model, GMM) b i (x) = M m=1 c im (2π)D Σ im exp [ 1 2 (x µ im) Σ 1 im (x µ im) ] (2-44) c im i m D x µ im Σ im 8 12 (Decision Tree) [13,14] [62 65] 2.5 HMM HMM HMM HMM HMM HMM HMM HMM 21

38 MLE MLE MLE EM Baum-Welch MLE ( ) MLE MLE MLE HMM ( ) MLE (Objective Function) (Criterion) 20 22

39 3 MLE MLE (Maximum Mutual Information Estimation, MMIE) (Minimum Classification Error, MCE) / (Minimum Word / Phone Error) (Generalized Probability Descent, GPD) Baum-Welch(Extened Baum-Welch, EB) (3.2) (3.3) (3.6) MMIE MCE MWE / MPE (3.7) (3.8) (3.9) (3.10) 3.2 c 1 c M M X M j i e i j C(x) = c i R R(c i x) = M e i j p(c j x) (3-1) j=1 L = R ( C(x) x ) dp(x) (3-2) 23

40 3 e i j e i j = (3-1) 0 i = j 1 i j (3-3) R(c i x) = 1 p(c i x) (3-4) (3-2) C(x) C(x) = arg min R(c i x) = arg max p(c i x) (3-5) i i MAP (Bayesian Ris) MAP p(c i x) = p(x c i) p(c i ) p(x) (3-6) p(x) p(x c i ) p(c i ) p(x c i ) p(x c i ) MLE p(x c i ) ( ) MAP MAP 24

41 3 3.3 MMIE MMIE MMIE [19] Bahl MMIE HMM 2000 MLE 18% MMIE [66] MMIE HMM E-Set 18% MMIE HMM [67] MLE MMIE MMIE [68] MMIE 1000 RM MMIE HMM Normandin [69] TI Digits MMIE HMM 50% MMIE [70] [71,72] MMIE MLE 6 WSJ MMIE 5% 10% [73,74] MMIE W ( ) W O O W O O W 25

42 3 H(W O) H(W O) = W,O p(w, O) log p(w O) = E[log p(w O)] (3-7) Λ p(w O) p(w O) p Λ (W O) H Λ (W O) = E[log p Λ (W O)] = W,O = W,O W,O = H(W O) p(w, O) log p Λ (W O) p(w, O) log p Λ(W O) p(w O) W,O p(w, O) log p(w O) [ pλ (W O) ] p(w, O) p(w O) 1 + H(W O) ( log x x 1) (3-8) H Λ (W O) H(W O) H Λ (W O) H(W O) p Λ (W O) p(w O) H Λ (W O) = H(W O) p Λ (W O) p(w O) I(W; O) I(W; O) = H(W) H(W O) (3-9) H(W) I Λ (W; O) = H(W) H Λ (W O) (3-10) H Λ (W O) I Λ (W; O) MMIE [75] MMIE (Conditional Maximum Lielihood) [76] H Λ (W O) Ĥ Λ (W O) 26

43 3 Ĥ Λ (W O) = 1 R R log p Λ (W r O r ) (3-11) r=1 HMM p Λ (W r O r ) = p Λ (O r W r )p(w r ) p Λ (O r W )p(w ) W M (3-12) F MMIE = 1 R R log p Λ (W r O r ) = 1 R r=1 R log r=1 MMIE p Λ (O r W r )p(w r ) W M p Λ (O r W )p(w ) (3-13) 3.4 MCE MCE MCE Juang [20] MCE MCE MCE MCE TIMIT MCE MLE 5% 14% [77] TI Digits MCE 25% [78,79] [80] MCE 10% MCE [24,81] MCE WSJ0 NAB MCE 5% 10% MMIE 27

44 3 MCE (Minimum Verification Error, MVE) [82,83] MVE MCE MVE MLE 25% [82] MVE MAP 50% MCE MCE MCE [77,84] MCE MCE MCE (Discriminant Function ) c 1 c M M {g i (x) i = 1,..., M} C(x) = arg max g i (x) (3-14) i MAP g i (x) {ĝi (x) i = 1,..., M } = arg min R ( C(x) x ) dp(x) (3-15) g i (x) (3-14) 0-1 R ( C(x) x ) = M i=1 δ ( g i (x) max j g j (x) ) δ(x c i ) (3-16) 28

45 3 (3-15) L = M i=1 δ ( g i (x) max j g j (x) ) δ(x c i ) dp(x) (3-17) MCE (Misclassification Measure) { 1 1/η d i (x) = g i (x) + log exp[g j (x) η]} (3-18) M 1 (Loss Function) j, j i l i (x) = e 2γd i(x)+ξ (3-19) L = M i=1 l i (x) δ(x c i ) dp(x) (3-20) (3-17) (3-20) MCE l i (x) 0-1 δ η γ ξ η η (3-18) d i (x) = g i (x) + max j, j i g j(x) (3-21) i η γ sigmoid (3.1) γ sigmoid γ γ γ sigmoid ξ sigmoid ξ (Margin) margin [85] η γ ξ = 0 (3-20) (3-17) 29

46 损损损损损损损 γ=3 γ=2 γ= γ=1, ξ=0 0.1 γ=2, ξ=0 γ=3, ξ= 自自自 d i (x) 3.1 γ sigmoid (3-20) L emp = R r=1 M l i (x r ) δ(x r c i ) (3-22) i=1 R R MCE (Embedded MCE Training) MCE (String-Model-Based MCE) HMM r W r { 1 g M MCE(O r r ) = log M MCE r g Wr (O r ) = log p Λ (O r W r )p(w r ) (3-23) M MCE r W M MCE r p η Λ (O r W )p η (W )} 1/η (3-24) W r M MCE r = M \ {W r } M MCE r M MCE r η M MCE r g M MCE(O r r ) = log max p Λ (O r W )p(w ) (3-25) W M MCE r nbest MCE η = 1 M MCE r 30

47 3 g M MCE r (O r ) = log W M MCE r p Λ (O r W )p(w ) (3-26) MCE (3-23) (3-26) (3-19) (3-22) MCE F MCE = 1 R R r=1 f (z) ( f log p Λ (O r W r )p(w r ) W M MCE p r Λ (O r W )p(w ) ) (3-27) 1 f (z) = (3-28) 1 + e 2γz (3-19) sigmoid f sigmoid MCE MMIE sigmoid MMIE 3.5 / MWE / MPE MWE / MPE MWE / MPE Povey [22] MPE 256 Switchborad MLE 10% MMIE 1% MWE / MPE [26,86,87] MWE / MPE (Minimum Divergence, MD) [88,89] (Minimum Exact Word Error) [90] [91] MWE / MPE (Discriminative Feature Extraction, DFE) [30,92] fmpe [26,87] MWE / MPE [93] MAP MLE-MAP 31

48 MWE / MPE MWE / MPE MMIE MCE ( ) Λ = arg max Λ E[A(W, W r )] = arg max Λ W M p κ Λ (W O)A(W, W r) (3-29) A(W, W r ) W W r ( ) A(W, W r ) MWE A(W, W r ) MPE p κ Λ (W O) p κ p Λ (W O) = κ (3-29) Λ = arg max Λ κ Λ (O W)pκ (W) W M p κ Λ (O W )p κ (W ) A { [ ] } arg max p(o W)p(W), Wr W M (3-30) (3-31) κ MWE / MPE κ (Acoustic Scaling) A(W, W r ) (Word Arc) (Phone Arc) q q 1 A(q) = 0 1 (3-32) 1 + 2e(q, z) q z A(q) = max (3-33) z 1 + e(q, z) q z 32

49 3 z q e(q, z) z [22,94] (3-30) (3-29) κ MWE / MPE F MWE,MPE = 1 R R W M p Λ (O r W)p(W)A(W, W r ) W M p Λ (O r W )p(w ) r=1 (3-34) 3.6 MMIE MCE (Corrective Training, CT) (Falsifying Training, FT) MMIE MCE CT FT MD [88,89] MD MWE / MPE MD MWE / MPE MD KL (Kullbac-Leibler Divergence, KLD) [95] margin (Large Margin Estimation, LME) [96,97] (Discriminative Margin) [85] (Soft Margin Estimation, SME) [98 100] SVM margin 33

50 3 f (z) M r α G(W, W r ) ML - MMIE z M 1 CT arg max W p(o r W)p(W) MCE 1 M \ {W r } - FT 1 + e 2ρz arg max W,W Wr p(o r W)p(W) MWE MPE exp(z) M 1 δ(w, W r ) A(W, W r ) MD D Λ (W W r ) MMIE MCE MWE / MPE [81,101] Schlüter [24] F Unified = 1 R ( [ W M p α Λ f log (O r W) p α ] (W) G(W, W r ) 1/α ) R W M r p α Λ (O r W ) p α (W ) r=1 (3-35) f (z) M r α G(W, W r ) (3.1)

51 3 (Gradient Descent, GD) Baum-Welch(EB) MCE GPD [20,102,103] MMIE [19] Quic-Prop R-Prop [75,104] MCE GPD (3-20) MCE x t ɛ t ɛ t, t=1 ɛt 2 < (3-36) t=1 Λ t+1 = Λ t ɛ t U l(x t ; Λ) Λ=Λt (3-37) L 1 ( U U ) MCE MCE l(x) x ( ) ɛ GPD GPD MCE 35

52 3 GPD GPD EB GPD EB EB [105] [106] EB D D EB [105,107] HMM Normandin EB HMM [69] HMM EB D [108] EB TIMIT EB [71,72] D [23,69,94,105] EB [109] EB MMIE MCE (3-35) α = 1 G(W, W r ) = δ(w, W r ) ( EB ) F = 1 R R ( [ p Λ (O r W r ) p(w r ) ]) f log W M r p Λ (O r W ) p(w ) r=1 (3-38) [23] F λ = 1 R R ( [ f p Λ (O r W r ) p(w r ) ]) log W M r p Λ (O r W ) p(w ) r=1 T r [γ tr (; W r ) γ tr ()] log p(o rt λ ) λ t=1 (3-39) λ Λ γ tr (; W r ) t γ tr () M r 36

53 3 t F / λ = 0 F S S F S S(Λ, Λ (0) ) = 1 R R ( [ f p log Λ (0)(O r W r ) p(w r ) ]) W M r p Λ (0)(O r W ) p(w ) r=1 T r [γ (0) tr (; W r) γ (0) tr ()] log p(o rt λ ) t=1 (3-40) Λ Λ (0) (0) S F F Λ Λ=Λ (0) = S Λ Λ=Λ (0) (3-41) (Wea-Sense Auxiliary Function) [94] S F 0 (3-40) EM γ S(Λ, Λ (0) ) (Concave) S sm (Λ, Λ (0) ) S(Λ, Λ (0) ) + S sm (Λ, Λ (0) ) [23,69,94] S sm D ( N(µ, σ 2 ) ) S sm (Λ, Λ (0) ) = 1 (D log(2πσ 2 2 ) + D (µ 2(0) + σ 2(0) ) 2D µ (0) µ + D µ 2 ) (3-42) σ 2 S sm (Λ, Λ (0) ) S sm Λ Λ=Λ (0) 37 = 0 (3-43)

54 3 F Λ Λ=Λ (0) = (S + Ssm ) Λ Λ=Λ (0) (3-44) S + S sm F S + S sm EM µ = Γ (O) + D µ (0) (3-45) γ + D σ 2 = Γ (O 2 ) + D (µ 2(0) + σ 2(0) ) γ + D µ 2 (3-46) γ = 1 R R ( [ f p log Λ (0)(O r W r ) p(w r ) ]) W M r p Λ (0)(O r W ) p(w ) r=1 T r [γ (0) tr (; W r) γ (0) tr ()] t=1 Γ (O) = 1 R Γ (O 2 ) = 1 R R ( [ f p log Λ (0)(O r W r ) p(w r ) ]) W M r p Λ (0)(O r W ) p(w ) r=1 T r [γ (0) tr (; W r) γ (0) tr ()] O rt t=1 R ( [ f p log Λ (0)(O r W r ) p(w r ) ]) W M r p Λ (0)(O r W ) p(w ) r=1 T r [γ (0) tr (; W r) γ (0) tr ()] O2 rt t=1 (3-47) (3-48) (3-49) D [23,69,94,105,109] 3.9 MLE CT FT 38

55 3 MCE n-best MLE I/O MLE uni-gram [23,94] MWE / MPE ( i-smoothing [22] ) MLE (Fine Structure) 3.10 MLE MMIE MCE MWE / MPE 39

56 3 40

57 4 MWCE 4 MWCE (MMIE) (MCE) / (MWE / MPE) MWE / MPE [94] MWE / MPE MMIE MMIE (Sub-String Level) (String Level) (word) (syllable) (phone) MWE MPE (Word Error Rate, WER) MMIE HMM MCE [78] MCE WER 1 100% 2 ( ) 50% 0% 0% 41

58 4 MWCE MCE MCE (General MCE Loss Function) [110] MCE(Label-Based Phoneme-Level MCE) [77] MCE(Phone- Discriminating MCE) [84] MCE MCE (Minimum Word Classification Error, MWCE) [111] MCE (Discriminant Function) (Misclassification Measure) (Loss Function) MCE MWCE WER MWCE MCE MCE [77,84,110] MWCE MWCE MWCE MWCE MWCE [23,24,81,101] MWCE MMIE MCE MWE MPE TIMIT WSJ0 MWCE MCE MMIE (4.2) MCE (4.3) 42

59 4 MWCE MCE MWCE (4.4) MWCE TIMIT WSJ0 (4.5) 4.2 MCE MCE MWCE 3 MCE MCE r (3-23) (3-26) d r = g Wr (O r ) + g M MCE r (O r ) { = log p Λ (O r W r )p(w r ) + log 1 M MCE r W M MCE r } 1/η p η Λ (O r W )p η (W ) (4-1) d r sigmoid L(d r ) = e 2γd r+ξ (4-2) (4-2) d r d r < 0 d r 0 L(d r ) 0 d r > 0 d r 0 L(d r ) 1 η γ MCE r L(d r ) 4.3 MWCE MCE (string-level MCE) (WER) 43

60 4 MWCE 分子词图 W 1 W 2 W3 W 0 W 2 W 3 分母词图 W 1 W 2 W 3 W1 W4 W 3 W M K w n r W 0 W 2 W 3 W 1 W 2 W 3 W 1 W 2 W 3 W 5 MWCE MWCE MWCE r N r W r = {w 1 r, w 2 r,..., w N r r } w n r M K w n r MJ w n r W M K w n r, w W, w wn r; W M J w n r, w W, w w n r (4-3) (4-3) w w n r w w n r M K w n r w M J w n r w n r (4.1) (4.2) M K w n MJ r w = M K n r w n MJ r w = M (4-3) n r [ 1 g K (Λ) = log M K w n r W M K w n r 44 p η Λ (O r W) p η (W)] 1/η (4-4)

61 4 MWCE 分子词图 W 1 W 2 W3 W0 W2 W3 分母词图 W1 W2 W 3 W1 W4 W 3 W M J w n r W 1 W 4 W 3 W 1 W 4 W 3 W 5 [ 1 g J (Λ) = log M J w n r W M J w n r p η Λ (O r W ) p η (W )] 1/η (4-5) MCE w n r d w n r = g K (Λ) + g J (Λ). (4-6) sigmoid L(d w n r ) = 1, (4-7) 1 + e 2γ(d w n +ξ) r MWCE (4-7) MWCE w n r w n r MCE MWCE w n r w n r MWCE MCE η γ (4-6) 45

62 4 MWCE d w n r = log max W M K w n r p Λ (O r W) p(w) + log max W M J w n r p Λ (O r W ) p(w ) (4-8) (4-7) 0 d w n L(d w n r ) = r < 0 1 d w n r > 0 (4-9) W w n r M K w M J n r w d n r w n r MWCE d w n r < 0 L = 0 M K w n r M J w W M K n r w (4-3) n r M K w W w n n r r w n r MWCE 0 d w n r > 0 L = 1 M K w n r M J w W M J n r w n r W w n r w n r MWCE 1 MWCE MCE η γ ( MCE MMIE MWE MPE) MWCE MWCE 46

63 4 MWCE α = η γ = η ρ ξ (4-4) (4-5) F MWCE = R N r r=1 n=1 ( f log W M K w n r W M J w n r p α Λ (O r W) p α (W) p α Λ (O r W ) p α (W ) ) (4-10) f (z) MCE ( ) 1 f (z) = (4-11) 1 + e 2ρz (3-27) (4-10) MCE ( ) MWCE MWCE MCE MWCE MCE w n r MWCE MCE HTK 3.4 MMIE MWE MPE [21] MLE HTK EB CPU MCE MWCE 47

64 4 MWCE i-smoothing MMIE MWE MPE i-smoothing τ [21] MCE τ = 0 MWCE [91] τ = 100 α 1/15 ρ 0.04 [24] TIMIT TIMIT MWCE [61,112] MLE (Core- Test) 12 MFCC+ 48 tri-phone 39 [61] HMM GMM MLE 62.76% [112] TIMIT MWE MPE MWCE 5 (4.3) (4.1) MCE MWCE MCE MCE MCE ( ) MWCE MLE MWCE 22.0% MWCE 48

65 4 MWCE MMIE MCE MWE/MPE MWCE 音音音音音 PER(%) 迭迭迭迭 TIMIT (Phone Error Rate, PER) 4.1 MLE MMIE MCE MW(P)E MWCE PER(%) (%) TIMIT (PER) MLE WSJ0 WSJ0 SI Nov 92 (non-verbalized) 5000 ( ) CMN(Cepstral Mean Normalization) 12 MFCC+ MLE [113] ( WSJ0 [114] ) 2774 cross-word tri-phone 8 GMM bi-gram tri-gram MLE 7.34% 4.89% [113] uni-gram tri-gram 5 (4.4) (4.2) MWE MPE ( MPE MWE ) [94] MCE 8.4% 49

66 4 MWCE 词词词词 WER(%) MMIE MCE MWE MPE MWCE 迭迭迭迭 4.4 WSJ0 Nov 92 5 (WER) MLE MMIE MCE MWE MPE MWCE WER(%) (%) WSJ0 Nov 92 5 (WER) MLE MMIE 13.3% MWE 18.2% MPE 16.0% MCE MWCE MLE 22.9% 4.5 MWCE MCE MWCE MWCE MWCE TIMIT WSJ0 MCE MWCE MWCE MMIE MWE MPE 50

67 5 MMIE Trust Region HMM 5 MMIE Trust Region HMM 5.1 GPD EB EB GPD EB EB D [69] D D EB HMM [23,69,94] D EB HMM (Quadratic Function) EB (Unbounded Optimization) i-smoothing EB EB HMM (Concave) (Constraint) 51

68 5 MMIE Trust Region HMM Trust Region MMIE MMIE (Objective Function) (Auxiliary Function) Trust Region Trust Region Trust Region (5.2) MMIE (5.3) KLD Trust Region (5.4) Trust Region (5.5) (5.6) 5.2 MMIE Trust Region (3-13) MMIE F MMIE = 1 F r (Λ; O r, M r ) = 1 [ log p(or Λ; M + r ) log p(o r Λ; M r ) ] R R r r (5-1) M + r = { } W r M r = M log p(o r Λ; M + r ) = log p Λ (O r W r ) p(w r ) log p(o r Λ; M r ) = log p Λ (O r W ) p(w ) (5-2) W M r (5-1) EM log p(o r Λ) Q H log p(o r Λ) = Q r (Λ Λ (0) ) H r (Λ Λ (0) ) (5-3) Q r (Λ Λ (0) ) = [ E log p(or, Λ) Or, Λ (0)] = log p(o r, Λ) p( O r, Λ (0) ) 52 (5-4)

69 5 MMIE Trust Region HMM H r (Λ Λ (0) ) = [ E log p( Or, Λ) Or, Λ (0)] = log p( O r, Λ) p( O r, Λ (0) ) (5-5) Λ (0) Λ log p(o r Λ; M + r ) = Q + r (Λ Λ (0) ) H + r (Λ Λ (0) ) log p(o r Λ; M r ) = Q r (Λ Λ (0) ) H r (Λ Λ (0) ) (5-6) F r (Λ; O r, M r ) F r (Λ; O r, M r ) = [ Q + r (Λ Λ (0) ) H + r (Λ Λ (0) ) ] [ Q r (Λ Λ (0) ) H r (Λ Λ (0) ) ] (5-7) A A r (Λ; O r, M r ) = [ Q + r (Λ Λ (0) ) H + r (Λ (0) Λ (0) ) ] [ Q r (Λ Λ (0) ) H r (Λ (0) Λ (0) ) ] (5-8) F A H H H F r (Λ; O r, M r ) Λ=Λ (0) = A r (Λ; O r, M r ) Λ=Λ (0) (5-9) F r (Λ; O r, M r ) Λ Λ=Λ (0) = A r(λ; O r, M r ) Λ Λ=Λ (0) (5-10) F r (Λ; O r, M r ) = Λ Λ=Λ (0) [ Q + r (Λ Λ (0) ) H r + (Λ Λ (0) ) ] [ Q r (Λ Λ (0) ) Hr (Λ Λ (0) ) ] Λ Λ=Λ (0) 53 (5-11)

70 5 MMIE Trust Region HMM A r 5.1 (0) Λ F r Λ A F H r (Λ Λ (0) ) Λ Λ=Λ (0) = log p( O r, Λ) p( O r, Λ (0) ) Λ Λ=Λ (0) p( O r, Λ (0) ) = p( O r, Λ) p( O r, Λ) Λ=Λ (0) Λ p( O r, Λ) = Λ Λ=Λ (0) = p( O r, Λ) Λ Λ=Λ (0) = 1 Λ Λ=Λ (0) = 0 Λ=Λ (0) (5-12) F r (Λ; O r, M r ) Λ H + r (Λ Λ (0) ) Λ Λ=Λ (0) Λ=Λ (0) = H r (Λ Λ (0) ) Λ Λ=Λ (0) = [ Q + r (Λ Λ (0) ) Q r (Λ Λ (0) ) ] Λ H r + (Λ (0) Λ (0) ) Hr (Λ (0) Λ (0) ) A r (Λ; O r, M r ) = [ Q + r (Λ Λ (0) ) Q r (Λ Λ (0) ) ] Λ Λ=Λ (0) Λ (5-10) = 0 (5-13) Λ=Λ (0) Λ=Λ (0) (5-14) (5-15) (5-9) (5-10) A F A F ( ) ( ) (5.1) A F 54

71 5 MMIE Trust Region HMM X X A r A r (0) Λ F r Λ (0) Λ F r Λ 5.2 A F 5.3 A F 5.3 Trust Region A F ( ) (5.2) (5.3) A F (5.2) ( ) A ( X ) F (5.3) A F F A A F KLD Trust Region KLD KLD 55

72 5 MMIE Trust Region HMM ρ 2 /2 D(λ λ (0) ) ρ2 2 D(λ λ (0) ) = 1 [ (µ µ (0) 2 ) Σ (0) 1 (µ µ (0) ) + tr( Σ Σ 1(0) ) Σ (0) + log Σ ] D (5-16) (5-17) D HMM (5-16) KLD HMM ρ ρ (5-17) KLD KLD Trust Region HMM Σ Σ (0) tr ( Σ Σ 1(0) ) = D log Σ(0) Σ = 0 (5-18) (5-16) KLD [ (µ µ (0) ) Σ (0) 1 (µ µ (0) )] ρ 2 (5-19) [( µ µ (0) ρ µ = Σ (0) 1 2 µ µ (0) = Σ (0) 1 2 ) ( µ µ (0) ] ) 1 (5-20) ρ µ (0) 56

73 5 MMIE Trust Region HMM x = ( µ µ (0) )/ρ X = x 1 x 2. x K (DK 1) (5-21) (5-20) X X 1 ( K HMM ) Trust Region HMM µ µ (0) (5-16) KLD tr ( Σ Σ 1(0) ) log Σ Σ 1(0) D ρ 2 (5-22) σ d σ 1(0) d log(σ d σ 1(0) d ) 1 ρ 2 (5-23) σ d = σ d σ 1(0) d d σ 2 d log σ2 d 1 ρ2 (5-24) d v d = log σ 2 d (5-24) exp(v d ) v d 1 ρ 2 (5-25) d σ d σ (0) d v d 0 exp(v d ) v d = 0 exp(v d ) 1 + v d + v 2 d /2 (5-26) (5.4) exp(v d ) 20 (v d 1.5) 1 1/20 (v d 1.5) 0.5 (5-26) (5-26) (5-25) d v 2 d 2 ρ2 (5-27) 57

74 5 MMIE Trust Region HMM 函函函 exp(v 0.5 d ) 1 + v d + v 2 d / (0) 自自自 v d = log σ d σ d 5.4 exp(v d ) v d = 0 log σ 2 1 ˆΣ = log σ 2 2. log σ 2 D (D 1) (5-28) ˆΣ (0) = log σ 2(0) 1 log σ 2(0) 2. log σ 2(0) D (D 1) (5-29) y = ( ˆΣ ˆΣ (0) )/ 2ρ 2 y 1 Y = y 2. y K (DK 1) (5-30) (5-27) Y Y Trust Region Trust Region Trust Region 58

75 A 5 MMIE Trust Region HMM A C MMIE F MMIE = 1 F r (Λ; O r, M r ) R r 1 A r (Λ; O r, M r ) R r = 1 [ Q + R r (Λ Λ (0) ) Q r (Λ Λ (0) ) ] + C r = 1 log p(o r, Λ) p( O r, Λ (0) ; M + r ) R r log p(o r, Λ) p( O r, Λ (0) ; M r ) (5-31) (5-31) C ( ) log p(o r, Λ) p( O r, Λ (0) ; M + r ) r = r [ log t ] p(o rt, t Λ) p( O r, Λ (0) ; M + r ) + C [ ] = log p(o rt, t Λ) p( O r, Λ (0) ; M + r ) + C r r t [ ] = log p(o rt, Λ) δ(, t ) p( O r, Λ (0) ; M + r ) + C t [ = log p(o rt, Λ) r t ] δ(, t ) p( O r, Λ (0) ; M + r ) + C (5-32) t t δ(, t ) Kronecer δ(, t ) = 0, t 1, = t (5-33) γ rt + = δ(, t ) p( O r, Λ (0) ; M + r ) (5-34) γ + rt t r M+ r 59

76 5 MMIE Trust Region HMM (5-32) log p(o r, Λ) p( O r, Λ (0) ; M + r ) r [ = r t ] log p(o rt, Λ) γ rt + + C = γ rt + log p(o rt, Λ) + C r t (5-31) log p(o r, Λ) p( O r, Λ (0) ; M r ) r = γrt log p(o rt, Λ) + C r t (5-35) (5-36) (5-31) F MMIE 1 R r t ( γ + rt γrt ) log p(ort, Λ) + C (5-37) log p(o rt, Λ) = log { 1 exp [ 1 (2π) D Σ 2 (O rt µ ) Σ 1 (O rt µ ) ] } = 1 2 log Σ 1 2 (O rt µ ) Σ 1 (O rt µ ) + C (5-38) (5-37) F ( = γ + rt γrt)[1 2 log Σ (O rt µ ) Σ 1 (O rt µ ) ] (5-39) r t Trust Region ( ) (5-39) F mean = = = ( γ + rt γrt)[1 2 (O rt µ ) Σ 1(0) (O rt µ ) ] + C r t ( γ + rt γrt)[1 µ O rtσ 1(0) ] µ + C r r t 2 µ Σ 1(0) ( γ + rt γrt)[1 2 µ µ Õ rt µ ] + C t 60 (5-40)

77 5 MMIE Trust Region HMM Õ rt = Σ 1 2 (0) O rt (5.3.2) x = ( µ µ (0) )/ρ F mean ( = γ + rt γrt)[1 2 (ρx + µ (0) ) (ρx + µ (0) ) Õ rt (ρx + µ (0) )] + C r t ( = γ + rt γrt)[1 2 (ρ2 x x ) + ρ( µ (0) Õ rt ) ] x + C r t (5-41) Γ 1 = r Γ 0 1 I D D Γ 0 = r t ρ 2 (γ + rt γ rt ) t ρ(γ + rt γ rt )( µ(0) Õ rt ) (5-42) Γ 0 = Γ 0 2 I D D... (5-43) Γ 0 K I D D (DK DK) Γ 1 1 Γ 1 = Γ 1 2. Γ 1 K (DK 1) ( X ) (5-44) F mean = 1 2 X Γ 0 X + Γ 1 X s.t. X X 1 (5-45) (5-39) F var = = ( γ + rt γrt)[1 2 log Σ (O rt µ (0) r t ( γ + rt γ ) rt r t d ) Σ 1 [1 2 log σ2 d + (O rtd µ (0) 61 2σ 2 d d )2 (O rt µ (0) )] + C ] + C (5-46)

78 5 MMIE Trust Region HMM (5.3.3) y = ( ˆΣ ˆΣ (0) )/ 2ρ 2 d σ 2 d = exp[ 2ρ 2 y d + log σ 2(0) d ] (5-46) F var = ( γ + rt γrt ) r t [ 2ρ 2 2 y d + (O rtd µ (0) d 2σ 2(0) d d )2 exp( 2ρ 2 y d ) ] + C (5-47) Trust Region y d = ( ˆσ 2 d ˆσ2(0) )/ 2ρ 2 0 exp( 2ρ 2 y d ) y d = 0 exp( 2ρ 2 y d ) 1 2ρ 2 y d + ρ 2 y 2 d (5-48) (5-47) F var ( = γ + rt γrt ) = d r t { (Ortd µ (0) d )2 ρ 2 2σ 2(0) d ( γ + rt γrt ) r t y 2 d + 2ρ 2 2 [ 1 ( Ortd µ (0) ) 2 ] } d y σ (0) d + C d (5-49) { 1 2 y diag[ ρ 2 (Õ rt µ (0) ) (Õ rt µ (0) )] y + 2ρ 2 2 [1 (Õ rt µ (0) ) (Õ rt µ (0) )] y } + C Ψ 0 = ( [ γ + rt γrt) diag ρ 2 (Õ rt µ (0) ) (Õ rt µ (0) )] (5-50) r t Ψ 1 = 2ρ 2 2 r t ( γ + rt γ rt) [ 1 (Õrt µ (0) ) (Õ rt µ (0) )] (5-51) Ψ 0 1 Ψ 0 = Ψ (5-52) Ψ 0 K (DK DK) 62

79 5 MMIE Trust Region HMM Ψ 1 1 Ψ 1 = Ψ 1 2. Ψ 1 K (DK 1) (5-53) ( Y ) F var = 1 2 Y Ψ 0 Y + Ψ 1 Y s.t. Y Y 1 (5-54) (5-54) (5-45) Trust Region (5-54) (5-45) [115] 5.1 Trust Region min p R n m(p) = f + g p p Bp s.t. p (5-55) p p λ 0 (B + λi)p = g, λ( p ) = 0, (B + λi) (5-56) 5.2 w λ 1 λ 2 λ 0 < λ 1 < λ 2 p 1 (B + λ 1 I)p 1 = w p 2 (B + λ 2 I)p 2 = w p 1 > p 2 [115] p B p = B 1 g p p Trust Region λ = 0 (5.1) p = B 1 g B p = B 1 g p > p Trust Region (5.2) λ > λ 0 = 0 p = (B + λi) 1 g λ p 63

80 5 MMIE Trust Region HMM p = 0 (5.1) p = (B + λi) 1 g B λ 0 > 0 (B+λ 0 I) (B+λ 0 I) 1 p 0 = (B + λ 0 I) 1 g p 0 < p = p 0 λ > λ 0 λ p p = 0 (5.1) p = (B + λi) 1 g p p (5.1) B (5-45) Γ 0 (5-54) Ψ 0 (B + λ 0 I) 1 (5.2) p λ λ p p = 0 (Binary Search) p 5.5 WSJ0 MMIE Trust Region HMM MLE 4 MMIE EB Trust Region MMIE Trust Region Trust Region ρ ρ (5.5) Trust Region ρ ρ ρ ( ρ = ) ρ ( ρ = ) ρ ( ρ = ) 64

81 5 MMIE Trust Region HMM 仅仅仅仅仅 (ρ=12) 仅仅仅仅仅 (ρ=24) 仅仅仅仅仅 (ρ=48) 仅仅仅仅仅 (ρ=96) 词词词词 WER(%) 迭迭迭迭 5.5 ρ 仅仅仅仅仅 (ρ=12) 仅仅仅仅仅 (ρ=24) 仅仅仅仅仅 (ρ=48) 词词词词 WER(%) 迭迭迭迭 5.6 ρ Trust Region ρ ρ = 12 ρ = ( ρ 2 /2 (5-16) ) Trust Region ρ ρ ρ ρ ρ (5.6) 65

82 5 MMIE Trust Region HMM 输入模型 输入模型 收集统计量 收集统计量 更新均值 更新均值 更新方差 收集统计量 更新方差 合并均值方差 输出模型输出模型 5.7 ( TR TR(ad-hoc) ) Trust Region ( TR ) ( TR(ad-hoc) ) TR Trust Region TR(ad-hoc) TR(ad-hoc) (5.7) Trust Region ρ = 24 EB TR TR(ad-hoc) MMIE MMIE (5.8) Trust Region EB EB Trust Region TR TR Trust Region EB Trust Region EB Trust Region TR 66

83 5 MMIE Trust Region HMM -2 x MMIE 准准 MMIE+EB(i-smth=100) MMIE+TR MMIE+TR(ad-hoc) 迭迭迭迭 ρ = 24 MMIE MMIE+EB(i-smth=100) MMIE+TR MMIE+TR(ad-hoc) 4.6 词词词词 WER(%) 迭迭迭迭 5.9 ρ = 24 TR(ad-hoc) (5.9) Trust Region EB (5.1) MMIE+EB 13.3% Trust Region 27.8% 30.5% EB MWE+EB ( 18.2% ) TR TR(ad-hoc) TR(ad-hoc) TR 67

84 5 MMIE Trust Region HMM 词词词词 WER(%) TR(ad-hoc, ρ=12) TR(ad-hoc, ρ=24) TR(ad-hoc, ρ=48) TR(ad-hoc, ρ=96) 迭迭迭迭 TR(ad-hoc) ρ 5.1 MLE MMIE+EB MMIE+TR MMIE+TR (ad-hoc) MWE+EB WER(%) (%) WSJ0 Nov 92 5 MLE MMIE+TR(ad-hoc) Trust Region ρ ρ (5.10) ρ ρ Trust Region 5.6 EB MMIE Trust Region KLD Trust Region EB Trust Region MMIE EB MMIE+TR WSJ0 68

85 5 MMIE Trust Region HMM MWE+EB 69

86 6 SME 6 SME 6.1 MMIE MCE MWE / MPE MWCE GPD EB Trust Region (Fine Structure) (Empirical Ris) (Expected Ris) (Actual Ris) (Generalization Ability) [23] MCE sigmoid [20] MWE / MPE i-smoothing [22] Trust Region (Support Vector Machine, SVM) margin margin [116] (Large Margin Estimation, LME) [96,97] (Discriminative Margin) [85] (Soft Margin Estimation, SME) [98,99] SME 70

87 6 SME SVM soft margin SME [98] SME ( MCE) [99] SME 1 MLE (within-word tri-phone) (cross-word tri-phone) 2 n-best 3 GPD 4 SME SME (string-level) (word-level) MWE / MPE SME 4 cross-word tri-phone MLE GPD EB SME 3 MCE SME (6.2) SME (6.3) MCE SME (6.4) 71

88 6 SME 6.2 SME SME SME SME MCE SME SME SME [99] l emp (Λ) = 1 l r (O r, Λ) (6-1) R r Λ R l r (O r, Λ) r MCE l r (O r, Λ) = e 2γd r(o r,λ)+ξ (6-2) d r (O r, Λ) γ ξ sigmoid [116] 1 ( ( 2R l test (Λ) l emp (Λ) + VC dim log + 1 ) log δ ) R VC dim 4 (6-3) VC ( VC dim ) (6-3) (6-3) ( R 72

89 6 SME ) (6-3) VC dim [116] VC dim margin VC dim margin (6-3) margin SME [98] L SME (Λ) = λ ρ + l emp(λ) = λ ρ + 1 l r (O r, Λ) (6-4) R ρ soft margin λ ( λ λ ) ρ VC dim λ/ρ (6-4) (6-3) SME r SME l r (O r, Λ) (6-4) ( ) ( margin ) 1 2 margin SME l r (O r, Λ) = ρ d r (O r, Λ) ρ > d r (O r, Λ) 0 (6-5) d r (O r, Λ) (Misclassification Measure) SME (Separation Measure) margin ρ (6.1) ɛ 1 6 margin ρ d r (O r, Λ) ρ d r (O r, Λ) 73

90 6 SME ε 4 ε 5 ε 1 ε 2 ε 3 ε ρ ρ margin 分类面 SME margin ( ) ( 0 < d r (O r, Λ) < ρ ) ( d r (O r, Λ) < 0 ) (6-5) d r (O r, Λ) l r (O r, Λ) = ρ d r (O r, Λ) 0 ρ > d r (O r, Λ) > τ (6-6) τ LME [96] SME LME d r (O r, Λ) < 0 margin ( ρ > d r (O r, Λ) > 0 ) d r (O r, Λ) > 0 SME soft margin 0 > d r (O r, Λ) > τ d r (O r, Λ) SME MCE 74

91 6 SME d r (O r, Λ) = log p Λ (O r W r )p(w r ) W M r p Λ (O r W)p(W) (6-7) W r M r = M \ {W r } (6-7) (6-5) (6-6) (6-4) SME MCE d r (O r, Λ) sigmoid (6-5) (6-6) MCE sigmoid SME SME MWE / MPE MMIE MWCE MCE SME SME l(o r, Λ) = l rt (O rt, Λ) (6-8) t margin P t P t = w,w W r, tw start t tw end p Λ (O r W r )p(w r ) p Λ (O r W)p(W) W M (6-9) t P t t (Word Posterior Probability, WPP) ρ d rt (O rt, Λ) ρ > P t l rt (O rt, Λ) = 0 75 (6-10)

92 6 SME l rt (O rt, Λ) = ρ d rt (O rt, Λ) 0 ρ > P t > τ (6-11) d rt (O rt, Λ) = log w,w W r,t start w t t end w,w W,W W r t start w w p Λ (O r W r )p(w r ) t t end w p Λ (O r W )p(w ) (6-12) t w (6-10) (6-11) (6-10) (6-11) (6-11) τ SME (6-12) (6-10) (6-11) (6-8) SME SME SME 6.3 SME WSJ0 Nov 92 MLE 7077 SI-84 MLE HTK 2818 cross-word tri-phone 8 GMM CMN 12 MFCC tri-gram MLE 5.06% 1 SME [99] bi-gram uni-gram (Re-Scoring) 1/13 1 (Georgia Institute of Technology)

93 6 SME MCE SME_u SME_uc 4.9 词词词词 WER (%) 迭迭迭迭 WSJ0 Nov 92 5 SME (WER) 6.1 MLE MCE SME u SME uc WER(%) (%) WSJ0 Nov 92 5 SME SME MCE GPD EB SME i-smoothing MCE [24] sigmoid γ = 0.04 MCE 4.60% 9.1% [24] 4 SME SME SME (6-5) (6-6) SME u SME uc ρ = 2 τ = 30 MLE SME (6.2) (6.1) 7-10 SME MCE MCE MCE 9.1% SME uc 77

94 6 SME SME (SME u) d r (O r, Λ) 13.2% SME MCE SME MCE [99] SME (6-6) τ SME τ SME d r (O r, Λ) SME (6.3) 10 SME u -10 SME u 0 SME uc SME u τ τ SME SME SME f SME fc (6-10) (6-11) ρ = 0.8 τ = 0.1 MLE SME (6.4) (6.2) SME f 11.9% SME fc 18.8% SME 78

95 6 SME MCE SME_f SME_fc 词词词词 WER (%) 迭迭迭迭 WSJ0 Nov 92 5 SME (WER) MLE MCE SME f SME fc WER(%) (%) WSJ0 Nov 92 5 SME SME SME τ SME f SME u SME uc τ SME fc SME τ SME P t SME (6.5) SME [0, 0.1] [0.8, 1.0] [0.1, 0.9] 0 (6-11) τ (6.5) SME fc [0, 0.1] τ [0.1, 0.8] 0.8 [0.8, 1.0] ρ SME 79

96 6 SME 10 x x SME (SME fc) P t (6.5) SME [0.1, 0.8] [0, 1] SME margin 6.4 SME SME MCE SME MLE (cross-word tri-phone) EB SME SME MCE SME τ SME SME SME SME 80

97 7 MMIE HMM 7 MMIE HMM 7.1 HMM HMM MLE GMM GMM HMM HMM HMM GMM (Gaussian Kernel) ( ) HMM HMM HMM Aaie (Aaie Information Criterion, AIC [117] ) (Bayesian Information Criterion, BIC [118] ) (Minimum Description Length, MDL [119] ) BIC / MDL [120] MMIE MCE 81

98 7 MMIE HMM [121] Normandin MMIE HMM Schlüter [23] HMM HMM BIC / MDL HMM MMIE (7.2) BIC / MDL (7.3) MMIE (7.4) (7.5) 7.2 BIC / MDL MMIE BIC / MDL HMM BIC / MDL [119] J HMM j b j m j b j (O t ) = c j N(O t ; µ j, Σ j ) (7-1) =1 m j c j µ j Σ j BIC / MDL 82

99 7 MMIE HMM j m j BIC m j = arg max m j BIC(m j ) (7-2) BIC BIC(m j ) = γ + jrt log b j(o rt ) 1 2 λ p (#Φ m j ) log[ γ + jrt ] (7-3) γ + jrt r t r t j r t ( ) #Φ m j m j λ p BIC / MDL BIC BIC m j j BIC / MDL 7.3 MMIE MMIE MMIE (3-13) MMIE (5-37) F MMIE = 1 [γ + jrt R γ jrt ] log b j(o rt ) + C j r t = 1 m j (7-4) [γ + jrt R γ jrt ] log[ c j N(O rt ; µ j, Σ j )] + C γ + jrt j r t =1 j r t γ jrt (7-4) γ+ jrt j γ jrt j 83

100 7 MMIE HMM BIC / MDL MMIE (7-4) γ rt MMIE j H j = F MMIE c j = 1 R [γ + jrt γ jrt ] N(O rt ; µ j, Σ j ) r t = 1 [γ rt + R γ rt ]/c j r t m j =1 c jn(o rt ; µ j, Σ j ) (7-5) γ rt (7-5) H j H j 0 γ rt + γ rt H j 0 H j > 0 γ rt + > γ rt H j > 0 H j > 0 H j H j < 0 84

101 7 MMIE HMM γ rt + < γ rt γ rt + 0 γ rt H j < 0 H j H j MMIE 3 1 H j 2 N ( H j ) 3 N ( H j ) HMM s s P s Q s a ps = 0, p P; a sq = 0, q Q a pq = a pq + a 1 ps 1 a a sq, p P, q Q ss (7-6) a s a s (7.1) 85

102 7 MMIE HMM a 14 1 a 13 a a a 23 a a 15 5 a a 13 3 a a 13 3 a 36 6 a a 23 a a 23 a (7.2) s s 1 s 2 s 1 s 2 s 1 s (7.1) MMIE 100 ( ) ( 6209 ) 50 ( 6199 ) HTK 10 HMM 11 3 silence 86

103 7 MMIE HMM MLE MMIE H j H j n = 0 N n < N MLE 7.1 H j i = arg min H j i H j H j i = arg max H j i H j n = n + 1 MMIE short pause (sp) BIC 32 BIC MLE % 5 4 MLE BIC MMIE (7.3) (7.4) 1 2 MMIE BIC 87

104 7 MMIE HMM 平均均均均均均均均基 BIC 准准准准平均均均均基 MMI 准准准准平均均均 句句句句句 (%) 平平平平平平平平平 平均均均均均均均均基 BIC 准准准准平均均均均基 MMI 准准准准平均均均 5 平数数数数 (%) 平平平平平平平平平 2/3 1/2 ( 12 MMIE 4 8 ) 4 (7.2) BIC MMIE BIC MMIE ( ) 88

105 7 MMIE HMM (%) BIC (%) MMIE (%) BIC MMIE (%) (%) MMIE BIC BIC MMIE BIC ( BIC MMIE ) MMIE MMIE (7.3) BIC 16.2% 16.1% MMIE 24.9% 23.9% 7.5 MMIE BIC / MDL MMIE 89

106 7 MMIE HMM 90

107 MWCE 2 Trust Region HMM MMIE 3 SME SME Soft Margin 4 MMIE MWCE MWCE MCE WER MWCE MWE MPE MWCE WSJ0 TIMIT MWCE MMIE Trust Region EB 91

108 8 EB D WSJ0 Trust Region Trust Region EB SME SME SME SME SVM soft margin WSJ0 SME MCE SME MCE τ SME MCE SME MMIE HMM HMM HMM MMIE 8.2 MWCE Trust Region 92

109 8 MMIE Trust Region SME 1 HMM ( ) margin 1 Georgia Institute of Technology SME MWCE WSJ0 Soft Margin Estimation with Various Separation Levels for LVCSR (Jinyu Li, Zhi-Jie Yan, Chin-Hui Lee and Ren-Hua Wang) ICSLP

110 8 94

111 HMM Viterbi HMM γ sigmoid M K w n r W 0 W 2 W 3 W 1 W 2 W 3 W 1 W 2 W 3 W M J w n r W 1 W 4 W 3 W 1 W 4 W 3 W TIMIT (Phone Error Rate, PER) WSJ0 Nov 92 5 (WER) A F A F A F exp(v d ) v d = ρ ρ ( TR TR(ad-hoc) ) ρ = 24 MMIE ρ = TR(ad-hoc) ρ SME WSJ0 Nov 92 5 SME (WER) SME (SME u) d r (O r, Λ) WSJ0 Nov 92 5 SME (WER) SME (SME fc) P t

112

113 TIMIT (PER) MLE WSJ0 Nov 92 5 (WER) MLE WSJ0 Nov 92 5 MLE WSJ0 Nov 92 5 SME WSJ0 Nov 92 5 SME MMIE

114 [1] Davis K H, Biddulph R, Balashe S. Automatic Recognition of Spoen Digits. The Journal of the Acoustical Society of America, 1952, 24(6): [2] Vintsyu T K. Speech Discrimination by Dynamic Programming. Cybernetics and Systems Analysis, 1968, 4(1): [3] Itaura F, Saito S. A Statistical Method for Estimation of Speech Spectral Density and Formant Frequencies. Electronics and Communications in Japan, 1970, 53(A): [4] Lowerre B. The Harpy Speech Understanding System. Morgan Kaufmann Publishers Inc., 1990: [5] Erman L D. Overview of the Hearsay Speech Understanding Research. ACM SIGART Bulletin, 1976, (56):9 16. [6] Klatt D H. Review of the ARPA Speech Understanding Project. The Journal of the Acoustical Society of America, 1977, 62(6): [7] Jeline F, Bahl L, Mercer R. Design of a Linguistic Statistical Decoder for the Recognition of Continuous Speech. IEEE Trans. on Information Theory, 1975, 21(3): [8] Schwartz R, Chow Y, Kimball O, et al. Context-Dependent Modeling for Acoustic-Phonetic Recognition of Continuous Speech. Proceedings of ICASSP1985, Vol. 10, [9] Juang B H, Levinson S, Sondhi M. Maximum Lielihood Estimation for Multivariate Mixture Observations of Marov Chains. IEEE Trans. on Information Theory, 1986, 32(2): [10] Rabiner L. A Tutorial on Hidden Marov Models and Selected Applications in Speech Recognition. Proceedings of the IEEE, 1989, 77(3): [11] Gauvain J L, Lee C H. Maximum a Posteriori Estimation for Multivariate Gaussian Mixture Observations of Marov Chains. IEEE Trans. on Speech and Audio Processing, 1994, 2(2): [12] Leggetter C, Woodland P. Maximum Lielihood Linear Regression for Speaer Adaptation of Continuous Density Hidden Marov Models. Computer Speech and Language, 1995, 9(2): [13] Young S, Odell J, Woodland P. Tree-Based State Tying for High Accuracy Acoustic Modelling. Proceedings of ARPA Worshop on Human Language Technology, [14] Odell J. The Use of Context in Large Vocabulary Speech Recognition[D]. Cambridge University, [15] Lee K F. Large-Vocabulary Speaer-Independent Continuous Speech Recognition: The Sphinx System[D]. Carnegie Mellon University, [16] Chow Y, Dunham M, Kimball O, et al. BYBLOS: The BBN Continuous Speech Recognition System. Proceedings of ICASSP1987, Vol. 12, [17] Murveit H, Cohen M, Price P, et al. SRI s DECIPHER System. Proceedings of Worshop on Speech and Natural Language,

115 [18] Huang X, Acero A, Alleva F, et al. Microsoft Windows Highly Intelligent Speech Recognizer: Whisper. Proceedings of ICASSP1995, Vol. 1, [19] Bahl L R, Brown P F, Souza P V, et al. Maximum Mutual Information Estimation of Hidden Marov Model Parameters for Speech Recognition. Proceedings of ICASSP1986, [20] Juang B H, Katagiri S. Discriminative Learning for Minimum Error Classification. IEEE Trans. on Signal Processing, 1992, 40(12): [21] Young S, et al. The HTK Boo (Revised for HTK version 3.4) [22] Povey D, Woodland P. Minimum Phone Error and I-Smoothing for Improved Discriminative Training. Proceedings of ICASSP2002, Vol. 1, [23] Schlüter R. Investigations on Discriminative Training Criteria[D]. RWTH Aachen University, [24] Macherey W, Haferamp L, Schlüter R, et al. Investigations on Error Minimizing Training Criteria for Discriminative Training in Automatic Speech Recognition. Proceedings of EuroSpeech2005, [25] Hasegawa-Johnson M, Baer J, Borys S, et al. Landmar-Based Speech Recognition: Report of the 2004 Johns Hopins Summer Worshop. Proceedings of ICASSP2005, Vol. 1, [26] Soltau H, Kingsbury B, Mangu L, et al. The IBM 2004 Conversational Telephony System for Rich Transcription. Proceedings of ICASSP2005, Vol. 1, [27] Soltau H, Saon G, Kingsbury B, et al. The IBM 2006 GALE Arabic ASR System. Proceedings of ICASSP2007, Vol. 4, [28] Seide F, Yu P, Ma C, et al. Vocabulary-Independent Search in Spontaneous Speech. Proceedings of ICASSP2004, Vol. 1, [29] Biem A, Katagiri S, Juang B H. Pattern Recognition Using Discriminative Feature Extraction. IEEE Trans. on Signal Processing, 1997, 45(2): [30] Povey D, Kingsbury B, Mangu L, et al. fmpe: Discriminatively Trained Features for Speech Recognition. Proceedings of ICASSP2005, Vol. 1, [31] Yan Z J, Soong F K, Wang R H. Word Graph Based Feature Enhancement for Noisy Speech Recognition. Proceedings of ICASSP2007, Vol. 4, [32] Davis S, Mermelstein P. Comparison of Parametric Representations for Monosyllabic Word Recognition in Continuously Spoen Sentences. IEEE Trans. on Acoustics, Speech, and Signal Processing, 1980, 28(4): [33] Hermansy H. Perceptual Linear Predictive (PLP) Analysis of Speech. The Journal of the Acoustical Society of America, 1990, 87(4): [34] Picone J. Signal Modeling Techniques in Speech Recognition. Proceedings of the IEEE, 1993, 81(9): [35] Hunt M, Lefèbvre C. A Comparison of Several Acoustic Representations for Speech Recognition with Degraded and Undegraded Speech. Proceedings of ICASSP1989, Vol. 1, [36] Kumar N. Investigation of Silicon-Auditory Models and Generalization of Linear Discriminant Analysis for Improved Speech Recognition[D]. Johns Hopins University,

116 [37] Bahl L R, Jeline F, Mercer R L. A Maximum Lielihood Approach to Continuous Speech Recognition. IEEE Trans. on Pattern Analysis and Machine Intelligence, 1983, 5: [38] Katz S. Estimation of Probabilities from Sparse Data for the Language Model Component of a Speech Recognizer. IEEE Trans. on Acoustics, Speech, and Signal Processing, 1987, 35(3): [39] Ney H, Essen U, Kneser R. On Structuring Probabilistic Dependences in Stochastic Language Modelling. Computer Speech and Language, 1994, 8(1):1 38. [40] Viterbi A. Error Bounds for Convolutional Codes and an Asymptotically Optimum Decoding Algorithm. IEEE Trans. on Information Theory, 1967, 13(2): [41] Lee A, Kawahara T, Doshita S. An Efficient Two-Pass Search Algorithm Using Word Trellis Index. Proceedings of ICSLP1998, [42] Jeline F. Fast Sequential Decoding Algorithm Using a Stac. IBM Journal of Research and Development, 1969, 13(6): [43] Bocchieri E. Vector Quantization for the Efficient Computation of Continuous Density Lielihoods. Proceedings of ICASSP1993, Vol. 2, [44] Ney H, Mergel D, Noll A, et al. A Data-Driven Organization of the Dynamic Programming Beam Search for Continuous Speech Recognition. Proceedings of ICASSP1987, Vol. 12, [45] Ney H, Haeb-Umbach R, Tran B H, et al. Improvements in Beam Search for Word Continuous Speech Recognition. Proceedings of ICASSP1992, Vol. 1, [46] Steinbiss V, Tran B H, Ney H. Improvements in Beam Search. Proceedings of ICSLP1994, [47] Schwartz R, Chow Y L. The N-Best Algorithms: An Efficient and Exact Procedure for Finding the N Most Liely Sentence Hypotheses. Proceedings of ICASSP1990, Vol. 1, [48] Soong F, Huang E F. A Tree-Trellis Based Fast Search for Finding the N-Best Sentence Hypotheses in Continuous Speech Recognition. Proceedings of ICASSP1991, Vol. 1, [49] Ney H, Aubert X. A Word Graph Algorithm for Large Vocabulary, Continuous Speech Recognition. Proceedings of ICSLP1994, [50] Mangu L. Finding Consensus in Speech Recognition[D]. Johns Hopins University, [51] Lo W K, Soong F. Generalized Posterior Probability for Minimum Error Verification of Recognized Sentences. Proceedings of ICASSP2005, Vol. 1, [52] Qian Y. Use of Tone Information in Cantonese LVCSR Based on Generalized Character Posterior Probability Decoding[D]. The Chinese University of Hong Kong, [53] Baum L, Eagon J. An Inequality with Applications to Statistical Estimation for Probabilistic Functions of Marov Processes and to a Model for Ecology. Bulletin of American Mathematical Society, 1967, 73: [54] Touda K, Yoshimura T, Masuo T, et al. Speech Parameter Generation Algorithms for HMM-Based Speech Synthesis. Proceedings of ICASSP2000, Vol. 3, [55] Brown P F, Pietra S D, Pietra V J D, et al. The Mathematic of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 1993, 19(2):