2

Size: px

Start display at page:

Download "2"

客韶侃熊
5 years ago
Views:

1 Hao Zhang

2 2

3 Softmax softmax / i

4 ii (conv) (pool) (relu) (fc) Softmax im2col

5 iii

6 iv

7 Chapter 1 (image classification), (categories)(,,,, ),, (label)., (object detection), (object segmentation). 1,,,,., 0 255, (semantic gap).,,,,,.,,,, (data-driven approach). (convolutional neuralnetwork, CNN), [28].,, [8], [22], [15], [36], [32], [5], [14]. : 2 1 (image recognition). 1

8 2 CHAPTER 1. ; 3 softmax ; 4 softmax ; 5 ; 6.

9 Chapter 2, ,., , R, G, B,, = 2, 359, 296, 0( ) 255( ).,.,.,, ,..,..,... 3

CHAPTER 2. 图像识别问题的挑战及数据驱动过程 4 图 2.1: 图像识别问题的挑战. 图像来源于 [7]. 背景融合. 目标可能会和背景混为一体, 使得它非常难以被认出. 类内变化. 相同类的不同个体之间可能会变的完全不同. 2.2 数据驱动过程我们该怎样写出一个算法对图像进行分类呢?

10 CHAPTER 2. 图像识别问题的挑战及数据驱动过程 4 图 2.1: 图像识别问题的挑战. 图像来源于 [7]. 背景融合. 目标可能会和背景混为一体, 使得它非常难以被认出. 类内变化. 相同类的不同个体之间可能会变的完全不同. 2.2 数据驱动过程我们该怎样写出一个算法对图像进行分类呢? 和写出一个排序算法不同, 我们不知道如果通过指定一系列识别规则的方法来识别图像中的目标并且能应对上述的这些挑战. 回想我们人类能对图像内容进行有效的识别, 是因为我们之前已经积累了许多经验, 通过对经验的利用, 从而对新情况做出判断 [37]. 在计算机系统中, 经验通常是以数据的形式存在. 我们将提供给计算机每个类别的许多实例 (examples), 它们组成了训练集 (training set), 利用学习算法 (learning algorithms) 从训练集中产生分类器 (classifier) 或模型 (model). 在面对新情况时 (例如看到一张以前未出现的图像), 模型会提供相应的判断. 这个过程, 叫做数据驱动过程.

11 Chapter 3 softmax,.. 3.1, m, K D = {( x (i), y (i) )} m i=1, (3.1) x (i) R n, i, (3.2) y (i) {0, 1, 2, 3,..., K 1}, i. (3.3), i. x (i) i. n, , n = = y (i) x (i). 3.2, h, (hypothesis), h( x) x, y, 3.1., h. 5

12 6 CHAPTER :. [25] , (hypothesis) h h( x; W, b) = W x + b, (3.4) W R K n, b R K. (3.5) W, b, W (weights), b (bias vector). W, b (ground truth), W, b.,, W, b K h( x; W, b) = W x + b (3.6) = w T 0 w T 1. x + b 0 b 1 w K 1 T b K 1. (3.7)

13 w 0 T x + b 0 w 1 T x + b 1 =. (3.8). w K 1 T x + b K 1 d W. w d T x + b d x d. w d T x, w d (prototype). x, x Softmax Softmax, s = h( x) = W x + b exp(s k) K 1 j=0 exp(s j) Pr(y = k x) = softmax. exp(s k ) K 1 j=0 exp(s [0, 1], k. (3.9) j) 3.3 h, x (i). y (i) ( ),. (loss function), W, b, W, b (maximum likelihood estimate) softmax. W, b = arg max L(W, b) (3.10) W, b = arg max p(d; W, b) (3.11) W, b

14 8 CHAPTER 3. = arg max W, b = arg max W, b = arg max W, b = arg max W, b = arg min W, b = arg min W, b = arg min W, b m p(y (i) x (i) ; W, b) i=1 log ( (iid)) (3.12) m p(y (i) x (i) ; W, b) (3.13) i=1 m log p(y (i) x (i) ; W, b) (3.14) i=1 m log i=1 m log i=1 1 m 1 m def 1 = arg min W, b m def = arg min W, b exp(s (i) y (i) ) K 1 j=0 exp(s(i) j ) ( s (i) = W x (i) + b) (3.15) m log i=1 ( m i=1 exp(s (i) y (i) ) K 1 j=0 exp(s(i) j ) (3.16) s (i) exp(s (i) y (i) ) K 1 j=0 exp(s(i) j ) (3.17) ) K 1 + log( exp(s (i) y (i) j )) (3.18) j=0 m err(w, b; x (i), y (i) ) (3.19) i=1, err(w, b; x (i), y (i) ) = log J(W, b). (3.20) exp(s (i) y (i) ) K 1 j=0 exp(s(i) j ) = s(i) +log( K 1 y (i) j=0 exp(s(i) j )) (hinge loss).,, err(w, b; x (i), y (i) ) err (i) softmax, (W, b) J(W, b) = 1 m m log i=1 exp(s (i) y (i) ) K 1 j=0 exp(s(i) j ) (3.21) (W, b). (W, b ), W b c, c > , (W, b),

15 (regularization)ω(w ). l 2 (norm)( l F ) Ω(W ) = W 2 F = K 1 n 1 W dj. (3.22) d=0 j=0 l 2, x, x. (generalization)., J(W, b) = 1 m = 1 m m log i=1 exp(s (i) y (i) ) K 1 j=0 exp(s(i) j ) + λ 2 W 2 F (3.23) m err (i) + λ 2 W 2 F. (3.24) i=1, λ,, (W, b),. W, b = arg min J(W, b). (3.25) W, b 3.4.1, J (W, b). (W, b), (W, b) J., (θ 0, θ 1 ), J (θ 0, θ 1 ), J(W + c 0 θ 0 + c 1 θ 1, b) (c 0, c 1 ), (W, b), b.,, 3.2. (W, b) (, J(W, b) )., (W, b),

10 CHAPTER 3. 3.2:. [26]. (W, b) J(W, b). 3.2, J,. W b θ, θ K n + K.,. min J( θ). (3.26) θ, v = 1, J, η > 0. η, J : min J( θ + η v). (3.27) v =1 v = arg min J( θ + η v) (3.

16 10 CHAPTER :. [26]. (W, b) J(W, b). 3.2, J,. W b θ, θ K n + K.,. min J( θ). (3.26) θ, v = 1, J, η > 0. η, J : min J( θ + η v). (3.27) v =1 v = arg min J( θ + η v) (3.28) v =1 arg min J( θ) + η v T J( θ) (3.29) v =1 = arg min J( θ) + η v J( θ) cos v, J( θ) (3.30) v =1 = arg min J( θ) + η J( θ) cos v, J( θ) (3.31) v =1

17 = arg min cos v, J( θ) (3.32) v =1 = J( θ) J( θ). (3.33), v J( θ). η. η, η. η,. η = α J( θ). J( θ), J( θ) = 0. θ θ η J( θ) J( θ) = θ α J( θ) J( θ) J( θ) (3.34) (3.35) = θ α J( θ). (3.36) 3.36 (gradient descent, GD), α (learning rate). (W, b) W W α W J(W, b) = W α 1 m m W err (i) αλw, (3.37) i=1 b b α b J(W, b) = b α 1 m m b err (i). (3.38) i=1 3.37, 3.38 (W, b), W, b , 3.38,,, (batch gradient descent). m, ImageNet[3] m,. J J, W W α W err (i) αλw, (3.39) b b α b err (i). (3.40)

18 12 CHAPTER 3. (stochastic gradient descent, SGD). SGD GD.,, GD. SGD,,. SGD [21]., GD SGD, GD m, SGD 1. (mini-batch gradient descent), m, m (batches). SGD,,. 1 m,, m, softmax,. s (i) = W x (i) + b, i, (3.41) exp(s (i) ) err (i) y = log (i) K 1 j=0 exp(s(i) j ) (3.42) K 1 = s (i) + log( exp(s (i) y (i) j )). (3.43) err (i) s (i) j = s (i) j = err (i) = s (i) y (i) s (i)err (i) = K 1 log( j=0 j=0 exp(s (i) j )), j y(i) (3.44) exp(s (i) j ) K 1 k=0 exp(s(i) k ), (3.45) exp(s (i) y (i) ) K 1 k=0 exp(s(i) k ) 1. (3.46) exp( s (i) ) K 1 k=0 exp(s(i) k ) e y (i). (3.47)

19 e y (i) K, y (i) 1, 0. W err (i) = ( s (i)err (i) ) x (i)t, (3.48) b err (i) = s (i)err (i). (3.49) 3.37, 3.38 W, b., softmax 1. Algorithm 1 softmax init. W N(0, ), b = 0 0. for t = 1 to T // T. sample a batch of data {( x (i), y (i) )} m i=1 for i = 1 to m s (i) = h( x (i) ) = W x (i) + b exp(s (i) y (i) ) K 1 j=0 exp(s(i) exp( s (i) ) K 1 k=0 exp(s(i) k // W, b //. err (i) = log // j ) s (i)err (i) = e ) y (i) W err (i) = ( s (i)err (i) ) x (i)t // err (i). b err (i) = s (i)err (i) ) J = 1 m m i=1 err(i) + λ 2 W 2 F //. W J = 1 m m i=1 W err (i) + λw b J = 1 m m i=1 b err (i) W W α W J b b α b J //. 3.5,..

20 14 CHAPTER x, ŷ = arg max k h( x) k = arg max s k. (3.50) k x D test = {( x (i), y (i) )} m test i=1, (3.51) x (i), 3.50 ŷ (i). Acc = 1 m test 1{ŷ (i) = y (i) } (3.52) m test i=1. 1{ }, 1{true statement} = 1, (3.53) 1{false statement} = 0. (3.54), Acc = 1, Acc < 1.

21 Chapter 4 (multilayer feedforward nueral network), (multilayer perceptrons, MLP),..,, / softmax.,,..,,, , (feature engineering) (representation)ϕ( x) x, ϕ( x). 4.3, x, ( ),. 15

22 16 CHAPTER 4.,. ϕ( x),,,. SIFT HOG. SIFT[23], (, DoG) ( DoG, ),,. SIFT,, : SIFT. [23]. HOG[2],,, 4.2. SIFT,. ϕ( x),.,,,.

4.1. / 17 4.2: HOG. [2]. 4.3: ϕ ϕ( x). [19]. 4.1.3 ϕ x,.

23 4.1. / : HOG. [2]. 4.3: ϕ ϕ( x). [19] ϕ x,. ϕ( x) VC (VC dimension), (overfitting). ϕ( ), (kernel engineering). (Gaussian kernel)k( x, x ) = exp( γ x x 2 ),, (learning from data).,

24 18 CHAPTER 4.?,,,, AI. (representation learning). (autoencoder)[1], , ϕ. (deep learning),,. 4.4,. ϕ( x) = ϕ 3 (ϕ 2 (ϕ 1 ( x))) (4.1) : ϕ 1 :. ϕ 2 :. ϕ 3 :.. (connectionism).,.,,, 4.5,., ,, h( x) = W x + b (4.2)

4.2. 19 4.4:. [9]. x ϕ( x). h( x) = W ϕ( x) + b. (4.3) ϕ( x) L 1 ϕ( x) = ϕ L 1 (ϕ L 2 (... ϕ 2 (ϕ 1 ( x)))). (4.4) ϕ l ( a).

25 :. [9]. x ϕ( x). h( x) = W ϕ( x) + b. (4.3) ϕ( x) L 1 ϕ( x) = ϕ L 1 (ϕ L 2 (... ϕ 2 (ϕ 1 ( x)))). (4.4) ϕ l ( a). (affine) z = W (l) a + b (l), l = 1, 2,..., L 1. (4.5) W (l), b (l). (activation function) ϕ l ( a) = max(0, z), l = 1, 2,..., L 1. (4.6) max(0, s) = 1{s > 0}s (4.7)

26 20 CHAPTER :,,. [9]. (rectified linear units, ReLU, relu). s 0 0, 4.6. z..,, h( x) x. relu,, sigmoid

27 , σ(s) = exp( s), (4.8) 4.7. sigmoid relu, relu[18], LSTM[12],. 4.6: ReLU. [11]. 4.7: Sigmoid. [25]. a (0) = x. (4.9) a (l) = max(0, W (l) a (l 1) + b (l) ), l = 1, 2,..., L 1, (4.10) h( x) = W (L) a (L 1) + b (L). (4.11) W (L), b (L) 4.3 W, b. a (L 1) ϕ( x).,.

28 22 CHAPTER , 4.8. (neuron) (dendrites), (nucleus), (axon)., (synapses). 4.8.,,,. x, w, n 1 z = w T x + b = w j x j + b, (4.12) j=0 a = max(0, z) = max(0, w T x + b). (4.13) 4.8:. [7]. n,,,, a k = max(0, z k ) = max(0, w d T x) + b d, d = 0, 1, 2,..., n 1. (4.14) a = a 0 a 1. a n 1 (4.15)

29 max(0, z 0 ) max(0, z 1 ) =. max(0, z n 1) max(0, w 0 T x + b 0 ) max(0, w 1 T x + b 1 ) =. max(0, w n T 1 x + b n 1) = max(0, w T 0 w T 1. x + b 0 b 1 w n T 1 b n 1. (4.16) (4.17) ) (4.18) = max(0, W x + b). (4.19) a (l) = max(0, W (l) a (l 1) + b (l) ), l = 1, 2,..., L 1, (4.20) h( x) = W (L) a (L 1) + b (L). (4.21) L., l ( a (l) ) l + 1,,,, (fullyconnected layer, fc)., L, 4.9. L,, , ( L ).

24 CHAPTER 4. 4.3 4.9:. [7]. J(W, b) = 1 m = 1 m m log i=1 m err (i) + i=1 exp(s (i) y (i) ) K 1 j=0 exp(s(i) j ) + L l=1 L l=1 λ 2 W (l) 2 F (4.22) λ 2 W (l) 2 F. (4.23) s (i) = h( x (i) ).

30 24 CHAPTER :. [7]. J(W, b) = 1 m = 1 m m log i=1 m err (i) + i=1 exp(s (i) y (i) ) K 1 j=0 exp(s(i) j ) + L l=1 L l=1 λ 2 W (l) 2 F (4.22) λ 2 W (l) 2 F. (4.23) s (i) = h( x (i) ). (4.24) h( x (i) ) ,, W (l) W (l) α W (l)j = W (l) α 1 m m W (l)err (i) αλw (l), l, (4.25) i=1 b (l) b (l) α b (l)j = b (l) α 1 m m b (l)err (i), l. (4.26) i=1

31 , W (l)err (i) b (l)err (i). s (i)err (i) = exp( s (i) ) K 1 k=0 exp(s(i) k ) e y (i), (4.27) s (i) = h( x (i) ) = W (L) a (L 1) + b (L). (4.28) W (L)err (i) b (L)err (i). W (L)err (i) = ( s (i)err (i) ) x (i)t, (4.29) b (L)err (i) = s (i)err (i). (4.30) W (l)err (i) b (l)err (i), l < L? W (l)err (i) b (l)err (i) l, a (l)err (i), z (l) = W (l) a (l 1) + b (l), (4.31) a (l) = max(0, z (l) ). (4.32) z (l)err (i) = ( a(l) z (l) )T a (l)err (i) (4.33) = 1{ z (l) > 0} a (l)err (i). (4.34), Hadamard (Hadamard product)., a (l 1)err (i) = ( z(l) a (l 1) )T z (l)err (i) (4.35) = W T z (l)err (i), (4.36) W (l)err (i) = ( z (l)err (i) ) a (l 1)T, (4.37) b (l)err (i) = z (l)err (i). (4.38) a (l)err (i), W (l)err (i), b (l)err (i), a (l 1)err (i). W (l)err (i) b (l)err (i) l. a (l 1)err (i) l 1,

32 26 CHAPTER 4. l 1., l = L,, 1,. (error back-propagation, BP),., , s (i) = h( x (i) ) 4.21.

33 Algorithm 2 for l = 1 to L // W, b 0. init. W (l) N(0, ), b (l) = 0 for t = 1 to T // T. sample a batch of data {( x (i), y (i) )} m i=1 for i = 1 to m a (0) = x (i) for l = 1 to L 1 z (l) = W (l) a (l 1) + b (l) a (l) = max(0, z (l) ) s (i) = h( x (i) ) = W (L) a (L 1) + b (L) exp(s (i) y (i) ) //. //. err (i) = log K 1 // k=0 exp(s(i) k ) s (i)err (i) exp( s = (i) ) e K 1 k=0 exp(s(i) k ) y (i) W (L)err (i) = ( s (i)err (i) ) x (i)t b (L)err (i) = s (i)err (i) a (L 1) = W (L)T s (i)err (i) for l = L 1 to 1 z (l)err (i) = 1{ z (l) > 0} a (l)err (i) W (l)err (i) = ( z (l)err (i) ) a (l 1)T b (l)err (i) = z (l)err (i) // err (i). a (l 1)err (i) = W T z (l)err (i) J = 1 m m i=1 err(i) + L λ l=1 W (l) 2 2 F //. for l = 1 to L W (l)j = 1 m m i=1 W (l)err(i) + λw (l) b (l)j = 1 m m i=1 b (l)err (i) W (l) W (l) α W (l)j b (l) b (l) α b (l)j //.

34 28 CHAPTER 4.

35 Chapter 5 (convolutional neural networks, CNN, ConvNets)., CNN,, BP. CNN, CNN, CNN. 5.1 CNN, CNN, CNN. 1,, CNN (tensor). D = {(X (i), y (i) )} m i=1, (5.1) X (i) R H W D, i, (5.2) y (i) {1, 2, 3,..., K}, i. (5.3), X (i) i, H W D. : W, ; D,. 1, CNN. (natural language processing, nlp), CNN [13]. 29

30 CHAPTER 5. 5.2 CNN,, (,, ). ( L ) 1 1 K,,, 5.1. 5.1: ( ) CNN( ). [7].

36 30 CHAPTER CNN,, (,, ). ( L ) 1 1 K,,, : ( ) CNN( ). [7]. ( +relu ), CNN 4 : (convolutional layer, conv), 2 (pooling layer, pool), (relu), (fc) (conv),. l, x R n l 1. 3 x : a = W x + b (5.4) = w T 0 w T 1. w T n l 1 x + b (5.5) 2. 3, x a (l 1).

37 w 0 T x + b 0 w 1 T x + b 1 =. (5.6). w n T l 1 x + b nl 1 n l w dl, b dl x, w d x. a dl = w T d l x + b dl = w dl x + b dl, d l. (5.7) x R n l 1 X RH l 1 W l 1 D l 1,, W d R H l 1 W l 1 D l 1. a dl = W dl X + b dl = H l 1 1 i=0 W l 1 1 j=0 D l 1 1 d=0 X(i, j, d)w dl (i, j, d) + b dl, d l. (5.8),, X(i, j, d) X (i, j, d).. CIFAR-10[17], , = 3072,., , = 120, 000.,,.,. (1).., a dl W dl (i, j, d) X(i, j, d).,,,,., a dl, 5.2. F 1 F 2 (receptive field),.,,, 5.3.

38 32 CHAPTER :.,. [9]. 5.3:. 32, 3. [7]. ( ),, (activation map), :. [6].

5.2. 33 W dl F 1 F 2 D l 1, W dl,, D l D l. D l, A R H l W l D l, 5.5. D l W dl, W R H F 1 W F 2 D l 1 D l. 5.5:. [7]. (2)..,.,,.,.,., d l i l, j l d l (convolution), 4 W dl (filter) (kernel).

39 W dl F 1 F 2 D l 1, W dl,, D l D l. D l, A R H l W l D l, 5.5. D l W dl, W R H F 1 W F 2 D l 1 D l. 5.5:. [7]. (2)..,.,,.,.,., d l i l, j l d l (convolution), 4 W dl (filter) (kernel). A(i l, j l, d l ) = F 1 1 i=0 F 2 1 j=0 D l 1 1 d= , (cross-correlation). A dl = X W dl + b dl. (5.9) X(i l + i, j l + j, d)w(i, j, d, d l ) + b dl, i l, j l, d l. (5.10)

40 34 CHAPTER :. [9]. A H l W l D l : D l l ; H l W l : H l = H l 1 F 1 + 1, (5.11) W l = W l 1 F (5.12) : D l, F 1 F 2.. (stride)s,, S, 5.7., S. A(i l, j l, d l ) = F 1 1 i=0 F 2 1 j=0 D l 1 1 d=0 X(i l S+i, j l S+j, d)w(i, j, d, d l )+b dl, i l, j l, d l. (5.13)

5.2. 35 5.7:. 1, 2. [7]., 5.10 S = 1. 0- (zero-padding)p., 0, P 1, P 2. 0-,. 0-, A. S = 1, H l = H l 1 F 1 + 2P 1 S W l = W l 1 F 2 + 2P 2 S P 1 = F 1 1 2 P 2 = F 2 1 2 + 1, (5.14) + 1. (5.15), (5.

41 :. 1, 2. [7]., 5.10 S = (zero-padding)p., 0, P 1, P 2. 0-,. 0-, A. S = 1, H l = H l 1 F 1 + 2P 1 S W l = W l 1 F 2 + 2P 2 S P 1 = F P 2 = F , (5.14) + 1. (5.15), (5.16). (5.17) H l = H l 1, (5.18) W l = W l 1. (5.19) (pool),,. pool, (max-pooling). A(i l, j l, d l ) = max X(i l S + i, j l S + j, d l ), i l, j l, d l. (5.20) 0 i<f 1,0 j<f 2 2 2, S = 2, max, 75%, 5.8.

36 CHAPTER 5. 5.8:. [7]. 5.2.3 (relu) relu, relu,.

42 36 CHAPTER :. [7] (relu) relu, relu,. A(i l, j l, d l ) = max(0, X(i l, j l, d l )), i l, j l, d l. (5.21) (fc),,. 5.3, s (i) = h(x (i) ) (5.22) h,, relu,. 5.4, BP.. relu,.

43 A(i l, j l, d l ) = F 1 1 i=0 F 2 1 j=0 D l 1 1 d=0 X(i l + i, j l + j, d)w(i, j, d, d l ) + b dl, i l, j l, d l. (5.23) A err (i), W err (i) b err (i), X err (i). err (i) Hl 1 W(i, j, d, d l ) = = err (i) b dl = = i l =0 H l 1 i l =0 H l 1 i l =0 H l 1 i l =0 err (i) Hl 1 X(i l + i, j l + j, d) = = i l =0 H l 1 i l =0 W l 1 j l =0 W l 1 j l =0 W l 1 j l =0 W l 1 j l =0 W l 1 j l =0 W l 1 j l =0 err (i) A(i l, j l, d l ) A(i l, j l, d l ) W(i, j, d, d l ) (5.24) err (i) A(i l, j l, d l ) X(i l + i, j l + j, d), i, j, d, (5.25) d l, err (i) A(i l, j l, d l ) (5.26) A(i l, j l, d l ) b dl err (i) A(i l, j l, d l ), d l, (5.27) err (i) A(i l, j l, d l ) A(i l, j l, d l ) X(i, j, d) (5.28) err (i) A(i l, j l, d l ) W(i, j, d, d l), i, j, d. (5.29),, ,, l l 1. max. z = max(x, y) = 1{x y}x + 1{y > x}y. (5.30) err(i), err(i) z x err(i) y, err (i) x = err(i) z z x (5.31)

44 38 CHAPTER 5. err (i) y = err(i) 1{x y}, (5.32) z = err(i) z z y (5.33) = err(i) 1{y > x}. (5.34) z, max. err(i) z,, 0. A(i l, j l, d l ) = max X(i l S + i, j l S + j, d l ), i l, j l, d l. (5.35) 0 i<f 1,0 j<f 2, F 1 F 2., l, l 1 0.,. 5.5 CNN,.

45 Chapter 6,,,, Softmax softmax f( s) k = exp(s k ) K 1 j=0 exp(s, k = 0, 1, 2,..., K 1 (6.1) j),, [9]. s j = c, j, f( s) k = : exp(s k ) K 1 j=0 exp(s j) = 1, k. (6.2) K (1). c. exp(c) 0., 0 0, (underflow). f( s) k = exp(c) = 0, (6.3) exp(c) K 1 j=0 exp(c) = 0, k. (6.4) 0 39

46 40 CHAPTER 6. (2). c. exp(c).,, (overflow). f( s) k = softmax f( s) k = = = = exp(c) =, (6.5) exp(c) K 1 j=0 exp(c) =, k. (6.6) exp(s k ) K 1 j=0 exp(s j) (6.7) exp(s k )c K 1 j=0 exp(s, c > 0 j)c (6.8) exp(s k + log c) K 1 j=0 exp(s j + log c) (6.9) exp(s k + c ) K 1 j=0 exp(s j + c ), c. (6.10) softmax,. c = max s l. (6.11) l s, 0,., 1, A(i l, j l, d l ) = F 1 1 i=0 F 2 1 j=0 D l 1 1 d=0 X(i l + i, j l + j, d)w(i, j, d, d l ) + b dl, i l, j l, d l. (6.12), 3 3 for, i l, j l, d l 3 for,, 1 for, 7 for. 7

47 for, 3 3 for, 4 for,., (Fourier transform), f g f g F(f g) = F(f) F(g). (6.13). A dl = X W dl + b dl, d l. (6.14), A dl = F 1 (F(X) F(W dl )) + b dl, d l. (6.15) (fast Fourier transform, FFT), O(N lg N) N, FFT 2. FFT, 3 3 [34]. FFT S > 1, im2col, im2col. :, X. F 1 F 2 D l 1, H l W l, X (H l W l ) (F 1 F 2 D l 1 ). X X [35]: (p, q) X, (i, j, d, d l ) W, (i l, j l, d l ) A, (i l 1, j l 1, d l 1 ) X, i l 1 = i l + i, (6.16)

48 42 CHAPTER 6. j l 1 = j l + j, (6.17) p = i l + H l j l, (6.18) q = i + F 1 j + F 1 F 2 d l 1. (6.19) p A (i l, j l ), q X d l 1 (i, j),. (i, j) (i l, j l ), X (i l 1, j l 1 ). W dl, W. F 1 F 2 D l 1, D l, W (F 1 F 2 D l 1 ) D l. X W, b. Ã = X W + 1 b T. (6.20) Ã (H l W l ) D l. Ã, A RH l W l D l. X, X X. im2col,.,, im2col. 6.3 θ θ α J( θ). (6.21),,., (hyperparameter)α, α., θ θ j. RMSprop[33], v j βv j + (1 β) ( J( θ) θ j ) 2, j, (6.22) θ j θ j α vj + ε J( θ) θ j, j. (6.23)

6.4. 43 v, θ,. v., ;,. β β {0.9, 0.99, 0.999}. ε ε [10 8, 10 4 ], 0.

49 v, θ,. v., ;,. β β {0.9, 0.99, 0.999}. ε ε [10 8, 10 4 ], 0. RMSprop,, Momentum[27], Nesterov Momentum[24], Adagrad[4], Adam[16]. 6.4 (dropout)[30]. p, 0, 6.1.,., (ensemble). 6.1:.. [30]. 6.5,,. ˆX = 1 m X (i), (6.24) m i=1 X (i) X (i) ˆX, i. (6.25)

50 44 CHAPTER , 0, 0. 0,,,,,., 0 0, N(0, ). 6.7,,. [18] (data augmentation).,,., 5 (4 + ),.

51 [1] Yoshua Bengio. Learning deep architectures for ai. Foundations and trends in Machine Learning, 2(1):1 127, [2] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In Computer Vision and Pattern Recognition, CVPR IEEE Computer Society Conference on, volume 1, pages IEEE, [3] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei- Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, CVPR IEEE Conference on, pages IEEE, [4] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12: , [5] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Advances in neural information processing systems, pages , [6] Andrew Ng et al. UFLDL tutorial. Stanford. [7] Li Fei-Fei, Andrej Karpathy, and Justin Johnson. CS231n: Convolutional neural networks for visual recognition. Stanford, [8] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmen- 45

52 46 tation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages , [9] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. Book in preparation for MIT Press, [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arxiv preprint arxiv: , [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pages , [12] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): , [13] Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. arxiv preprint arxiv: , [14] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages , [15] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages , [16] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arxiv preprint arxiv: , [17] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, [18] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages , 2012.

53 47 [19] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553): , [20] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11): , [21] Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages Springer, [22] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages , [23] David G Lowe. Object recognition from local scale-invariant features. In Computer vision, The proceedings of the seventh IEEE international conference on, volume 2, pages Ieee, [24] Yurii Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, volume 27, pages , [25] Andrew Ng. CS229: Machine learning. Stanford. [26] Andrew Ng. Machine learning. Coursera. [27] Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1 17, [28] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3): , [29] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arxiv preprint arxiv: , 2014.

54 48 [30] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1): , [31] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1 9, [32] Yaniv Taigman, Ming Yang, Marc Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages , [33] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURS- ERA: Neural Networks for Machine Learning, 4:2, [34] Nicolas Vasilache, Jeff Johnson, Michael Mathieu, Soumith Chintala, Serkan Piantino, and Yann LeCun. Fast convolutional nets with fbfft: A gpu performance evaluation. arxiv preprint arxiv: , [35] Jianxin Wu. CNN for dummies [36] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In Advances in neural information processing systems, pages , [37].., 2016.

RPN 2 DeepParts 22 part pool 2 3 HOG 12 LBP 13 Harr - like 15 DPM 16 Deformable Parts Model VGG16 X. Wang 14 VGG Convolutiona

RPN 2 DeepParts 22 part pool 2 3 HOG 12 LBP 13 Harr - like 15 DPM 16 Deformable Parts Model VGG16 X. Wang 14 VGG Convolutiona 2017 22 2 CHINESE JOURNAL OF STEREOLOGY AND IMAGE ANALYSIS Vol. 22 No. 2 June 2017 209 1007-1482 2017 02-0209 -0215 DOI 10. 13505 /j. 1007-1482. 2017. 22. 02. 014 RPN 1 2 3 3 1 1 2 3 1. 100090 2. 100090