97 7 DSP Otsu 24.39% 25.54% 6, 9, S 0, D, 8, B 82.92% 80.85% 96% XScale-PXA270 624 Mhz / Linux 7 ~ 8 fps I
Embedded Real-Time Multiple Vehicle License Plate Recognition System Student : Chih-Wen Lee Advisor : Mon-Chau Shie ABSTRACT In Taiwan, density of vehicles is come out top in the world. However, it is short of embedded recognition system for license plates of cars and motorcycles in domestic study. In this paper, we proposed a DSP-less embedded system of Real-Time multi-target recognition without professional video camera to solve license plate images in conditions of low-contrast, blur, or skew. The system is composed of three modules: localization, processing of vehicle plates, and character recognition. For the unclear vehicle plate images, we proposed a method which is suitable to binarize the license plate. Compared with localized Otsu s method of 3 3 block processing, our method achieved 24.39% improvement for cars license plates recognition accuracy, and 25.56% improvement for motorcycles license plates. To improve template matching, we separated the character templates into some subsets by character s outline, and could effectively avoid miss-recognition of the characters 6, 9, S. Moreover, by calculating the ratio of left and right side s background pixels of character s contour, the characters 0, D, 8, B could be authentically recognized. The experimental videos were captured upon a sky bridge on Roosevelt Road, Taipei. The accuracies of car s and motorcycle s license plate recognition are 82.92% and 80.85%, and the accuracy of localization is 96%. The performance on embedded system (XScale-PXA270 624 MHz / Linux) is 7~8 fps. Keyword: license plate recognition, embedded, real-time, license plate binarization II
III
ABSTRACT.. i. ii iii... iv...viii... xi... 1 1.1...1 1.2...1 1.3...2 1.4...2 1.4.1...3 1.4.2...4 1.5...8 1.6...8... 10 2.1 ( License Plate Localization )...10 2.1.1 ( Texture Feature )...10 2.1.2 ( Color Feature )... 11 2.2 ( Edge Detection )... 11 2.2.1... 11 2.2.2...12 2.3 ( Binarization )...13 2.3.1 ( Fixed Threshold )...13 IV
2.3.2 ( Average Threshold )... 14 2.3.3 ( Statistical Threshold )...14 2.4 (Character Segmentation)...16 2.4.1 (Projection Histogram)...16 2.4.2...17 2.4.3 ( Connected Components Labeling )...17... 19 3.1 (Preprocessing)... 20 3.2 ( License Plate Localization )...21 3.2.1 ( Edge Detection )...21 3.2.1.1...21 3.2.1.2... 22 3.2.1.3...23 3.2.2...23 3.2.2.1... 23 3.2.2.2...24 3.2.3...25 3.2.3.1...25 3.2.4...27 3.2.4.1... 27 3.2.4.2... 29 3.2.4.5... 32 3.3...32 3.3.1... 33 3.3.2...34 3.3.3 ( Binarization )... 36 V
3.3.3.1 Otsu...36 3.3.3.2...37 3.3.3.3...40 3.3.4...40 3.3.5 ( Character Segmentation )...41 3.3.6...42 3.3.6.1 ( Skew Correction )...43 3.3.6.2...44 3.4 ( Character Identification )...45 3.4.1 ( Template Matching)...46 3.4.2...47 3.4.3...48 3.4.4... 49 3.4.4.1 0, D, 8, B... 50 3.4.4.2 Y, V...52... 53 4.1...53 4.2...53 4.3...54 4.4...56 4.5... 58 4.6 Otsu...59 4.7...60 4.8 Otsu...60 4.9... 61 VI
... 65....66 VII
1.1 Panasonic VDR-D200... 3 1.2 HP ipaq hx4700... 3 1.3... 4 1.4... 5 1.5... 6 1.6... 8 2.1 AUSTRIA... 10 2.2... 10 2.3 ( 4-neighborhood )... 17 2.4 ( 8-neighborhood )... 18 3.1 ( Interlaced effect )... 20 3.2 ( Deinterlaced effect)... 20 3.3... 21 3.4 8... 22 3.5... 22 3.6 1... 24 3.7 3.5(b)... 24 3.8 3.7... 25 3.9... 25 3.10... 26 3.11... 26 3.12... 27 3.13... 28 3.14... 29 VIII
3.15... 30 3.16... 32 3.18... 33 3.19... 34 3.20... 35 3.21... 35 3.22... 36 3.23 3.22 Otsu... 36 3.24 3.22 3x2 Otsu... 37 3.25 3x4 Otsu... 37 3.26... 39 3.27... 39 3.28... 40 3.29... 40 3.30... 42 3.31... 43 3.32... 44 3.33... 44 3.34... 45 3.35... 45 3.36... 46 3.37 (a) f (b) g (c) f g... 47 3.38... 48 3.39... 50 3.40... 50 3.41 8, B... 51 IX
3.42 0, D... 51 3.42 Y,V... 52 4.1... 54 4.2... 55 4.3... 55 4.4... 56 4.5... 57 4.6... 57 4.7 (1)... 58 4.8 (1)... 58 4.9 (2)... 59 4.9 (1)... 62 4.10 (2)... 62 4.11 (1)... 63 4.12 (2)... 63 4.13... 64 X
1.1 Panasonic VDR-D200... 3 1.2 HP ipaq hx4700... 4 1.3... 7 2.1... 12 3.1... 49 4.1... 54 4.2... 56 4.3 Otsu... 59 4.4... 60 4.5 Otsu... 61 XI
1.1 1.2 1
1.3 1. 2. 3. DSP 4. 5. 1.4 2
1.4.1 Panasonic VDR-D200 F 1/6 CCD (68 ) D200: F1.8 ( ) / F3.9 ( ) 30x 1.9-57mm 30.5mm 1.1 Panasonic VDR-D200 HP ipaq hx4700 3
Intel XScale-PXA270 (ARMv5TE) at 624 Mhz 64 MB Mobile SDRAM WLAN Bluetooth IrDA VGA (480x640) transfective AK4641 audio codec 1.2 HP ipaq hx4700 Ångström embedded Linux distribution OpenCV 1.4.2 9 25X 50 704x480 4
20 60 70 5
1. 135 ~ 155 2. 100 ~ 115 100~160 100 155 6
[4] 250cc 250mm x 140mm 35mm x 65mm 320mm x 150mm 45mm x 90mm 1.3 7
1.5 CCD ( Interlaced ) 1.6 8
9
2.1 ( License Plate Localization ) 2.1.1 ( Texture Feature ) ( Morphology ) DWT [6] [2-5] [7-8] [11-14] [1, 15-19] 10
2.1.2 ( Color Feature ) HSI ( neural-network classifier ) [20-21] genetic [22] fuzzy aggregator [23] 2.2 ( Edge Detection ) 2.2.1 Roberts Prewitt Sobel Roberts 0 +1 1 0 1 +1 0 0 1 0 +1 0 0 1 +1 0 Prewitt 1 0 +1 1 0 +1 1 0 +1 1 1 1 0 0 0 +1 +1 +1 11
Sobel 1 0 +1 2 0 +2 1 0 +1 1 2 1 0 0 0 +1 +2 +1 2.1 Sobel A Gx Gy : Gx = 1 0 +1 2 0 +2 1 0 +1 * A and Gy = +1 +2 +1 0 0 0 1 2 1 * A (2-1) g(x, y) = f(x+1, y+1) + 2 x f(x+1, y) + f(x+1, y-1) f(x-1, y+1) 2 x f(x-1, y) f(x-1, y-1) (2-2) 2.2.2 [-1, +1] 12
[5] [-1, 0, 1] [7] [-1, 0, 0, 1] Sobel Prewitt 3x3 2.3 ( Binarization ) ( Threshold Value ) ( Adaptive ) Otsu 2.3.1 ( Fixed Threshold ) 255 ( ) 0 ( ) f(x, y) TH G G(x, y) = 1 if f(x, y) > 0 if f(x, y) < (2-3) 13
2.3.2 ( Average Threshold ) [7] =0.88 TH*= TH W H G T = (, ) (2-3) 2.3.3 ( Statistical Threshold ) ( Minimum within-group variance ) Otsu [24] 1979 ( Class ) ( Variance ) Otsu M x N 0 ~ 255 i 14
P(i) = (2-4) k C 0 C 1 C 0 0 ~ k C 1 k+1 ~ 255 W 0 W 1 W = Pr(C ) = P (2-5) W = Pr(C ) = P = 1 W (2-6) (mean) = (2-7) = (2-8) σ = (i μ ) (2-9) σ = (i μ ) (2-10) σ = W x σ + W x σ (2-11) 15
k k Otsu Otus 2.4 (Character Segmentation) [6] (Projection Histogram) (Connected Components Labeling) 2.4.1 (Projection Histogram) 16
2.4.2 2.4.3 ( Connected Components Labeling ) 17
18
1. CCD 2. 3. 4. Otsu 24.39% 25.54% 19
3.1 ( Preprocessing ) ( Interlaced effect ) CCD ( Progressive-scan ) ( Interlaced-scan ) frame rate 30 fps 1/60 f f f f f f f (x, y) = f (x, y) (3-1) 20
3.2 ( License Plate Localization ) 3.2.1 ( Edge Detection ) 3x3 Roberts [-1, 0-1] (3-2) g(x, y) = f(x+1, y) - f(x-1, y) = f(x, y) * [ -1, 0, -1] (3-3) 21
Sobel Prewitt 704x330 frame 2x702x330 Roberts [-1, 0, 1] 22
( VALID_T ) 24 VE VE(x, y) = 1 if f(x + 1, y) f(x 1, y) > VALID_T 0 otherwise (3-4) 3.2.2 CONN_W CONN_T E x, y E(x, y) CONN_W > CONN_T (3-5) 1, I, 7, L, F CONN_W 0.75 23
1.5 CONN_C CONN_W 1/2 1.5 1/4 5 4 1 7 L 24
3.2.3 1:2 1. 2. 3. 4. WIDTH HEIGHT WIDTH/HEIGHT) >= 2 WIDTH/HEIGHT) <= 6 25
WIDTH >= 95 WIDTH <= 350 (3-6) 10 26
3.2.4 1. 2. 27
( 10 ) (3-7) 28
3.14 1. 7 2. L 7 L 54 115 100 x 20 29
12 30
( 3.10 ) 155 160 WIDTH * HEIGHT 14.2% * WIDTH ~ 16.3% * WIDTH 19.1% * WIDTH ~ 22% * WIDTH 7 L 1/4 3/4 31
(a) 7 (b) L 45mm x 90mm 35mm x 65mm 1. 45mm x 90mm 2. 35mm x 65mm 1 : 1.28 1 : 1.38 1.5 3.3 32
3.3.1 [-1 0 1] [0, 0, 0, 255, 255, 255] (3-11) [0, 0, 255, 255, 0, 0] (3-12) 2 33
4 1, L, 7, C 4 8 3/5 8 24 Y H, 8, B 3.3.2 CCD 34
64 g(x, y) g(x, y) 64 2 32 g(x, y) 5 2 200 (3-13) 200 2 1 (a) (b) 35
3.3.3 ( Binarization ) Otsu Otsu ( ) Otsu Otsu Otsu 3x2 36
Otsu 3.25 D D D [-1, 0, 1] 37
G VE FORE_T BACK_T COUNT VALID_T VE(x, y) = 1 if f(x + 1, y) f(x 1, y) > VALID_T 0 otherwise (3-14) 4*2 8 count VE FORE_T BACK_T FORE_T = G(x 1, y) if VE(x, y) = 1 (3-15) BACK_T = G(x + 1, y) if VE(x, y) = 1 (3-16) 38
FORE_T BACK_T 20% FORE_T = FORE_T + ((BACK_T - FORE_T)*0.2) (3-17) B B(x, y) = 1 if G(x, y) < FORE_T 0 otherwise (3-18) 39
Otsu 23.65% 23.41% 1 3.3.4 6 2 40
G WIDTH_TH X = WIDTH_TH G X = - WIDTH_TH G (3-19) 3.3.5 ( Character Segmentation ) 1.86 1. 22 <= <= 40 2. >= 6 3. / >=1.3 41
3.3.6 42
hdistance vdistance [1] ratio = hdistance / vdistance (3-20) x = x + y/ratio y = y - x/ratio (3-21) x y 43
4 6 < 6 >= 1.5 44
Y g y1 y2 g y1 y2 Y (3-22) 3.4 ( Character Identification ) ( Template Matching ) 45
3.4.1 ( Template Matching) ( Character Normalization ) 19x35 NMI Normalize Match Index [26] g f g f M+ 46
M- M+ = f(x, y) g(x, y) = 0 M- = f(x, y) g(x, y)!= 0 NMI = ( ) ( ) ( ) ( ) (3-23) NMI (a) (b) (c) 0.77 = ( ) ( ) ( ) ( ) (3-24) NMI 3.4.2 NMI 47
6, 9, S, E, L, F, 8, B, 0, D 0, D, 8, B 6, 9, S 6 9 S NMI 3.4.3 17 6, 9, S, E, F, L 6, 9, S,,E F 2, 4, 7, M, N 48
0 A, K, N, X, M, R 1 C, E 2 F 3 6, G 4 H, K, M, N, X 5 3, 7, J 6 K, N, U, V, W, X, Y 7 L 8 0, D 9 7, P a 5, S b 4, 9 c 2, Z d 2, 7 e 4, 7, 8, B, T, K, N, Q, V, W, X, Y f 1, I >= 3 g A ~ Z, 0~9 3.1 3.4.4 8, B, 0, D, Y, V 49
8 5 b < a b < c d < c d <e B 1.5 50
0 B, D 8, 0 100 8, B 13% 7% 2% 0 D 100 13% 2% 1% 51
Y, V 3/4 52
Otsu 4.1 Intel XScale-PXA270 624 Mhz ARM Linux OpenCV 120ms 10ms ~ 20ms 4.5 4 190 ms 40 ms / frame 4.2 3 frame 5 3 4 3 2 53
4.3 4 3 82 47 68 38 82.92% 80.85% 84.44% 78 44 87.17% 86.36% 4.1 54
4.3 (b), (f) 4.3 (c), (e) 47 2 55
2 38/45 x 100% = 84.44% 4.4 2 3 4 5 8 92 54 89 52 54 96.73 96.29 100% 4.2 56
100. (a) (b) ( c) 4.6 (b) C, 7 4.6 (c) frame 57
4.5 3 4 190ms 4-1 120ms 10ms ~ 20ms 58
8 10ms 190ms 4.6 Otsu Otsu [1, 2, 4, 27-29] Otsu Otsu 82 47 48 68 26 38 58.53% 82.92% 55.31% 80.85% 66 78 33 44 72.72% 87.17% 78.78% 86.36% 4.3 Otsu 59
Otsu 24.39% 25.54% 4.7 82 47 62 68 38 38 75.60% 82.92% 80.85% 80.85% 73 78 44 44 84.93% 87.17% 86.36% 86.36% 7.32% / 2.24% 0% 4.4 7.32% 4.8 Otsu 60
3x3 Otsu 82 47 43 48 26 26 52.43% 58.53% 55.31% 55.31% 63 66 33 33 68.25% 72.72% 78.78% 78.78% 6.1% / 4.47% 0% 4.5 Otsu 6.1% 4.4 1.22% 4.9 61
62
63
17 64
6, 9, S 0, D, 8, B XScale-PXA270 DSP 96% 80% 100ms 30ms 5 4 fps Adaptive 65
[1] 2006 [2] 2000 [3] 2005 [4] 2005 [5] 2003 [6] 2003 [7] 2000 [8] 2003 [9] Clemens Arth, Florian Limberger, Horst Bischof, Real-Time License Plate Recognition on an Embedded DSP-Platform, Computer Vision and Pattern Recognition, 2007. CVPR '07. IEEE Conference on, pp. 1-8. [10] J. S. Kang, M. H. Kang, C. H. Park, J. H. Kim, Y. S. Choi, Implementation of Embedded System for Vehicle Tracking and License Plates Recognition using Spatial Relative Distance, Information Technology Interfaces, 2004. 26th International Conference on, vol.1, 2004, pp. 167-172. [11] S.-Z. Wang and H.-J. Lee, Detection and recognition of license plate characters 66
with different appearances, Intelligent Transportation Systems, 2003. Proceedings. 2003 IEEE, vol. 2, 2003, pp. 979-984. [12] S. Kim, D. Kim, Y. Ryu, and G. Kim, A robust license-plate extraction method under complex image conditions, Pattern Recognition, 2002. Proceedings. 16th International Conference on, vol. 3, 2002, pp. 216-219. [13] H. Bai, J. Zhu, and C. Liu, A fast license plate extraction method on complex background, Intelligent Transportation Systems, 2003. Proceedings. 2003 IEEE, vol. 2, 2003, pp. 985-987. [14] H. Bai and C. Liu, A hybrid license plate extraction method based on edge statistics and morphology, Pattern Recognition, 2004. ICPR 2004. Proceedings of the 17th International Conference on, vol. 2, 2004, pp. 831-834. [15] V. Kamat and S. Ganesan, An efficient implementation of the Hough transform for detecting vehicle license plates using DSP S, Real-Time Technology and Applications Symposium, 1995. Proceedings, 1995, pp. 58-59. [16] S. Gendy, C. L. Smith, and S. Lachowicz, Automatic car registration plate recognition using fast Hough transform, Security Technology, 1997. Proceedings. The Institute of Electrical and Electronics Engineers 31st Annual 1997 International Carnahan Conference on, 1997, pp. 209-218. [17] Jianlong Zhu, Yannan Zhao, Vehicle license image segmentation using wavelet transform, Intelligent Multimedia, Video and Speech Processing, 2001. Proceedings of 2001 International Symposium on, 2001, pp. 267 270. [18] Ching-Tang Hsieh, Yu-Shan Juan, Kuo-Ming Hung, Multiple license plate detection for complex background, Advanced Information Networking and Applications, 2005. AINA 2005. 19th International Conference on, vol. 2, 2005, pp. 389-392. [19] Huiqiong Chen, Derek Rivait and Qigang Gao, Real-Time License Plate I 67
dentification by Perceptual Shape Grouping and Tracking, Intelligent Transportation Systems Conference, 2006. ITSC '06. IEEE, pp.1352-1355. [20] W. Wei, M. Wang, and Z. Huang, An automatic method of location for number-plate using color features, Image Processing, 2001. Proceedings. 2001 International Conference on, vol. 1, 2001, pp. 782-785. [21] S. H. Park, K. I. Kim, K. Jung, and H. J. Kim, Locating car license plates using neural networks, Electronics Letters, vol. 35, 1999, pp. 1475-1477. [22] S. K. Kim, D. W. Kim, and H. J. Kim, A recognition of vehicle license plate using a genetic algorithm based segmentation, Image Processing, 1996. Proceedings, International Conference on, vol. 1, 1996, pp. 661-664. [23] Shyang-Lih Chang, Li-Shien Chen, Yun-Chung Chung, and Sei-Wan Chen, Automatic license plate recognition, Intelligent Transportation Systems, IEEE Transactions on, vol. 5, Issue 1, 2004, pp. 42-53. [24] N. Otsu, A Threshold Selection Method from Gray-Level Historgams, IEEE Transactions on System, Man, and Cybernetics, vol. SMC-9, 1979, pp. 62-66. [25] Rafael C. Gonzalez and Richard E. Woods, Digital Image Processing, 2nd Editio, Prentice-Hall, New Jersey, 2002 [26] J. R. Parker, Algorithms for Image Processing and Computer Vision, Wiley; Pap/Cdr edition, November 25, 1996 [27] Naito, T.; Tsukada, T.; Yamada, K.; Kozuka, K.; Yamamoto, S, Robust license-plate recognition method for passing vehicles under outside environment, Vehicular Technology, IEEE Transactions on, vol. 49, Issue 6, 2000, pp. 2309-2319 [28] Hongliang Bai; Junmin Zhu; Changping Liu, A fast license plate extraction method on complex background, Intelligent Transportation Systems, 2003. 68
Proceedings. 2003 IEEE, vol. 2, 2003, pp. 985-987 [29] Naito, T.; Tsukada, T.; Yamada, K.; Kozuka, K.; Yamamoto, S, Robust recognition methods for inclined license plates under various illumination conditions outdoors, Intelligent Transportation Systems, 1999. Proceedings. 1999 IEEE/IEEJ/JSAI International Conference on, 1999, pp. 697-702 [30] The Ångström Distribution, http://www.angstrom-distribution.org/ [31] Open Source Computer Vision Library, http://www.intel.com/technology/computing/opencv/ 69