Embedded Real-Time Multiple Vehicle License Plate Recognition System Student : Chih-Wen Lee Advisor : Mon-Chau Shie ABSTRACT In Taiwan, density of veh

Size: px

Start display at page:

Download "Embedded Real-Time Multiple Vehicle License Plate Recognition System Student : Chih-Wen Lee Advisor : Mon-Chau Shie ABSTRACT In Taiwan, density of veh"

徽程
6 years ago
Views:

1 97 7 DSP Otsu 24.39% 25.54% 6, 9, S 0, D, 8, B 82.92% 80.85% 96% XScale-PXA Mhz / Linux 7 ~ 8 fps I

2 Embedded Real-Time Multiple Vehicle License Plate Recognition System Student : Chih-Wen Lee Advisor : Mon-Chau Shie ABSTRACT In Taiwan, density of vehicles is come out top in the world. However, it is short of embedded recognition system for license plates of cars and motorcycles in domestic study. In this paper, we proposed a DSP-less embedded system of Real-Time multi-target recognition without professional video camera to solve license plate images in conditions of low-contrast, blur, or skew. The system is composed of three modules: localization, processing of vehicle plates, and character recognition. For the unclear vehicle plate images, we proposed a method which is suitable to binarize the license plate. Compared with localized Otsu s method of 3 3 block processing, our method achieved 24.39% improvement for cars license plates recognition accuracy, and 25.56% improvement for motorcycles license plates. To improve template matching, we separated the character templates into some subsets by character s outline, and could effectively avoid miss-recognition of the characters 6, 9, S. Moreover, by calculating the ratio of left and right side s background pixels of character s contour, the characters 0, D, 8, B could be authentically recognized. The experimental videos were captured upon a sky bridge on Roosevelt Road, Taipei. The accuracies of car s and motorcycle s license plate recognition are 82.92% and 80.85%, and the accuracy of localization is 96%. The performance on embedded system (XScale-PXA MHz / Linux) is 7~8 fps. Keyword: license plate recognition, embedded, real-time, license plate binarization II

3 III

4 ABSTRACT.. i. ii iii... iv...viii... xi ( License Plate Localization ) ( Texture Feature ) ( Color Feature ) ( Edge Detection ) ( Binarization ) ( Fixed Threshold )...13 IV

5 2.3.2 ( Average Threshold ) ( Statistical Threshold ) (Character Segmentation) (Projection Histogram) ( Connected Components Labeling ) (Preprocessing) ( License Plate Localization ) ( Edge Detection ) ( Binarization ) V

6 Otsu ( Character Segmentation ) ( Skew Correction ) ( Character Identification ) ( Template Matching) , D, 8, B Y, V Otsu Otsu VI

7 VII

8 1.1 Panasonic VDR-D HP ipaq hx AUSTRIA ( 4-neighborhood ) ( 8-neighborhood ) ( Interlaced effect ) ( Deinterlaced effect) (b) VIII

9 Otsu x2 Otsu x4 Otsu (a) f (b) g (c) f g , B IX

10 3.42 0, D Y,V (1) (1) (2) (1) (2) (1) (2) X

11 1.1 Panasonic VDR-D HP ipaq hx Otsu Otsu XI

13 DSP

14 1.4.1 Panasonic VDR-D200 F 1/6 CCD (68 ) D200: F1.8 ( ) / F3.9 ( ) 30x mm 30.5mm 1.1 Panasonic VDR-D200 HP ipaq hx4700 3

15 Intel XScale-PXA270 (ARMv5TE) at 624 Mhz 64 MB Mobile SDRAM WLAN Bluetooth IrDA VGA (480x640) transfective AK4641 audio codec 1.2 HP ipaq hx4700 Ångström embedded Linux distribution OpenCV X x480 4

17 ~ ~ ~

18 [4] 250cc 250mm x 140mm 35mm x 65mm 320mm x 150mm 45mm x 90mm 1.3 7

19 1.5 CCD ( Interlaced ) 1.6 8

20 9

21 2.1 ( License Plate Localization ) ( Texture Feature ) ( Morphology ) DWT [6] [2-5] [7-8] [11-14] [1, 15-19] 10

22 2.1.2 ( Color Feature ) HSI ( neural-network classifier ) [20-21] genetic [22] fuzzy aggregator [23] 2.2 ( Edge Detection ) Roberts Prewitt Sobel Roberts Prewitt

23 Sobel Sobel A Gx Gy : Gx = * A and Gy = * A (2-1) g(x, y) = f(x+1, y+1) + 2 x f(x+1, y) + f(x+1, y-1) f(x-1, y+1) 2 x f(x-1, y) f(x-1, y-1) (2-2) [-1, +1] 12

24 [5] [-1, 0, 1] [7] [-1, 0, 0, 1] Sobel Prewitt 3x3 2.3 ( Binarization ) ( Threshold Value ) ( Adaptive ) Otsu ( Fixed Threshold ) 255 ( ) 0 ( ) f(x, y) TH G G(x, y) = 1 if f(x, y) > 0 if f(x, y) < (2-3) 13

25 2.3.2 ( Average Threshold ) [7] =0.88 TH*= TH W H G T = (, ) (2-3) ( Statistical Threshold ) ( Minimum within-group variance ) Otsu [24] 1979 ( Class ) ( Variance ) Otsu M x N 0 ~ 255 i 14

26 P(i) = (2-4) k C 0 C 1 C 0 0 ~ k C 1 k+1 ~ 255 W 0 W 1 W = Pr(C ) = P (2-5) W = Pr(C ) = P = 1 W (2-6) (mean) = (2-7) = (2-8) σ = (i μ ) (2-9) σ = (i μ ) (2-10) σ = W x σ + W x σ (2-11) 15

27 k k Otsu Otus 2.4 (Character Segmentation) [6] (Projection Histogram) (Connected Components Labeling) (Projection Histogram) 16

28 ( Connected Components Labeling ) 17

29 18

30 1. CCD Otsu 24.39% 25.54% 19

31 3.1 ( Preprocessing ) ( Interlaced effect ) CCD ( Progressive-scan ) ( Interlaced-scan ) frame rate 30 fps 1/60 f f f f f f f (x, y) = f (x, y) (3-1) 20

32 3.2 ( License Plate Localization ) ( Edge Detection ) 3x3 Roberts [-1, 0-1] (3-2) g(x, y) = f(x+1, y) - f(x-1, y) = f(x, y) * [ -1, 0, -1] (3-3) 21

33 Sobel Prewitt 704x330 frame 2x702x330 Roberts [-1, 0, 1] 22

34 ( VALID_T ) 24 VE VE(x, y) = 1 if f(x + 1, y) f(x 1, y) > VALID_T 0 otherwise (3-4) CONN_W CONN_T E x, y E(x, y) CONN_W > CONN_T (3-5) 1, I, 7, L, F CONN_W

35 1.5 CONN_C CONN_W 1/ / L 24

36 : WIDTH HEIGHT WIDTH/HEIGHT) >= 2 WIDTH/HEIGHT) <= 6 25

37 WIDTH >= 95 WIDTH <= 350 (3-6) 10 26

39 ( 10 ) (3-7) 28

40 L 7 L x 20 29

41 12 30

42 ( 3.10 ) WIDTH * HEIGHT 14.2% * WIDTH ~ 16.3% * WIDTH 19.1% * WIDTH ~ 22% * WIDTH 7 L 1/4 3/4 31

43 (a) 7 (b) L 45mm x 90mm 35mm x 65mm 1. 45mm x 90mm 2. 35mm x 65mm 1 : :

44 3.3.1 [-1 0 1] [0, 0, 0, 255, 255, 255] (3-11) [0, 0, 255, 255, 0, 0] (3-12) 2 33

45 4 1, L, 7, C 4 8 3/ Y H, 8, B CCD 34

46 64 g(x, y) g(x, y) g(x, y) (3-13) (a) (b) 35

47 3.3.3 ( Binarization ) Otsu Otsu ( ) Otsu Otsu Otsu 3x2 36

48 Otsu 3.25 D D D [-1, 0, 1] 37

49 G VE FORE_T BACK_T COUNT VALID_T VE(x, y) = 1 if f(x + 1, y) f(x 1, y) > VALID_T 0 otherwise (3-14) 4*2 8 count VE FORE_T BACK_T FORE_T = G(x 1, y) if VE(x, y) = 1 (3-15) BACK_T = G(x + 1, y) if VE(x, y) = 1 (3-16) 38

50 FORE_T BACK_T 20% FORE_T = FORE_T + ((BACK_T - FORE_T)*0.2) (3-17) B B(x, y) = 1 if G(x, y) < FORE_T 0 otherwise (3-18) 39

51 Otsu 23.65% 23.41%

52 G WIDTH_TH X = WIDTH_TH G X = - WIDTH_TH G (3-19) ( Character Segmentation ) <= <= >= 6 3. / >=1.3 41

54 hdistance vdistance [1] ratio = hdistance / vdistance (3-20) x = x + y/ratio y = y - x/ratio (3-21) x y 43

55 4 6 < 6 >=

56 Y g y1 y2 g y1 y2 Y (3-22) 3.4 ( Character Identification ) ( Template Matching ) 45

57 3.4.1 ( Template Matching) ( Character Normalization ) 19x35 NMI Normalize Match Index [26] g f g f M+ 46

58 M- M+ = f(x, y) g(x, y) = 0 M- = f(x, y) g(x, y)!= 0 NMI = ( ) ( ) ( ) ( ) (3-23) NMI (a) (b) (c) 0.77 = ( ) ( ) ( ) ( ) (3-24) NMI NMI 47

59 6, 9, S, E, L, F, 8, B, 0, D 0, D, 8, B 6, 9, S 6 9 S NMI , 9, S, E, F, L 6, 9, S,,E F 2, 4, 7, M, N 48

60 0 A, K, N, X, M, R 1 C, E 2 F 3 6, G 4 H, K, M, N, X 5 3, 7, J 6 K, N, U, V, W, X, Y 7 L 8 0, D 9 7, P a 5, S b 4, 9 c 2, Z d 2, 7 e 4, 7, 8, B, T, K, N, Q, V, W, X, Y f 1, I >= 3 g A ~ Z, 0~ , B, 0, D, Y, V 49

61 8 5 b < a b < c d < c d <e B

62 0 B, D 8, , B 13% 7% 2% 0 D % 2% 1% 51

63 Y, V 3/4 52

64 Otsu 4.1 Intel XScale-PXA Mhz ARM Linux OpenCV 120ms 10ms ~ 20ms ms 40 ms / frame frame

65 % 80.85% 84.44% % 86.36%

66 4.3 (b), (f) 4.3 (c), (e)

67 2 38/45 x 100% = 84.44% %

68 100. (a) (b) ( c) 4.6 (b) C, (c) frame 57

69 ms ms 10ms ~ 20ms 58

8 10ms 190ms 4.6 Otsu Otsu [1, 2, 4, 27-29] Otsu Otsu 82 47 48 68 26 38 58.

70 8 10ms 190ms 4.6 Otsu Otsu [1, 2, 4, 27-29] Otsu Otsu % 82.92% 55.31% 80.85% % 87.17% 78.78% 86.36% 4.3 Otsu 59

71 Otsu 24.39% 25.54% % 82.92% 80.85% 80.85% % 87.17% 86.36% 86.36% 7.32% / 2.24% 0% % 4.8 Otsu 60

72 3x3 Otsu % 58.53% 55.31% 55.31% % 72.72% 78.78% 78.78% 6.1% / 4.47% 0% 4.5 Otsu 6.1% %

73 62

74 63

75 17 64

76 6, 9, S 0, D, 8, B XScale-PXA270 DSP 96% 80% 100ms 30ms 5 4 fps Adaptive 65

77 [1] 2006 [2] 2000 [3] 2005 [4] 2005 [5] 2003 [6] 2003 [7] 2000 [8] 2003 [9] Clemens Arth, Florian Limberger, Horst Bischof, Real-Time License Plate Recognition on an Embedded DSP-Platform, Computer Vision and Pattern Recognition, CVPR '07. IEEE Conference on, pp [10] J. S. Kang, M. H. Kang, C. H. Park, J. H. Kim, Y. S. Choi, Implementation of Embedded System for Vehicle Tracking and License Plates Recognition using Spatial Relative Distance, Information Technology Interfaces, th International Conference on, vol.1, 2004, pp [11] S.-Z. Wang and H.-J. Lee, Detection and recognition of license plate characters 66

78 with different appearances, Intelligent Transportation Systems, Proceedings IEEE, vol. 2, 2003, pp [12] S. Kim, D. Kim, Y. Ryu, and G. Kim, A robust license-plate extraction method under complex image conditions, Pattern Recognition, Proceedings. 16th International Conference on, vol. 3, 2002, pp [13] H. Bai, J. Zhu, and C. Liu, A fast license plate extraction method on complex background, Intelligent Transportation Systems, Proceedings IEEE, vol. 2, 2003, pp [14] H. Bai and C. Liu, A hybrid license plate extraction method based on edge statistics and morphology, Pattern Recognition, ICPR Proceedings of the 17th International Conference on, vol. 2, 2004, pp [15] V. Kamat and S. Ganesan, An efficient implementation of the Hough transform for detecting vehicle license plates using DSP S, Real-Time Technology and Applications Symposium, Proceedings, 1995, pp [16] S. Gendy, C. L. Smith, and S. Lachowicz, Automatic car registration plate recognition using fast Hough transform, Security Technology, Proceedings. The Institute of Electrical and Electronics Engineers 31st Annual 1997 International Carnahan Conference on, 1997, pp [17] Jianlong Zhu, Yannan Zhao, Vehicle license image segmentation using wavelet transform, Intelligent Multimedia, Video and Speech Processing, Proceedings of 2001 International Symposium on, 2001, pp [18] Ching-Tang Hsieh, Yu-Shan Juan, Kuo-Ming Hung, Multiple license plate detection for complex background, Advanced Information Networking and Applications, AINA th International Conference on, vol. 2, 2005, pp [19] Huiqiong Chen, Derek Rivait and Qigang Gao, Real-Time License Plate I 67

79 dentification by Perceptual Shape Grouping and Tracking, Intelligent Transportation Systems Conference, ITSC '06. IEEE, pp [20] W. Wei, M. Wang, and Z. Huang, An automatic method of location for number-plate using color features, Image Processing, Proceedings International Conference on, vol. 1, 2001, pp [21] S. H. Park, K. I. Kim, K. Jung, and H. J. Kim, Locating car license plates using neural networks, Electronics Letters, vol. 35, 1999, pp [22] S. K. Kim, D. W. Kim, and H. J. Kim, A recognition of vehicle license plate using a genetic algorithm based segmentation, Image Processing, Proceedings, International Conference on, vol. 1, 1996, pp [23] Shyang-Lih Chang, Li-Shien Chen, Yun-Chung Chung, and Sei-Wan Chen, Automatic license plate recognition, Intelligent Transportation Systems, IEEE Transactions on, vol. 5, Issue 1, 2004, pp [24] N. Otsu, A Threshold Selection Method from Gray-Level Historgams, IEEE Transactions on System, Man, and Cybernetics, vol. SMC-9, 1979, pp [25] Rafael C. Gonzalez and Richard E. Woods, Digital Image Processing, 2nd Editio, Prentice-Hall, New Jersey, 2002 [26] J. R. Parker, Algorithms for Image Processing and Computer Vision, Wiley; Pap/Cdr edition, November 25, 1996 [27] Naito, T.; Tsukada, T.; Yamada, K.; Kozuka, K.; Yamamoto, S, Robust license-plate recognition method for passing vehicles under outside environment, Vehicular Technology, IEEE Transactions on, vol. 49, Issue 6, 2000, pp [28] Hongliang Bai; Junmin Zhu; Changping Liu, A fast license plate extraction method on complex background, Intelligent Transportation Systems,

80 Proceedings IEEE, vol. 2, 2003, pp [29] Naito, T.; Tsukada, T.; Yamada, K.; Kozuka, K.; Yamamoto, S, Robust recognition methods for inclined license plates under various illumination conditions outdoors, Intelligent Transportation Systems, Proceedings IEEE/IEEJ/JSAI International Conference on, 1999, pp [30] The Ångström Distribution, [31] Open Source Computer Vision Library, 69

P(x,y) P(x-1,y) P(x,y-1) P(x,y+1) P(x+1,y) 1 3.2 Sobel LaplacePrewittRoberts Sobel [2] Sobel 3.3 2 [6] 0 1 1: P(x,y) t (4-connectivity) 2: P(x,y) t 3:

P(x,y) P(x-1,y) P(x,y-1) P(x,y+1) P(x+1,y) 1 3.2 Sobel LaplacePrewittRoberts Sobel [2] Sobel 3.3 2 [6] 0 1 1: P(x,y) t (4-connectivity) 2: P(x,y) t 3: chdong@npic.edu.tw rola@mail.stu.edu.tw (ITS) (LPR) ITS 2. [1][8] [3][9][12] [9] frequency table Sobel [3][12] RGB HSB color model [11] Hough transformation Sobel [4] [10] 1. Intelligent Transportation