A novel license plate character segmentation method for different types of vehicle license plates

License plate character segmentation (LPCS) is a very important part of vehicle license plate recognition (LPR) system. The accuracy of LPR system widely depends on two parts; namely license plate detection (LPD) and LPCS. Different country has different types and shapes of LPs are available. Based on character position on LP, we can find two types of LPs over the world, single row (SR) and double rows (DR) LP. Most of the LPCS methods are generally used for SRLP. This paper proposed a novel LPCS method for SR and DR types of LPs. Experimental results shows the real-time effectiveness of our proposed method. The accuracy of our proposed LPCS method is 99.05% and the average computational time is 27ms which is higher than other existing methods.


I. INTRODUCTION
The numbers of vehicles are increasing consequently on the highway with the rapid economic development and social improvement. Nowadays, LPR is an important research field in the era of computer vision, image processing and pattern recognition. Vehicle LPR is one of the most important subjects in traffic surveillance system. We are able to obtain useful information about vehicles through LPR. Recently, LPR is widely used in highway tollbooth, traffic monitoring, stolen vehicle recognition, intelligent parking management and so on.
The LPR system is composed of three parts, LPD, LPCS and license plate character recognition (LPCR). In order to recognize the vehicle LP in real-time, however, LP should be quickly and robustly detected in advance. Unsatisfying results of LPD affect performance of LPR. Therefore, detecting LPs under various complex environments remain a challenging problem. LPCS is the necessary procedure after LPD and it is most important part for LPR system. The accuracy of LPR is entirely depends on LPD and LPCS.
Many researchers have been making great efforts on LPCS. Lots of studies have been done on the LPCS. The most common LPCS method is based on the projection information of plate characters [1] [2]. Pixel distribution density and region pixel concentration for LPCS are presented by P.Sandhya Rani et al., [3] and K. Romić et al., [4]. Youngwoo Yoon et al., [5] demonstrated a combination of multimethod binarization methods, Jaedo Kim et al., [6] presented topological transform such as twist, rotation, W. J. Li et al., [7] and W. Q. Yuan et al., [8] illustrated template matching , L. Chen et al., [9] and J. Yu et al., [10] presented clustering methods for LPCS.
This research mainly consists of the following parts; 1) In the part of image pre-processing, LP image normalization; the gray scale; contrast-limited adaptive histogram equalization (CLAHE); Bilateral filter and Otsu's threshold were applied in this research.
2) To deal with different types of LP, Verifying LP type after image pre-processing and classifies them in to two categories; SR and DR type. Histograms of binary image were used to verify the LP type.
3) In the part of image post-processing, CLAHE threshold; Sauvola threshold; a bitwise NOT (negation) operation; Bilateral filter; horizontal projection map; Blob labeling or connected component analysis (CCA); Euclidean distance were performed with verified LP image and find the LP characters. This paper is organized as follows: Challenges are illustrated in Section 2, the proposed LPCS method is described in Section 3 and the experimental results in Section 4 show that proposed method is able to ensure fast LPCS as well as achieve the high segmentation accuracy than other existing methods. Finally, conclusion is summarized in Section 5.

II. CHALLENGES
There are a number of possible difficulties are available for LPR system affected by images. If LPD is affected by images then it provided low accuracy results for LPCS. All factors that may influence the ability of LPR units to accurately read and match LPs under consideration are summarized as follows: 1) Bad image quality (resolution) problem caused by using a low-quality camera or long distance between a camera and a vehicle.
2) Blurry image problem caused by mainly motion blur.
*Corresponding Author: Moon Kyou Song, IEEE senior member 3) Illumination and low contrast due to overexposure, reflection or shadows, caused by vehicle headlight or other light sources during the image acquisition. 4) Occlusion problem; an object obscured or dirt on the LP during the image acquisition. 5) Partial LP image problem caused by a distorted LP or only some part of LP.
6) Environmental problems caused by snowing, raining, etc. during the image acquisition. Examples of difficult images for LPR system Most of the countries LP have SPLP. To properly work with LPR systems, we must manage a large variety of LPs (SR and DR with different size), so we selected LPs from South Korea because each province in Korea has its own LP color, pattern, and formats of numbers and other characters. Different colors represent different types of vehicles. Moreover, there are two different types of LPs available in Korea, such as SRLP (white or yellow) and DRLP (green or yellow or orange) type based on character position information. Fig. 2 shows the different types and sizes of LPs available in Korea. Several image processing techniques are applied for LPCS because it is very important for obtaining a high accuracy of character recognition. If the character is segmented well then the recognition part acquires a high performance otherwise not. Our proposed system consists of three parts; image preprocessing, verification of LP type and image post-processing with character extraction. Fig. 3 demonstrates our proposed LPCS process as below;

A. Image Pre-processing
In this section, some image pre-processing algorithms are used for finding LP type. The LP image pre-processing processes are as below; 1) Resizing the input image: Firstly, resized the all input LP images with the resolution of 200x70.
2) Image enhancement: To enhance the LP image contrast quality, the contrast-limited adaptive histogram equalization (CLAHE) [11] method is utilized. It is different from the normal histogram equalization (HE) with respect that calculates many histograms, each corresponding to a discrete section of an image and uses them to rearrange the lightness values of the LP image. Therefore it is appropriate for improving local contrast of LP image and bringing out more in detail. Fig.4(c) shows much clearer and more details than the original LP image ( Fig.4 (a)).
3) Image filtering: For filtering the image, the Bilateral filter [12] is applied. This was proposed by Tomasi & Manduchi et al. as a non-iterative method for edge-preserving smoothing. The Bilateral filter, which is defined as; spatial extent of the kernel or size of the considered neighborhood, range r  is minimum amplitude of an edge.
The main goal of filtering is to smooth an input image. However, sometimes the filters do not only dissolve the noise, but also smooth away the edges. To solve this problem, we can use a filter called bilateral filter, which is an advanced version of Gaussian filter, it introduces another weight that represents how two pixels can be close (or similar) to one another in value, and by considering both weights in image, Bilateral filter can keep edges sharp while blurring image. Bilateral filtering with parameters r  =3 pixels and r  =100 intensity values are applied to the image in Fig. 4(d).

4) Region of interest (ROI) selection:
After finishing the filtering, set the region of interest (ROI) which contains maximum intensity value of the image. In our input LP image case, the ROI starting pixel is x=50 and y=17 with the width of 100 and height of 35 from the starting pixel point. Fig. 4(e) shows the ROI image from original LP image. Fig. 4.
Results of image pre-processing methods

5) Thresholding of ROI image:
Otsu's method [13] is used to perform clustering-based image thresholding for the reduction of a gray level image to a binary image from the ROI image shows in Fig. 4(f).

6) Save the ROI image for LP type verification:
Finally, save the binary ROI image for verifying LP type.

B. Verification of LP Types
Binary image is a digital image that has only two possible values for each pixel. Typically the two colors used for a binary image are black and white though any two colors can be used. The color used for the object(s) in the image is the foreground color while the rest of the image is the background color. Binary images are also called bi-level or two-level. This means that each pixel is stored as a single bit-0(black) or 1(white). Compute the histograms of binary ROI image which we obtained after applying image pre-processing techniques and find the LP type.   1) Image enhancement: CLAHE threshold value for contrast limiting parameter is 10 be used. The effectiveness is showed in Fig. 6(b).
2) Image thresholding: Sauvola method [14] performs local thresholding of a two-dimensional array SRLP image with Sauvola algorithm. In Sauvola algorithm, the threshold th(x,y) is calculated using the mean mn(x,y) and the standard deviation sd(s,y) of the pixel intensity in a m×m window centered on the pixel (x,y). ( Where R is the maximum value of the standard deviation (R=128 for a greyscale SRLP image), and k is a parameter which range is [0.2, 0.5] takes a positive values and we use 0.34. The local mean mn(x,y) and standard deviation sd(s,y) adjust the value of the threshold according to the contrast in the local neighborhood of the pixel.
3) Negation operation: Perform a bitwise NOT (negation) operation after Sauvola Thresholding. Any pixel that is a 1 in the expression becomes a 0 in the result and any pixel that is a 0 in the expression becomes a 1; the result showed in Fig.6 (d).

4) ROI image selection and filtering:
Fourthly, as we know the resized LP size is 200x70 pixels. The 20 pixels value from lower part of LP region contains no character information. So we can eliminate that lower part of the LP region and keep only 200x50 pixels. Again the Bilateral filter [12] is applied for removing noise from image and finds the ROI image by using the horizontal projection information. The horizontal projection shows that the LP character region has only high frequency than the upper region of LP. So again we eliminate upper region with the horizontal frequency information and keep the LP character region only as a ROI image shows in Fig.6 (e).

5) Removing small components from ROI image:
Morphological operation [15]; dilation and erosion are used to remove small components from the ROI images.
6) Finding SRLP character regions: Connected component labeling or connected component analysis (CCA) or blob labeling is used in LP image to detect connected regions in binary LP image [16]. Assume that the segmented LP image R consists of m disjoint regions R i . The LP image R often consists of objects and a background.
Where R C is the set complement, R b is considered as background, and other regions are considered as objects. Input to a labeling algorithm is usually either a binary or multi-level image, where background may be represented by zero pixels, and objects by non-zero values. A multi-level image is often used to represent the labeling result, background being represented by zero values, and regions represented by their non-zero labels. After applying blob labeling, find the character blobs in ROI image and bounded by 2D bounding box in LP image as shown in Fig.6 (f). Hence, we are able to obtain LP character region. 7) Character extraction and save the character images: Finally, calculate the distance between the centers of the blobs by using Euclidean distance as below, Once we obtained the distance between the centers of the blobs, the blob can be segmented with the distance information and we need to save the segmented blobs for LPCR. The results of segmented blobs are shown the LP characters in Fig.6 (g). 1) Image enhancement: CLAHE threshold value for contrast limiting parameter is 3 be used; the result showed in Fig.7(b).
2) Negation operation: Performs a bitwise NOT operation after CLAHE as shown in Fig.7(c).

4) Repeat negation operation:
Again performs a bitwise NOT operation after Sauvola thresholding showed in Fig.7 (e).

5) ROI image selection:
Eliminating the 20 pixels value from lower part of LP because those regions have no character information and keep only 200x50 pixels.
6) ROI image separation: Using horizontal projection information, the LP can be splits into two parts; upper plate and lower plate, which is defined as a two ROI images.

7) Character extraction and save the character images:
The rest of the parts are same as SRLP image post-processing; perform CCA with both ROI images (upper plate & lower plate) and find the character blobs in both ROI image and bounded by 2D bounding box in LP image as shown in Fig.7 (g). Euclidean distance computes the distance between the centers of the blobs. The blobs are segmented with their distance information and we need to save the segmented blobs for LPCR. The results are showing the LP characters in Fig.7 (h).

IV. EXPERIMENTAL RESULTS
To test the LPCS using our proposed method in this paper, we applied the method to a database of 1000 SRLPs and 1000 DRLPs with the resolution of 200x70 pixels that were captured at different times and weather conditions. The experiment is based on the conditions of a system with CPU 3.40-GHz Intel Core i7-2600 and 8.00 GB of RAM, and implemented using Microsoft Visual Studio 2012 with OpenCV library. Table I shows the performance measures of different LPCS techniques. Our proposed method shows the best performance than other existing methods. We achieve the fast computational time with compare to the latest result for LPCS in [5]. The numbers of properly segmented SRLPs are 998 and DRLPs are 983. So the total perfectly segmented LP images are 1981 from 2000 LP images. Part of the experiment results are shown in Fig. 8. We demonstrated a procedure for LPCS algorithms. We used a robust method for work with different types of LPs in our LPCS system. Our proposed LPCS system is separated into two stages, image pre-processing, which make our LPs simple for verifying the LP type and image post-processing, which is segmented our LPs character effectively. In this paper, we demonstrated that such simplicity and effectiveness allow our method to provide better performance than other existing methods. Most of the existing techniques are performed with simple or SRLP and are not suitable for real-time applications; however, our proposed algorithm is not only for SRLP but also for DRLP, thus rendering it suitable for real-time applications. Using our proposed method, experimental results show that the test accuracy is 99.05% with a computational time of 27ms, which shows that the accuracy is higher than other existing methods with faster computational time compare to the latest results. Moreover, LPCR is our principal future work.