1. Introduction

Most solar salt in Korea is produced in Shinan, which in in Jeollanam-do province. Jeollanam-do had an average aging index (i.e., “elderly population per 100 youth population”) of 196.1 in 2020. By contrast, the aging index of Shinan was 555.7, which indicates that Shinan is aging quite quickly (Korean Statical Information Service, 2021). The aging of the population is directly related to the paucity of workers available for solar salt production. The number of workers decreased from 2,918 in 2015 to 2,305 in 2017. Among solar salt production companies, 46.3 % have two workers and 36.8 % have one worker, accounting for the majority of all companies (Korean Ministry of Ocean and Fisheries, 2019). Solar salt production companies face a declining trend as the workers in the production of solar salt are aging and there is no inflow of workers.

In solar salt production, salt is obtained by trapping seawater in a salt field and then allowing the water to evaporate in sunlight. The produced salt is collected using traditional tools or electric machines. The salt collection process consumes considerable manpower even when collection equipment is used. In such a scenario, automatic salt collection devices can reduce manpower.

Image processing extracts features such as color, texture, shape, and structure from a photograph or video. The image processing procedure consists of several steps such as preprocessing, Region of Interest (RoI) setting, sampling, threshold setting, feature extraction, recognition, and interpretation (Park and Bae, 2019). Algorithms process information from digitized images using several techniques, including: perspective transformation (Neven et al., 2018;Muthalagu et al., 2020) that transforms an image or corrects distortion, color models used to detect color or object, morphological operations that analyze and process image shapes, and contouring that identifies the outline and shape of objects. The floor of the saltern uses black tiles to absorb sunlight more effectively. Therefore, when salt particles are formed, it is easy for a color model to perform object detection by comparing the color of the salt particles against the black floor. Furthermore, if an image is converted to binary image and a threshold is set, image processing algorithms can easily detect areas having contrasting colors (Kim et al., 2021) but fing it difficult to detect areas for various colors when multiple colors are involved. In this study, a detection method using the HSV color model was developed because during the salt collection process, it is necessary to detect three colors representing directions in addition to salt.

Machine learning has improved tremendously with the advancement of computer and graphic processing unit (GPU) technologies. Object detection technology is one such machine learning applications, and among machine learning, deep learning shows good performance in object detection. The main deep learning object detection algorithms include YOLO, R-CNN, RetinaNet, and VGGNet. Most deep learning algorithms are convolutional neural networks (CNNs), and when performing convolutional operations, they use a filter with the same parameters to move the pixel positions while repeatedly performing multiple matrix multiplications, resulting in extensive computation (Park, 2021). This amount of computation requires a GPU or high-performance equipment that can perform fast computation in parallel, which would make an unmanned solar salt collection vehicle too expensive. Our proposed image processing algorithm is not complicated in structure and does not require much computation, so it can be applied using low-performance, low-cost equipment, which has the advantage of being competitive in mass production and distribution. Furthermore, equipment used in saltern is operated in extreme conditions under the influence of strong solar heat, high humidity, and high salinity, so equipment failures are frequent and the equipment replacement cycle is short. In this light, low-cost equipment becomes a critical factor.

In the present work, an experiment was performed to improve the color detection performance of an unmanned solar salt collection vehicle using image processing. The dataset was obtained by directing a scene showing the collection of solar salt in a saltern, as an around-view image using image processing. Color was detected only in the RoI by setting the RoI in the around-view image. A mask was created by setting the HSV upper and lower limit value, followed by performing a morphological operation and an AND operation with the original image. The detected area in color detection represents the outline through contours, and the contours are calculated using an image moment. The case of the collection vehicle within the saltern is determined using the contour area. The performance was evaluated by comparing the detection area in each algorithm step and the final detection area.

2. Method

The image processing algorithm for the unmanned solar salt collection vehicle is shown in Figure 1. The input image is transformed into an around-view image through resizing, rotation, and perspective transformation step. RoI is set in the around-view to reduce the amount of computation, and HSV color model conversion is performed only in the RoI area for color detection. A mask is created by setting the HSV upper and lower limit values of the color to be detected, and the color area is detected through the AND operation. A morphological operation is performed on the detected color area to remove noise, and the area is calculated using the contour and image moment. Finally the detected area is compared with a set area to determine the location case of the collection vehicle in the salt plate.

2.1 Preprocessing

Preprocessing reduces unnecessary data in an image and refines meaningful data. There is considerable data in images captured by a camera. For example, there are 10,000 kinds of pixels in an image of size 100×100. For color images, an RGB value ranging between 0 to 256×256×256 is allocated to each pixel, so there are 16,777,216 possible value for each pixel (Gonzalez and Woods, 2013). Analyzing all these pixels without additional processing is difficult and time-consuming.

In addition to reducing the number of pixels, image preprocessing simplifies data using transformations of the size, angle, and color of the image, leaving only necessary data. Preprocessing improves image processing accuracy and precision while reducing computation time.

2.1.1 Around-view

Recently, the around-view system, which is an image as if looking down from the sky, has been used in vehicles by installing cameras on all sides. The image to be used for the experiment is input through the camera, and is resized to 512×512 for consistency.

With reference to the collection vehicle, (a) is the front, (b) is the right, (c) is the back, and (d) is the left in Figure 2. The resized input images are rotated clockwise by (a) 0°, (b) 90°, (c) 180°, and (d) 270° to align the direction.

The rotated images are mapped to the coordinates of the images using perspective transformation. Perspective transformation transforms two-dimensional (x, y) coordinates into three-dimensional coordinates (X, Y, W) and then transforms them back into two-dimensional coordinates (u, v) (Gonzalez and Woods, 2013).

Perspective transformation has eight degrees of freedom, so at least four tie points are needed to determine transformation. The around-view image by combining the preprocessed images is shown in Figure 3.

2.1.2 Region of Interest

The RoI is used when the goal is to find a specific object or singularity on the image. Setting the RoI is al critical task that can prevent image processing for an irrelevant region, thereby reducing the amount of computation and improving the accuracy of the algorithm. The algorithm operation works only on the RoI; it does not consider the entire image. The RoI is set to a size of 40×100 for the front and back sides and 100×40 for the left and right sides.

2.1.3 Color model conversion

The RGB color model represents colors as a combination of red, green, and blue, whereas the HSV color model represents colors intuitively as humans see them. Therefore, the HSV color model is preferred for color detection because it is more similar to the human color recognition process than the RGB color model (Gonzalez and Woods, 2013).

We did not convert the entire around-view image from the RGB color model to the HSV color model, but only the RoI. As a result, it has RoI in HSV state and RoI (original image) in RGB state.

2.2 Color detection and contour

2.2.1 Color detection

A mask for color detection is created by setting the upper and lower limits value of H, S, and V of the color to be detected. Color detection is performed via an AND operation representing the original image and the mask. Figure 4 shows color detection, (a) is the original image, (b) is the mask for red detection and (c) shows that overlapping parts detected by the AND operation of (a) and (b).

In our experiment, red (representing the front), green (representing the back), blue (representing the left and right sides), and white (the salt) were detected.

2.2.2 Morphological operation (Gonzalez and Woods, 2013)

Morphological operations use structuring elements to remove noise, fill holes in images, and connect broken lines (Park et al., 2017) Examples of morphological operations are erosion, dilation, open, close, gradient, tophat, and blackhat.

In Figure 5 (a), there is an area that is not accurately detected because there is black noise in the middle portion of the detected color. For red, blue, and green, the opening operation was applied to remove the finely detected part among the parts that should not be detected, and the closing operation was applied to remove the noise in the black portion among the detected parts. Figure 5 (b), to which the morphological operation is applied, shows that the noise in (a) has been removed.

In Figure 6 (a), white represents salt and black represents the salt plate tile. White color is used for salt detection, but the particles of salt are small, so when it is detected as shown in Figure 6 (b), it is detected as fine dots. In white, the dilation operation was repeated thrice times to dilate the detected portion of the image, and then noise was removed by applying an opening operation and a closing operation. The result is (c).

2.2.3 Contour

Image contours connect the borders of areas with the same color or intensity in the image. Figure 7 (a) and (b) are images of the area obtained through contouring. The area obtained by the contour is the area of the color detected in the image.

The feature value for a contour is called image moment. The image moment consists of 24 values and is largely composed of spatial moments, central moments, and central normalized moments. The 24 values are divided into 10 spatial moments, 7 central moments, and 7 central normalized moments.

Using the image moment, we can calculate the center of gravity of the contour, the length of the contour in the form of a closed curve, the length of the curve rather than the closed curve, and the area of the region surrounded by the contour. Among the above image moments, m00 of the spatial moment represents the area of the detected contour area. In this experiment, the spatial moment m00 among the image moments was used, through which the contour detection area was obtained.

3. Results

3.1 Dataset

The experiment was performed using a video filmed in a saltern. The salt plate of the saltern was a square having area 1530cm×1530 cm. The four sides were separated using various-colored tents. The front is red, the back is green, and the left and right are blue. Figure 8 shows the colored tent installed on the salt plate.

Table 1 summarizes the weather conditions at the time of the experiment (14:00HR). A place where there is little rain, high radiation, and high temperature is suitable for salterns. Thus, salterns have similar high temperature, radiation, and weather conditions during the salt production period.

Assuming that cameras are installed on all sides of the collection vehicle, the experimental dataset was obtained by photographing four images i.e., the front, back, left, and right at the corresponding position of the salt plate. The obtained image was made into an around-view image through image preprocessing.

3.2 Cases of collection vehicle location

Figure 9 shows nine cases depending on the location of the collection vehicle in the salt plate. Color detection was performed with color representing each direction and white representing salt. The detected color represents an area using a contour and calculates the area using an image moment. The area of the detected color was compared with a set area to determine the case.

3.3 Experimental result

The experiment used the around-view image created after performing image preprocessing on the dataset. The around-view image sets the RoI and changes it to the HSV color model. For color detection, a mask is created by setting the HSV upper and lower limit values of the color to be detected, and is detected by performing an AND operation with the original image.

The limit value of HSV was set while changing the hue values. Figure 10 shows the difference in salt detection according to the change in the upper limit of hue in white. The lower limit value of HSV was fixed at (0, 0, 0), and only the upper limit value was changed by 10 from (90, 255, 255) to (120, 255, 255). Comparing hue values (100, 255, 255) and (110, 255, 255), it can be seen that salt is incorrectly detected on the right and back, which should not have been detected when the hue values are (110, 255, 255). Looking at (b) and (c) of Figure 10, it is clear that the upper limit values should be set to (100, 255, 255).

The HSV lower limit value was experimentally determined by fixing the upper limit value to (100, 255, 255) and changing the hue lower value from (0, 0, 0) to (30, 0, 0) by 10. Figure 11 shows the salt detection according to the lower limit value. Looking in front of (a) in figure 11, it can be seen that red is also detected. However, in (b) in figure 11, the front red color is not detected. Therefore, the lower limit value of the HSV was determined as (10, 0, 0).

The upper and lower limit value were determined for red, blue, and green in the same way as for white. Table 2 presents the final red, white, green, and blue HSV upper and lower limit values.

In the case of white color, salt particles are small, so it is detected in very small areas and needs to be expanded. So, dilation operation is performed to expand the small detection areas. The dilation operation filter used a 3×3 size, and there were three consecutive dilation operations. Figure 12 shows the change of the detection area depending on the increase in the number of dilation operations. As the number of dilation operations increases, it can be seen that the detection area becomes larger as small particles expand and combine.

Figure 13 shows how the number of particles changes with the number of dilation operations. The number of particles decreases as the number of dilation operations increases. Looking at (d) of figure 12, the detected salt particles on the front and left have a sufficiently large size owing to the dilation operation. However, spurious salt particles are detected on the right and back. Therefore, the opening operation is performed to remove small noises, and close operation is performed to connect the remaining unconnected areas.

Figure 14 shows the closing operation after the opening operation. Compared with (d) of figure 12, it can be seen that the front and left sides are connected smoothly, and minor instances of noise on the right and back sides are removed. In the case of red, green, and blue, a large enough area can be detected only by detecting HSV, so the dilation operation is not required. However, the opening operation is performed after the closing operation to fill in the black spots of the detection area and to remove minor noise.

Figure 15 illustrates the performance evaluation of applying white to each step of the proposed image processing algorithm. The y-axis is expressed as the ratio of the area obtained in each image processing algorithm step and the area finally obtained. The x-axis represents the algorithm step for each case. Looking at the graph, when only HSV was used (in case3), extremely minute particles are detected, so the area is very small. At this time, the number of particles is 121ea and only 0.2 % of the final area was included. However, it can be seen that the area increases through the dilation operation. When only HSV was used, case7 detected the best, but only 74.36 % of the final detection area. In the case of white, there was a performance improvement from a minimum of 25 % to a maximum of more than 99 %.

Figure 16 shows graphs showing the performance when the algorithm is applied to red, green, and blue. In the case of red, there was a detection area increase of at least 44 % and a maximum of 68 %, and the average detection area increased by 53 %. In the case of green, detection was good from the HSV step, and there was no large area change, but an average increase of 7 %. In the case of blue, it increased by an average of 15 %.

4. Conclusions

This study developed an image-based algorithm to improve the color detection performance of an unmanned solar salt collection vehicle. Nine cases were considered depending on the location of the collection vehicle on the salt plate. The dataset was obtained by capturing images in four directions (front, back, left, and right of the collection vehicle) assuming various locations within the salt plate. The dataset was created as an around-view after image preprocessing and color detection. The color was detected by creating a mask by setting the HSV upper and lower limit value, and performing morphological operation and an AND operation with the original image. The detected color represented the detection area through contour, while the detection area was calculated using the image moment. The case of the collection vehicle was detected by comparing the area of the detection area with the set area. The performance was evaluated by comparing the detection area in each algorithm step and the final detection area.

When evaluating the results through the experiment, white color detection (salt detection) showed a minimum 25 % improvement in the detection area, and a maximum increase of 99 % or more was confirmed. When all white cases were averaged, the area increased by approximately 76 %. In the case of red, there was detection area increase of at least 44 % and a maximum of 68 %, with the average increase being 53 %. In the case of green, detection was good from the HSV step, and there was no large area change, but an average increase of 7 %. In the case of blue, it increased by an average of 15 %. These results confirmed the accuracy of the proposed algorithm.

In future research, it is necessary to acquire more images and diverse datasets. In addition, an image of the four sides used as the dataset needs to be captured using a fixed camera for an accurate around-view image, for which it is necessary to develop a collection vehicle prototype. Finally, it is necessary to use real-time videos, not images, in order for a proposed solution to be applicable to real-world collection vehicles.