Image analysis-based recognition and quantification of grain number per panicle in rice

Background The number grain per panicle of rice is an important phenotypic trait and a significant index for variety screening and cultivation management. The methods that are currently used to count the number of grains per panicle are manually conducted, making them labor intensive and time consuming. Existing image-based grain counting methods had difficulty in separating overlapped grains. Results In this study, we aimed to develop an image analysis-based method to quickly quantify the number of rice grains per panicle. We compared the counting accuracy of several methods among different image acquisition devices and multiple panicle shapes on both Indica and Japonica subspecies of rice. The linear regression model developed in this study had a grain counting accuracy greater than 96% and 97% for Japonica and Indica rice, respectively. Moreover, while the deep learning model that we used was more time consuming than the linear regression model, the average counting accuracy was greater than 99%. Conclusions We developed a rice grain counting method that accurately counts the number of grains on a detached panicle, and believe this method can be a huge asset for guiding the development of high throughput methods for counting the grain number per panicle in other crops.


Background
Phenomics involves the gathering of high-dimensional phenotypic data to screen mutants with unique traits and identify the corresponding genes [1]. Current methods for obtaining phenotypic data are generally manual [2], making them time-consuming, labor-intensive, and less accurate. Therefore, such approaches have been impractical for high-throughput measurements during plant growth and development.
The number of rice grains per panicle is a key trait that effects grain cultivation, management, and subsequent yield [3][4][5], as well as being an important parameter for evaluating the potential of new rice cultivars [6]. Rapid measurement of grain number per panicle could improve the efficiency of scientific research and cultivar development.
Image analysis-based methods have been widely used in many aspects of plant phenotyping. Image-analysis based high-throughput phenotyping platforms have also been applied to measure phenotypic traits of rice, including: plant height, the green leaf area, and rice tiller number [7]. Yang et al. [8] measured the number of panicles on plants using multi-angel color images and an artificial neural network algorithm. The authors reported a reliable, automatic, high-throughput leaf scorer (HLS) for the evaluation of leaf traits, including leaf number, size, shape, and color [9]. Feng

Open Access
Plant Methods  [10] developed a hyperspectral imaging system for the accurate prediction of the above-ground biomass of individual rice plants in the visible and near-infrared spectral regions. Zhou et al. [11,12] used image analysis techniques to assess plant nitrogen and water status. Huang et al. [13] developed a prototype for the automatic measurement of panicle length using dual-cameras, which were equipped with a long-focus lens and a short-focus lens to capture a detailed and complete image of the rice panicle. In addition, image-based methods have been used to characterize seed morphology, including: seeds size, shape, color, and endosperm structure [14][15][16]. With the advancement of modern optical imaging and automation technology, hardware is no longer a bottleneck for phenotyping. Instead, the analysis and processing of multi-disciplinary optical images have become the new bottleneck [17].
The research on the rapid counting of grain number per panicle has been carried out in different ways. Generally, the panicle is spread out on a white background and held in place by metal pins so that branches and grains are nonoverlapping [14,18]. It is also an effective way to spread the grains after threshing [16]. These methods are not suitable for rice panicles with severe adhesions in the Yangtze River Basin. Currently, there are two primary methods for the determination of grain number per panicle. The first method is to count the number of grains manually after threshing, which is an incredibly time-consuming and labor intensive process. During the processing of threshed grains, due to the existence of a large amount of awns and overlaid and clustered grains, it is very challenging for traditional algorithms to identify individual rice kernels when they are touching [19,20]. Husking the grains would make them smoother and easy to separate, but husking also produces broken rice kernels and complicates the counting procedure.
The second methods for determining grain number on each panicle is the most common method and is called on-panicle counting method, which involves counting the number of grains in a spikelet. Collecting an image of the entire panicle is also problematic due to overlaid and clustered grains. To some extent, threedimensional image acquisition may solve the problem of the touching grains, but equipment to conduct this analyses is expensive and complicated to use.
In this study, we proposed a new counting method that uses image processing and deep learning algorithm to detect rice grain from the image of the primary branch was acquired using digital scanner. Our method would solve grain overlap or clustering problems, be more cost-effective and user-friendly, and facilitate high throughput counting of grain number per panicle in rice.

Field experiment
Since panicle morphology is effected by variety and species, we used two varieties of Indica rice (Yangliangyou No. 6 and Fengyouxiangzhan) and two varieties of Japonica (Wuyunjing No. 27 and Nanjing No. 9108). In addition, due to the fact that cultivation conditions can affect the grain size and grain number of per panicle, the experiment was organized as two-factor randomized complete block design with seeding density (150, 225, and 300 × 10 4 plant ha −1 ) and fertilizer (150, 225, and 300 × kg ha −1 ). At the full-heading stage, fifteen panicles per sample group were randomly harvested ( Table 1). Total of 540 panicles, which included the spikelet and panicle of different sizes and shapes, were collected in the study.

Image acquisition
The different postures of rice panicles have great influences on image recognition. We divided the panicles into three groups based on manual shaping (Fig. 1). Shape A: Panicles were not manually shaped and panicle images were obtained in the natural state. Shape B: the primary branches were manually separated. The moisture content of panicle branch in the mature stage was low and the branch would undergo inert deformation. However, the stem and branch were not completely fractured, and the panicle image was taken in a natural unfolded state. Shape C: the primary branches were removed from panicle stem. The panicle stem and branch were completely fractured, and the panicle image was taken in a natural scattered state. A two-image acquisition method was used for the panicles. The first way is that the samples were placed on a black light-absorbing fabric and the rice panicle images were acquired using digital camera (SONY, model ILCE-6300; equipped with E PZ 16-50 mm, f 3.5-5.6 lens) at a distance of 20 cm above the samples. The image size was 4032 × 3024. The second way is the panicle samples were placed on a scanner (Canon, model LiDE 400, resolution 4800 dpi, and scan speed: color, A4, 300 dpi, 8 s). The scanner was connected to a computer and data were transferred via a high-speed USB2.0 Type-C device. The panicle images were acquired through digital scanner and the image size was 2480 × 3507. The scanner was covered with black light-absorbing fabric to avoid light noise caused by reflection and projection. The basic information of image data was shown in Table 2.

Image pre-processing
As shown in Fig. 2a, the contrast between the foreground and background was clear, and the foreground panicles were easily detected using the Otsu Image segmentation algorithm ( Fig. 2b) [21]. During image acquisition, black light-absorbing fabric was used to prevent noise caused by light reflection and projection. However, repeated use of same fabric caused contamination by the impurities on the panicle surface and produced some noise. The original image size is 4032 × 3024 or 2480 × 3507, the size of the noise connection area is less than 100 pixels, and the panicle size is more than 100,000 pixels. So, we removed the noise by setting a threshold of 1000-pixels and extracting the connected area (Fig. 2c). The stem of the panicle was not investigated and would be removed. The width of the unfocused stem is 5-10 pixels, only less than half of the width of rice grains (30-50 pixels). Using a 5 × 5 disk mask to erode the denoising image for three times and then dilating it for three times (Fig. 2d). The branch image (without stem) ( Fig. 2d) was subtracted from the denoised binary image (Fig. 2c) to obtain a stem image with noise ( Fig. 2e) which was generated due to the excessive operation of eroding panicles in the previous step. Similarly, we removed the noise by setting a 200-pixels threshold and extracting the connected area ( Fig. 2f ). Next, the stem image (without noise) ( Fig. 2f ) was subtracted from the denoised binary image (Fig. 2c) to obtain a binary image without the stem or noise (Fig. 2g). The RGB image without stems or noise ( Fig. 2h) were used for further image processing.

Algorithm for the calculation of grain number per panicle
For untouched grains, the general counting method is to calculate the number of connected regions in binary images. When it touched, several methods were used to split the clustered kernels, including: dilation and erosion operation, the watershed method, corner detection, and feature matching. However, each method has its limitations in our study, which will be discussed in detail later in this paper. We designed two methods as follows.

Linear regression algorithm
Coverage, corner point, etc. are effective feature parameters in image processing and analysis [22]. In this study, three parameters were used as candidates for the construction of linear regression models to count the The parameter of all primary branch was first extracted from each panicle images. The sum of each branch parameter was used as the entire panicle parameter. All three parameters using the following equation: where S im is the size of original image (Fig. 2a); N cd is the number of pixels with value of 1 in coverage image (Fig. 3b); N sk is the number of pixels with value of 1 in 3c); and N co is the number of pixels with value of 1 in contour image (Fig. 3d). The parameters were normalized using min-max normalization method as shown in the following equation: where CDʹ is the normalized CD, CD min is the minimum value of CD, CD max is the maximum value of CD, Skʹ is the normalized Sk, Sk min is the minimum value of Sk, Sk max is the maximum value of Sk, Coʹ is the normalized Co, Co min is the minimum value of Co, Co max is the maximum value of Co.  Since the actual measurement was an integer and the model-predicted number was not always an integer, we rounded predicted number to integers for comparison. Table 2 is the image information used for regression model training and verification. In order to increase the sample size, the 45 shape A panicles were processed into 20 shape B panicles and 30 shape C panicles to acquire images again. The constructed model was evaluated using R 2 and RMSE.

Deep learning algorithm
The second method used in this study was the deep learning method (Fig. 4), which is popular and offers high accuracy performance. In this study, the superior Faster RCNN + ResNet101 network was used for grain identification [23,24]. Due to the heavy manual labeling work, only 120 original images were randomly selected for the model training and validation ( Table 2). After preprocessing, images containing multiple branches and stems were separated into images containing a single branch, and saved as a new image. Total of 1337 images were obtained (Fig. 4a). These images were labeled manually and are available from the author. We divided 70% of the dataset into training sets and 30% of them into validation sets. 400 original images were used for testing. Specifically, an original image was segmented into sub-images. The sub-images were separately identified and then aggregated into test results for one image. The images were sent to the Resnet101 feature extraction network (Fig. 4b) to generate the feature map. The authors of ResNet101 proposed a residual structure (Fig. 4c) to resolve the degradation problem. The selective search (SS) was replaced by region proposal network (RPN) (Fig. 4d). The RPN considers nine possible reference windows (Fig. 4e) at each sliding window (SW) position which can improve the speed and accuracy of object extraction. Finally, training and validation progress is performed by minimizing classification loss and box regression loss. Detailed information about the hardware, software, and model hyperparameters are provided in Table 3. The other hyperparameters were consistent with the original research [24].

Comparison on image manually counting method
We compared the counting accuracy of image manual counting methods on two image acquisition ways, two different species of rice and three manually shaped panicles. The ground truth (GT) is panicle grain manual counting. We found that regardless of the image acquisition method, for Shape A, the manual counting accuracy remained below 80%, and was the lowest one of the rice shapes (Table 4). The results indicated that Shape A was not useful because it led to low panicle grain counting accuracy. The maximum counting accuracy for Shape A was 80%, even with the most potent and optimized algorithm. In addition, a 95% counting accuracy could be achieved in the digital camera acquired images, which was at least 3% lower than those in scanner acquired images. This is primary due to the grain distribution on the panicle, which is not even and there being a high quantity of touching grains that are spread out during scanning. The counting accuracies for Shape B and Shape C were both greater than 95%.

Linear model analysis
We established 3 univariate linear regression models for counting the grain number on each panicle using CDʹ, Skʹ, and Coʹ. We also compared these 3 models in images that contained different panicle shape manipulation and were acquired using different digital devices. As shown in Fig. 5, for all three models, the R 2 was > 0.90. The CDʹbased linear regression model had an R 2 > 0.95, and had a better performance than Skʹ-or Coʹ-based models. Overall, images that were acquired from a scanner had lower RMSE than images acquired form a digital camera. Furthermore, the RMSE of Shape C + Indica rice images was lower than that of Shape B + Japonica rice images.
We validated the CDʹ-based linear regression model by comparing the model predictions for grain number with the actual measured number (Fig. 6). With an R 2 = 0.9831 and RMSE of 5.9481 for Indica rice, and R 2 = 0.975 and RMSE of 6.4405 for Japonica rice, the Scanner + Shape C method had the best performance in all ways. The Scanner + Shape B also had better performance as compared with the Camera + Shape B and Shape C. Therefore, scanner-acquired Shape C, in combination with CDʹbased linear regression model, provided the most accurate grain counting.
We also constructed a multiple linear regression model based on the three parameters. Table 5 shows the optimal model results obtained after screening. Compared to the univariate linear model (Figs. 5, 6), the multiple linear regression model requires more input factors, however their accuracy has not been greatly improved. Therefore, this study is more inclined to the univariate linear regression model.

Deep learning model
The deep learning model gave satisfactory performance for grain detection. The grains of both Indica and Japonica rice were easily detectable (Fig. 7). The counting accuracies of the deep learning models were all > 98%, and were unlikely to be affected by devices and panicle shape (Table 6). Specifically, the Scanner + Shape C method had the highest counting accuracy of 99.38% and a false detection rate of 0%, as well as a very low miss detection rate.

Discussion
In this study, we not only developed a linear regression model and a deep learning model to count the grain number per rice panicle, but also tested the grain counting efficiency of traditional image-based counting  methods. Traditional grain counting methods rely primarily on the separation of touching grains, and mainly algorithms include the dilation and erosion method [25], the improved watershed algorithm [26], and feature point matching method [27,28]. The commonly encountered problems of these traditional methods are  Fig. 8. The dilation and erosion method often fail to separate touching kernels (Fig. 8c). The improved watershed algorithm was able to detect the kernel edges, but the kernel surface was over-separated and could not account kernel number accurately (Fig. 8d). The corner detection and feature matching methods (Fig. 8e) had an unsymmetrical detection capability and no corner point was detected on the yellow line side, while more corners were detected on the other side.
Therefore, we do not believe that this method can provide satisfactory separation. The removal of stems during image preprocessing improves the counting accuracy. We compared the model performance in the presence and absence of stems using accuracy as comparison criteria (Table 7). Both the linear model and the deep learning model improved counting accuracy after stem removal, with a maximum improvement of 3%. The deep learning model had significantly higher counting accuracy compared to the linear model, which exhibited as high as a 10% accuracy difference between different rice types. Meanwhile, the deep learning model had a similar performance regardless of rice types. These results demonstrated that the impact of rice type on the model performance was significant for the linear model, but had less of an effect on the deep learning model.
To further validate the robustness of linear regression models, we used two varieties of Indica rice (Yangdao No. 6 and Liangyoupeijiu) and two varieties of Japonica (Lianjing No. 7 and Huaidao No. 5) as new materials to analyze subspecies varieties. Take the Scanner + Shape   C method as an example, and the verification results are shown in Fig. 9. The R 2 and RMSE of the Indica rice model were 0.9737 and 7.1120 respectively. The R 2 and RMSE of the Japonica rice model were 0.9623 and 6.4746 respectively. Compared with the results of Fig. 6, there is a slight decrease. So, the robustness of linear regression models is greatly affected by the shape of the panicle and grain, and is less affected by the size of the panicle and grain. The shape is mainly controlled by the subspecies, and the size is controlled by cultivation measures such as variety, density and nitrogen application rate. The deep learning model did have a slightly higher performance for the Japonica rice than the Indica rice. We analyzed images from test set that had a low counting accuracy (Fig. 10). The Indica rice grains were long and slender, which make them more likely to form a cluster and consequently, can't be detected as effectively by the deep learning model. Therefore, the counting accuracy of Indica rice is lower than that of Japonica rice regardless of model.
When using the image-based algorithm, the running time of the model was also an important factor. We took 20 panicle images as a group and calculated the time spent. Repeat 5 times and take the average as results ( Table 8). The time needed for counting was twofold: the time needed for image acquisition and the time needed to run the algorithm. The Shape A image acquisition time was the fastest among these conditions. However, the results in Table 4 showed that the accuracy of using Shape A images was low. The use of X-ray technique or three-dimension (3D) image will have a better performance on Shape A. Charytanowicz et al. [29] used X-ray images to evaluate geometric features for wheat grain classification. However, X-ray equipment is expensive. Long-term acceptance of X-rays can cause a lot of damage to human body. Generally, the accuracy of the    3D scanner is high, which is suitable for obtaining large objects such as forest trees [30]. For small targets such as rice, expensive higher precision instruments are required. In addition, the acquisition and processing of 3D images may take more time. In summary, we take Shape B and Shape C as research objects rather than Shape A. The time needed to acquire images using a digital camera was 50% of that using scanner, and the time needed for Shape C was shorter than Shape B. During the time experiment, Shape B needs to carefully separate the intertwined branches, and the violent operation will cause the individual grains to fall off and affect the subsequent treatment. Shape C can be separated from the stem by cutting a knife. Therefore, Shape B takes more time than Shape C. When running the linear model, most of the time was used for image processing and for the extraction of parameters. Meanwhile, when running deep learning model, most time was spent on parameter loading. The linear regression model required significantly less time than the deep learning model due to the fact that the deep learning model must load millions of parameters and involves large amount of data execution. The Scanner + Shape B + Deep Learning method takes 8 min, which is only about one-third of the manual counting time. The brain power expended by manual counting was not included yet. Thus, using multiple sets of high-performance graphics processing unit (GPU) could significantly accelerate data execution [31]. In addition, model compression or the establishment of a simpler deep learning model could also reduce the model running time [32].

Conclusion
In summary, we established two models to count the grain number per panicle, a linear regression model and a deep learning model, which had a counting accuracy greater than 96% and 99%, respectively. However, the deep learning model required more time than the linear regression model. If we consider the time cost, linear regression model is recommended for counting the rice grain number per panicle. Otherwise, the deep learning model would be best to optimizing accuracy. We believe our high-throughput and rapid method for counting the number of rice grains per panicle is a useful tool for rice phenomics research.