A novel machine-vision-based facility for the automatic evaluation of yield-related traits in rice

The evaluation of yield-related traits is an essential step in rice breeding, genetic research and functional genomics research. A new, automatic, and labor-free facility to automatically thresh rice panicles, evaluate rice yield traits, and subsequently pack filled spikelets is presented in this paper. Tests showed that the facility was capable of evaluating yield-related traits with a mean absolute percentage error of less than 5% and an efficiency of 1440 plants per continuous 24 h workday.


Background
Rice is the staple food for a large number of countries and regions in the world, particularly in Asia [1]. Because the world's population is increasing, obtaining higher yields has been the primary breeding target of rice cultivation [2]. As a complex agronomic trait, rice yield is determined by the product of the grain weight, the number of grains per panicle and the number of panicles per plant. The number of total spikelets per panicle and the seed setting rate are two traits that multiplicatively determine the number of grains per panicle, and the grain weight is largely determined by the grain size, including the grain length, the grain width, and the grain thickness [3].
The evaluation of yield traits, including the number of total spikelets (including filled and unfilled spikelets), the number of grains (also known as the number of filled spikelets), the seed setting rate (the number of filled spikelets divided by the number of total spikelets), the 1000-grain weight, the grain length, and the grain width, is an essential step in rice breeding, genetic research and functional genomics research [4][5][6]. Currently, rice yield trait evaluation is mainly performed by experienced workers. When investigations of large numbers of plants are needed, the manual measurement process is very subjective, inefficient, tedious, and errorprone. Most importantly, manual measurements are greatly affected by worker fatigue, which is a major problem in conducting mass measurements and renders the evaluation results questionable. In addition to trait extraction and evaluation, data logging and seed management are two instrumental steps in rice research. Traditionally, the processing of data, seed packaging, and seed coding are preformed manually and are thus error-prone and unreliable. A mistake in data management and seed management would lead to incorrect decisions and treatment of the seeds and is thus intolerable in rice research. For this reason, at least three workers are normally needed to check and verify the data to avoid the mistakes.
Modern plant breeding technologies are able to produce hundreds to thousands of new varieties, creating the need for rapid evaluation of plant materials to provide pertinent information prior to entering the next cycle of selection [7]. The low efficiency of manual trait evaluation makes it unsuitable for meeting the increasing demand for higher evaluation speeds. Automated assessment and measurement of plant phenotypes is therefore indispensable [8]. Several efforts have been made to automate plant phenotyping, such as automated analysis of plant leaves [9], roots [10,11], hypocotyls [12], tillers [13], and shoot biomasses [14] and whole adult plants [15,16]. Plant phenotyping facilities have also been established in large research centers and universities in Australia [17,18], Germany [19], the UK [20], and France [21]. However, to the best of our knowledge, very little information is available on rice yield trait evaluation. Previously, our group developed a method to identify and count filled/unfilled spikelets [22]. We integrated visible-light imaging and X-ray imaging to simultaneously calculate the total spikelet number and the filled spikelet number. Nevertheless, the accuracy and efficiency of the method was limited by the performance of the X-ray system. Moreover, because of the utilization of an X-ray system, the prototype was expensive and posed a radiation hazard; thus, it was not suitable for widespread use. Research on single-trait evaluation, mainly grain dimension measurement, has also been reported [23,24]. However, these studies have the disadvantage of measuring only a single trait.
This work aimed to develop an integrated and laborfree engineering solution for automatic panicle threshing, rice yield-related trait evaluation (including the number of total spikelets (NTS), the number of filled spikelets (NFS), the 1000-grain weight (TGW), the grain length (GL), and the grain width (GW)), and seed packing. The task involved the development of an automated threshing machine for the threshing of spikelets and the separation of spikelets and other unwanted materials; the design of mechanisms for separating the filled spikelets from the unfilled spikelets; the design of a machine vision system for imaging rice spikelets; the development of real-time algorithms for trait evaluation; the construction of a data logging and management system for data tracking; and the design of control and communication procedures to supervise the whole system, including a user-friendly interface.

Development of the SEA facility
The facility consisted of three major elements: a threshing unit, an inspection unit, and a packing-weighing unit ( Figure 1). The control center used software developed using LabVIEW 8.6 (National Instruments, USA) to control the whole system. The operating procedure includes the following steps: (1) the barcode of the rice plant being evaluated was obtained with a barcode scanner or by manual input, depending on the user's selection. (2) The threshing machine was started as panicles to be processed by the machine were detected. Spikelets were transferred to the inspection unit while impurities were collected at the impurity outlet. (3) The "total-spikelet-vision camera" collected images of the total spikelets (total-spikelet image) that came from the threshing unit (including filled and unfilled spikelets). A wind separator separated the filled spikelets from the unfilled spikelets. The 'filled-spikelet-vision camera' collected images of the filled spikelets (filled-spikelet image). The images were analyzed to determine yield traits. Unfilled spikelets were collected at the unfilled-spikelet outlet. (4) After inspection, lifting equipment raised the collected filled spikelets and delivered them to a packing machine for packaging. (5) A code-jetting machine printed the barcode of the recently examined rice plant on the packing bag. (6) An electronic balance weighed the filled spikelets and sent the data to the computer. (7) A collecting device collected the packed filled spikelets. An additional movie file shows the operation procedure in more detail [see Additional file 1].The developed prototype, dubbed the Seed-Evaluation Accelerator (SEA), is shown in Figure 2. The control flowchart is shown in Figure 3. The times required for panicle feeding (T f ), threshing (T t ), inspection (T i ), and packing-weighing (T p ) were designed to be 30, 50, 60, and 40 seconds, respectively. T idle was the idle time between subsequent measurements and depended on the operator. The threshing unit and the inspection unit worked in parallel with the packing-weighing unit, and T p was less than T i . Thus, the total processing time per rice plant was determined by Eq. 1:

Threshing unit
Panicles were fed into the threshing unit via a panicle inlet. The threshing of the spikelets was triggered by pulses received from a photoelectric sensor attached to the panicle inlet. The machine threshed the spikelets through roller-compaction processes. A sieve was used to separate the spikelets from branches and other unwanted materials. A tilted, vibrating plate under the sieve disaggregated the spikelets as they reached the end of the plate. After the spikelets were vibrated into the spikelet outlet, impurities were blown out through the impurity outlet by an air blower. Figure 4 shows the details of the inspection unit. The prototype used two line-scan cameras (Spyder 3 GIGE vision SG-11-02k80-00-R, Teledyne DALSA Company, Germany) to acquire 5000 × 2048 pixel grayscale images with a resolution of 0.23 mm/pixel. The cameras were controlled by the computer workstation (HP z600, Hewlett-Packard Development Company, USA) through an Ethernet card (NI PCIe 8235, National Instruments Corporation, USA) that digitized the images into 8-bit files. Two line-array LED light sources served as the illumination system. Spikelets coming from the threshing unit were transferred onto the first conveyor (420 mm wide), where the ellipsoidal spikelets lay flat. When the spikelets were transferred to the second conveyor, they were imaged by the so-called "total-spikelet-vision camera" as they passed through the field of view of the camera. Then, the filled spikelets and the unfilled spikelets were separated by a wind separator mounted between the second and third conveyors. Unfilled spikelets were blown out and collected at an unfilled-spikelet outlet, whereas filled spikelets fell onto the third conveyor and were imaged by the "filled-spikelet-vision camera". To spread out the spikelets, the second conveyor was designed with a higher speed than that of the first conveyor, and the Figure 1 Scheme of the SEA facility. Panicles were threshed by the threshing unit, and spikelets were transferred to the inspection unit. A wind separator separated filled spikelets from unfilled spikelets. One camera acquired images of the total spikelets (including filled spikelets and unfilled spikelets) and one acquired images solely of the filled spikelets. The images were subsequently analyzed to obtain yield traits. After inspection, the filled spikelets were packed and weighed.

Inspection unit
speed of the third conveyor was higher than that of the second. This design produced a separation of individual spikelets and consequently facilitated image analysis. A black conveyor belt was chosen, as it generated good contrast between the belt and spikelets and thus was beneficial for image segmentation.

Packing-weighing unit
As shown in Figure 5, after inspection, the filled spikelets fell into a grain-collecting tank placed under the third conveyor. Next, lifting equipment raised the tank and poured the filled spikelets into a packing machine. After packing, a code-jetting machine printed the barcode of the plant being evaluated onto the packing bag. Subsequently, an electronic balance weighed the filled spikelets and sent the grain weight (W grain ) to the computer. The TGW was obtained using Eq. 2:

Communication interface
The communication interface is illustrated in Figure 6. The resultant total-spikelet image and the filled-spikelet image, along with the yield trait data, were displayed on the interface. When the "Input barcode manually?" button was clicked, a dialog box was shown to allow users to input the barcode manually.

Automated data and seed management
At the beginning of the yield trait evaluation for each rice plant, the user chose to either scan the barcode of the plant using a barcode scanner or input the barcode manually. This barcode was transferred to the code-jetting machine, which then sprayed the barcode on the Figure 2 The developed prototype of the SEA facility. The facility mainly consisted of three units: a threshing unit for removing spikelets from the panicles, an inspection unit for assessing and measuring yield traits, and a packing-weighing unit for packing and weighing filled spikelets.  packing bag to facilitate seed management. Yield trait data of the rice plant were stored in a Microsoft Excel file along with the barcode of the plant for indexing and data management.

Performance evaluation of the hardware
Rice panicles from 214 harvested Huageng 295 rice plants were tested to evaluate the performance of the prototype. All of the spikelets, including spikelets at the impurity outlet, the unfilled-spikelet outlet and the filled-spikelet outlet (packing machine), were collected. The number of total spikelets and the number of filled spikelets at the three outlets were counted separately and recorded. For the NTS, NFS and TGW, each rice plant was evaluated three times by different personnel, and the average values were computed as reference data.
Manual observations were defined and computed as given in Table 1.
The threshing machine worked well during the tests. The absolute threshing error for total/filled spikelets was calculated as the total/filled spikelet number at the impurity outlet. The percentage threshing error for total/filled spikelets was calculated as the absolute threshing error divided by the number of total/filled spikelets of the rice plant being evaluated. Figure 7 illustrates the percentage error of the threshing unit for total spikelets and filled spikelets. As shown in Figure 7, the threshing error for total spikelets was higher than for filled spikelets, indicating that it was more difficult to thresh unfilled spikelets than filled spikelets. It can be observed from Figure 7 that some samples had threshing errors that were significantly higher than the average error. This was because the feeding speed had an important influence on the threshing of the spikelets. Two panicles entering the threshing machine at the same time would lead to fewer spikelets being threshed. These spikelets that remained on the panicles would be blown out with the panicle branches as impurities and, consequently, increase the threshing error. The mean absolute error and mean absolute percentage error of  the threshing unit were 34 and 3.33%, respectively, for total spikelets and 19 and 2.27%, respectively, for filled spikelets.
The multi-stage conveyor proved able to segregate most of the spikelets. As a result, most kernels in both the total-spikelet image and the filled-spikelet image were observed to be isolated kernels. The packingweighing unit worked properly during the test, with average weighing differences of less than 0.01 g between automatic measurements and manual measurements.

Performance evaluation of the image analysis algorithms
To test the performance of the image analysis algorithms, manually threshed spikelets of 90 Huageng 295 rice plants were fed into the inspection unit and imaged by the cameras. Manually threshed spikelets were used to exclude measuring errors caused by threshing. Figure  8 shows the results of manual observation versus image analysis of manually threshed spikelets. In comparison, the image-analysis performance with automatically threshed spikelets coming from the threshing unit was also investigated ( Figure 9). The mean absolute error and the mean absolute percentage error with manually threshed spikelets were 22 and 1.36%, respectively, for the NTS and 7 and 0.54%, respectively, for the NFS. In comparison, the mean absolute error and the mean absolute percentage error with automatically threshed spikelets were 27 and 2.81%, respectively, for the NTS and 15 and 1.77%, respectively, for the NFS. As expected, the measuring accuracy for the manually threshed spikelets was higher than for the automatically threshed spikelets. This was because the threshing machine removed the hulls of some spikelets, and the   broken hulls would appear in the image as impurities.
The external appearance of some broken hulls was similar to a spikelet, and consequently, they would be mistakenly treated as spikelets. In addition, some brown rice spikelets (spikelets without hulls) were much smaller than normal spikelets and thus would be mistakenly treated as impurities.
As measuring the GL and the GW of a grain is troublesome to perform, the mean GL and GW of only 50 rice plants were measured manually. For each rice plant, 10 filled spikelets were randomly chosen. Five workers measured the grain length and the grain width of each spikelet using a Vernier caliper, and the average value was regarded as the grain length and the grain width of one spikelet. To eliminate measuring errors caused by sampling, the facility measured the same 10 spikelets of each rice plant that were used for the manual measurements. The GL and GW of each rice plant were computed as the mean grain length and grain width values of the selected 10 spikelets.
Large variances were noted for the GL and GW estimation among different personnel. The lack of a mathematical definition of GL and GW and worker fatigue under continuous measuring conditions were believed to be two primary reasons for the huge variance among workers. Manual measurements and automatic measurements for GL and GW are illustrated in Table 2. Generally, automatically measured GL values were slightly larger than the values from manual measurements. However, GW values that were measured automatically matched well with those measured manually. The larger GL value from automatic measurement than from manual measurements was because there were errors in the manual location of the maximum length (GL), and inaccurate location of the length invariably leads to underestimation of the GL. Unlike GL measurements, imprecise location of the width may result in underestimation or overestimation of the GW. As a result, the average manually measured GW values matched well with automatically measured values.

Performance evaluation of the whole facility
Scatter plots of manual measurements versus automatic measurements of the whole facility for the number of total spikelets, the number of filled spikelets and the 1000-grain weight are shown in Figure 10. As weighing differences between automatic measurements and manual measurements were minor, the discrepancy in the TKW between manual and automatic measurements was chiefly caused by NFS measurement differences. Table 3 summarizes the mean absolute error (MAE, defined by Eq. 3) and the mean absolute percentage error (MAPE, defined by Eq. 4) of the whole facility for the evaluated yield traits. As shown in the table, the facility was capable of evaluating yield traits with mean absolute percentage errors of 4.02%, 3.33%, 1.47%, 1.31% and 1.08% for number of total spikelets, number of filled spikelets, grain length, grain width, and 1000-grain weight, respectively. The MAE and MAPE were computed using Eqns. (3) and (4) where n was the number of samples, x i.a was the i th automatically measured value, and x i.m was the i th manually measured value.
As observed from the results, the measuring error of the NTS was larger than that of the NFS. This was because broken hulls caused by threshing were also imaged by the "total-spikelet-vision camera". Additionally, larger errors can result from broken hulls, which  are very similar in appearance to complete spikelets and, consequently, may be treated as spikelets by the vision system but are ignored in manual observation. These broken hulls were blown out into the impurity outlet by the wind separator and thus would not influence the NFS measurement. Anther reason for the larger measuring error of the NTS was that the threshing error of the total spikelets was larger than that of the filled spikelets. The threshing error directly decreases the number of spikelets that pass through the inspection unit and, consequently, influences the measuring accuracy of the NTS and the NFS. The relation between the threshing error and the measuring error of the NTS and the NFS were investigated. Figure 11 shows the variation of the percentage measuring error of the facility for the NTS and the NFS as a percentage threshing error changes. As shown in Figure 11, the measuring error for both the NTS and the NFS presented an upward trend as the threshing error increased. Compared with the measuring error for the NFS, the measuring error for the NTS had a weaker relationship with the threshing error. This was because broken hulls had a considerable effect on the NTS measuring error.  . As the prototype needs no human operation except for panicle feeding, it is feasible to run the facility continuously for 24 hours for mass measurements. From this point of view, the facility was capable of evaluating rice yield traits with an efficiency more than 70 times greater than that of manual operation at its maximum throughput. The efficiency of the whole facility was determined by the inspection unit and the time elapsed, which was limited by the threshing time. More effective threshing methods will be developed in the future to improve the efficiency of the whole facility. The software was designed to allow the images to be processed concurrently as the cameras were acquiring images. Image processing requires less time than image acquisition. The shorter time needed for image processing also opened up the possibility of applying more sophisticated algorithms for more sophisticated solutions. For instance, statistical classifiers such as the distance classifier, discriminant analysis and artificial neural networks could be adapted in the future to discriminate broken hulls from spikelets.

Automation and integration of threshing, multi-trait measurement and seed packing
With the only manual operation being panicle feeding, the SEA facility automated the entire process of threshing, fast multi-trait measurement and seed packing. The yield traits were automatically observed and stored with a unique code after system inspection. Meanwhile, the seeds were automatically packed with the relevant code. The automation and integration of the entire process will substantially improve the yield trait evaluation process for rice researchers. With the data tracking ability, it was convenient for the user to manage and analyze data. Data tracking also allowed the user to combine yield traits with other traits such as the tiller number, the leaf area, the plant height, thus allowing an integrated understanding. Moreover, data tracking is beneficial for seed management. Compared with manual data logging and seed management, the data tracking of the facility made data management and seed management more robust and reliable.

Conclusions
This paper described an engineering prototype for the automatic evaluation of rice yield traits, including the number of total spikelets, the number of filled spikelets, the grain length, the grain width, and the 1000-grain weight. The prototype comprised three major units: the threshing unit, the inspection unit, and the packingweighing unit. The mean absolute percentage error was less than 5% for all of the evaluated yield traits, and the efficiency was approximately 1440 plants per 24-hours continuous workday. The facility will be helpful for improving the accuracy and efficiency of rice yield trait evaluation and will serve as a powerful tool in rice plant phenotyping, which will eventually benefit rice breeding, genetic research, functional genomics research and other rice research. With some modifications, the application could be extended and generalized to other crops, such as wheat, corn and barley. Other compound yield traits such as the seed setting rate and the lengthwidth ratio can also be deduced from the extracted traits. In summary, using agricultural photonics, the high-throughput facility, dubbed the Seed-Evaluation Accelerator, gives plant scientists a novel tool to unlock the phenotypic information coded in rice genome [25].

Image acquisition
The control software was designed for evaluating yield traits of one rice plant at a time. The continuous acquisition of the images was controlled by the pulses received from a photoelectric sensor attached to the panicle inlet of the threshing unit. Images were acquired and stored using the NI-IMAQ Virtual Instruments (VI) Library for LabVIEW (National Instruments Corporation, USA). For each plant, 14 images were acquired by the "total-spikelet-vision camera" (called the total-spikelet image) and 20 images were acquired by the "filledspikelet-vision camera" (called the filled-spikelet image). This design was applicable to most rice varieties with less than 20 panicles per plant. Figure 12 shows typical total-spikelet and filled-spikelet images acquired by the two cameras. Note that the images shown in Figure 12 have been cropped for better visualization, as the original images are too large (5000 × 2048 pixels).

Image analysis
The image-analysis software was programmed using NI Vision for LabVIEW 8.6 (National Instruments Corporation, USA). The software was designed to allow the two cameras (the "total-spikelet-vision camera" and the "filled-spikelet-vision camera") to work at the same time, and the images were analyzed in the computer simultaneously while the cameras were acquiring new images, thus optimizing the measurement efficiency. Figure 13 outlines the flowchart of the algorithm for determining the spikelet number in an image (a totalspikelet image or a filled-spikelet image). Image segmentation was performed to determine the background and objects of interest. For better processing speed, a pixel-oriented segmentation algorithm was applied in this study. A pixel was considered to belong to a background point if its grayscale fell below a predefined, fixed threshold. In a subsequent step, a median filter (3 × 3 neighborhood) was used to remove isolated pixels. As some small pieces of branches may appear on the conveyor, objects with a length-width ratio greater than three times that of the spikelets were treated as branches and removed. Branches with a length-width ratio less than three times that of the spikelets were subsequently removed using the "IMAQ detect line" operation. Small regions with an area less than half of the average area of spikelets were regarded as impurities and removed from the image. Spikelets may be touching during on-line processing. The shape of an isolated spikelet is roughly elliptical, so the "IMAQ detect Ellipse" operation was used to identify the isolated spikelet in the image (NI Vision Concepts Manual, National Instruments Corporation, USA). After identification, the original image was divided into two images, one image with only the isolated spikelet (isolated image) and the other with only the touching spikelets (touching image). From an efficiency perspective, we opted for a simple areadetermination method to determine the spikelet number in the touching image N touching , which was computed by Eq. 5 and Eq. 6: N touching = T j=1 N j (6) where N j was the actual spikelet number for a given touching region j in the touching image, the function round(x) rounded x to the nearest integer, A j was the area of the touching region j, Ā was the average spikelet area calculated from the isolated image, and T was the number of touching regions.
The spikelet number (N) in the original image was determined by summing up the spikelet numbers in the  The length and width of each isolated spikelet in the filled-spikelet images were calculated. The GL is defined as the maximum Euclidean distance between two boundary points of a filled spikelet, and the GW is defined as the maximum length of straight lines perpendicular to the line of the GL. Figure 14 shows the structure of the image-processing program in LabVIEW for total-spikelet images and filled-spikelet images. Illustrations showing block diagrams of the VIs developed in this research are attached in Additional files 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, and 17.