Image-based plant wilting estimation

Background Environmental stress due to climate or pathogens is a major threat to modern agriculture. Plant genetic resistance to these stresses is one way to develop more resilient crops, but accurately quantifying plant phenotypic responses can be challenging. Here we develop and test a set of metrics to quantify plant wilting, which can occur in response to abiotic stress such as heat or drought, or in response to biotic stress caused by pathogenic microbes. These metrics can be useful in genomic studies to identify genes and genomic regions underlying plant resistance to a given stress. Results We use two datasets: one of tomatoes inoculated with Ralstonia solanacearum, a soilborne pathogen that causes bacterial wilt disease, and another of soybeans exposed to water stress. For both tomato and soybean, the metrics predict the visual wilting score provided by human experts. Specific to the tomato dataset, we demonstrate that our metrics can capture the genetic difference of bacterium wilt resistance among resistant and susceptible tomato genotypes. In soybean, we show that our metrics can capture the effect of water stress. Conclusion Our proposed RGB image-based wilting metrics can be useful for identifying plant wilting caused by diverse stresses in different plant species.


Introduction
Ralstonia solanacearum (Rs) [2] is a soil-borne bacterium that infects plant roots and ultimately causes susceptible plants to wilt and die.Bacterial wilt caused by Rs [2] is a major threat to crop production around the world [15,18,34].In the United States, some of the most important vegetable crops, such as tomatoes and potatoes [1], can suffer significant yield loss from bacterial wilt [3,34].Figure 1 shows an example of a healthy tomato plant, while Figure 2 shows the growth of tomato plants exhibiting different levels of resistance after inoculation with the bacterium Rs.
One of the best bacterial wilt controlling methods is through developing genetically resistant plants, but resistance genes that act on US strains of Rs have not been identified [17].To identify resistance genes, metrics to quantify Rs-induced plant wilt are desirable.In this paper, we propose a set of computational wilting metrics using several RGB images of the plant.We tested our metrics on high Rs resistance and low Rs resistance tomato plants, both with and without bacterial infection.In addition, we use machine learning-based methods to show that expert-labeled wilting scores can be predicted using our metrics.

Review and Assumptions
There are many ways for plant scientists to quantify the effect of plant diseases [10,18,19,20], but each method has its own shortcomings.Some studies use crop yield (amount of a crop grown per unit area of land) as the metric [13].In [18], Ji et al. use crop yield as an indicator to study the effectiveness of using the chemical Thymol as a biofumigant to control Rs-induced tomato bacterial wilt.In [19],  The downside of using crop yield is that it requires long waiting periods for the plants to be ready for harvesting.Many studies also measure the amount of pathogen directly from samples of the plants [12,20,21].To do this, either the whole plant or parts of the plant are collected to be examined for the presence of the pathogen.This destructive sampling method often hinders plant growth and sometimes even kills the plants.In addition, both crop yield and destructive sampling methods are difficult to use to show the long term wilting trend of the same plant.Another approach is to have experts visually examine the plants to determine wilting.For example, in [10], Engelbrecht et al. measure leaf water potential using visual assessment that is very subjective and difficult to reproduce.For tomato plants, experts rate each plant on its degree of wilting, taking into consideration many features of the plant such as the the overall loss of plant mass and the shift in color [3].In most cases only trained experts are capable of assigning a proper visual score.We choose to design our wilting metrics using several RGB images of the plant.RGB images are commonly used for other plant studies [6,9,11,23].Using RGB images is also non-destructive and easily accessible.
Several sensor-based wilting metrics in the past have been proposed.In [7], Caplan et al. use manually measured leaf angles as an indicator of drought stress.In [5], Bock et al. determine disease severity with RGB images of individual leaves.These methods require acquiring individual leaves so they are very labor intensive.Other methods such as [22,31] also use RGB images for estimating wilting, but they require special equipment such as guided rail cameras and laser sensors [31] or field servers [22].Our method uses RGB images from multiple views using a single RGB camera which requires much less equipment and labor.
In [26], Sancho et al. use RGB image-based color information as part of the metrics to estimate Verticillium wilt of olive plants.Sancho et al. incorporated eighteen color measurements into their metrics, but some of the metrics require cutting the olive leaves.Similar to Sancho et al., the metric we proposed also uses RGB image-based color information, but we reduce the number of color based metrics from eighteen to one and our method does not require cutting physical leaves.We also add a color correction step to account for the difference in imaging conditions.
From our observations, bacterial wilt has significant impact on the color and shape of plants.Examples can be seen in Figure 1 and Figure 2. Our primary approach is to estimate color and shape information from RGB images.For color information, we estimate the color distribution of the plant image pixels.For shape analysis, we first use the convex hull [4] to capture the outline of the plant.Since the convex hull only captures information related to the outer edge of the plants, we developed several stem-based metrics which estimate the distribution of the plant materials relative to the stem.

Proposed Wilting Metrics
In this section, we first give an overview of our wilting metrics.For our experiment in Section 4 we use two types of plants: (1) Solanum lycopersicum, or Hawaii 7996 (HA), which is a high Rs resistance plant; and (2) Solanum pimpinellifolium, or West Virginia 700 (WV), which is a low Rs resistance plant [14].Both types of plants are divided into an "inoculated" (the bacteria Rs is introduced to the plant via water) group and a "mock" (the control group, no Rs is introduced) group.

Overview
Figure 3 shows a block diagram of our proposed method.Initially we use color correction, plant segmentation, and stem segmentation on the original RGB images.Color correction is used to address the image color inconsistency caused by the camera settings and acquisition conditions (e.g., lighting).Plant and stem segmentation are used to All metrics can be categorized into color, shape (nonstem) based, and stem-based metrics.Color and shape based metrics use color corrected images and the corresponding plant mask.Stem-based metrics require the plant mask and stem mask.To show the utility of our metrics, we also trained a random forest to predict a visual wilting score for a plant using our metrics.

Initial Processing
Color Correction: Since we are capturing wilting information across multiple days and for multiple plants, it is important to ensure that the lighting condition variations are suppressed as much as possible.We color correct our images using a reference object that is known as a Fiducial Marker (FM) which can be seen in Figure 5.The FM is a colored checkerboard that has known physical dimensions and known colorimetric pixel values for each of its color squares.We detect the FM in the image and estimate the average RGB pixel values for each of the color squares in the image.Since we know the actual RGB values of the color squares, we can transform each pixel in the image to correct the color of each pixel.Consider a 9 × 3 matrix C image , which consists of the average R, G, and B pixel values for the 9 color squares of the FM.We also know the actual colorimetric pixel values values of the color squares of the FM, represented by the 9 × 3 matrix C real .We estimate the 3 × 3 transformation matrix T : An example of color correction is shown in Figure 5.
Plant Segmentation: After the color correction is completed, we segment the plant from the background.After examining the images in multiple color spaces, we selected two channels that captured the plants in our image sufficiently: (1) the V channel from the HSV color space; and (2) the B* channel from the L*A*B* color space.We obtain two preliminary plant segmentation masks from these channels by empirically determining separate thresholds for each image.For our experiments, we used a threshold of 140 for the V channel and 130 for the B* channel, assuming the pixel values are between 0 and 255.These preliminary binary masks are then combined with the logical 'OR' function to obtain a single mask containing the information from both the V and B* channels.To ensure any undesired objects in the mask are removed, we use sequential opening and closing operations [33] with a structuring element of ones in a 3 × 3 matrix to remove noise and fill holes in the mask.
Stem Segmentation: For stem segmentation, we use a deep neural network-based solution similar to what was used by Yang et al. [32].A set of stem masks are manually labeled and used to train the stem segmentation networks.Two convolutional neural network (CNN) [27] models, Mask R-CNN [16] and U-Net [25], are used for our stem segmentation.The Mask R-CNN network is used for the majority of the stem segmentation tasks.For the plants where Mask R-CNN fails to detect stems, we use the U-Net to detect the stem.Mask R-CNN produces better quality [32] masks compared to U-Net but it sometimes fails to detect the stem and outputs nothing.Since U-Net always produces a mask, it is used as a complementary stem segmentation network.To summarize, with the trained neural networks, our stem segmentation step takes the original image as input and generates a stem mask (Figure 4) as output.

Color and Shape Based Metrics
Color: Using the plant segmentation masks described above, we next examine plant shape and color.We are able to segment the relevant parts of the image, consisting of the plant, and examine the distribution of pixel values in the A* channel of the L*A*B* color space.Since wilting plants tend to change color from green to brown, we want to capture the trend of the histogram over time.We compare the differences in the pixel distribution over time using the Bhattacharya distance [8].Our assumption is that resistant plants will have very small deviation from their original color distribution, while the wilting plants have a larger deviation.
Shape: We capture the general shape of the plant using the convex hull [4] around the plant mask.We define two convex hull metrics: perimeter of the plant object; and area of the convex hull of the plant object.We can also find area of the plant object, width of the plant object, and height of the plant object using the plant mask p msk (x, y) (Figure 4).
Plant Height, Area, and Width: Our indexing orientation is shown in Figure 7 and let the image pixel resolution be C pres .From the plant mask we can find plant area P area which is the total area of the plant material.
A horizontal profile h hor and a vertical profile h ver are es- Plant width P width is defined as the difference between the leftmost pixel and rightmost pixel of the plant mask.
where h ver (P max−l ) ≥ 1, h ver (P max−r ) ≥ 1, For plant height P height , we remove the top 5% of plant material and label the y-coordinate of the 5% plant material cutoff line as Y T op , where We then define the upper edge of the pot as the bottom of the plant, denoting its average y-coordinate as Y Bot .P height is defined as the difference between the 5% plant material line and the bottom of the plant (Figure 7).We remove the top 5% of plant material so a small leaf at the top of the plant will not affect the plant height metric.Figure 7 and Figure 8 show a visualization of the P height and P width .Plant Height, Area, and Width give an overview of the general shape of the plant.As a plant wilts, we expect almost every metric listed here to decrease.The plant loses volume and turgidity, reducing its area, convex hull area, height, and width.The

Stem-Based Metrics
Figure 2 shows the change in plant material distribution for a healthier plant and a more infected plant.For a healthy plant in its early growth stage, the distribution of the plant material tends to be further away from the stem.Once the plants becomes infected and starts to wilt, the plant material will get closer to the stem due to the decreasing structural rigidity [29].Based on our experimental observation and plant physics, we consider the stem of plant to be a good reference point for plant material distribution estimation.
The inputs required for stem-based metrics are the plant mask p msk (x, y), and stem mask p stem (x, y).Both the plant mask and stem mask are binary images with size M × N pixels.
Center of Mass: From the stem mask p stem , we use linear regression [28] to form the function s lin (y).Using s lin (y) we can then separate plant mask p msk (x, y) into a left plant mask p l−msk (x, y) and right plant mask p r−msk (x, y) (Figure 6).
We then estimate the left center of mass CM lef t and right center of mass CM right . x,y x • p l−msk (x, y) x,y Vertical Distribution: The goal of the vertical distribution is to capture the plant material distribution along the y-axis for each half of the plant mask.We sample the distribution at 33%, 66%, and 90%.Lets use 90% as an example.For the left plant mask, we estimate the horizontal profile h l−hprof .
Then we find the y-coordinate of the 90% plant material line Y l−90v , where The same steps are used to find Y r−90v using right plant masks.The average 90% distribution V 90y is defined as Horizontal Distribution: The goal of the horizontal distribution is to capture the plant material distribution along the x-axis for each half of the plant mask.We sample the distribution at 33%, 66%, and 90% as well.Because the stem is not always vertical, we are not able to find a xcoordinate for the stem.We first do a horizontal shift to p r−msk (x, y) using the stem separation line, resulting in a shifted R r−msk (x, y).We then shift and flip p l−msk (x, y), resulting a flipped and shifted R l−msk (x, y) (Figure 10).
The rest of the steps are the same as vertical distribution.Lets again use 90% as an example.For left half plant masks, we estimate the horizontal profile h l−vprof .Then we find the x-coordinate of the 90% plant material line X l−90h .Similar steps are taken to find X r−90h using right plant masks.The horizontal 90% distribution H 90 is the sum of X r−90h and X l−90h .Figure 10 shows a example shifted maskR l−msk (x, y) and X r−90h .
All the shape-based and stem-based metrics above are then converted from pixels to metric units using the pixel resolution.Pixel resolution can be estimated from the dimensions of the Fiducial Marker.

Experimental Design
We first demonstrate that our metrics can distinguish different plant genetics as well as how they been infected.We are also interested in comparing our methods to the way experts visually score a plant.Experts examine the plants visually eight days after inoculation and rate each plant on its degree of wilting using a value between 0 and 1.We describe a random forest network method below to generate a predicted visual score from the proposed metrics.We then compare the predicted visual scores with visual scores assigned by expert plant scientists.
Dataset: Two species of tomato, Hawaii 7996 (HA) and West Virginia 700 (WV), are planted in a growth chamber For all images, we use the same camera positioned at the same location and under controlled lighting conditions.Fiducial markers are used for color correction.Each time a plant is imaged, eight side-view images are acquired from eight angles.For the inoculated group, expert observers assign a wilting score between 0 and 1 to each plant eight days after inoculation.In summary, there are 122 plants in the inoculated group.Of the 122 plants, 61 are HA and 61 are WV.We have the expert visual scores for all 122 plants.The mock group contains 36 plants (18 HA and 18 WV).Each image is 5496 × 3670 pixels with pixel resolution of 0.052 cm / pixel.
Statistical Analysis: The goal of the statistical analysis is to show that our metrics can capture group differences.For example, in general, since HA should wilt less than WV after infection, we can check if our metrics are showing significant statistical difference between inoculated HA and WV.For each plant, we estimate our metrics from all eight side-images, using the average for each metric as the final value.There are cases where no metrics can be estimated from an image.This usually occurs if the plant is too small and plant segmentation or stem detection fails due to the image capture angle.For these plants, the metrics estimated from the remaining sides are still averaged and used.
Random Forest: We also compare our metrics to visual scores obtained by experts.Our goal is to use our image-based metrics to generate a score that is similar to the expert's visual score.Experts visually examine the plants eight days after inoculation, which is two days after the last images are captured.They rate each plant on its degree of wilting using a score between 0 and 1, taking into consideration many features of the plant such as the turgidity of the petioles, the overall loss of plant mass, and the shift in color.
For our experiments, we treat the problem as a binary decision: either the plant has wilted or not.Any plant with an expert wilting score above 0.5 is considered as "wilted", and any plant with an expert score below would be "not wilted".We split the image-based metrics and the associated expert visual scores in a 6:4 ratio for training and testing.We then train a random forest consisting of 1000 decision trees.The random forest is evaluated by using the testing data to predict a visual score using our image-based metrics and then comparing it to the actual expert score.

Statistical Analysis
CM hor−dis Difference: In this section we compare the change of CM hor−dis from -1dpi to 3dpi. Figure 11 shows the results for CM hor−dis difference between 3dpi and -1dpi.Table 1 shows the Welch's t-test [30] results for some of the pairs.From the statistical test results (Table 1) and the distribution chart (Figure 11), ∆CM hor−dis shows no statistical difference between HA and WV when grown without Rs.Once the plants are inoculated with bacteria, both HA and WV show decrease in ∆CM hor−dis .This demonstrates that our metric is showing the effect of the inoculation on the plant as early as 3 dpi.
We can also observe that once the plants are inoculated, the more Rs-resistant HA plants have a much smaller decrease in ∆CM hor−dis than WV plants.This is a indicator that our metric is capturing the plant resistance to Rs rather than some other inherent genetic difference because ∆CM hor−dis only shows a statistical difference between HA and WV when the plants are inoculated with Rs.Overall, difference of CM hor−dis matches our prior knowledge about the resistance of the plants, and it demonstrates the effect of inoculation at an early stage.
Bhattacharya Distance: In this section we examine the impact of bacterial wilt in the inoculated group.Figure 12 shows the Bhattacharya Distance (BD) [8] of pixel color distributions on different dpis for inoculated HA and WV plants.Here, the BD measures the difference in the pixel distribution of color for each day post inoculation from the pixel distribution of color pre-inoculation.The Kruskal-Wallis [30] test for inoculated HA plants returns a p-value of 0.235 and for inoculated WV plants, it returns a p-value of 3.52e − 12.
As we can see from Figure 12, the distribution of the color pixels in inoculated HA plants do not have significant

Random Forest
Recall that we have obtained expert visual scores representing the degree of wilting for each of the plants eight days post inoculation.Our goal is to use our image-based metrics to generate a score that is similar to the expert visual score.We want to demonstrate that our metrics contain enough information to reach the same conclusions as a trained expert.As discussed previously, we train a random forest consisting of 1000 decision trees to generate a score using our image-based metrics.We evaluate the scores output by the random forest using Precision, Recall, and F1 Score [24]: These metrics can be used for either class of "wilted" or "not wilted", where the definitions of True Positive (TP), False Positive (FP), and False Negative (FN) will change depending on the class.The results are shown in Table 2.For the testing dataset of 48 plants, the random forest model was able to use our metrics to predict the final state of the plant with an accuracy (F1 Score) of 100%.The predicted visual scores from our metrics match the scores provided by the experts for every plant.Note that the expert visual scores are based on the state of the plant at eight days post inoculation.The performance of our model is achieved using metrics from images up to six days post inoculation, meaning that the random forest is able to predict the final state of the plant two days in advance.

Conclusion and Future Work
In this paper, we proposed image-based metrics for estimating the amount of wilting in a diseased plant.We demonstrate that our metrics are able to distinguish between diseased plants and healthy plants of the same species.We can also distinguish between resistant plant species and less resistant plant species.We also use our metrics to predict visual wilting scores using a random forest.We conclude that our metrics are effective for measuring bacterial wilt.For future work we will extend our methods to other species and investigate new features to improve our wilting estimation.

Figure 1 :
Figure 1: An example of the growth of a plant without bacterial inoculation (a) -1 dpi (days post inoculation) (b) 3 dpi

Figure 2 :
Figure 2: Plants inoculated with Rs bacteria.Here, dpi means days post inoculation.The Hawaii 7996 (HA) plants are resistant to infection while the West Virginia 700 (WV) plants are susceptible to infection.

Figure 3 :
Figure 3: Block Diagram for Our Method

Figure 4 :
Figure 4: An example of a mock WV plant, all images are cropped : (a) Original Plant (cropped).(b) Plant Mask.(c) Stem Mask.(d) Stem Mask overlaid on original

Figure 6 :
Figure 6: An example of the segmentation mask process: (a) Original Plant (b) Plant Mask (c) Left plant Mask (d) Right Stem Mask (e) Left CM (f) Right CM

Figure 7 :
Figure 7: Plant Height and Index Orientation

x
• p r−msk (x, y) x,y p r−msk (x, y) , x,y y • r−msk (x, y) x,y p r−msk (x, y) coordinate difference between the left and right center of mass (CM) is set to be Center of Mass Horizontal Distance CM hor−dis .The average of the difference between the CM y-coordinates and the bottom of the plant Y Bot is defined as Center of Mass Height CM height .Figure 9 shows a visualization of the metrics.

Figure 11 :
Figure 11: Change of CM hor−dis from -1dpi to 3dpi in cm

Figure 12 :
Figure 12: Bhattacharya Distance for inoculated HA and WV plants

Table 1 :
Center of Mass (CM) Distance t-test results

Table 2 :
Random Forest Classification Results