A method for automatic segmentation and splitting of hyperspectral images of raspberry plants collected in field conditions
© The Author(s) 2017
Received: 30 May 2017
Accepted: 11 September 2017
Published: 1 November 2017
Hyperspectral imaging is a technology that can be used to monitor plant responses to stress. Hyperspectral images have a full spectrum for each pixel in the image, 400–2500 nm in this case, giving detailed information about the spectral reflectance of the plant. Although this technology has been used in laboratory-based controlled lighting conditions for early detection of plant disease, the transfer of such technology to imaging plants in field conditions presents a number of challenges. These include problems caused by varying light levels and difficulties of separating the target plant from its background. Here we present an automated method that has been developed to segment raspberry plants from the background using a selected spectral ratio combined with edge detection. Graph theory was used to minimise a cost function to detect the continuous boundary between uninteresting plants and the area of interest. The method includes automatic detection of a known reflectance tile which was kept constantly within the field of view for all image scans. A method to split images containing rows of multiple raspberry plants into individual plants was also developed. Validation was carried out by comparison of plant height and density measurements with manually scored values. A reasonable correlation was found between these manual scores and measurements taken from the images (r2 = 0.75 for plant height). These preliminary steps are an essential requirement before detailed spectral analysis of the plants can be achieved.
Plant breeders are constantly striving to improve crop productivity through the breeding of plants with desirable agronomic traits that are also tolerant of abiotic stresses such as drought, mineral deficiency or heat stress and biotic stresses caused by pests and diseases. Raspberry is a perennial crop species grown widely in Europe and North America, and raspberry production faces a number of specific challenges. In terms of biotic stress, the spread of root rot disease caused by Phytophthora rubi has devastated many growers’ plantations and, once present, there is no effective treatment. The development of resistant raspberry varieties is therefore a key priority for breeders and a molecular markers breeding strategy is currently underway [1, 2]. Other challenges include combating insect pests, particularly root-feeding vine weevil larvae, and producing quality fruit [3, 4]. Recently, developmental disorders have also affected raspberry yields with increased incidence of crumbly fruit , lack of evenness and variable timings of bud break, and inconsistency in flowering and fruiting [5, 6]. Therefore, resilience in new variety development is essential to ensure future success in the raspberry industry.
Breeders therefore require tools for detecting the biochemical, physiological, and developmental responses to such stresses. These ‘phenotyping’ tools need to be capable of high-throughput screening of the huge number of plants that are typically generated and characterised as part of plant breeding programmes, whether for the mapping of quantitative trait loci (QTL) or for direct selection, and they need to be applicable in the field, rather than in controlled environments alone [7–9]. Sensors that detect reflected light from plants provide a technology that could satisfy these requirements. A wide range of sensors is now available for field use [10, 11]; these are most efficiently deployed on mobile field platforms [12, 13], while lighter sensors may also be deployed on unmanned aerial vehicles (UAVs) [14–16].
Among the many sensors available, there has been particular interest in the past few years in hyperspectral sensing of plant reflectance in the visible (400–700 nm), near infrared (NIR, 700–1000 nm) and the shortwave infrared (SWIR, 1000–2500 nm). This makes use of characteristic features of the reflectance spectrum that depend on biochemical or structural features of plant leaves and can be used for quantification of biochemical and physiological responses [17–19]. The power of hyperspectral remote sensing for detection and characterisation of plant responses to stress has been amply demonstrated by using in-field non-imaging spectroradiometry for the characterisation of traits as diverse as leaf biochemical composition , leaf water status  and responses to biotic stresses, however, the use of hyperspectral imagery has potential to greatly refine the power of this technique.
This study reports the development of an image processing method to support the application of a novel field phenotyping platform that incorporates two hyperspectral scanners: (1) SWIR scanner covering region 1000–2500 nm; and (2) a visible and near infra-red scanner (VNIR) covering the region 400–1000 nm with a combined capacity of recording 400 different wavelength bands. The scanners were mounted on a trolley pulled behind a tractor in such a way as to provide vertical scans of the lateral view of the plant canopy. Individual scans as the trolley moved forward were combined in a push-broom manner  to generate images of the lateral aspect of the plant row. A number of image analysis steps are required to extract the relevant data from the images for use in plant phenotyping.
Unfortunately, the shift to imagery as a phenotyping tool generates enormous amounts of data, necessitating the development of automated image analysis and statistical techniques for the required data reduction and synthesis [22, 23]. Image analysis techniques range from those that retain spatial information about the patterns of spectral variation (of particular interest for plant diseases or mineral disorders where there may be characteristic patterns of leaf colour [24, 25]) to the straightforward extraction of average spectra for the class of plant material of interest. The use of this latter approach is equivalent to the use of non-imaging spectrometers , but has the added power of eliminating error caused by those areas of the sensor field of view that do not correspond to the material of interest (i.e. plant leaf surfaces or even ‘sunlit’ leaf surfaces alone). Much previous work using spectral sensing for plant phenotyping has been conducted in highly controlled conditions , or focusses on relatively few wavelengths from red, green, blue (RGB) or multispectral sensors . While growing plants in growth chambers or glasshouses allows environmental conditions to be closely controlled, it is important that phenotyping methods are applicable to the field conditions experienced by plants in a commercial setting. Imaging in field conditions produces additional challenges including the rapidly varying light exposure, influence of wind turbulence and the need to distinguish the target plants from a complex and varying background: in the field we are unable to move plants to an ideal position for imaging.
This study focusses on the critical step in image analysis of segmenting raw images to separate data linked to the leaf material of interest from that linked to other material such as stems, soil or other background features. Segmentation approaches for hyperspectral images can be split into two categories: those that attempt a pixel level classification based on the spectral signature of each pixel; and object based methods where location of the pixels is taken into account [28, 29]. Pixel based approaches have been commonly used for remote sensing data from either satellite or aircraft mounted scanners. These work well in situations with a variety of pixel classes that are well defined but where spatial resolution is poor. As the spatial resolution of hyperspectral imagers has improved, object based segmentation methods have become more common. These exploit information about the location of pixels in images to improve segmentation accuracy, detecting continuous objects that can then be allocated to a particular class. For field based systems, the limited number of potential object classes and high spatial resolution provides additional options for segmentation of image data.
The aim of this study was to develop an automatic method to segment the hyperspectral images identifying and labelling plants of interest and detecting reflectance standard. The approach chosen was to produce a single channel image from a combination of wavelength bands to segment important objects in the image. One major advantage of this approach is that dealing with only a few wavelength bands, reduces the image size and memory requirements for the image processing. It also allows generic machine vision techniques that are not specific to hyperspectral imaging to be used. The objectives were to overcome the main challenges associated with (1) detection of the white reference tile and (2) the accurate splitting of the image into individual plants.
Plants in the field
A replicated mapping population of 188 offspring with parents of Glen Moy and Latham raspberry varieties was planted at a field site at the James Hutton Institute, Dundee, UK. The advantage of using a mapping population for this work is that the image data can be related to molecular markers for traits via QTL analysis. Several linkage maps have been developed for traits of interest using this mapping population [2–5, 30, 31]. The plants were subject to two different biotic stresses (root rot and vine weevil, together with a stress-free control) and three different irrigation levels (control, drought, overwatered). The root rot plots were located on a site with known infection. Vine weevil plots were infested with eggs produced from live cultures held at the James Hutton Institute. Due to the extra space the need for biosecurity containment, the vine weevil plot only had two different irrigation levels. These treatments were combined to give eight different treatment combinations. Two replicates of each plant genotype were assigned to random positions within each block. The plots were uncovered in the field so control of water levels was limited by weather conditions. The plots were weeded and the raspberry plants were trained to grow on wires held up by wooden posts in line with standard practice for growing raspberry.
Imaging set up
The SWIR and VNIR scanners were mounted at the same height with a 30 cm horizontal offset between them. They were oriented to cover the same area of the studied plants. The images from each scanner were analysed independently rather than attempting a registration of the two images. Registration was not attempted as it would add in an extra source of potential error to the data analysis. A slightly differing vertical resolution would mean any registered results would need to be interpolated. The horizontal offset between the two scanners meant there was a time delay between imaging the same plant by the two different scanners adding in potential error on any days with sufficient wind speed to disturb the plants.
Dark current was removed from the SWIR images by subtracting the spectrum acquired when no light was reaching the detector from the spectrum returned in each image. The dark current spectrum is unique to each vertical pixel within the image. The dark current for the VNIR scanner was much smaller and removal was not necessary to achieve good quality images.
The imaging method (described above) ensured that all plants of interest were contained in the images. The first step of image processing involved manually cropping the ends of each image to remove non-target image data, which required the user to click on the image, at points indicating the start and end of the row of plants. A three band true colour representation of images was used for this purpose. No further manual intervention was required for image processing.
Further processing of image data differed slightly between the VNIR and SWIR scanners.
Automatic plant detection
The space between the rows of raspberry plants was grassed, which means that grass appeared at the bottom of each of the images. Our initial segmentation included this grass as part of the plant. An approach was developed to remove this non-target plant material by detecting the bottom of the cardboard background. Graph theory was used to find an edge across the NDVI image. Graph based segmentation has been used to solve a number of different image segmentation problems , including both plant segmentation problems  and much wider applications [36, 37]. The main advantages of this technique are the low computational power required and the lack of need for training data.
The area below the boundary found was removed from detected plant material.
The problem of the removal of grass from the foreground is also presents itself in the SWIR images. Here a different approach was used because attempts to replicate the method used in the VNIR images did not produce good results, due to noise in the SWIR images with a horizontal consistency that persisted even after dark current removal; this noise created false edges that were picked up frequently. Instead, a fixed part of the bottom of the image was removed. Although this resulted in removal of a larger section of the image than the VNIR method, the approach ensured the grass was reliably removed.
Automatic detection of white reference
A method was developed for automatic detection of the white reference tile in each image to allow correction of changes in light levels between images.
Following the initial detection of the white reference between the two bounding lines, further image corrections were carried out. In principle, the white reference was continuous across the image. In reality, the white reference was occluded in parts of the image by plant material and wooden posts.
Splitting into individual plants
Individual plants in each row of the trial were separated by a distance of 1 m (48 plants per row), which is larger than commercial plantations, but facilitated the task of splitting image files into individual plants. Separating individual plants was necessary as each plant in the row of an image was a unique genotype in the mapping population. Since the image was collected at a constant speed, the simplest approach would be to split each image into 48 equal sized sections, with each containing a separate plant. In practice, this approach was not robust due to errors caused by variation in tractor speed or plant growth that deviated from vertical, so a method based on our prior detection of plant material was used instead.
The number of pixels assigned as ‘plant’ material for each column was calculated. Peaks in this should correspond to plants and minima should correspond to gaps between plants. A peak finding function was used to identify peaks, while enforcing a minimum distance of 80% of the expected plant separation. The points midway between each peak were used to split the image into individual plants. Any areas where plant separation was too large were split into multiple regions to account for any plants that had been missed by the peak finding method. The number of plants identified and the known number of plants in the image was matched by either merging small regions or splitting large regions. Finally, the splitting points were adjusted to identify a local minimum, in the number of plant pixels, corresponding to an optimum point for splitting the image.
This method was applied to both VNIR and SWIR images.
Using the segmented image data, the spectrum of the white reference tile was used to normalise the image from pixel intensity values to reflectance. The image was divided into sections of fixed size then dividing the spectrum of each pixel in that section by the mean white reference spectrum from that section.
Finally, the normalised reflectance spectrum of each plant was extracted, together with measures of the plant height and mean density of the plant, using the binary segmented plant image. These final two measures were used to evaluate performance of the segmentation by comparison with visual scoring of the plants in each image for height, density and diameter.
Results and discussion
Comparison of height measures
Although the main purpose of the imaging platform was not to collect plant height and density data specifically, the comparison of manual measurements with automated measurements for height and plant density provided a means of validating the methods used to process image data collected using the platform. Future work will focus on the spectrum produced by imaging. This requires an accurate segmentation method to ensure that image data provides information about the spectral response of the target plants, rather than the background or neighbouring plants. Establishing the automated image processing methods for segmenting and splitting the image data will facilitate the analysis of the plant reflectance spectrum and responses to a range of biotic and abiotic conditions.
The study aim and objectives were achieved by developing a method for the automatic segmentation of target plant material in hyperspectral images of raspberry plants and the splitting of these images into individual plants. This is the first time that the segmentation of ground based hyperspectral images of bush crops has been attempted. Segmentation was carried out on images generated from a normalised difference between particular bands. Novel bands were selected for the segmentation of SWIR images. The segmentation was carried out using thresholds and graph theory. A small amount of manual intervention was required as an initial step to remove irrelevant sections at the start and end of images.
The performance of this technique has been partially validated by comparison of manual and automated measures of plant shape. The good correlation between manual and automated measurements confirmed the value of our segmentation method. The evaluation of segmentation performance was carried out in this way, as manual segmentation of the images would have being labour-intensive and also subject to observer bias. Manual inspection of segmented images indicated good quality segmentation in all images.
This method has been developed to solve a particular problem encountered with imaging perennial plants in field plantations. Challenges of irregular plant growth and inconsistent lighting had to be overcome to enable the segmentation to succeed. Due to the relative novelty of using ground based hyperspectral scanners in field environments, there are no widely applicable segmentation tools available at present. The methods described here have been developed with a particular imaging set-up in mind, although using details of the approach, other researchers may be able to alter and adapt the methods to solve their own image segmentation challenges.
This study reports development of the imaging platform and associated imaging processing methods. In future, we aim to implement these methods in further work to analyse the utility of the extracted image data for a range of applications.
DW and AB carried out the imaging. DW developed the image analysis technique and was a major contributor in writing the manuscript. HJ, AP and DW were involved in development of the imaging platform. SM and JG visually scored the plants. AB, SM, KL, AK, AP and JG were involved in design of the field trial layout. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Availability of data
The datasets analysed during the current study are available from the corresponding author on reasonable request.
This project is co-funded by Innovate UK project 101819 Imaging sensor solutions in the soft fruit industry for high throughput phenotyping and monitoring of abiotic and biotic stresses for premium variety production and maximised yields. The project partners are James Hutton Limited, Soil Essentials, Delta-T devices, Thomas Thomson Ltd, Total World Fresh, ADHB and James Hutton Institute. Insect cultures at James Hutton Institute are maintained with funding from the Scottish Government’s Rural and Environment Science and Analytical Services Division (RESAS).
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Jennings S, et al. New developments in raspberry breeding in Scotland. In: XI international rubus and ribes symposium 1133; 2015.Google Scholar
- Graham J, et al. Towards an understanding of the nature of resistance to Phytophthora root rot in red raspberry. Theor Appl Genet. 2011;123(4):585–601.View ArticlePubMedGoogle Scholar
- Graham J, et al. Genetic and environmental regulation of plant architectural traits and opportunities for pest control in raspberry. Ann Appl Biol. 2014;165(3):318–28.View ArticleGoogle Scholar
- Graham J, et al. Towards an understanding of the control of ‘crumbly’ fruit in red raspberry. Springerplus. 2015;4(1):223.View ArticlePubMedPubMed CentralGoogle Scholar
- Graham J, et al. Mapping QTLs for developmental traits in raspberry from bud break to ripe fruit. Theor Appl Genet. 2009;118(6):1143–55.View ArticlePubMedGoogle Scholar
- Graham J, Jennings N. Raspberry breeding, in breeding plantation tree crops: temperate species. Berlin: Springer; 2009. p. 233–48.View ArticleGoogle Scholar
- Araus JL, Cairns J. Field high-throughput phenotyping: the new crop breeding frontier. Trends Plant Sci. 2014;19:52–61.View ArticlePubMedGoogle Scholar
- Fiorani F, Schurr U. Future scenarios for plant phenotyping. Annu Rev Plant Biol. 2013;64:17.1–17.25.View ArticleGoogle Scholar
- Furbank RT, Tester M. Phenomics—technologies to relieve the phenotyping bottleneck. Trends Plant Sci. 2011;16:635–44.View ArticlePubMedGoogle Scholar
- Jones HG, Vaughan RA. Remote sensing of vegetation: principles, techniques, and applications. Oxford: Oxford University Press; 2010.Google Scholar
- Li L, Zhang Q, Huang D. A review of imaging techniques for plant phenotyping. Sensors. 2014;14:20078–111.View ArticlePubMedPubMed CentralGoogle Scholar
- Deery D, et al. Proximal remote sensing buggies and potential applications for phenotyping. Agronomy. 2014;4:349–79.View ArticleGoogle Scholar
- Busemeyer L, et al. BreedVision—a multi-sensor platform for non-destructive field-based phenotyping in plant breeding. Sensors. 2013;13:2830–47.View ArticlePubMedPubMed CentralGoogle Scholar
- Holman FH, et al. High throughput field phenotyping of wheat plant height and growth rate in field plot trials using UAV based remote sensing. Remote Sens. 2016;8:1031.View ArticleGoogle Scholar
- Perry EM, et al. Field-based rapid phenotyping with unmanned aerial vehicles (UAV). In: 16th Australian agronomy conference. University of New England, Armidale; 2012.Google Scholar
- Zaman-Allah M, et al. Unmanned aerial platform-based multi-spectral imaging for field phenotyping of maize. Plant Methods. 2015;11:35.View ArticlePubMedPubMed CentralGoogle Scholar
- Blackburn GA. Hyperspectral remote sensing of plant pigments. J Exp Bot. 2007;58:855–67.View ArticlePubMedGoogle Scholar
- Jones HG. Plants and microclimate: a quantitative approach to environmental plant physiology. 3rd ed. Cambridge: Cambridge University Press; 2014.Google Scholar
- Kuska M, et al. Hyperspectral phenotyping on the microscopic scale: towards automated characterization of plant-pathogen interactions. Plant Methods. 2015;11(1):1–15.View ArticleGoogle Scholar
- Cheng T, Rivard B, Sánchez-Azofeifa A. Spectroscopic determination of leaf water content using continuous wavelet analysis. Remote Sens Environ. 2011;115(2):659–70.View ArticleGoogle Scholar
- Jones HG, Vaughan RA. Remote sensing of vegetation: principles, techniques, and applications. Oxford: Oxford University Press; 2010.Google Scholar
- Minervini M, Scharr H, Tsaftaris SA. Image analysis: the new bottleneck in plant phenotyping [applications corner]. IEEE Signal Process Mag. 2015;32:126–31.View ArticleGoogle Scholar
- Wahabzada M, et al. Metro maps of plant disease dynamics—automated mining of differences using hyperspectral images. PLoS ONE. 2015;10(1):e0116902.View ArticlePubMedPubMed CentralGoogle Scholar
- Camargo A, Smith JS. An image-processing based algorithm to automatically identify plant disease visual symptoms. Biosys Eng. 2009;102:9–21.View ArticleGoogle Scholar
- Camargo A, Smith JS. Image pattern classification for the indentification of disease causing agents in plants. Comput Electron Agric. 2009;66:121–5.View ArticleGoogle Scholar
- Delalieux S, et al. Detection of biotic stress (Venturia inaequalis) in apple trees using hyperspectral data: non-parametric statistical approaches and physiological implications. Eur J Agron. 2007;27:130–43.View ArticleGoogle Scholar
- Andrade-Sanchez P, et al. Development and evaluation of a field-based high-throughput phenotyping platform. Funct Plant Biol. 2014;41:68–79.View ArticleGoogle Scholar
- Yan G, et al. Comparison of pixel-based and object-oriented image classification approaches—a case study in a coal fire area, Wuda, Inner Mongolia, China. Int J Remote Sens. 2006;27(18):4039–55.View ArticleGoogle Scholar
- Shackelford AK, Davis CH. A combined fuzzy pixel-based and object-based approach for classification of high-resolution multispectral data over urban areas. IEEE Trans Geosci Remote Sens. 2003;41(10):2354–63.View ArticleGoogle Scholar
- Graham J, et al. The construction of a genetic linkage map of red raspberry (Rubus idaeus subsp. idaeus) based on AFLPs, genomic-SSR and EST-SSR markers. Theor Appl Genet. 2004;109(4):740–9.View ArticlePubMedGoogle Scholar
- Graham J, et al. Mapping gene H controlling cane pubescence in raspberry and its association with resistance to cane botrytis and spur blight, rust and cane spot. Theor Appl Genet. 2006;112(5):818–31.View ArticlePubMedGoogle Scholar
- Carlson TN, Ripley DA. On the relation between NDVI, fractional vegetation cover, and leaf area index. Remote Sens Environ. 1997;62(3):241–52.View ArticleGoogle Scholar
- Haralick RM, Sternberg SR, Zhuang X. Image analysis using mathematical morphology. IEEE Trans Pattern Anal Mach Intell. 1987;4:532–50.View ArticleGoogle Scholar
- Felzenszwalb PF, Huttenlocher DP. Efficient graph-based image segmentation. Int J Comput Vision. 2004;59(2):167–81.View ArticleGoogle Scholar
- Pound MP, et al. AutoRoot: open-source software employing a novel image analysis approach to support fully-automated plant phenotyping. Plant Methods. 2017;13(1):12.View ArticlePubMedPubMed CentralGoogle Scholar
- Dong Y, et al. Nondestructive analysis of automotive paints with spectral domain optical coherence tomography. Appl Opt. 2016;55(13):3695–700.View ArticlePubMedGoogle Scholar
- Williams D, et al. Reconstruction of 3D surface maps from anterior segment optical coherence tomography images using graph theory and genetic algorithms. Biomed Signal Process Control. 2016;25:91–8.View ArticleGoogle Scholar
- Otsu N. A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern. 1979;9:62–6.View ArticleGoogle Scholar