A novel 3D imaging system for strawberry phenotyping
© The Author(s) 2017
Received: 31 July 2017
Accepted: 23 October 2017
Published: 8 November 2017
Accurate and quantitative phenotypic data in plant breeding programmes is vital in breeding to assess the performance of genotypes and to make selections. Traditional strawberry phenotyping relies on the human eye to assess most external fruit quality attributes, which is time-consuming and subjective. 3D imaging is a promising high-throughput technique that allows multiple external fruit quality attributes to be measured simultaneously.
A low cost multi-view stereo (MVS) imaging system was developed, which captured data from 360° around a target strawberry fruit. A 3D point cloud of the sample was derived and analysed with custom-developed software to estimate berry height, length, width, volume, calyx size, colour and achene number. Analysis of these traits in 100 fruits showed good concordance with manual assessment methods.
This study demonstrates the feasibility of an MVS based 3D imaging system for the rapid and quantitative phenotyping of seven agronomically important external strawberry traits. With further improvement, this method could be applied in strawberry breeding programmes as a cost effective phenotyping technique.
A successful strawberry breeding programme generates and selects genotypes with traits suitable for the industry in its target geographic region . As often genotypes cannot be directly observed, traditional breeding selects on the basis of a weighted selection index of phenotypes . In order to maximise the accuracy of selection, heritable traits of interest must be measured precisely and accurately. Currently, most external fruit quality phenotyping approaches in strawberry breeding relies on the human eye to make assessments . This approach is labour-intensive, prone to human bias and typically generates ordinal data less suitable for the most powerful quantitative statistical models .
Use of image analysis has the potential to overcome some of these limitations, with previous studies showing success in utilising 2D high-throughput imaging systems to assess external fruit quality . Most studies were focussed on colour analysis of fruits, including apple , citrus , mango  and banana , but some systems have assessed morphological attributes, including the size of apples  and the shape of oranges . For strawberry, an automated grading system was developed by Liming et al.  that assesses colour, size and four degrees of shape. In another 2D strawberry imaging system, developed by Nagata et al., the maximum fruit diameter could be derived by automatically identifying the axis from the top of the calyx to the tip of the nose . However, 2D image analysis is not always a reliable fruit phenotyping method due to uneven colour distribution of fruit and occlusion of morphology from different viewing perspectives .
Recently, 3D imaging has been increasingly explored, as the cost of hardware decreases and reassembly techniques improve , with a range of sensors deployed for plant phenotyping. Light detection and ranging (LiDAR) was used to generate detailed 3D models of plants [15, 16], but is currently expensive, time-consuming and complex to implement . Binocular stereovision is a low-cost solution for 3D plant canopy reconstruction , but with only two viewing perspectives, is insufficient to model the entire target. Other techniques, including time-of-flight (TOF) [19, 20] and structured light , have similar limitations in gathering 360° information from the target.
Studies of 3D imaging based phenotyping of fruit are limited. A 3D model of mango was generated using four cameras and the shape-from-silhouettes reconstruction method, but it did not encompass 360° of the fruit. Five parameters were extracted from the 3D model including length, width, thickness, volume and surface area in order to sort the mangoes by size. Image based sorting accuracy was comparable to manual sorting, but no comparison of individual trait data to a “gold standard” was shown . Shape-from-silhouettes was also successfully applied to the 3D reconstruction of tomato seedlings with ten calibrated cameras. Stem height and leaf area were accurately measured after geometry based segmentation .
In this study, a novel 3D imaging based approach for phenotyping strawberry fruit was explored. MVS and SfM were applied to generate a 3D model of strawberry and software was developed to measure seven agronomically important external strawberry traits. This method is promising to facilitate strawberry breeding by providing a high-throughput, objective and low-cost phenotyping system.
100 strawberries were purchased from local supermarkets, including 10 different varieties, to represent the diverse range of commercially available strawberry phenotypes. All fruits were assessed before their “best before” dates. Fruits would likely have been subjected to chilling to 4 °C within 4 h of harvest and kept at that temperature throughout the supply chain until sale. Fruits were stored at 4 °C until assessment.
Manual scoring metrics for seven external fruit quality traits
External quality parameter
Number of achenes visible, without disturbing calyx
Maximum Euclidean distance between any pair of points on the calyx
Scale 1–8 (strawberry colour chart for experimental ends, Ctifl, France)
Dimension of fruit from centre of calyx to tip of nose
Greatest dimension of fruit orthogonal to the height
Greatest dimension of the fruit orthogonal to both height and length
Volume of displaced water when fruit was completely submerged
3D point cloud reconstruction
A Dell desktop computer (CPU Xeon® CPU X5560 @2.80 GHz x 16, Intel Co., Santa Clara, CA, USA) with a graphics card (Quadro K2200 GPU, NVIDIA Co., Santa Clara, CA, USA) operating on Linux Ubuntu 14.04 was used in this study for both software development and point cloud processing.
The point cloud reconstruction was implemented with commercial software (Agisoft Photoscan, Agisoft, LLC, St. Petersburg, Russia; licence required), utilising the Structure from Motion (SfM) algorithm  (Fig. 2b). Due to the high overlap between adjacent images and high resolution (5148 × 3456) of each image, pre-processing software was developed to automatically reduce the number of images by discarding three frames from every four. This was found to be the minimum number to reconstruct all the 3D models successfully. Additionally, each image was rescaled to the resolution of 1000 × 1450, which greatly increased the processing speed with satisfactory point cloud quality.
Automated 3D image analysis
The automated point cloud analysis software was developed in C++ with Point Cloud Library (PCL) . The software is programmed to automatically load all point cloud files in order and process them in a batch by implementing the point cloud segmentation and external quality attributes measurement algorithms.
Point cloud segmentation
Orienting bounding box (oBB) fitting
The OBBs were fitted to the segments of holder and the combination the holder and the fruit body for the size measurement. The major eigenvectors of the covariance matrix of points in a point cloud define the major axis of its OBB . The second axis was determined by calculating the maximum Euclidean distance of the points in the point cloud orthogonal to the major axis. The final axis was orthogonal to both other axes.
Height, length and width
An OBB was fitted to the point cloud of the combination of the fruit body and holder segments. The OBB was not fitted directly to the body, as its irregular shape often resulted in misidentification of the vertical axis. The height of the combination of fruit body and holder was always the largest dimension, so the magnitude of the OBB major axis was assumed to be equivalent to the height of the fruit body and holder model. As the fruit body was always longer and wider than the holder, the second and third dimensions of the OBB represented length and width respectively. The height of the holder was estimated by fitting an OBB to its point cloud and the difference in height between it and the combination of fruit body and holder OBB was assumed to be the height of the fruit. Ratios between the three fruit body dimensions and the height of the holder were multiplied by the true height of the holder to derive the strawberry height, width and length.
The mesh of the strawberry body (Fig. 3e) was constructed from the point cloud using Poisson Surface Reconstruction , which produces an enclosed mesh without any edges or large holes. The mesh volume was calculated by summing the volume of every triangle based pyramid formed from each face of the mesh and the origin of the point cloud .
The edges of the calyx segment were identified by applying convex hull , enabling rapid calculation of the maximum Euclidean distance (Fig. 2d). The ratio between the calyx maximum distance and the height of the holder OBB was multiplied by the true height of the holder to estimate calyx size.
The segmentation of achenes from the point cloud was based on identifying points in the body segment with an arbitrary range in the hue channel of HSV space. A clustering algorithm based on Euclidean distances between points was implemented to group points corresponding to the same achene (Fig. 3f)  and the number of clusters was counted automatically.
As hue value in HSV space represents visual sensation of perceived colour , the mean intensity of the hue channel of every point in the body segment was calculated for the assessment of the strawberry colour.
In this study, the concordance correlation coefficient (CCC) was used to measure the concordance between the manually derived and the 3D image derived external fruit quality traits . Additionally, the coefficient of determination (r2) was calculated to estimate correlation between the sets of values. Statistical analysis was performed using R . Linear models and associated coefficients were derived using the “lm” function, the root mean square error (RMSE) was derived using the “Metrics” package  and the concordance correlation coefficient (CCC) was derived using the “Agreement” package .
In order to evaluate the measurement of seven external strawberry fruit quality parameters using 3D imaging (hereafter referred to as automated assessment), 100 berries were automatically and manually assessed. Reliable reconstruction could be achieved by taking a minimum of 37 images per berry with 100% successful reconstruction, though the nose of the fruit was often missing due to occlusion from the shooting angle. With the described setup, data capture took approximately 60 s, including 10 s of operator action per sample. Model reconstruction took approximately 15 min and parameter derivation took approximately 50 s. Both these operations were fully automated.
In order to validate the 3D reconstruction, the point cloud of the holder segment was manually measured in Meshlab , an open source software for 3D mesh visualisation. Although their absolute sizes in image space were inconsistent (range 0.36–1.73; mean 0.78; SD 0.27), ratios among the height, width and length were the similar to the true ratios. As there was no evidence of distortion, the absolute height of the holder was used as a standard for fruit dimension measurements. Moreover, incorporation of the holder point cloud ensured that the vertical axis was always greater than any other axis, allowing the major eigenvector of the point cloud covariance matrix to consistently define the vertical axis.
Good concordance between manual and automated measurements of calyx size, height, length, width and volume, and promising results for achene number and colour were achieved. It is suggested that the qualitative traits of strawberry currently used in breeding can be represented using the measurements generated from this study. For instance, a “long conic”  fruit has a large ratio of height to width and measurement of “Cap size”  can be defined by the ratio of calyx size to fruit width and length.
With further development, automated assessment could be suitable for integration into existing strawberry breeding programmes, bringing a range of advantages. Firstly, the quantitative, accurate and unbiased measurements would increase the accuracy of the selection in strawberry breeding. The precise measurements would be particularly suitable for input into models of genomic selection, which attempt to quantify small effect quantitative trait loci (QTLs) associated with polygenic traits [39, 40]. Secondly, automated assessment has the potential to improve the speed of assessment. The described setup requires approximately 10 s of human operator time per sample, approximately 20-fold faster than making the equivalent manual measurements. Thirdly, the low cost and wide availability of hardware mean that this approach can be easily introduced into existing breeding programmes with minimal capital expenditure.
Measurement error may have arisen from a range of sources. During manual assessment, the axis of measurement was determined by eye, potentially resulting in non-maximal distances or non-orthogonal axes. As the calyx is soft, errors may have been induced in the operation of the callipers. Correlation between the measurements of colour may be weak as manual assessment is subjective and it is difficult to assess fruit with uneven colour distributions.
This imaging system can be developed to reduce the duration of data capture by using alternative imagers such as scientific cameras or webcams with programmable shutter speeds and resolutions. Reducing resolution to 1000 × 1450 greatly increases the processing speed compared to the original images, but further investigation is needed to identify the minimum resolution to generate satisfactory point clouds. Use of multiple calibrated cameras to capture information from different viewpoints simultaneously could also be explored to further improve the quality and efficiency of 3D reconstruction, particularly from the nose of the fruit and the data capture speed.
As both fruit body and achenes have a range of colours, our current algorithm of arbitrary hue thresholding is unlikely to be reliable in identifying achenes from a range of cultivars. More sophisticated adaptive or texture based thresholding algorithms would likely improve the cluster identification.
It is believed that more traits could be derived from the gathered dataset. Firstly, algorithms exist that can calculate the surface area of a 3D mesh , which together with reliable achene counts could be used to quantify achene density. Secondly, rotational symmetry could be quantified by considering the distribution of the Euclidean distance of points to the principal axis in 2D slices of the point cloud orthogonal to the principle axis. Thirdly, it may be possible to quantify the morphology of the fruit body at the neck of the fruit to derive information regarding the neck line.
In this study, a MVS based 3D reconstruction pipeline was developed and utilised to generate in silico model of strawberries. Automated 3D image point cloud analysis software was developed in house to derive berry achene counts, calyx size, colour, height, length, width and volume of the model. This study found good correlation between the automated and manual assessment techniques for dimension measurements and volume, suggesting that automated assessment is a promising technique to be utilised in place of manual assessment for these traits.
The focus of this study was the investigation of the use of 3D imaging to phenotype strawberries for commercial breeding. This system, with further improvement, can be quantitative, accurate, rapid and require little capital investment to be integrated into existing strawberry breeding programmes. This approach can also be further developed for strawberry quality control as its high precision is particularly suited for assessing differences within single cultivars, a situation frequently encountered in pack houses.
BL devised experiments, developed software and guided the data analysis. JQH assisted in experimental design, conducted the manual assessment and analysed the data. BL and JQH prepared the manuscript. RJH assisted in experimental design and editorial oversight. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Availability of data and materials
The software developed and datasets generated and analysed during the current study are available from the Image-processing repository at the East Malling Github site (www.github.com/organizations/eastmallingresearch/).
Consent for publication
All authors of the manuscript have read and agreed to its content and are accountable for all aspects of the accuracy and integrity of the manuscript.
Ethics approval and consent to participate
This research was supported by the Agriculture and Horticulture Development Board (AHDB) through Studentship Project CP163, Biotechnology and Biological Sciences Research Council (BBSRC) project BB/M01200X/1 awarded to RJH and Engineering and Physical Science Research Council (EPSRC) EP/R005583/1.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Mathey MM, Mookerjee S, Gündüz K, Hancock JF, Iezzoni AF, Mahoney LL, et al. Large-scale standardized phenotyping of strawberry in RosBREED. J Am Pomol Soc. 2013;67:205–16.Google Scholar
- Chandler C, Folta K, Dale A, Whitacker VM, Herrington M. Chapter 9—strawberry. In: Badeneses ML, Byrne DH, editors. Fruit breeding. Berlin: Springer; 2012. p. 305–25.View ArticleGoogle Scholar
- Goddard ME, Hayes BJ. Genomic selection. J Anim Breed Genet. 2007;124:323–30.View ArticlePubMedGoogle Scholar
- Dadwal M, Banga VK. Color image segmentation for fruit ripeness detection: a review. In: 2nd International Conference on Electrical Electronics and Civil Engineering; 2012. p. 190–3.Google Scholar
- Throop JA, Aneshansley DJ, Anger WC, Peterson DL. Quality evaluation of apples based on surface defects: development of an automated inspection system. Postharvest Biol Technol. 2005;36:281–90.View ArticleGoogle Scholar
- Blasco J, Aleixos N, Moltó E. Computer vision detection of peel defects in citrus by means of a region oriented segmentation algorithm. J Food Eng. 2007;81:535–43.View ArticleGoogle Scholar
- Kang SP, East AR, Trujillo FJ. Colour vision system evaluation of bicolour fruit: a case study with “B74” mango. Postharvest Biol Technol. 2008;49:77–85.View ArticleGoogle Scholar
- Mendoza F, Aguilera JM. Application of image analysis for classification of ripening bananas. Food Eng Phys Prop. 2004;69:471–7.Google Scholar
- Blasco J, Aleixos N, Moltó E. Machine vision system for automatic quality grading of fruit. Biosyst Eng. 2003;85:415–23.View ArticleGoogle Scholar
- Costa C, Menesatti P, Paglia G, Pallottino F, Aguzzi J, Rimatori V, et al. Quantitative evaluation of Tarocco sweet orange fruit shape using optoelectronic elliptic Fourier based analysis. Postharvest Biol Technol. 2009;54:38–47.View ArticleGoogle Scholar
- Liming X, Yanchao Z. Automated strawberry grading system based on image processing. Comput Electron Agric. 2010;71S:32–9.View ArticleGoogle Scholar
- Nagata M, Bato PM, Mitarai M, Qixin C, Kitahara T. Study on sorting system for strawberry using machine vision (part 1). Development of software for determining the direction of strawberry (Akihime variety). J Jpn Soc Agric Mach. 2000;62:100–10.Google Scholar
- Paulus S, Behmann J, Mahlein AK, Plümer L, Kuhlmann H. Low-cost 3D systems: suitable tools for plant phenotyping. Sensors. 2014;14:3001–18.View ArticlePubMedPubMed CentralGoogle Scholar
- Vázquez-Arellano M, Griepentrog HW, Reiser D, Paraforos DS. 3-D imaging systems for agricultural applications—a review. Sensors. 2016;16:1–24.View ArticleGoogle Scholar
- Kjaer KH, Ottosen C-O. 3D laser triangulation for plant phenotyping in challenging environments. Sensors. 2015;15:13533–47.View ArticlePubMedPubMed CentralGoogle Scholar
- Paulus S, Dupuis J, Mahlein A, Kuhlmann H. Surface feature based classification of plant organs from 3D laserscanned point clouds for plant phenotyping. BMC Bioinformatics. 2013;14:238.View ArticlePubMedPubMed CentralGoogle Scholar
- Zhang Y, Teng P, Shimizu Y, Hosoi F, Omasa K. Estimating 3D leaf and stem shape of nursery paprika plants by a novel multi-camera photography system. Sensors. 2016;16:874.View ArticlePubMed CentralGoogle Scholar
- Klodt M, Herzog K, Töpfer R, Cremers D. Field phenotyping of grapevine growth using dense stereo reconstruction. BMC Bioinformatics. 2015;16:143.View ArticlePubMedPubMed CentralGoogle Scholar
- Alenyà G, Dellen B, Torras C. 3D modelling of leaves from color and ToF data for robotized plant measuring. In: Proceedings of IEEE international conference on robotics and automation; 2011. p. 3408–14.Google Scholar
- Klose R, Penlington J, Ruckelshausen A. Usability study of 3D time-of-flight cameras for automatic plant phenotyping. Image Analysis for Agricultural Products and Processes. 2011;69:93–105Google Scholar
- Chéné Y, Rousseau D, Lucidarme P, Bertheloot J, Caffier V, Morel P, et al. On the use of depth camera for 3D phenotyping of entire plants. Comput Electron Agric. 2012;82:122–7.View ArticleGoogle Scholar
- Chalidabhongse T, Yimyam P, Sirisomboon P. 2D/3D vision-based mango’s feature extraction and sorting. In: 9th international conference on Control, Automation, Robotics and Vision, 2006, ICARCV’06; 2006.Google Scholar
- Golbach F, Kootstra G, Damjanovic S, Otten G, van de Zedde R. Validation of plant part measurements using a 3D reconstruction method suitable for high-throughput seedling phenotyping. Mach Vis Appl. 2015;27:663–80.View ArticleGoogle Scholar
- Rose JC, Paulus S, Kuhlmann H. Accuracy analysis of a multi-view stereo approach for phenotyping of tomato plants at the organ level. Sensors. 2015;15:9651–65.View ArticlePubMedPubMed CentralGoogle Scholar
- Fonstad MA, Dietrich JT, Courville BC, Jensen JL, Carbonneau PE. Topographic structure from motion: a new development in photogrammetric measurement. Earth Surf Proc Land. 2013;38:421–30.View ArticleGoogle Scholar
- Westoby MJ, Brasington J, Glasser NF, Hambrey MJ, Reynolds JM. “Structure-from-motion” photogrammetry: a low-cost, effective tool for geoscience applications. Geomorphology. 2012;179:300–14.View ArticleGoogle Scholar
- Rusu RB, Cousins S. 3D is here: point cloud library. In: Point Cloud Library http://pointclouds.org/. Accessed 15 July 2017.
- Wu D, Sun DW. Colour measurements by computer vision for food quality control—a review. Trends Food Sci Technol. 2013;29:5–20.View ArticleGoogle Scholar
- Ding S, Mannan MA, Poo AN. Oriented bounding box and octree based global interference detection in 5-axis machining of free-form surfaces. Comput Aided Des. 2004;36:1281–94.View ArticleGoogle Scholar
- Kazhdan M, Bolitho M, Hoppe H. Poisson surface reconstruction. In: Eurographics symposium on geometry processing; 2006. p. 61–70.Google Scholar
- Zhang C, Chen T. Efficient feature extraction for 2D/3D objects in mesh representation. In: Proceedings of international conference on image processing; 2001, p. 1–4.Google Scholar
- Cupec R, Nyarko E, Filko D. Fast 2.5D mesh segmentation to approximately convex surfaces. In: 5th European conference on mobile robotics; 2011. p. 3–8.Google Scholar
- Dixon SJ, Brereton RG. Comparison of performance of five common classifiers represented as boundary methods: Euclidean distance to centroids, linear discriminant analysis, quadratic discriminant analysis, learning vector quantization and support vector machines, as dependent on data structure. Chemom Intell Lab Syst. 2009;95:1–17.View ArticleGoogle Scholar
- Lawrence I, Lin K. A concordance correlation coefficient to evaluate reproducibility. Biometrics. 1989;45:255–68.View ArticleGoogle Scholar
- R Core Team. R: A language and environment for statistical computing; 2017.Google Scholar
- Hamner B. Package “Metrics” version 0.1.2. In: The Comprehensive R Archive Network. 2017. http://cran.r-project.org/web/packages/Metrics/Metrics.pdf. Accessed 12 July 2017.
- Yu Y, Lawrence L. Package “Agreement” version 0.8-1. In: The Comprehensive R Archive Network. 2015. http://cran.r-project.org/web/packages/Agreement/Agreement.pdf. Accessed 12 July 2017.
- Cignoni P, Callieri M, Corsini M, Delepiane M, Ganovelli F, Ranzuglia G. MeshLab: an open-source mesh processing tool. Sixth Eurographics Italian chapter conference; 2008. p. 129–36.Google Scholar
- Meuwissen THE, Hayes BJ, Goddard ME. Prediction of total genetic value using genome-wide dense marker maps. Gene Soc Am. 2001;157:1819–29.Google Scholar
- Gezan SA, Osorio LF, Verma S, Whitaker VM. An experimental validation of genomic selection in octoploid strawberry. Hortic Res. 2017;4:1–9.View ArticleGoogle Scholar