Table 3 Well-established methods and algorithms used for 3D plant phenotyping
Method | Plant | Short description |
|---|---|---|
3D Image Acquisition and Registration | ||
3D Laser Scanning (LiDAR) | Aubergine [252] Bamboo-leaf oak tree [252] Benth (Nicotiana benthamiana) [183] Birch [174] Botanic trees [168] Cereal plants [27] Chickpea [328] Elm tree [182] Grape [329] Grapevine [35] Horse Chestnut [174] Japanese cedar [252] Japanese larch [252] Maize (Corn) [34, 129, 160, 186, 330, 331] Orchard tree [175] Poplar [174] Rapeseed (Brassica sp.) [32] Red Oak [174] Rosebush [264] Sorghum [328] Soybean [25] Sugar maple [182] Sweet Chestnut [174] Thale cress (Arabidopsis) [10, 28, 156] Yellow birch [182] | Measures accurately the distance between the sensor and a target based on the elapsed time between the emission and return of laser pulses (’Time-of-Flight’ (ToF) method) or based on trigonometry (the ’optical probe’ or ’light section’ methods). |
Electrical Resistance Tomography (ERT) | Chicory [56] Maize (Corn) [56] Switchgrass [56] | Is a geophysical technique for imaging sub-surface structures from electrical resistivity measurements made at the surface, or by electrodes in one or more boreholes. |
Gaussian Mixture Model (GMM) | Apple tree [185] Barley [121] Cherry [185] Thale cress (Arabidopsis) [185] | Represents discrete point sets by continuous density functions [157]. |
Generalized Voxel Coloring | Variant of Voxel Coloring which allows the cameras to completely surround the scene [119]. | |
Iterative Closest Point (ICP) | Barley [156] Maize (Corn) [34, 38, 47, 129, 227] Pepper [155] Thale cress (Arabidopsis) [156] Tomato [34] | Minimizes distances between two point clouds. Often used to obtain a full 3D reconstruction from multiple 3D scans which capture the object from different angles [112, 146,147,148,149]. |
Magnetic Resonance Imaging (MRI) | Barley [63] Bean plant [62] Maize (Corn) [63] | Is a type of scan that uses strong magnetic fields and radio waves to produce detailed images of the inside of the body. |
Marching Cubes | Transforms a voxel grid into a polygon mesh. The algorithm walks through the voxel grid and chooses a certain surface configuration composed of triangles from a table, based on the values of neighboring voxels. The individual polygons are then fused into a surface [162]. | |
Multi-view Stereo (MVS) | Anthurium andraeanum [40] Banana [90] Basil [317] Calathea makoyana [111] Epipremnum aureum [111] Grapevine [333] Hedera nepalensis [111] Ipoestia [317] Ixora [317] Litchi [88] Maize (Corn) [107, 133, 227, 330, 334] Monstera deliciosa [111] Rape [89] Sugar beet [108] Wheat [169] | Uses two or more cameras to generate parallax from different perspectives, and obtaining the distance information of the object and then reconstructs a 3D shape from calibrated overlapping images captured from different viewpoints [170]. In case of having two cameras, it is called stereo vision. |
Patch-based Multi-View Stereo (PMVS) | Artificial plant [75] Bambara groundnut [78] Proso millet [78] | Reconstructs a 3D point cloud model based on multiple color input images. A requirement of this algorithm is that the intrinsic (such as focal length) and extrinsic (3D position and orientation) camera parameters are known. |
Photometric Stereo (PS) | Dock (Rumex Obtusifolius) [12] Potato [12] Thale cress (Arabidopsis) [17] | Estimates the surface normals of objects by observing the object under different lighting conditions [50]. |
Shape-from-silhouette (SFS) (Shape-from-contour) | Artificial plant [209] Barley [121] Sorghum [209] Tomato [101] Wheat [209] | Reconstructs the visual hull of an object, by means of the intersection of silhouette cones determined by the objects’ silhouettes captured from different angles [335]. |
Space Carving (Shape-from-contour) | Aloe vera [81] Banana [104] Bromeliad species [81] Chili [81] Cordyline species. [81] Cotton [99] Pumpkin (Cucurbita pepo) [81] Rapeseed [81] Sorghum [105] Other [102] | Reconstructs the maximal shape of an object that is photo-consistent with the object. Photo-consistency includes consistency with the objects silhouettes, but also with its projected surface colors. The algorithm iteratively ’carves’ space away from an enclosing volume in a well-defined way, until the shape is photo-consistent with all the views [98]. |
Structured Light | Anthurium [48] Cabbage [26] Cucumber [26] Dishlia [48] Tomato [26] | Extracts the 3D surface shape based on the information from the distortion of the projected structured-light pattern without ionizing radiation [52]. |
Structure from Motion (SfM) | Barley [18] Basil [317] Brussels sprout [79] Chili plant [11] Ipoestia [317] Ixora [317] Maize (Corn) [9, 13, 82, 334, 336,337,338] Nephthytis [94] Olive [223] Physalis sp. [9] Poinsettia [94] Brassica sp. (Rapeseed) [9] Savoy cabbage [79] Schefflera [94] Soybean [14, 83, 84, 84, 334, 339] Sugar beet [13] Sunflower [11, 13, 79, 83, 240] Thale cress (Arabidopsis) [9] Wheat [9] | Reconstructs the 3D structure using a series of 2D images with a high degree of overlap, taken from different angles. It identifies matching features which are tracked from image to image to produce estimates of the camera positions and orientations, as well as the coordinates of the features to create a point cloud. |
Time of Flight (ToF) (Including Microsoft Kinect sensors) | Calathea makoyana [111] Cyclamen [40] Epipremnum aureum [111] Hedera nepalensis [111] Hydrangea [40] Lettuce [44] Monstera deliciosa [111] Orchidaceae [40] Paprika [109] Pelargonium [40] Pepper [155] Pumpkin [46] Rapeseed [33] Rosebush [110] Sorghum [39] Soybean [43] Sunflower [13] Tomato [45] Wheat [36] Yucca [110] | ToF cameras use time between emitted light and reflected light from thousands of points to conduct 3D images. |
Voxel Coloring (Shape-from-photoconsistency) | Rose [97] | Reconstructs a photo-consistent 3D shape not by carving away voxels, but by identifying voxels that have a unique coloring which is constant across all possible photo-consistent interpretations of the scene. Has the limitation that all cameras have to be placed on one side of the scene. |
X-Ray (Micro) Computed Tomography (CT / µCT) | Barley [64] Cassava [59] Chickpea [64] Duckweed [57] Sinningia [342] Sorghum [343] Tomato [61] Other [344] | Is a non-destructive imaging tool for the production of high-resolution three-dimensional (3D) images composed of two-dimensional (2D) trans-axial projections, or ‘slices’, of a target specimen [345]. |
3D Image Processing | ||
Bilateral smoothing techniques | Maize (Corn) [15] Tomato [346] | Is a non-linear filtering technique and a simple, non-iterative scheme for edge-preserving smoothing [137, 142]. |
Clustering-based Segmentation | Olive [223] Wheat [35] | Uses clustering algorithms to group data points that are more similar to one another in order to obtain a segmented image. |
Color-based Filter | Brassica sp. [9] Calathea makoyana [111] Epipremnum aureum [111] Hedera nepalensis [111] Maize (Corn) [9] Monstera deliciosa [111] Physalis sp. [9] Thale cress (Arabidopsis) [9] Wheat [9] | Distinguishes between foreground and background and removes background pixels based on the RGB color information. |
Dart Throwing Filter | Sequentially add points from the original point cloud to a downsampled point cloud if they don’t have a neighbor in the output point cloud within a certain radius [136]. | |
Density-based Spatial Clustering of Applications with Noise (DBSCAN) | Removes clusters of size less than a predetermined threshold if they are located further away than a certain distance from any other point cluster. Can be used as noise filtering. | |
Mean Shift | Paprika [109] | Iteratively shifts each data point to the average of data points in its neighborhood by using kernel density estimation. |
Moving Least Squares (MLS) | Pepper [155] | Iteratively projects points on weighted least squares fits of their neighborhoods to cause the points to lie closer to an underlying surface [131]. |
M-Estimator Sample Consensus (MSAC) | Maize (Corn) [130] | Is a variant of the RANSAC algorithm which adopts bounded loss of RANSAC by using a different loss function [347] |
Radius-based Outlier Filter (RBOF) | Calathea makoyana [111] Epipremnum aureum [111] Hedera nepalensis [111] Monstera deliciosa [111] | Modifies the elemental criterion of a specific element based on a weighted average of the criteria in a fixed neighborhood. |
Random Sample Consensus (RANSAC) | Grape [329] Maize (Corn) [82] | Fits geometric primitives to point clouds by choosing the best among fits to numerous random samplings of the data [128]. |
Spatial Region Filter | Calathea makoyana [111] Epipremnum aureum [111] Hedera nepalensis [111] Maize (Corn) [133] Monstera deliciosa [111] | Removes all points outside a region defined in a 3D coordinate system. |
Statistical Outlier Removal (SOR) Filter | Calathea makoyana [111] Epipremnum aureum [111] Hedera nepalensis [111] Monstera deliciosa [111] Sugar beet [25] Other [135] | Removes points if the mean distance to its neighbors surpasses a threshold based on the mean and standard deviation of all neighbor distances. |
Surface Boundary Filter (SBF) | Calathea makoyana [111] Epipremnum aureum [111] Hedera nepalensis [111] Monstera deliciosa [111] | Identifies and removes boundary points using a threshold on the angle between a projected vector in the normal plane to the first two principal components and one of the principal components [111]. |
Voxel Grid Downsampling | Brassica sp. [9] Calathea makoyana [111] Epipremnum aureum [111] Hedera nepalensis [111] Monstera deliciosa [111] Physalis sp. [9] Thale cress (Arabidopsis) [9] Wheat [9] | Divides the point cloud into a 3D voxel grid and replaces points within each voxel by the centroid of all the points within the voxel [134]. |
3D Image Analysis | ||
α-shape Triangulation | Thale cress (Arabidopsis) [156] Barley [156] | Transforms a point cloud into a polygon mesh. The shape is determined by connecting sets of 3 points into a triangle if they can be circumscribed by a sphere with radius α which doesn’t contain any other points [163]. |
Breath-first flood-fill algorithm | Tomato [101] | Determines the area connected to a given node in a multi-dimensional array. |
Constrained Region-growing | Cotton [164] | Segments a surface mesh segmentation by growing regions from seed points to adjacent mesh faces, constrained by changes in curvature, which correspond to sharp edges [207, 208]. |
Delaunay Triangulation (Advancing Front) | Aloe vera [81] Brassica sp. [81] Bromeliad sp. [81] Chili [81] Cordyline sp. [81] Pumpkin (Cucurbita pepo) [81] Rape [81] Rice [229] | Creates a triangulation (in 2D) or covering by tetrahedra (in 3D) of a point cloud, such that no point lies in the circumcircle of any triangle or tetrahedron. Delaunay triangulations tend to maximize the minimum interior angle of each triangle or tetrahedron, and hence avoid sharp angles (“sliver triangles”). Used in the context of this paper to grow a surface from a set of seed triangles. [233,234,235]. |
Dense Conditional Random Field (CRF) | Maize (Corn) [227] | Acquires an accurate and spatially consistent labeling of pixels after the application of a unary classifier which doesn’t take the spatial context of pixel labels into account. The model establishes pairwise potentials on all pairs of pixels in the image. An energy function consisting of both unary and pairwise components is minimized [228]. |
Dijkstra’s algorithm | Berryless grape [247] Maize (Corn) [130] Pine tree [180] | Finds the shortest path between vertices in a graph [171]. |
(Fast) Point Feature Histogram (FPFH / PFH) | Barley [226] Benth (Nicotiana benthamiana) [183] Grapevine [226] Rapeseed (Rape) [125] Wheat [226] | Describes the local geometry around a point in point clouds using features based on the angular relationships between pairs of points and their normals, within a certain radius around each query point. The features are counts within histogram bins of these values. FPFH is a more efficient version of PFH which reduces the number of pairs for which angles have to be calculated while retaining most of the discriminative power of PFH. |
Locally Estimated Scatterplot Smoothing (LOESS) | Maize (Corn) [229] Rice [229] | Reconstructs a continuous surface even with the presence of the discontinuity of surface points. |
Minimum Oriented Bounding Box (MOBB) | Rapeseed (Rape) [125] | Determines the smallest bounding box for a point set (i.e., smallest area, volume or hyper-volume in higher dimensions) within which all points lie. |
Minimum Spanning Tree | Finds a subset of edges in a graph which connects all the vertices, and which has a minimum total length [172]. | |
Non-Uniform Rational B-splines (NURBS) | Mint [238] Maize (Corn) [38] Sunflower [239] Other [237] | Defines smooth curves and surfaces by a list of 3D coordinates of surface control points and associated weights . |
Randomly Intercepted Nodes (RAIN) | Maize (Corn) [82] | Simulates the behavior of randomly placed rain drops to find the routes of these drops while moving from point to point. Performs segmentation based on the points considered as potential path candidates. |
Spectral Clustering | Thale cress (Arabidopsis) [9] Birch [174] Brassica sp. [9] Horse Chestnut [174] Maize (Corn) [9] Physalis sp. [9] Poplar [174] Red Oak [174] Sweet Chestnut [174] Wheat [9] | Is a technique with roots in graph theory, where the approach is used to identify communities of nodes in a graph based on the edges connecting them. |
Voxel verlapping Consistency Check | Cotton [99] Maize (Corn) [99] | Encloses the voxel grid by a bounding box. Considers the (area of) constituent voxels at different cross-sections of this bounding box to segment between stem and leaves. |
Machine Learning Techniques | ||
Boosting | Soybean [84] | Seeks to improve the prediction power by training a sequence of weak models, each compensating the weaknesses of its predecessors. |
Deep Learning (DL) | Banana [90] Maize (Corn) [160, 299, 310, 313] Rice [311] Rosebush [264] Rosette plants [1] Sorghum [276] Thale cress (Arabidopsis) [17] Tobacco [276] | DL is a very commonly employed algorithm in the ML algorithms, and it is derived from the conventional neural network but considerably outperforms its predecessors. DL employs transformations and graph technologies simultaneously in order to build up multi-layer learning models. The most famous types of deep learning networks are CNNs, RNNs, and RvNNs. |
Hidden Markov Models (HMMs) | Maize (Corn) [34] Tomato [34] | Are probabilistic models in which an unobservable (“hidden”) Markov process influences an observable process in a specific way. The goal is to est the hidden states from the observations. |
K-Means Clustering | Barley [226] Grapevine [226] Maize (Corn) [117] Sorghum [117] Soybean [84] Wheat [226] | Is one of the simplest and most popular unsupervised machine learning algorithms and aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster. |
K-Nearest Neighbors (KNN) | Aloe vera [81] Birch [174] Brassica sp. [81] Bromeliad species [81] Chili [81] Cordyline sp. [81] Horse Chestnut [174] Maize (Corn) [34, 133, 161, 186] Poplar [174] Pumpkin [81] Rapeseed [81] Red Oak [174] Sweet Chestnut [174] | Is a simple, supervised machine learning algorithm that can be used to solve both classification and regression problems and clusters the point set into a series of k nearest neighbors. |
Random Forest Classifier (RFC) | Rosebush [264] | Uses a combination of tree predictors such that each tree depends on the values of a random vector sampled independently and with the same distribution for all trees in the forest [263]. |
Self-Organizing Map (SOM) | Is an unsupervised neural network using the concept of competitive learning instead of back-propagation. | |
Support Vector Machine (SVM) | Soybean [84] | Is a popular and commonly used choice for binary classification problems which can perform nonlinear classification. |
Miscellaneous Techniques | ||
Maize (Corn) [95] Mango [122] Olive [122] Peach [122] Pine tree [180] Tomato [242] Walnut [122] | Methods developed for specific plant applications. | |