Deep phenotyping: deep learning for temporal phenotype/genotype classification

Plant productivity must increase dramatically this century, while using resources more efficiently, to accommodate the ever-growing demand of a more affluent and growing human population. Precision breeding, via selecting advantageous genomic variants, will help improve plant productivity and efficiency but it relies on a detailed understanding of the genotype to phenotype relationship [1]. Here, a framework for automatic feature (phenotype) extraction and classification during the plant growth time period can greatly facilitate these studies. We have developed climate chambers, which maintain diurnal and seasonal climate signals but remove the weather noise plaguing field studies. These chambers have automated image capture capability to constantly monitor plants throughout their entire life cycle [2].

Arabidopsis thaliana is one of the model organisms used for studying plant biology, and it now has genomes sequences from 1000s of accessions [3]. Since the growth patterns of this plant are easily observable (especially from top-view), it is a very useful model for automated phenotyping. Previous work on phenotyping different accessions (genotypes) have mostly used biologist specified, ‘hand-crafted’ image features such as number of leaves, leaf area, compactness, roundness, etc. [4–8]. These features are computed either manually or via custom image processing algorithms. Their output may then be passed to a classifier. The main weakness of using hand-crafted descriptors is that although they are readily interpretable, they may be missing or incorrectly measuring the actual features that are variable among accessions. Furthermore, the custom image processing methods to extract the hand crafted features may not work as well when run on other experiments and may be difficult to generalize to more heterogeneous datasets [9].

Problems with hand crafted features have been addressed in the past few years by harnessing the power of deep learning Convolutional Neural Networks (CNNs) in particular [10–14], although difficulties with interpretation of the machine learned traits and over-fitting to a particular experiment remain. CNNs automatically find and extract the most descriptive features from the data during the training process. In other words, both feature extraction and training steps are performed simultaneously and hence, the system tries to find the features that minimize the loss criterion of the phenotyping problem. As a result, novel features for accession recognition are revealed in this process. However, in order for a machine to learn a good set of features, a very large training dataset is required.

CNNs are great for classification and segmentation of images, but they are unable to properly model dynamic systems, such as time-lapse video in our case. Although CNNs can not encode temporal dependency of successive image frames, this problem can be addressed by using a Recurrent Neural Network (RNN) in which, each image frame is processed and analyzed by a neural cell and the information of each cell is circulated to the succeeding cells. RNNs, and in particular Long Short-Term Memories (LSTMs, which are explained in detail in "LSTM" section) have demonstrated potential in computer vision for analysis of dynamic systems [15–19]. In this study we utilize LSTMs to carefully model the growth patterns of plants.

In this work we investigate the capability of CNN features for describing the visual characteristics (phenotypes) of different accessions (genotypes), and compare these deep features with hand-crafted descriptors that were primarily used in previous works. In particular we present a plant analysis framework that automatically extracts and utilizes most descriptive features for each application and exempts us from manual feature selection and tuning for different tasks and experiments. More importantly, we propose to use LSTMs to automatically take into account the growth and temporal behavior of plants in their classification. By incorporating the temporal information into the analysis, it is revealed how phenotypes that distinguish different accessions change over days of plant growth. This framework can also be used for classification of the plants with different genotypes, plants grown in different environment conditions (e.g. soil, temperature, humidity and light), or detection of plant diseases. Furthermore, plant detection and classification using robotics and automation for improved plant production and care is another potential application.

In addition, we release a new challenging dataset that contains time-lapse recordings of top-view images of Arabidopsis accessions, to evaluate the proposed method in this paper for accession classification task. Note that there is a substantial similarity between the appearance of different accessions in this dataset, which is even very hard for biologists to distinguish them. Nonetheless, our model outperformed traditional methods based on hand-crafted image features and other accession classification frameworks, by using deep features as well as by encoding temporal information. A primary extension of this work in the future is to study new accessions and their behavioural and appearance association with parental reference accessions. This can vastly help us to better find relationships between phenotypes and genotypes. This is briefly described in "Conclusion" section.

Research has focused on automatic plant phenotyping and classification using high-throughput systems. Classification of growth phenotypes based on data from known planted genotypes represents a typical experimental design where the aim is to obtain measures that maximize signal between genotypes relative to environmental error within biological replicates of the same genotype. Advanced image processing using machine learning techniques have become very popular in phenotyping qualitative states [20–24] while there are still many prospective needs and goals [25–29] to be experimentally explored in plants. A number of recent studies have presented high-throughput systems for plant phenotyping [2, 30–33] and also plant/leaf segmentation and feature extraction [34–37].

Plant classification has attracted researchers from the computer vision community [38–41] given its importance in agriculture and ecological conservation. There are several studies of plant classification built on the pictures of individual plant leaves [42–45]. Approaches to recognize plant disease [46, 47], symptoms of environmental stress [31, 48], and differentiation of crops from weeds [49, 50] have been previously studied. Normally three primary steps of plant/leaf segmentation, feature extraction, and classification are involved in these studies. The performance of the whole phenotyping pipeline depends on the performance and interaction among each of the three elements.

In the past few years, deep learning methods and in particular, Convolutional Neural Networks have achieved state-of-the-art results in various classification problems, and have motivated scientists to use them for plant classification [51–57] and plant disease detection tasks as well [58, 59]. CNNs are able to learn highly discriminative features during the training process and classify plants, without any need for segmentation or hand-crafted feature extraction. In particular, [54] used a CNN for root and shoot feature identification and localization. The authors in [52] proposed Deep Plant framework which employs CNNs to learn feature representation for 44 different plant species using the leaves. However, all the above-mentioned studies in plant phenotyping, feature extraction and classification are all based on individual static images of the plants of different species. In other words, temporal information, such as the growth patterns, one of the key distinguishing factors between varieties within plant species, has not been previously taken into account. Temporal cues can be very helpful, especially for distinguishing between different plants that have similar appearances, e.g. for separating different accessions of a particular plant, which is often a very challenging task.

In order to account for temporal information, various probabilistic and computational models (e.g. Hidden Markov Models (HMMs) [60–62], rank pooling [63–65], Conditional Random Fields (CRFs) [66–68] and RNNs [69–72]) have been used for a number of applications involving sequence learning and processing.

RNNs (and LSTMs in particular) are able to grasp and learn long-range and complex dynamics and have recently become very popular for the task of activity recognition. For example, The authors in [73, 74] used CNN and LSTM for generating image descriptions and multi-lable image classification, respectively. More specifically, [15–19] used LSTM in conjunction with CNN for action and activity recognition and showed improved performance over previous studies of video data. In this paper, we treat the growth and development of plants as an action recognition problem, and use CNN for extracting discriminative features, and LSTM for encoding the growth behavior of the plants.

In this section, we explain the fundamentals of deep structures used in this paper, including CNN, RNN and LSTM.

CNN

Figure 1 depicts the schematic of a Convolutional Neural network (Alexnet [75]). Each layer in this network consists of a set of parameters, which are trainble in general, either from scratch or by benefiting from pretrained networks (refer to "CNN training" section for further explanation). The output of each layer might pass through some non-linear activations such as sigmoid or Relu functions [75]. The CNN structure takes a tensor of three-dimensional data as its input, passes it through multiple sets of layers and then outputs a score that represents the semantic class label of the input data. For instance in a simple cat vs. dog classification task, the input could be the image of a kitty and the correct output would be a high score for the cat class.

In our application, we feed the CNN with top-view images (with three color channels) from plants. Next we introduce the main layers of a CNN.

Convolutional layer

This layer is computed by applying multiple filters to the input image, i.e. sliding the filter window over the entire input image. Different filters can have different parameters, which lets them detect and learn different image features. For example, one filter could be in charge of spotting vertical edges, while another one could detect horizontal edges [76]. The output of this layer is called a feature map, which is depicted in Fig. 2. It shows class activation maps that identify image important regions.

Filters are normally designed to be small (\(3\times 3\), \(5\times 5\), \(7\times 7\), ...), to reduce the number of parameters in the system. As a result, regardless of the size of the input image, the parameter size remains limited. Moreover, multiple back-to-back small filters in successive layers can cover a larger receptive field and consequently, more context information can be encoded. This is in contrast to the design of a fully connected neural network where all the units in the previous layer are connected to every unit in the next layer with unique parameters, which leads to a sizable parameter set.

Max pooling layer

Each feature map obtained from the convolutional layer, is an indicator of a particular feature in different locations of the input image. We normally want our descriptors to be robust against minor displacements of the input data. This is addressed by adding a max pooling layer to the network, which downsamples the feature maps. In other words, it reduces small patches of the feature map into single pixels. If a feature is detected anywhere within the patch, the downsampled patch fires a detection of that feature (local invariance).

A more practical benefit of the pooling layer is that, reducing the size of the feature maps leads to a significant decrease in the number of parameters, which in turn controls overfitting and also speeds up the training process. Another advantage of pooling layer is that it helps the network to detect more meaningful and high-level features as it moves on to the deeper layers. In this structure, the first layer has detected low level features like edges, whereas the next layer could grab more sophisticated descriptors like leaves or petiole, and the layer after has learned high-level features that are able to describe the whole plant.

Fully connected layer

After a sequence of multiple convolution and pooling layers, the size of input data is shrunk dramatically which is suitable as input to a fully connected (dense) layer. The resulting feature maps up to this point of the network are vectorized and feed a multi-layer fully connected neural network, whose last layer (a.k.a classification layer or softmax layer) denotes the scores of the class labels in our problem.

The last fully connected layer is in charge of computing the scores for each class label. Each neuron in this layer represents a category in the classification problem, and its class probability can be computed by applying a softmax function to its inputs from the previous layer.

CNN structure

The structure of a CNN (number of different layers, size of the filters, size of the fully connected layers, etc.) may vary depending on the application and the size of the training data. During the past few years, several architectures have been proposed and shown to work quite well for image classification and segmentation problems, among which Alexnet [75], VggNet [77] and ResNet [78] are the most notable ones.

Figure 1 shows the schematic of Alexnet, which has five convolution layers, three of which are followed by max pooling layers. It also features three fully connected layers. This is the network that first attracted the attention of researchers to the potential of CNNs, by winning the ImageNet Large Scale Visual Recognition Competition (ILSVRC) by a big margin [79], compared to the models with hand-crafted features.

RNN

Figure 3 illustrates a simple RNN [80] that models a temporal data with three time points. In this representation, each time step is portrayed by a block of neurons, which receives two inputs respectively from the observed frame at that time, and the temporal cues propagated from previous times points. A fully connected neural network is embedded within each RNN cell to analyze the visual information of each frame together with the information that is received from previous times, to obtain the system state at each time frame. Let \(\mathbf{x}(t)\), \(\mathbf{h}(t)\) and \(\mathbf{y}(t)\) denote the visual input data, the output of RNN cell and the class label of the sequential data, respectively, at time t. Then the RNN can be expressed as

$$\begin{aligned} \mathbf{h}(t)&= {} f\Big (\mathbf{W}_{xh}{} \mathbf{x}(t) + \mathbf{W}_{hh}\mathbf{h}(t-1) + \mathbf{b}\Big ) \end{aligned}$$

(1)

$$\begin{aligned} \mathbf{y}(t)&= {} g\Big (\mathbf{W}_{hy}{} \mathbf{h}(t)\Big ) \end{aligned}$$

(2)

where \(\mathbf{W}_{xh}\), \(\mathbf{W}_{hh}\) and \(\mathbf{W}_{hy}\) are the neural network parameters, \(\mathbf{b}\) is a bias vector, and f and g are element-wise non-linear functions which are often set to hyperbolic tangent (\(\phi\)) and sigmoid (\(\sigma\)), respectively.

What makes this structure more interesting is that we can readily integrate RNN with a CNN, by feeding the visual input of the RNN cell with the pre-trained CNN features of the image frame at that time point.

LSTM

The main shortcoming of standard RNNs (Fig. 3) is that they can not encode temporal dependencies that prolong to more than a limited number of time steps [81]. In order to address this problem, a more sophisticated RNN cell named Long Short-Term Memory (LSTM) has been proposed to preserve the useful temporal information for an extended period of time.

An LSTM [82], as depicted in Fig. 4, is equipped with a memory cell and a number of gates. The gates control when a new piece of information should be written to the memory or how much of the memory content should be erased. Similar to a standard RNN, the state of the system at each time point is computed by analyzing the visual input at that time point, together with the output of previous cell and also the content of the LSTM memory, which is referred to as \(\mathbf{c}(t)\). Given \(\mathbf{x}(t)\), \(\mathbf{h}(t)\) and \(\mathbf{c}(t)\), the LSTM updates are defined as

$$\begin{aligned} \mathbf{i}_{t} & = \sigma \Big (\mathbf{W}_{xi}{} \mathbf{x}(t) + \mathbf{W}_{hi}\mathbf{h}(t-1) + \mathbf{b}_{i}\Big ) \end{aligned}$$

(3)

$$\begin{aligned} \mathbf{f}_{t} & = \sigma \Big (\mathbf{W}_{xf}{} \mathbf{x}(t) + \mathbf{W}_{hf}\mathbf{h}(t-1) + \mathbf{b}_{f}\Big ) \end{aligned}$$

(4)

$$\begin{aligned} \mathbf{o}_{t} & = \sigma \Big (\mathbf{W}_{xo}{} \mathbf{x}(t) + \mathbf{W}_{ho}\mathbf{h}(t-1) + \mathbf{b}_{o}\Big )\end{aligned}$$

(5)

$$\mathbf{c}(t) = \mathbf{f}_t\odot \mathbf{c}(t-1) +\mathbf{i}_t\odot \phi \Big (\mathbf{W}_{xc}{} \mathbf{x}(t) + \mathbf{W}_{hc}{} \mathbf{h}(t-1) + \mathbf{b}_{c}\Big )$$

(6)

$$\begin{aligned} \mathbf{h}(t) & = \mathbf{o}_t\odot \phi \Big (\mathbf{c}(t)\Big ) \end{aligned}$$

(7)

In these equations, \(\mathbf{i}_t\), \(\mathbf{f}_t\) and \(\mathbf{o}_t\) denote input gate, forget gate and output gate respectively. The input gate controls how much of the new input data should be recorded into the memory, whereas the forget gate decides how much of the old memory should be preserved at each time. The output of the LSTM cell is also computed by applying the output gate to the memory content. This sophisticated structure enables LSTM to perceive and learn long-term temporal dependencies. Note that \(\odot\) in Eq. 3 indicates an element-wise multiplication.

After seeing a sufficient number of data sequences in the training phase, LSTM learns when to update the memory with new information or when to erase it, fully or partially. LSTMs can model various sequential data very easily, unlike other complicated and multi-step pipelines. Furthermore, they can be fine-tuned similar to CNNs. These benefits has made LSTMs very popular in the recent years for modelling data sequences. In this paper, we propose a CNN-LSTM structure (Fig. 5) to build a plant classification system, which is explained in more detail in "CNN-LSTM network" section.

We aim to propose an automatic accession classification framework, using the deep visual features of the plants (which are trained specifically for the accession categories) as well as the temporal cues of the plant growth sequences. To this end, in this section we introduce the CNN-LSTM model and then explain how to train this model.

CNN-LSTM network

In this section, we describe the proposed framework for genotype classification, which is composed of a deep visual descriptor (using a CNN), and an LSTM which can recognize and synthesize temporal dynamics in an image sequence as well as the texture changes. As depicted in Fig. 5, our approach is to first pass each individual frame of the plant image sequence through the deep visual descriptor (CNN) to produce a fixed-length vector representation. This fixed-length vector embodies the features of each individual plant, which are extracted after fine-tuning step (as explained in "CNN training" section). In this work, we have used Alexnet as our CNN.¹ The outputs of CNN for the sequence of pot images are then passed onto a sequence learning module (LSTM). At this stage, the LSTM attempts to classify the plants via analyzing the sequences of the features that are extracted from image frames and by taking into account their temporal variations. Although there is no quantitative measurement (among the deep features and their variations) for some important phenotypes, such as number of leaves or growth rates, these information are implicitly encoded throughout the time by the network to better distinguish different accessions. In other words, the proposed CNN-LSTM structure captures the activity of the plants during their growth period to model the relationships between their phenotypes and genotypes.

The proposed model can automatically classify plants into the desired categories, given only the plant images. Note that our approach can be easily extended to the cases, where more classes are involved, just by performing the training phase for the new set of classes. Extending the model to applications other than plant classification is just as easy, where one can simply modify the target layer of the network to fit that particular problem. This is counter to the conventional phenotyping methods, where one is required to find relevant hand-crafted features for each individual application.

CNN training

The goal of training is to find the values of network parameters such that the predicted class labels for the input data are as close as possible to their ground truth class labels. This, however, is a very challenging task since CNNs normally have a vast number of parameters to be learned. Alexnet for instance is built on more than 60 millions parameters. Training a system with this many parameters requires a massive number of training images as well.

There are a few publicly available datasets that provide sufficient number of images for training CNN architectures, among which ImageNet-ILSVRC is very popular. It is a subset of much larger ImageNet dataset and has about 1.2 millions images selected from 1000 different categories. However, in many problems we do not have access to a large dataset, and this prevents us from properly training a CNN for them.

It is shown if we initialize the network using the parameters of a pre-trained CNN (a CNN that is already trained on a big dataset like ImageNet), and then train it using the limited dataset in our problem, we can achieve very good performance. In particular, we can rely on the basic features that the CNN has learned in the first few layers of the network on ImageNet, and try to re-train the parameters in the last few layers (normally fully connected layers) such that the network could be fit to our specific problem. This method is often referred to as fine-tunning, which speeds up the training process and also prevents overfitting of the network to a relatively small dataset.

Note that in many image classification problems, it is very common to preserve all the layers and parameters of a pre-trained CNN, and only replace the last layer that represents the 1000 class labels of ImageNet with the class labels in our specific problem. Then only the parameters of the classification layer are learned in the training phase, and the rest of the parameters of the network are kept fixed to the pre-trained settings. In fact here we assume that the deep features that are previously learned on ImageNet dataset can describe our specific dataset quite well, which is often an accurate assumption. The outputs of the layer before the classification layer of a CNN are sometimes refereed to as pre-trained CNN features.

In this work, we chose to fine-tune a pre-trained CNN using the top-view images of the plants, in order to learn more discriminant features for distinguishing different accessions.

Data augmentation

When a dataset has a limited number of images, which is not sufficient for properly training the CNN, it makes the network vulnerable to overfitting. In order to synthetically increase the size of the training data, we can use a simple and common technique, called Data Augmentation. In this procedure, we rotate each image in the dataset by \(90^\circ\), \(180^\circ\) and \(270^\circ\) around its center and add it to the dataset.

In this section, we first introduce the dataset and then explain the pre-processing and plant segmentation steps. Next, we report the accession classification results using the proposed CNN-LSTM method. In order to evaluate this method more thoroughly, we extract a set of hand-crafted features and investigate their performance in the accession classification task, compared to our CNN-LSTM framework that uses deep features. Furthermore, we report the results of a variant of our approach where the LSTM is replaced by a CRF, to have a more thorough temporal analysis of the proposed model. To the best of our knowledge, our dataset is the first publicly available dataset that provides successive daily images of plants while they are growing, together with their accession class information. Therefore we did not have access to other temporal data to further evaluate our model. We hope this could help other researchers in the field to have a more in-depth study of temporal variations of different accessions.

In this paper, we proposed a framework for automatic plant phenotyping based on deep visual features of the plants and also temporal cues of their growth patterns to classify them based on their genotypes. Classification of accessions using their images implies the difference in their appearances and indicates the ability of deep learning based methods in finding these differences. Moreover, to the best of our knowledge, this is the first work that studies the temporal characteristics and behaviors of plants using LSTMs and shows their potential for the accession classification task. Our experiments evidence the benefits of using deep features over hand-crafted features, and indicate the significance of temporal information in a plant classification task.

Despite the deep learning demand for a large input dataset and our limited sequential data from different accessions, we presented a sophisticated deep network and an efficient method to train it. In the future, we plan to augment our dataset with more varying visual and sequential data to enhance the robustness of our system when dealing with more challenging classifications.

The model obtained in this study can be used for analysis of unseen accessions, e.g. for finding their behavioral similarities with the accessions used in the training, which could reveal the relationships between the phenotypes and genotypes (our ongoing work). In fact, probabilistic classification of reference accessions is a holistic approach to plant phenotyping where unknown accessions can be typed as to their similarity to multiple references. This goes beyond traditional hand crafted measures of leaf size shape and color. One example is the classification of progeny accessions based on their similarity to parental reference accessions. We plan to apply our trained classifier to a large set of accessions. The probability of each genotype state, Sf-2, Cvi, Ler-1, Col-0, is a multivariate growth pattern phenotype of each accession, which can be decomposed into its causal genetic factors using Genome Wide Association Studies.

Furthermore, due to the generality of the proposed model, it can be used with no major modification for other tasks such as disease detection or for analyzing different environment conditions (e.g. soil, temperature, humidity and light) for plants. Studying the temporal behavior of the plants using the recorded image sequences of their first few days of growth and based on our CNN-LSTM model, can predict the crop yield of the plants as well as their health (our future work).