A vision language model for generating XML-based organ-level plant architecture representations of cowpea from simulated images

Abstract

Three-dimensional (3D) procedural plant architecture models are essential for simulating plant biophysical processes, but their parameters are difficult to measure at scale. We present a novel algorithm that generates 3D plant architecture from an image using a Vision Language Model (VLM). By encoding the detailed plant architecture into token sequences, the VLM can learn to predict organ-level geometric and topological parameters from synthetic imagery generated via the Helios 3D simulator.

Our results show that VLMs can predict plant architecture tokens with high accuracy (F1 score of 0.73) and outperform traditional regression methods in estimating bulk traits like leaf count and area. This work provides a scalable framework for reconstructing 3D plots for digital twins in agriculture.

Synthetic Data & Simulation

To train our model, we generated a large-scale synthetic dataset of cowpea plants using Helios 3D. This allows for perfect ground-truth labels of complex internal structures (internodes, petioles, and leaf angles) that are typically occluded in real-world imagery.

Figure 4. Integration of XML-based architecture definitions with the Helios 3D renderer to produce training data.

Proposed Methodology

Our approach moves beyond simple bounding box detection by predicting the underlying "procedural recipe" of a plant. We utilize a dual-stage process: first, converting the hierarchical XML architecture into a predictable token sequence; and second, training a VLM to map input images to these sequences.

Figure 2. Overall structure of the VLM-based procedural plant architecture generation algorithm.

Figure 3. Detailed process of mapping plant organ annotations to tokenized sequences for VLM training.

Experimental Results

Our model demonstrates superior performance in capturing bulk biophysical parameters that are traditionally difficult to estimate from 2D images. By understanding the 3D procedural logic, the VLM can "infer" occluded leaves and stems.

Figure 5. Comparison of bulk trait estimation (Leaf Area, Dry Mass) across different growth stages.

Figure 6. Qualitative results showing the reconstructed 3D architecture (right) versus input images (left).

DNA-Style Architecture Analysis

One of the unique advantages of our token-based representation is the ability to treat plant structure as a sequence of "genetic" tokens. This "DNA-style" visualization allows us to compare ground truth versus estimated architectures organ-by-organ, highlighting where the model captures regional structures versus where it makes specific parameter errors.

Sequence comparison for a young plant (Day 9).

Sequence comparison for a mature plant (Day 19).

BibTeX

@article{yun2026visionlanguagemodelgenerating,
      title={A Vision Language Model for Generating XML-Based Organ-Level Plant Architecture Representations of Cowpea From Simulated Images}, 
      author={Heesup Yun and Isaac Kazuo Uyehara and Ioannis Droutsas and Earl Ranario and Christine H. Diepenbrock and Brian N. Bailey and J. Mason Earles},
      journal={Frontiers in Artificial Intelligence},
      volume={9},
      year={2026},
      publisher={Frontiers Media SA},
      doi={10.3389/frai.2026.1844338},
      url={https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2026.1844338/full}
}