AIFS Sony Tech Challenge
1st Place Winner - 2025 AIFS × Sony AI AgTech Challenge
Our team (Heesup Yun, Jonathan Berlingeri, Ethan Truong, and Nathan Chiu) won 1st Place at the 2025 AIFS × Sony AI AgTech Challenge. We built a working technical demo within the two-day hackathon. Equipped with a Raspberry Pi Zero and a Raspberry Pi AI Camera, we built WeedTrackr: Precision Weed Management System.
Why weeds
Weed management is a major problem in agriculture, with large economic and environmental costs:
- $33 billion in annual U.S. crop yield losses due to weed competition.
- $6 billion in additional control measure costs each year.
- 70 million pounds of herbicides are lost to imprecise application and overuse each year.
Conventional methods rely on “blanket” treatments: equal amounts are applied to the entire field regardless of actual weed distribution. While precision equipment exists, it is expensive and built primarily for annual row crops. In vineyards and orchards, mowing is common for controlling weeds between rows, but weeds along rows remain another challenge.
WeedTrackr
WeedTrackr is a low-cost AI weed monitoring system designed specifically for planning herbicide sprays in orchards and vineyards. It automates weed scouting and herbicide application planning during regular maintenance activities, saving time and money, and mitigating environmental impacts by reducing herbicide use by up to 90% through targeted application.
Using it takes three steps:
- Mount: Mount WeedTrackr onto your mower or tractor.
- Power on: Activate the image collection system by powering on WeedTrackr.
- Work: Continue with planned activities. WeedTrackr runs autonomously in the background while you work.
How it works
WeedTrackr’s software stack integrates affordable, off-the-shelf hardware with custom integration software.
- Detect: We use a Raspberry Pi Zero with the AI-enabled camera and a Sony Spresense GPS module to capture geolocated images as you move through your field.
- Classify (Edge AI): The onboard AI model processes images in real-time to identify and categorize weeds (broadleaf vs. grass) and distinguish them from the soil. Our custom weed classification model is built on a modified MobileNetV2 architecture, trained on a diverse dataset (500+ broadleaf, 500+ grass, 1000+ soil/background)
The model was optimized for edge deployment. Using MCT (Model Compression Toolkit) for INT8 quantization and ONNX conversion to an IMX500 package, we reduced the model size by 75% while maintaining over 93% of the original accuracy.
- Plan (Web Dashboard): Built with React + TypeScript, the frontend dashboard uses the classified data to generate spray route plans and variable rate application maps. It visualizes GPS-accurate weed distribution heat maps (separating grass and broadleaf weeds), historical treatment analysis, and estimated chemical savings compared to blanket treatment.
Expected benefits
AI and robotics hardware is now cheap enough for this kind of tool. As agricultural input costs (chemicals, labor, and fuel) climb and government regulations tighten, minimizing chemical buildup and preserving long-term biodiversity is increasingly important.
By only spraying where it’s needed, WeedTrackr helps farmers:
- Stay compliant with evolving environmental standards
- Keep food cleaner by reducing herbicide over-usage
- Protect land health
- Reduce manual labor by automating weed scouting
- Boost crop yields with fewer weeds and better plant performance
Technical Details
The Raspberry Pi Zero isn’t typically powerful enough to handle deep learning right away. To make it work, I pushed the CPU and GPU limits through overclocking and tailored our image classification model for the Raspberry Pi AI Camera. This setup shifts the heavy lifting from the CPU to the camera’s dedicated AI accelerator, making it possible to run sophisticated models on such a small, low-power device.
We first set up the device with a 64-bit OS referring to this guide: Install a 64-bit OS on Raspberry Pi zero 2 - Q-engineering.
Below is the evolution of our hardware configurations, using resources like the MakeUseOf overclocking guide.
1. Stock Configuration
By default, the arm_freq is set to 1000 and gpu_freq to 300.
weed@raspberrypi:~ $ vcgencmd get_config int
... (shortened)
arm_freq=1000
core_freq=400
gpu_freq=300
sdram_freq=450
2. First Overclock
We increased the arm_freq to 1200 and core_freq to 500 for a noticeable performance bump without touching the voltage.
weed@raspberrypi:~ $ vcgencmd get_config int
... (shortened)
arm_freq=1200
core_freq=500
gpu_freq=300
sdram_freq=450
3. Pushing Further
Finally, we pushed arm_freq to 1300, gpu_freq to 500, and sdram_freq to 500. We also configured over_voltage=5 and over_voltage_sdram=3 to keep the system stable.
weed@raspberrypi:~ $ vcgencmd get_config int
... (shortened)
arm_freq=1300
core_freq=500
gpu_freq=500
sdram_freq=500
over_voltage=5
over_voltage_sdram=3
These settings can be permanently applied in /boot/config.txt as follows:
[all]
arm_freq=1300
core_freq=500
gpu_freq=500
sdram_freq=500
over_voltage_sdram=3
# over_voltage is done by the governor.
# set the parameter to overrule its moderate choice.
over_voltage=5
enable_uart=1
dtoverlay=pi3-miniuart-bt
Overclocking improved the VNC performance a lot. But it requires active cooling, so I added a heatsink and fan.
Running the AI Model on the Raspberry Pi Camera
Getting the deep learning model to run optimally on the Raspberry Pi AI Camera involved several rounds of testing.
1. Exploring TF Lite & Miniforge
We initially explored installing TensorFlow Lite and experimented with a few guides for object detection and image classification, such as building the tflite-runtime explicitly for the Pi Zero:
echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-get update
sudo apt-get install python3-tflite-runtime
We also set up a local Conda environment using Miniforge to manage dependencies cleanly. We reviewed the official TensorFlow Lite Raspberry Pi examples. To ensure the camera works, it’s also practical to check out the new libcamera-based Python library, picamera2.
2. Resolving Environment Conflicts
To use the specific Sony IMX500 AI sensor effectively, the necessary tools are installed via:
sudo apt install imx500-all
3. Execution
Once the environment was correctly pointed towards the native Raspberry Pi OS libraries without Conda’s interference, everything worked. We successfully executed our .rpk compiled model with:
python imx500_classification_demo_ucd.py --model weed.rpk --softmax
You can check out our repository and code here.