RSS · Harvard REACT Lab 2026

Seeing moving objects: next-best-view planning

K. Li, M. Mantovani, R. J. Wood, L. Sabattini, S. Gil

The robot predicts where the object will be and scores each reachable view by the new surface it should reveal. On a real robot, 79.8% of the object is reconstructed in 10 views, against 63.3% and 60.7% for the two baselines.

Problem

To reconstruct an object in 3D, a robot has to look at it from many angles. Most planners assume the object stays still. When it moves, the view you chose is already outdated by the time you get there. Tracking methods keep the object in sight, but they don’t care whether you’re seeing new surface.

Approach

Predict where the object will be, using a fixed-lag Gaussian Process smoother over its position and velocity, with uncertainty.
Generate candidate viewpoints around the predicted location and discard those the robot can’t reach in one step.
Score each view by expected new surface coverage across many sampled object positions (Monte Carlo), then go to the best one.

One planning step: current belief (purple), predicted belief (red), candidate views on an ellipse (orange), the ones inside the robot’s one-step reach (dashed blue), best expected score (green).

My contribution

I developed the GP-based motion prediction: the fixed-lag smoother that turns noisy position measurements into a position-and-velocity estimate, and the one-step predictive belief the planner samples from.
I’m second author. The candidate generation, the view scoring and the experiments are the team’s work, led by Karen Li.

Results

Real robot, 0.10 m position noise, one representative trial: 79.8% of the object reconstructed after 10 views, against 63.3% for non-predictive planning and 60.7% for tracking-only. The same ordering holds at every noise level, in simulation and on the robot.

Real robot, three planners on the same trial at views 1, 5 and 10, with completeness. Colour is the view at which a patch was first seen: the predictive planner keeps adding new surface, the baselines mostly re-see what they have. Bottom: mean ± std over four trials.

Stack

Robot: Mobile ground robot with an Intel RealSense D435i depth camera
Software: ROS 2 Jazzy, Python, Gazebo for simulation
Sensing: Object position from a separate source (noise injected, 0.10–0.20 m); motion capture only for ground truth
Compute: Planning on CPU, under a second per iteration

Done during my visiting PhD at Harvard’s REACT Lab, with Stephanie Gil and Robert Wood. Published as “Motion-Uncertainty-Aware Next-Best-View Planning for Moving Object Reconstruction”. The cover animation is a small illustrative simulation of the method, not experimental data.