Skip to content

Benchmarking

CI measures every number on this page on pinned hardware and renders it from committed data under docs/data/. Nothing here is hand-written. The data changes only through reviewed chore(data): refresh … pull requests that the benchmark workflows open, so a regression, or an improvement, is always a visible diff.

What CI measures

Steady Real Time Factor (RTF): simulated seconds per wall-clock second. The loop measures it over the whole run excluding the first 250 control steps, where lazy kernel compilation and initial settle happen. Whole-run wall time is never used as a speed metric. Alongside the steady figure, the loop profiler records an RTF window series of 1000-tick windows, first window dropped, whose min / avg / max / σ show how stable a run is.

The benchmark mission: one PX4 flight, the same for every matrix cell but one. It arms, runs AUTO.TAKEOFF, then flies a 4-waypoint DO_REPOSITION mission with confirmed arrival at each waypoint. The powerline cell flies an inspection of its power line instead, as a PX4 mission. It follows the line pole by pole at 1.5 m over the top conductor and lands at its far end. The script scripts/ci/benchmark_cell.py flies both through nexus_uav.px4.OffboardClient, the client it opens on PX4's offboard link. PX4 Software In The Loop (SITL) runs in the loop, in lockstep on :4560, so these figures come from a closed loop, not from physics-only throughput. All cells record their flight with --log on: recording is part of the measured configuration.

Real-time factor matrix

Physics only (a vehicle with no RTX sensor, no renderer)

Vehicle Scene Device Steady RTF Window min / avg / max
astro_max_base empty cpu 0.43× 0.43 / 0.43 / 0.44 (σ 0.00)
astro_max_base empty cuda 1.97× 1.94 / 1.99 / 2.07 (σ 0.05)

RTX rendering (the camera rendered by the Kit peer, GPU)

Vehicle empty cesium
astro_max_fpv 0.63× 0.59×

cesium streams Google 3D Tiles over the network: its RTF varies with tile traffic.

Measured on NVIDIA A10G · 2026-08-16 · commit f414a10ef37b
Mission: takeoff to 5.0 m, then waypoints 20,0,5; 20,-20,8; 0,-20,5; 0,0,5 (x,y,z world offsets [m] from the takeoff point; +x is north).

The scheduled gpu-benchmark-matrix workflow produces the matrix on CI, each cell on a g5.2xlarge box of its own with one A10G, and refreshes it via a data PR. Cells that structurally don't exist aren't shown. A visual-only scene changes nothing for a vehicle that renders nothing, since physics loads only UsdPhysics-authored geometry, so the physics table flies the empty scene alone. In the RTX table the vehicle's camera renders in the Kit peer container beside the loop. CPU is the bit-exact determinism authority rather than a performance configuration, so it's displayed only in the physics reference row.

Per-example performance

Every gpu-examples CI run scores each example on flight quality and steady RTF. Flight quality is the evo pose Absolute Pose Error (APE) against the example's reference, plus the example's own stats:

Example Steady RTF Pose APE rmse Other measured metrics
pid 6.24× 0.825 m reached 4, final_dist_m 0.284 m, ape_trans_max_m 1.642 m, final_err_m 0.284 m
gain_tuning 5.37× 0.656 m gradient_cosine 1, opt_hold_m 0.272 m, opt_final_m 0.272 m, base_final_m 3.155 m, ape_trans_max_m 1.370 m, final_err_m 0.272 m
mass_recovery mass_rel_err 0.0003, true_mass_kg 12.85, recovered_mass_kg 12.86, wall_s 41.9 s
sampling_mpc 0.30× 0.606 m reached 3, final_dist_m 0.002 m, min_clearance_m 0.938 m, max_tilt_deg 35.0°, path_wiggle 1.258, nose_err_deg 16.1°, ape_trans_max_m 1.167 m, final_err_m 0.002 m
acados_nmpc 0.43× 0.022 m final_dist_m 0.021 m, track_rmse_m 0.021 m, track_max_m 0.062 m, max_tilt_deg 15.8°, nose_err_deg 11.4°, ape_trans_max_m 0.058 m, ape_rot_rmse_deg 17.9°, ape_rot_max_deg 38.9°
px4_sitl 1.94× climb_m 5.040 m, time_to_takeoff_s 22.8 s, px4_warnings 0

Measured on NVIDIA A10G · 2026-08-16 · commit b7d6504465c6

sampling_mpc is exempt from the RTF ≥ 1 doctrine by design, since it runs 16 parallel rollouts of an 80-step horizon per control tick. Its gate is its recorded value instead.

Every gated metric over the last commits on main, one panel per metric under its example, with the gate as a dashed line. The page draws them in the browser from the per-commit bench feed every gpu-examples run on main uploads, so they're as fresh as the last merge. A metric with no gate has no panel: a metric whose drift matters takes a gate. The bucket serves the feed to the deployed site only, so a local mkdocs serve shows a note in their place.

Loading the trend feed…

The regression gates

The same harness that produces these numbers gates them. scripts/ci/evaluate_examples.py runs every example and compares each metric against the committed scripts/ci/examples_baselines.json. A bare value gates exactly, min/max gate absolutely, and min_frac/max_frac gate relative to the recorded baseline value once one exists. Baselines are hardware-pinned to the A10G runner, and only a deliberate --update-baselines dispatch re-records them. A metric without a baseline entry is report-only: the harness shows it and never gates on it.

Beyond the gates, every upload also writes a per-commit trend feed, bench/<sha>.json in the CI artifacts bucket in github-action-benchmark custom JSON. It lists the file in bench/latest.json, the index the trends read.

Reading numbers elsewhere

Example pages embed their own generated stat blocks from the same data. Prose figures quoted in design discussions, for example the dev-workstation RTFs in Execution & determinism, name the hardware they ran on and are illustrative, not tracked.