Skip to main content

The engagement viewer: single cases behind the G2 numbers

· 4 min read

The gates judged the models on medians over thousands of scenarios. The new engagement viewer shows one scenario at a time: the plan, both platforms' course and speed, the bearings, and each gate's carried models beside the classical solver and the naive baseline. It is the first step of SRD Annex A. Its first use is the owner's realism check of the generator, which comes before any change to the training data.

Iteration 2c: G2 failed on speed; the project stops at G2

· 9 min read

Iteration 2c was the last attempt the SRD allows under G2. It combined the two levers that each moved speed on validation, a speed-weighted loss and twice the model capacity, and judged G2 on a third fresh test split. Range and course are the best of the project. Speed is not within reach.

Verdict: G2 failed (agent review panel, round 2, no rejections). Under SRD §7 the project stops at G2: no-go.

  • Range passes for both models: 13.5% [13.0, 13.9] instantaneous and 12.1% [11.7, 12.6] time-late at zig + 300 s on the σ = 1° slice, against 20%.
  • Course passes for both: 7.36° [7.05, 7.70] and 6.12° [5.87, 6.41], against 10°.
  • Speed fails for both: 1.47 kt [1.37, 1.52] and 1.53 kt [1.47, 1.61] at zig + 600 s, against 1 kt. Only 37% and 33% of scenarios are under 1 kt. Classical least squares reaches 0.61 kt on the same scenarios, at the Cramér–Rao bound.

Experiment of record: experiments/20261016-i2d-g2/. Curves and slices: iteration-2c results.

Iteration 2b: speed-weighted loss, G2 failed again

· 8 min read

Iteration 2b asks one question: does weighting speed in the loss get the iteration-2 models to the G2 speed threshold? It changes nothing else, and judges G2 on a fresh test split.

Verdict: G2 failed (agent review panel, round 2, no rejections).

  • Range passes for both models: 14.6% [14.1, 15.2] instantaneous and 15.8% [15.2, 16.2] time-late at zig + 300 s on the σ = 1° slice, against 20%.
  • Course passes for both: 8.22° [7.88, 8.54] and 8.66° [8.29, 9.03], against 10°. The time-late course failure of iteration 2 is gone.
  • Speed fails for both, by less: 1.77 kt [1.68, 1.83] and 1.77 kt [1.67, 1.86] at zig + 600 s, against 1 kt. The same weights before 2b, scored on the same scenarios, were at 2.02 kt and 2.31 kt. Classical reaches 0.60 kt.

Experiment of record: experiments/20261015-i2c-g2/. Curves and slices: iteration-2b results.

Iteration 2: G2 failed on speed

· 8 min read

Iteration 2 asks whether the iteration-1 models give a useful estimate on realistic data: noisy bearings, dropouts and a rate-limited ownship turn.

Verdict: G2 failed (agent review panel, round 2, no rejections).

  • Range passes for both models. Median range error at zig + 300 s on the σ = 1° slice is 16.5% [16.1, 17.1] instantaneous and 18.5% [17.7, 19.1] time-late, against a 20% threshold.
  • Course passes for the instantaneous model only: 9.46° [9.02, 9.90]. The time-late model fails at 10.57° [10.16, 11.04], against 10°.
  • Speed fails decisively for both: 2.09 kt [2.00, 2.18] and 2.38 kt [2.28, 2.48] at zig + 600 s, against 1 kt. Classical reaches 0.62 kt on the same scenarios.

Experiment of record: experiments/20261014-i2-g2/. Curves and slices: iteration-2 results.

Iteration 1: G1 passed (panel round 3) with a caveat

· 25 min read

Iteration 1 asks the question the whole project exists for: given simulated bearings and the ownship track, can a small neural network estimate target range better than a guess from the prior?

Verdict: G1 passed (panel round 3) with a caveat. Both model variants beat the naive baseline by more than 2× at the gate point, with the whole 95% interval below 0.5: instantaneous 0.419 [0.407, 0.431], time-late 0.450 [0.438, 0.464] (medians 16.1% and 17.3%; naive 38.4%, classical 4.6%). The shuffled-label model scores at naive. The machine verdict is inconclusive, because the time-late model is measurably worse than the instantaneous one (1.075 [1.046, 1.108] × at zig + 300 s) although it sees more data; the agent review panel passed G1 under a rule fixed before the deciding run (pre-registered rule 1, SRD §5) and recorded that as the caveat. Experiment of record: experiments/20261012-g1-r2/.

It took two attempts. In attempt 1 (experiments/20261011-g1-r2/) the time-late model was cut off by a wall-clock cap after 25 of its 60 epochs and failed (0.551 [0.538, 0.565]). One pre-registered continuation of that model, to early stopping at 50 epochs, gave the pass. Full curves and slices: iteration-1 results.

Iteration 0: G0 passes — the pipeline is verified

· 2 min read

The first gate asks a narrow question: can we trust the data before training anything on it?

Verdict: G0 pass (machine verdict; team review pending). Experiment: experiments/20261011-g0/.

  • 100,000 noise-free scenarios generated (seed 42); the 10,000-scenario test split was scored.
  • The least-squares solver recovered truth on 10,000 / 10,000 scenarios. Worst cases: 2.9×10⁻⁴ relative range, 0.016° course, 0.002 kt speed (tolerances 10⁻³, 0.1°, 0.01 kt). Median range error 1.9×10⁻⁷.
  • The naive baseline reproduced exactly across two independent fits.

Yardsticks for iteration 1​

On the same noise-free test set, the naive baseline's median range error is 40% (P90 139%), its median course error 90° and its gross-error rate 37%. Any model must beat these to pass G1.

On noisy data (σ 0.5–2°, up to 10% dropouts) the classical solver's median range error is about 5%, with P90 ≈ 43% and gross-error rate ≈ 9%. None of those gross errors were optimiser failures: starting from the true state never found a lower cost. The tail comes from the information in the bearings, not from a weak solver.

Caveat​

Passing G0 shows that generator and solver agree. On its own it would not catch an error shared by both; hand-computed geometry tests cover that separately.

Generation plus the G0 run took about 70 seconds on a laptop-class machine.