Skip to main content

Iteration 2c: G2 failed on speed; the project stops at G2

· 9 min read

Iteration 2c was the last attempt the SRD allows under G2. It combined the two levers that each moved speed on validation, a speed-weighted loss and twice the model capacity, and judged G2 on a third fresh test split. Range and course are the best of the project. Speed is not within reach.

Verdict: G2 failed (agent review panel, round 2, no rejections). Under SRD §7 the project stops at G2: no-go.

  • Range passes for both models: 13.5% [13.0, 13.9] instantaneous and 12.1% [11.7, 12.6] time-late at zig + 300 s on the σ = 1° slice, against 20%.
  • Course passes for both: 7.36° [7.05, 7.70] and 6.12° [5.87, 6.41], against 10°.
  • Speed fails for both: 1.47 kt [1.37, 1.52] and 1.53 kt [1.47, 1.61] at zig + 600 s, against 1 kt. Only 37% and 33% of scenarios are under 1 kt. Classical least squares reaches 0.61 kt on the same scenarios, at the Cramér–Rao bound.

Experiment of record: experiments/20261016-i2d-g2/. Curves and slices: iteration-2c results.

Iteration 2b: speed-weighted loss, G2 failed again

· 8 min read

Iteration 2b asks one question: does weighting speed in the loss get the iteration-2 models to the G2 speed threshold? It changes nothing else, and judges G2 on a fresh test split.

Verdict: G2 failed (agent review panel, round 2, no rejections).

  • Range passes for both models: 14.6% [14.1, 15.2] instantaneous and 15.8% [15.2, 16.2] time-late at zig + 300 s on the σ = 1° slice, against 20%.
  • Course passes for both: 8.22° [7.88, 8.54] and 8.66° [8.29, 9.03], against 10°. The time-late course failure of iteration 2 is gone.
  • Speed fails for both, by less: 1.77 kt [1.68, 1.83] and 1.77 kt [1.67, 1.86] at zig + 600 s, against 1 kt. The same weights before 2b, scored on the same scenarios, were at 2.02 kt and 2.31 kt. Classical reaches 0.60 kt.

Experiment of record: experiments/20261015-i2c-g2/. Curves and slices: iteration-2b results.

Iteration 2: G2 failed on speed

· 8 min read

Iteration 2 asks whether the iteration-1 models give a useful estimate on realistic data: noisy bearings, dropouts and a rate-limited ownship turn.

Verdict: G2 failed (agent review panel, round 2, no rejections).

  • Range passes for both models. Median range error at zig + 300 s on the σ = 1° slice is 16.5% [16.1, 17.1] instantaneous and 18.5% [17.7, 19.1] time-late, against a 20% threshold.
  • Course passes for the instantaneous model only: 9.46° [9.02, 9.90]. The time-late model fails at 10.57° [10.16, 11.04], against 10°.
  • Speed fails decisively for both: 2.09 kt [2.00, 2.18] and 2.38 kt [2.28, 2.48] at zig + 600 s, against 1 kt. Classical reaches 0.62 kt on the same scenarios.

Experiment of record: experiments/20261014-i2-g2/. Curves and slices: iteration-2 results.

Iteration 1: G1 passed (panel round 3) with a caveat

· 25 min read

Iteration 1 asks the question the whole project exists for: given simulated bearings and the ownship track, can a small neural network estimate target range better than a guess from the prior?

Verdict: G1 passed (panel round 3) with a caveat. Both model variants beat the naive baseline by more than 2× at the gate point, with the whole 95% interval below 0.5: instantaneous 0.419 [0.407, 0.431], time-late 0.450 [0.438, 0.464] (medians 16.1% and 17.3%; naive 38.4%, classical 4.6%). The shuffled-label model scores at naive. The machine verdict is inconclusive, because the time-late model is measurably worse than the instantaneous one (1.075 [1.046, 1.108] × at zig + 300 s) although it sees more data; the agent review panel passed G1 under a rule fixed before the deciding run (pre-registered rule 1, SRD §5) and recorded that as the caveat. Experiment of record: experiments/20261012-g1-r2/.

It took two attempts. In attempt 1 (experiments/20261011-g1-r2/) the time-late model was cut off by a wall-clock cap after 25 of its 60 epochs and failed (0.551 [0.538, 0.565]). One pre-registered continuation of that model, to early stopping at 50 epochs, gave the pass. Full curves and slices: iteration-1 results.

Iteration 0: G0 passes — the pipeline is verified

· 2 min read

The first gate asks a narrow question: can we trust the data before training anything on it?

Verdict: G0 pass (machine verdict; team review pending). Experiment: experiments/20261011-g0/.

  • 100,000 noise-free scenarios generated (seed 42); the 10,000-scenario test split was scored.
  • The least-squares solver recovered truth on 10,000 / 10,000 scenarios. Worst cases: 2.9×10⁻⁴ relative range, 0.016° course, 0.002 kt speed (tolerances 10⁻³, 0.1°, 0.01 kt). Median range error 1.9×10⁻⁷.
  • The naive baseline reproduced exactly across two independent fits.

Yardsticks for iteration 1​

On the same noise-free test set, the naive baseline's median range error is 40% (P90 139%), its median course error 90° and its gross-error rate 37%. Any model must beat these to pass G1.

On noisy data (σ 0.5–2°, up to 10% dropouts) the classical solver's median range error is about 5%, with P90 ≈ 43% and gross-error rate ≈ 9%. None of those gross errors were optimiser failures: starting from the true state never found a lower cost. The tail comes from the information in the bearings, not from a weak solver.

Caveat​

Passing G0 shows that generator and solver agree. On its own it would not catch an error shared by both; hand-computed geometry tests cover that separately.

Generation plus the G0 run took about 70 seconds on a laptop-class machine.