Skip to main content

Iteration 5: a hybrid passes G3; the network adds robustness, not accuracy

· 4 min read

G2 left the models usable but about 3 × worse than the classical least-squares solver. G3 asks for parity: within 1.2 × classical on every metric, and a lower gross-error rate.

Iteration 5 trained nothing. It combined the two estimators: classical's estimate, except where classical fails or the network disagrees by more than 50% on range, where the network's is used.

G3 passes for that hybrid on a fresh test set of 20,000 scenarios (seed 207):

  • Parity. Every median and 90th percentile of range, course and speed is within 0.97–1.02 × classical.
  • Gross errors. Five minutes after the zig they fall from 1.19% to 0.57% (−0.62 pp [−0.82, −0.42]).
  • Validation agreed. It forecast 1.25% → 0.55% on seed 205.

Records: experiments/20261028-i5-g3/. Figures: iteration-5 results.

Iteration 4: velocity outputs pass G2, still 3 × classical

· 4 min read

Iteration 4 changed the models, not the problem. It kept iteration 3's three-leg engagements, made the models output a velocity (vx, vy) instead of a course and a speed, and added the derived features (bearing rate, its validity flag, the bearing-line intercept).

G2 passes. The seed-205 test split was inconclusive at 0.981 kt [0.940, 1.031]. The rules allow one re-score on a larger fresh-seed test set, and on seed 206, with 100,000 scenarios and the same frozen models, every check passes.

  • Speed passes. The instantaneous LSTM reaches 0.939 kt [0.926, 0.951] at zig + 600 s on the σ = 1° slice, against 1 kt. The BiLSTM reaches 0.740 kt.
  • Range and course pass easily: 7.3% and 4.7° for the LSTM, 5.3% and 3.1° for the BiLSTM.
  • Better than iteration 3 on the same rows. Range, course and speed are 16–27% better.
  • Still about 3 × classical. Classical reaches 0.32 kt on the same rows. The LSTM is 2.1 × classical on range, 3.5 × on course and 2.9 × on speed.

Records: experiments/20261025-i4-g2-rescore/ and experiments/20261024-i4-g2/. Figures: iteration-4 results.

Iteration 3: a second zig helps everyone, classical most

· 4 min read

Iteration 3 changed the problem, not the models. After the owner's realism check in the viewer, ownship now zigs twice, and both turns cross the line of sight. The 2c models were trained from scratch on the new engagements. Speed improved by almost half, but not enough: the iteration stopped at validation under its pre-registered rule, and the test split was not scored.

  • Speed fails the rule. The instantaneous LSTM reaches 1.10 kt [1.06, 1.16] at zig + 600 s on the σ = 1° slice. The rule needed an interval reaching below 1 kt. The forecast, recorded before training, was 1.1–1.3 kt.
  • The BiLSTM gets closest of any model so far: 1.02 kt [0.98, 1.05]. It is reported, not gated.
  • Range and course pass easily: 8.8% and 6.2° for the LSTM, 7.2% and 4.4° for the BiLSTM.
  • Classical gains more. On the same scenarios it reaches 0.34 kt, so the LSTM is now 3.3 × classical on speed, against 2.4 × in iteration 2c.

Record: experiments/20261021-i3a-val/. Figures: iteration-3 results.

The engagement viewer: single cases behind the G2 numbers

· 4 min read

The gates judged the models on medians over thousands of scenarios. The new engagement viewer shows one scenario at a time: the plan, both platforms' course and speed, the bearings, and each gate's carried models beside the classical solver and the naive baseline. It is the first step of SRD Annex A. Its first use is the owner's realism check of the generator, which comes before any change to the training data.

Iteration 2c: G2 failed on speed; the project stops at G2

· 9 min read

Iteration 2c was the last attempt the SRD allows under G2. It combined the two levers that each moved speed on validation, a speed-weighted loss and twice the model capacity, and judged G2 on a third fresh test split. Range and course are the best of the project. Speed is not within reach.

Verdict: G2 failed (agent review panel, round 2, no rejections). Under SRD §7 the project stops at G2: no-go.

  • Range passes for both models: 13.5% [13.0, 13.9] instantaneous and 12.1% [11.7, 12.6] time-late at zig + 300 s on the σ = 1° slice, against 20%.
  • Course passes for both: 7.36° [7.05, 7.70] and 6.12° [5.87, 6.41], against 10°.
  • Speed fails for both: 1.47 kt [1.37, 1.52] and 1.53 kt [1.47, 1.61] at zig + 600 s, against 1 kt. Only 37% and 33% of scenarios are under 1 kt. Classical least squares reaches 0.61 kt on the same scenarios, at the Cramér–Rao bound.

Experiment of record: experiments/20261016-i2d-g2/. Curves and slices: iteration-2c results.

Iteration 2b: speed-weighted loss, G2 failed again

· 8 min read

Iteration 2b asks one question: does weighting speed in the loss get the iteration-2 models to the G2 speed threshold? It changes nothing else, and judges G2 on a fresh test split.

Verdict: G2 failed (agent review panel, round 2, no rejections).

  • Range passes for both models: 14.6% [14.1, 15.2] instantaneous and 15.8% [15.2, 16.2] time-late at zig + 300 s on the σ = 1° slice, against 20%.
  • Course passes for both: 8.22° [7.88, 8.54] and 8.66° [8.29, 9.03], against 10°. The time-late course failure of iteration 2 is gone.
  • Speed fails for both, by less: 1.77 kt [1.68, 1.83] and 1.77 kt [1.67, 1.86] at zig + 600 s, against 1 kt. The same weights before 2b, scored on the same scenarios, were at 2.02 kt and 2.31 kt. Classical reaches 0.60 kt.

Experiment of record: experiments/20261015-i2c-g2/. Curves and slices: iteration-2b results.

Iteration 2: G2 failed on speed

· 8 min read

Iteration 2 asks whether the iteration-1 models give a useful estimate on realistic data: noisy bearings, dropouts and a rate-limited ownship turn.

Verdict: G2 failed (agent review panel, round 2, no rejections).

  • Range passes for both models. Median range error at zig + 300 s on the σ = 1° slice is 16.5% [16.1, 17.1] instantaneous and 18.5% [17.7, 19.1] time-late, against a 20% threshold.
  • Course passes for the instantaneous model only: 9.46° [9.02, 9.90]. The time-late model fails at 10.57° [10.16, 11.04], against 10°.
  • Speed fails decisively for both: 2.09 kt [2.00, 2.18] and 2.38 kt [2.28, 2.48] at zig + 600 s, against 1 kt. Classical reaches 0.62 kt on the same scenarios.

Experiment of record: experiments/20261014-i2-g2/. Curves and slices: iteration-2 results.

Iteration 1: G1 passed (panel round 3) with a caveat

· 25 min read

Iteration 1 asks the question the whole project exists for: given simulated bearings and the ownship track, can a small neural network estimate target range better than a guess from the prior?

Verdict: G1 passed (panel round 3) with a caveat. Both model variants beat the naive baseline by more than 2× at the gate point, with the whole 95% interval below 0.5: instantaneous 0.419 [0.407, 0.431], time-late 0.450 [0.438, 0.464] (medians 16.1% and 17.3%; naive 38.4%, classical 4.6%). The shuffled-label model scores at naive. The machine verdict is inconclusive, because the time-late model is measurably worse than the instantaneous one (1.075 [1.046, 1.108] × at zig + 300 s) although it sees more data; the agent review panel passed G1 under a rule fixed before the deciding run (pre-registered rule 1, SRD §5) and recorded that as the caveat. Experiment of record: experiments/20261012-g1-r2/.

It took two attempts. In attempt 1 (experiments/20261011-g1-r2/) the time-late model was cut off by a wall-clock cap after 25 of its 60 epochs and failed (0.551 [0.538, 0.565]). One pre-registered continuation of that model, to early stopping at 50 epochs, gave the pass. Full curves and slices: iteration-1 results.

Iteration 0: G0 passes — the pipeline is verified

· 2 min read

The first gate asks a narrow question: can we trust the data before training anything on it?

Verdict: G0 pass (machine verdict; team review pending). Experiment: experiments/20261011-g0/.

  • 100,000 noise-free scenarios generated (seed 42); the 10,000-scenario test split was scored.
  • The least-squares solver recovered truth on 10,000 / 10,000 scenarios. Worst cases: 2.9×10⁻⁴ relative range, 0.016° course, 0.002 kt speed (tolerances 10⁻³, 0.1°, 0.01 kt). Median range error 1.9×10⁻⁷.
  • The naive baseline reproduced exactly across two independent fits.

Yardsticks for iteration 1​

On the same noise-free test set, the naive baseline's median range error is 40% (P90 139%), its median course error 90° and its gross-error rate 37%. Any model must beat these to pass G1.

On noisy data (σ 0.5–2°, up to 10% dropouts) the classical solver's median range error is about 5%, with P90 ≈ 43% and gross-error rate ≈ 9%. None of those gross errors were optimiser failures: starting from the true state never found a lower cost. The tail comes from the information in the bearings, not from a weak solver.

Caveat​

Passing G0 shows that generator and solver agree. On its own it would not catch an error shared by both; hand-computed geometry tests cover that separately.

Generation plus the G0 run took about 70 seconds on a laptop-class machine.