Iteration 2: G2 failed on speed

Iteration 2 asks whether the iteration-1 models give a useful estimate on realistic data: noisy bearings, dropouts and a rate-limited ownship turn.

Verdict: G2 failed (agent review panel, round 2, no rejections).

• Range passes for both models. Median range error at zig + 300 s on the σ = 1° slice is 16.5% [16.1, 17.1] instantaneous and 18.5% [17.7, 19.1] time-late, against a 20% threshold.
• Course passes for the instantaneous model only: 9.46° [9.02, 9.90]. The time-late model fails at 10.57° [10.16, 11.04], against 10°.
• Speed fails decisively for both: 2.09 kt [2.00, 2.18] and 2.38 kt [2.28, 2.48] at zig + 600 s, against 1 kt. Classical reaches 0.62 kt on the same scenarios.

Experiment of record: experiments/20261014-i2-g2/. Curves and slices: iteration-2 results.

What changed from iteration 1

• Noise and dropouts.
  • Bearing noise σ ∈ {1°, 2°}, half the scenarios each.
  • Dropouts drawn per sample, at 0–10% per scenario.
  • G2 is judged on the σ = 1° slice. The σ = 2° slice and the two dropout bands are reported.
• A rate-limited turn.
  • Ownship's zig is now flown at 0.5–3°/s instead of instantaneously.
  • The zig time is the end of the turn, so "zig + 300 s" means five minutes after ownship is steady on its new leg.
  • This changed the generator (0.2.0), so G0 was re-run and passed again (panel rounds 1–3, 20261013-g0-i2-r2).
• Provenance.
  • Every dataset header names the generator's commit.
  • Every model records the header of the data it was trained on.
  • tma-train refuses to resume on different features. tma-g2 refuses to score the test split before the phase decision is committed.
• Confidence intervals.
  • Every gated median carries a seeded bootstrap 95% interval.
  • A check whose interval contains its threshold is inconclusive, not a pass or fail.

Two training phases, one test score

The test split was protected by a two-phase rule, fixed before training (research R5).

Phase A ran the G1 configurations unchanged: 60 epochs, patience 3, one seed per model. On validation, the instantaneous model passed range (17.4%) and failed course (10.47°) and speed (2.24 kt), so phase A stalled. The decision record was committed before phase B started: decision.md.

Phase B made one change, the training budget. Every model continued from its phase-A weights to 150 epochs with patience 6. This was the user's choice over the pre-registered default (derived features), and the record says what it tests: whether undertraining explains the gap.

Phase B improved range (16.6%) and course (9.56°) on validation, but not speed (2.12 kt). It was carried by the pre-declared rule: its range error was no worse than phase A's by more than phase A's interval half-width.

The test split was then scored once, for phase B, with phase A beside it.

Results on test (σ = 1°)

Estimator	Range, zig+300 s	Course, zig+300 s	Speed, zig+600 s
G2 threshold	< 20%	< 10°	< 1 kt
lstm-instantaneous (B)	16.5% [16.1, 17.1]	9.46° [9.02, 9.90]	2.09 kt [2.00, 2.18]
bilstm-time-late (B)	18.5% [17.7, 19.1]	10.57° [10.16, 11.04]	2.38 kt [2.28, 2.48]
lstm-instantaneous (A, reported)	17.7%	10.45°	2.26 kt
bilstm-time-late (A, reported)	20.0%	11.24°	2.64 kt
classical	8.8%	3.42°	0.62 kt
naive	37.0%	89.9°	4.25 kt

n = 4,707 scenarios at zig + 300 s (93% of the slice) and 2,995 at zig + 600 s (59%).

The repeated G1 checks

These are judged on all rows.

Check	Result
instantaneous / naive < 0.5	pass: 0.472 [0.458, 0.484]
time-late / naive < 0.5	fail: 0.514 [0.500, 0.529], on the boundary
time-late not worse than instantaneous	fail, as at G1
both models beat their controls	pass
shuffled-label model at naive	pass: 1.015 ×

The time-late fail sits on a knife-edge. The interval's lower bound is 0.50035, and with other bootstrap seeds it falls between 0.4995 and 0.5006. It does not decide the gate.

Why speed fails

The panel ruled out the pipeline:

• Three reviewers re-predicted phase B independently and got exactly 2.09 kt and 2.38 kt.
• The model's mean speed is right: 11.58 kt predicted against 11.69 kt true.

The errors come from shrinkage towards the mean instead:

• The slope of predicted against true speed is about 0.57.
• Slow targets are over-estimated and fast ones under-estimated: errors are 2.6 kt at 0–5 kt and 2.9 kt at 15–20 kt, against 1.5 kt at 10–15 kt.

The information is there. A per-scenario Cramér–Rao bound on the actual bearings gives a median achievable speed error of 0.64 kt, and classical reaches 0.62 kt. The models are about 3.3 times the bound.

Three things point to how the models are trained, not to a ceiling of the approach:

• Speed gets little weight in the loss. All four loss terms are weighted equally, and speed contributes about 15% of the loss.
• Training stopped early. In phase B the training loss was still falling (LSTM 0.400 → 0.377 over its last 14 epochs). Validation loss moved by about ±3% from epoch to epoch, more than the per-epoch gain, so patience 6 ended the run.
• More epochs barely moved speed. The phase-B change improved range and course but not speed.

Where the models do well

Range bands at zig + 300 s, all noise levels:

True range	Instantaneous	Classical	Naive
2–5 km (n 1,438)	20.0%	5.3%	147%
5–10 km (n 3,486)	15.3%	9.5%	23.0%
10–20 km (n 3,728)	16.7%, gross 2.3%	18.5%, gross 24.7%	31.9%
20–30 km (n 534)	31.4%	28.7%	58.6%

Short ranges carry classical's advantage. Beyond 10 km the models match or beat it on range, and fail grossly far less often. The same holds at σ = 2°: the instantaneous model's range error moves only from 16.5% to 18.4%, while classical's doubles from 8.8% to 17.1%.

On held-out geometry, with initial range 20–30 km (outside training, reported only), the models beat classical on every median:

Estimator	Range	Course	Speed
Models	43.8–44.8%	about 36°	about 3.6 kt
Classical	48.8%	50.9°	9.2 kt

What the panel found

Five reviewers: TMA geometry, estimation and numerics, evaluation statistics, reproducibility, and red team. All five returned approve-with-findings in round 1, and none rejected.

Two major findings, both resolved:

• Phase A's predictions overwrote phase B's on disk. tma-g2 wrote both phases' prediction files under the same names. The verdict was unaffected, because it was scored in memory, but re-scoring from disk would have given phase A's numbers. Fixed: the other phase's files are now tagged.
• The models were stopped by noise, not converged. Accepted. The verdict is stated for this architecture, these inputs and this budget.

A gate-rule fix. The G2 code failed a check whose interval straddled its threshold whenever the point estimate was at or above it. The spec says that case is inconclusive. The code now follows the spec, and a check passes only if its point estimate is also below the threshold.

Re-gating the committed metrics with the new code gave identical checks, and the reproducibility reviewer re-ran the whole gate byte for byte.

Round 2 approved with only minor findings open. Record: review.md.

What G2 cannot tell us

• The speed point covers 59% of scenarios. Only windows that run at least ten minutes past the turn are scored at zig + 600 s, so the figure describes longer windows (decision 13). At zig + 300 s, where 93% are scored, speed is still 2.30 kt.
• Medians hide the range dependence. The range pass is not carried by short ranges, which are the models' weak band. But a single median mixes a 10–20 km band where the models beat classical with a 2–5 km band where they are four times worse.
• One seed per model (decision 11). The intervals cover test-set sampling, not training-to-training variation.
• The course pass is narrow. Its interval's upper bound is 9.90° against a 10° threshold.
• Targets never manoeuvre. G2 says nothing about tracking a target's own zig.
• Not done: scoring the G1 models on a turn-only set (reported only, not gated). It is deferred to iteration 2b.

Next: iteration 2b

The SRD allows one more round after a stalled G2 (decision 5). The leads, in research R6, R18 and R19:

1. Derived features. Bearing rate and the bearing-line intercept, already built (feature_set: derived), give the network quantities classical TMA works from.
2. Speed in the loss. Weight it per term, or scale its target to unit variance.
3. Per-leg steady-track fit. Within each ownship leg, fit tan B(t) = (x₀ + vₓt)/(y₀ + v_yt) to the bearings. The fitted shape constrains relative velocity without needing range, and the residual flags a target manoeuvre.
4. Budget and stopping. A smoothed stopping criterion or a longer run, since training loss was still falling.

Before 2b scores a gate, three tooling fixes are due:

• tma-g2 checks the plan's content digests itself;
• the decision commit must be an ancestor of the weights' commit;
• the same-rows check compares row identity, not just counts.