Iteration 1: G1 passed (panel round 3) with a caveat
Iteration 1 asks the question the whole project exists for: given simulated bearings and the ownship track, can a small neural network estimate target range better than a guess from the prior?
Verdict: G1 passed (panel round 3) with a caveat. Both model variants beat the naive baseline
by more than 2× at the gate point, with the whole 95% interval below 0.5: instantaneous 0.419
[0.407, 0.431], time-late 0.450 [0.438, 0.464] (medians 16.1% and 17.3%; naive 38.4%, classical
4.6%). The shuffled-label model scores at naive. The machine verdict is inconclusive, because
the time-late model is measurably worse than the instantaneous one (1.075 [1.046, 1.108] × at
zig + 300 s) although it sees more data; the agent review panel passed G1 under a rule fixed
before the deciding run (pre-registered rule 1, SRD §5) and recorded that as the caveat.
Experiment of record:
experiments/20261012-g1-r2/.
It took two attempts. In attempt 1 (experiments/20261011-g1-r2/) the time-late model was cut
off by a wall-clock cap after 25 of its 60 epochs and failed (0.551 [0.538, 0.565]). One
pre-registered continuation of that model, to early stopping at 50 epochs, gave the pass. Full
curves and slices: iteration-1 results.
The question
G1 (SRD §5) passes if, for both model variants, the median relative range error 5 minutes after the ownship zig is below half of the naive baseline's, on the same test scenarios. The naive baseline predicts the training-set mean range, course and speed, so a model that beats it by 2× has learned something about the geometry, not just the prior.
The two variants answer different questions. The instantaneous model is causal: its estimate at a given time uses only bearings up to that time, as a real-time tracker would. The time-late model sees the whole recorded window, including bearings after the gate point, and so is a post-hoc reconstruction.
Each ratio is judged with a paired bootstrap 95% interval: pass if the whole interval is below 0.5, fail if it is wholly at or above, otherwise inconclusive. Further checks guard the result: a model trained on shuffled labels must score like naive (leakage), each recurrent model must beat a non-recurrent control, and time-late must not be worse than instantaneous. How these combine is on the evaluation page.
What was trained
Four small models over the same masked 240-row input window (see models),
plus one leakage run, trained on GitHub-hosted runners (4 cores, no GPU). Every model had
max_epochs 60, patience 3 on validation loss and a 5.5 h wall-clock cap (max_wall_s 19,800;
2 h for the shuffled run). Epochs are counted from 0, so "60 (best 58)" means the 59th epoch gave
the kept weights.
| Model | Role | Layers | Epochs (best) | Stop reason | Wall time | s / epoch | Runner CPU |
|---|---|---|---|---|---|---|---|
lstm-instantaneous | instantaneous variant (causal) | LSTM 96 | 60 (58) | max_epochs | 4 h 36 min | 276 | AMD EPYC 9V45 |
bilstm-time-late | time-late variant (whole window) | BiLSTM 96 | 25 (23) | wall_clock | 5 h 23 min | 775 | AMD EPYC 9V74 |
cnn-causal-control | control for instantaneous | 3 × causal Conv1D 64, k = 15 | 21 (17) | early stopping | 53 min | 152 | AMD EPYC 9V74 |
mlp-control | control for time-late | MLP 256-256 over the flattened window | 24 (20) | early stopping | 7 min | 17 | AMD EPYC 7763 |
lstm-instantaneous-shuffled | leakage check | LSTM 96 on shuffled labels | 8 (4) | early stopping | 1 h 22 min | 617 | Intel Xeon Platinum 8370C |
Data: 100,000 scenarios (seed 101), σ = 0.5°, no dropouts, initial range 2–20 km log-uniform; 80,000 train, 10,000 validation, 10,000 test, plus 10,000 held-out scenarios at 20–30 km.
The bilstm-time-late row is attempt 1. Under the protocol below it was continued once, from the
epoch-23 weights with no wall-clock cap, and stopped early after 50 epochs (best 46); the other
four models are unchanged. One configuration per model; nothing was tuned. In attempt 1 the
time-late configuration was started twice with identical settings (one job cancelled, one
completed).
Every run (FR-009)
- 38131616871,
train/i1, 11 Oct: all five models. Instantaneous, CNN, MLP and shuffled completed; these four are the models of record. The time-late job was cancelled after ≈ 5 h 15 min; the g1 job then stopped at once (time-late: training has not finished). Test split not scored. - 38152208243,
train/i1, 11 Oct: time-late only, attempt 1 (5 h 24 min job, AMD EPYC 9V74); the other four reused. Its g1 job scored the test split but could not push its results branch; informational only (same models as the attempt-1 re-score). - 38153046011 and
38154023864,
train/ckpt-smoke: checkpoint smoke tests (MLP); g1 stopped at model collection. Test split not scored. - 38165442543,
train/i1: cancelled 25 s after it was queued; no job ran. - 38178916962: re-score
(
evaluate-g1.yml) of the committed models, failed before any model was scored. The test features, regenerated at schema 0.5.0, carry a different header digest from the 0.4.0 training data, so prediction refused them ("normalisation differs"). Fixed by PR #27 (accepts a schema-version-only relabel). - 38182021648: re-score with PR #27,
attempt 1 of record,
experiments/20261011-g1-r2/. - 38208186337,
train/i1-continue, 12 Oct: the pre-registered time-late continuation, over two run attempts (jobs listed under attempt 2 below). Its final g1 job scored the test split intoexperiments/20261012-g1/, the training run's own folder (train/i1 code, no bootstrap; itsverdict.jsonreadsfail): informational only. - 38275763335: re-score of all five
committed models on main code (
evaluate-g1.yml), attempt 2 of record,experiments/20261012-g1-r2/.
Attempt 1: results at the gate point (zig + 300 s, test split)
Experiment 20261011-g1-r2, re-scored from the committed weights by run 38182021648. n = 9,784
scenarios; 216 excluded because their window ends before the gate point; the same scenarios for
every estimator.
| Estimator | Range median | Range P90 | Course median | Course P90 | Speed median | Speed P90 | Gross-error rate |
|---|---|---|---|---|---|---|---|
| naive | 38.4% | 161.5% | 90.0° | 163.0° | 4.30 kt | 7.67 kt | 35.7% |
| classical (per-timestep least squares) | 4.6% | 29.2% | 1.9° | 23.7° | 0.64 kt | 5.19 kt | 5.7% |
lstm-instantaneous | 16.1% | 39.6% | 9.4° | 41.7° | 2.16 kt | 5.59 kt | 5.4% |
bilstm-time-late | 21.2% | 49.6% | 11.5° | 48.4° | 2.84 kt | 6.28 kt | 9.9% |
cnn-causal-control | 29.3% | 69.7% | 23.0° | 78.4° | 3.81 kt | 7.33 kt | 20.1% |
mlp-control | 26.3% | 75.7% | 16.8° | 65.6° | 3.39 kt | 7.05 kt | 18.7% |
lstm-instantaneous-shuffled | 38.6% | 122.6% | 82.9° | 159.5° | 4.33 kt | 7.66 kt | 34.9% |
G1 threshold: 0.5 × naive median = 19.2%.
Classical non-converged rate at the gate point: 2.0% (non-converged rows count as +∞). Classical is far ahead of every model (a median 3.5× lower than the instantaneous model's); G1 asks only whether the models beat the prior, and closing that gap is G3's question.
Intervals
Paired bootstrap of the ratio model / naive median range error at zig + 300 s (2,000 resamples of the 9,784 test scenarios, seed 20261011). A G1 check passes only if the whole 95% interval is below 0.5.
| Variant | Model median | Naive median | Classical median | Ratio to naive | 95% interval | G1 check |
|---|---|---|---|---|---|---|
| instantaneous | 16.1% | 38.4% | 4.6% (0.119) | 0.419 | [0.407, 0.431] | pass |
| time-late | 21.2% | 38.4% | 4.6% (0.119) | 0.551 | [0.538, 0.565] | fail |
The classical ratio (in brackets) is a point value; no interval was computed for it. The interval reflects which test scenarios were drawn, not how a re-trained model would score (one training seed per model).
Checks
Machine verdict: fail, because g1-time-late and time-late-not-worse failed.
| Check | Result | Detail (median range error; naive and classical beside each) |
|---|---|---|
g1-instantaneous | pass | 16.1% vs limit 19.2%; ratio 0.419 [0.407, 0.431] (naive 38.4%, classical 4.6%) |
g1-time-late | fail | 21.2% vs limit 19.2%; ratio 0.551 [0.538, 0.565] (naive 38.4%, classical 4.6%) |
time-late-not-worse | fail | time-late worse at 20 of 20 gated curve points (every 60 s to 20 min); 3 later points skipped (n < 500) |
instantaneous-beats-control | pass | zig+300s: 16.1% vs CNN 29.3%; final: 16.4% vs 29.3% (naive 38.4% / 40.9%, classical 4.6% / 2.4%) |
time-late-beats-control | pass | zig+300s: 21.2% vs MLP 26.3%; final: 24.3% vs 28.2% (naive 38.4% / 40.9%, classical 4.6% / 2.4%) |
shuffled-labels-at-naive | pass | 1.005 × naive (38.6% vs 38.4%; allowed 0.9–1.1) |
Error against time since the zig. Naive is flat (38–49% median throughout). Classical falls fast: 22.6% one minute after the zig, 4.6% at five minutes, 1.6% at fifteen. The instantaneous model falls from 24.5% at one minute to 16.1% at five and levels off near 15% at ten to fifteen minutes. The time-late model has the same shape but sits above it everywhere (26.0%, 21.2%, 19.0%), although it sees more data at every point. Rounds 1–2 read this as the signature of an undertrained model, not of a pipeline that hides the future from it (see the panel's checks below); round 3 withdrew that explanation for the continued model, which trained to early stopping and is still worse (see the caveat). Interactive curves are on the results page.
By range band and zig angle
Median range error at zig + 300 s, with each model's ratio to naive in brackets. Bands are by the mean range over the window, so a few test scenarios fall outside 2–20 km. Slices with n < 200 are not interpreted.
| Slice | n | naive | classical | instantaneous | time-late |
|---|---|---|---|---|---|
| range 0–2 km | 132 | 603.7% | 0.7% (0.001) | 29.9% (0.050) | 53.5% (0.089) |
| range 2–5 km | 1,640 | 150.8% | 2.0% (0.013) | 17.2% (0.114) | 26.2% (0.174) |
| range 5–10 km | 3,705 | 23.2% | 3.8% (0.166) | 13.6% (0.588) | 18.6% (0.802) |
| range 10–20 km | 3,741 | 32.5% | 7.2% (0.222) | 16.2% (0.498) | 20.0% (0.615) |
| range 20–30 km | 557 | 58.5% | 10.4% (0.178) | 28.7% (0.490) | 33.0% (0.564) |
| range 30+ km | 9 | 71.8% | 8.2% (0.114) | 27.0% (0.377) | 37.9% (0.529) |
| zig 30–50° | 3,232 | 38.7% | 6.4% (0.166) | 17.1% (0.441) | 21.6% (0.557) |
| zig 50–70° | 3,286 | 37.9% | 4.3% (0.113) | 16.0% (0.423) | 21.0% (0.554) |
| zig 70–90° | 3,266 | 38.7% | 3.5% (0.091) | 15.3% (0.395) | 21.1% (0.545) |
The per-band ratios mostly measure naive, not the models: naive guesses the middle of the prior, so it is very wrong at short range and nearly right at 5–10 km. In absolute terms the models show the prior's U-shape (best at 5–10 km, worse at both ends), whereas classical's error grows steadily with range, as the geometry predicts (lower bearing rate at long range). Larger zigs help classical (6.4% → 3.5%) and, weakly, the instantaneous model (17.1% → 15.3%); the time-late model is flat across zig angle (21.0–21.6%).
Held-out geometry (20–30 km initial range, reported, not gated)
10,000 scenarios outside the training range; at zig + 300 s, n = 9,769 (231 excluded).
| Estimator | Range median | Range P90 | Gross-error rate | Ratio to naive |
|---|---|---|---|---|
| naive | 59.7% | 70.2% | 77.6% | 1 |
| classical | 18.8% | ∞ (12.6% non-converged) | 23.9% | 0.315 |
lstm-instantaneous | 42.4% | 57.6% | 28.5% | 0.710 |
bilstm-time-late | 45.9% | 59.9% | 37.1% | 0.768 |
Outside the training range the models lose most of their advantage over naive (ratios 0.71 and 0.77, against 0.42 and 0.55 in range), while classical keeps a median of 19% at the cost of more non-converged solutions. The panel read this as the models being held towards the training prior rather than extrapolating the geometry. G2 is to include a range-extrapolation check.
Attempt 1: why the time-late model was undertrained
All models shared the same budget in time, not in epochs: a 5.5 h wall-clock cap per model, inside GitHub's 6 h job limit. The bidirectional LSTM runs the window in both directions and has twice the instantaneous LSTM's parameters (82,948 vs 41,476); one epoch took about 775 s against 276 s. The cap therefore stopped it after 25 epochs while the instantaneous model ran all 60. Its validation loss was still falling (0.592, 0.565, 0.571 over the last three epochs; best at epoch 23).
The panel compared like with like. On the post-zig validation rows (the rows that are scored), the time-late model at epoch 23 had a loss of 0.406; the instantaneous model at the same epoch had 0.478 and needed until epoch 31 to reach 0.405. At equal training, time-late was ahead.
What the panel found in attempt 1 (rounds 1–2)
Five independent agent reviewers (TMA geometry, estimation and numerics, evaluation statistics,
reproducibility, red team) reviewed attempt 1 over two rounds; the full record, with every finding
and response, is in
experiments/20261011-g1-r2/review.md.
In plain words:
- The pipeline is sound. Padding does not leak into the bidirectional pass (padded and unpadded windows give identical outputs); predictions line up with their scenarios; the causal models ignore bearings after the time they estimate; the time-late model does use later bearings (cutting them off makes it worse, 21.2% → 27.2%).
- The instantaneous pass is real. No truth reaches the inputs or the normalisation, naive is not artificially weak, every comparison uses the same 9,784 scenarios, and the shuffled-label model trained down to the "know nothing" level and scores like naive.
- Reproducible. A reviewer regenerated the data and re-scored the committed weights locally: the verdict file came out byte-identical, naive and classical metrics byte-identical, model metrics within 1.3 × 10⁻⁵ relative.
- The time-late fail is a budget effect (one reviewer rejected in round 1 until this was recognised). Calling it "ML not extracting the signal" would overstate it. The first result is recorded as: G1 not passed as run; the instantaneous variant met the criterion; the time-late variant failed under a 25-epoch wall-clock budget and is plausibly undertrained; no evidence that the time-late signal is absent.
- But no rule allowed a re-run. Turning "undertrained" into "try again" needed an explicit amendment, decided after seeing the result and written down before any new training. That is what the user decided (below), and round 2 approved the protocol with all five reviewers at approve-with-findings.
Smaller findings, reported rather than fixed now: pre-zig rows, never scored, carry 64% of the time-late model's training loss; model manifests lack a git commit field (provenance is recorded in the review instead); training budgets should be set in epochs, not wall-clock time, because runner CPUs vary; the shuffled model's median course error (82.9°) is a little below chance (≈ 90°), a weak bearing-as-course prior rather than leakage.
The decision and the continuation protocol
The user decided on 12 October, after the first G1 result and before any further training, to
continue the time-late model once, to the same 60-epoch budget the instantaneous model had. SRD
§5 was amended to allow exactly this, and the protocol was fixed in
review.md
before the run started:
- One change only: no wall-clock cap. Training resumes from the committed epoch-23 weights (digest checked) and runs to epoch 60 or early stopping (patience 3). Epoch 24 of the first run is dropped.
- Unchanged: data, splits, training seed, architecture, loss, learning rate, batch size. The other four models are reused untouched; the instantaneous model is not retrained.
- Optimizer: Adam's state was never saved in attempt 1, so it restarts once, at epoch 24, and epochs 24–26 do not count towards patience. From then on Adam's state is saved with every checkpoint and restored if a job is interrupted, so a resumed run matches an uninterrupted one (tested: identical weights).
- Scoring: the verdict of record comes from re-scoring all five committed models with
evaluate-g1.ymlon main; the test split is not touched until training completes. - Outcomes, fixed in advance: both intervals wholly below 0.5 and shuffled labels at naive →
G1 pass, even if
time-late-not-worsestill fails (the record then says "time-late not shown better than instantaneous"). A ratio fail of the continued model is final: no further continuation, seed or architecture change, and the project stops per SRD §5. - Record: attempt 1 (0.551) stays recorded as a valid attempt. Every job of the continuation is listed with its run id, epochs covered and runner CPU.
Attempt 2: the continued time-late model
Experiment 20261012-g1-r2, re-scored by run
38275763335 on main code (commit
bb4525b). The four reused models are byte-identical to attempt 1; their metrics differ from
attempt 1's re-score by at most 8.4 × 10⁻⁶ relative (float32 inference on AVX2, then AVX-512), and
naive and classical are identical.
Training. Run 38208186337, branch train/i1-continue, commit 7c1c579. Before training, the
seed step checked the epoch-23 weights (digest 2c5f8784…) and a configuration that differed from
attempt 1 only in max_wall_s (19,800 → 86,400).
| Attempt | Job | Epochs | Runner CPU | Outcome |
|---|---|---|---|---|
| 1 | 114676289268 | 24–41 | AMD EPYC 7763 | 345-min step timeout; last checkpoint at epoch 41 |
| 1 | 114785366608 | – | – | g1 stopped before scoring; test split untouched |
| 2 | 114789157629 | 42–49 | Intel Xeon Platinum 8573C | early stopping, best epoch 46 |
| 2 | 114832540726 | – | – | g1 succeeded; informational folder 20261012-g1/ |
The first job's log reads "resuming bilstm-time-late after epoch 23 (optimizer state restarts;
patience not counted for epochs 24–26)", the second's "resuming bilstm-time-late after epoch 41
(optimizer state restored)". Epochs took about 1,100 s on the AMD runner and 870 s on the Intel one. Epochs 0–23 ran in attempt
1 (run 38152208243, AMD EPYC 9V74), so training crossed three CPU families and a retrain will not
reproduce the weights byte for byte. The committed weights (c52b628c…) are the artefact of record;
re-scoring them is deterministic, and a reviewer reproduced this experiment folder byte for byte.
Extracted log lines are in
experiments/20261012-g1/training-logs/.
Training stopped after 50 epochs (best 46), stop reason early_stopping, against the
instantaneous model's 60. Validation loss at the best epoch was 0.441 (attempt 1's best: 0.565 at
epoch 23).
The epoch-48 spike. Train and validation loss jumped at epoch 48 (validation 0.447 → 0.535, then 0.448 at epoch 49). It is a training instability, not a resume artefact: the instantaneous model, never resumed, had one at epoch 40. The scored weights are from epoch 46, before it. The spike contributed to the patience stop at epoch 49, so the "equal epoch budget" was in effect 50 against 60; whether epoch 48 would otherwise have improved is unknown. Gradient clipping is an iteration-2 candidate.
At the gate point (zig + 300 s, test split)
n = 9,784 scenarios (216 excluded), the same for every estimator, as in attempt 1.
| Estimator | Range median | Range P90 | Course median | Course P90 | Speed median | Speed P90 | Gross-error rate |
|---|---|---|---|---|---|---|---|
| naive | 38.4% | 161.5% | 90.0° | 163.0° | 4.30 kt | 7.67 kt | 35.7% |
| classical | 4.6% | 29.2% | 1.9° | 23.7° | 0.64 kt | 5.19 kt | 5.7% |
lstm-instantaneous | 16.1% | 39.6% | 9.4° | 41.7° | 2.16 kt | 5.59 kt | 5.4% |
bilstm-time-late (continued) | 17.3% | 43.5% | 9.6° | 41.1° | 2.39 kt | 5.71 kt | 7.3% |
cnn-causal-control | 29.3% | 69.7% | 23.0° | 78.4° | 3.81 kt | 7.33 kt | 20.1% |
mlp-control | 26.3% | 75.7% | 16.8° | 65.6° | 3.39 kt | 7.05 kt | 18.7% |
lstm-instantaneous-shuffled | 38.6% | 122.6% | 82.9° | 159.5° | 4.33 kt | 7.66 kt | 34.9% |
G1 threshold: 0.5 × naive median = 19.2%. Classical non-converged rate at the gate point: 2.0%. Classical is still far ahead: its median is 3.5× lower than the instantaneous model's and 3.8× lower than the time-late model's.
Intervals, both attempts
Paired bootstrap of model / naive median range error at zig + 300 s (2,000 resamples of the 9,784 test scenarios, seed 20261011). A G1 check passes only if the whole 95% interval is below 0.5. Medians are of range error; classical's ratio to naive is 0.119, a point value (no interval computed).
| Attempt, variant | Model | Naive | Classical | Ratio | 95% interval | G1 check |
|---|---|---|---|---|---|---|
| 1, instantaneous | 16.1% | 38.4% | 4.6% | 0.419 | [0.407, 0.431] | pass |
| 1, time-late | 21.2% | 38.4% | 4.6% | 0.551 | [0.538, 0.565] | fail |
| 2, instantaneous | 16.1% | 38.4% | 4.6% | 0.419 | [0.407, 0.431] | pass |
| 2, time-late | 17.3% | 38.4% | 4.6% | 0.450 | [0.438, 0.464] | pass |
The time-late upper bound is 0.036 below 0.5, about 5 bootstrap standard errors; its point estimate is about 7 below. The statistics reviewer's independent bootstrap agreed (time-late [0.4377, 0.4649], instantaneous [0.4074, 0.4313]). The intervals reflect which test scenarios were drawn, not how a re-trained model would score: each variant was trained once.
Checks
Machine verdict: inconclusive, from time-late-not-worse (tiered rule 2). verdict.json is
left exactly as the code wrote it; the pass is the panel's application of rule 1.
| Check | Result | Detail (median range error; naive and classical beside each) |
|---|---|---|
g1-instantaneous | pass | 16.1% vs limit 19.2%; ratio 0.419 [0.407, 0.431] (naive 38.4%, classical 4.6%) |
g1-time-late | pass | 17.3% vs limit 19.2%; ratio 0.450 [0.438, 0.464] (naive 38.4%, classical 4.6%) |
time-late-not-worse | fail | more than 5% worse than instantaneous at 14 of 20 gated points; 3 later points skipped (n < 500) |
instantaneous-beats-control | pass | zig+300s: 16.1% vs CNN 29.3%; final: 16.4% vs 29.3% (naive 38.4% / 40.9%, classical 4.6% / 2.4%) |
time-late-beats-control | pass | zig+300s: 17.3% vs MLP 26.3%; final: 18.5% vs 28.2% (naive 38.4% / 40.9%, classical 4.6% / 2.4%) |
shuffled-labels-at-naive | pass | 1.005 × naive (38.6% vs 38.4%; allowed 0.9–1.1) |
Error against time since the zig (median range error, test split):
| Time since zig | n | naive | classical | instantaneous | time-late (continued) | time-late / instantaneous |
|---|---|---|---|---|---|---|
| 60 s | 10,000 | 40.3% | 22.6% | 24.5% | 24.1% | 0.984 |
| 300 s | 9,784 | 38.4% | 4.6% | 16.1% | 17.3% | 1.075 |
| 600 s | 6,512 | 39.1% | 2.3% | 14.9% | 15.7% | 1.054 |
| 900 s | 3,116 | 43.4% | 1.6% | 14.5% | 15.5% | 1.069 |
| 1200 s | 516 | 48.9% | 1.4% | 17.0% | 16.5% | 0.973 |
The continuation closed most of the gap to the instantaneous model (attempt 1's time-late model: 26.0%, 21.2% and 19.0% at one, five and ten minutes) but did not cross it.
By range band and zig angle
Median range error at zig + 300 s, ratio to naive in brackets. Slices with n < 200 are not interpreted.
| Slice | n | naive | classical | instantaneous | time-late (continued) |
|---|---|---|---|---|---|
| range 0–2 km | 132 | 603.7% | 0.7% (0.001) | 29.9% (0.050) | 46.2% (0.077) |
| range 2–5 km | 1,640 | 150.8% | 2.0% (0.013) | 17.2% (0.114) | 21.2% (0.141) |
| range 5–10 km | 3,705 | 23.2% | 3.8% (0.166) | 13.6% (0.588) | 15.8% (0.683) |
| range 10–20 km | 3,741 | 32.5% | 7.2% (0.222) | 16.2% (0.498) | 15.6% (0.480) |
| range 20–30 km | 557 | 58.5% | 10.4% (0.178) | 28.7% (0.490) | 27.5% (0.471) |
| range 30+ km | 9 | 71.8% | 8.2% (0.114) | 27.0% (0.377) | 30.3% (0.422) |
| zig 30–50° | 3,232 | 38.7% | 6.4% (0.166) | 17.1% (0.441) | 18.3% (0.473) |
| zig 50–70° | 3,286 | 37.9% | 4.3% (0.113) | 16.0% (0.423) | 17.0% (0.449) |
| zig 70–90° | 3,266 | 38.7% | 3.5% (0.091) | 15.3% (0.395) | 16.6% (0.429) |
The time-late model's deficit is at 2–10 km; at 10–30 km it is level with, or slightly ahead of, the instantaneous model. Both models still follow the training prior's U-shape across range bands, whereas classical's error grows steadily with range.
Held-out geometry (20–30 km initial range, reported, not gated)
n = 9,769 at zig + 300 s (231 excluded).
| Estimator | Range median | Range P90 | Gross-error rate | Ratio to naive |
|---|---|---|---|---|
| naive | 59.7% | 70.2% | 77.6% | 1 |
| classical | 18.8% | ∞ (12.6% non-converged) | 23.9% | 0.315 |
lstm-instantaneous | 42.4% | 57.6% | 28.5% | 0.710 |
bilstm-time-late (continued) | 43.9% | 58.5% | 31.1% | 0.735 |
As in attempt 1, both models are held towards the training prior outside the training range.
What the panel decided (round 3)
The same five specialist reviewers judged the continuation. The full record is in
experiments/20261012-g1-r2/review.md.
| Reviewer | Attempt 1 (rounds 1–2) | Round 3 | Round 3 re-invitation |
|---|---|---|---|
| TMA geometry | approve-with-findings | approve-with-findings | approve-with-findings |
| Estimation and numerics | reject → approve-with-findings (round 2) | approve-with-findings | approve |
| Evaluation statistics | approve-with-findings | approve-with-findings | approve-with-findings |
| Reproducibility | approve-with-findings | approve-with-findings | approve |
| Red team | approve-with-findings | approve-with-findings | approve-with-findings |
All five said the evidence supports G1 pass under pre-registered rule 1. The reproducibility
reviewer re-scored the committed weights locally and got verdict.json, bootstrap.json,
metrics, curves and held-out results byte-identical; the red team found no protocol deviation,
leakage or selection between attempts (one continuation, one re-score).
One major finding (M1, red team; geometry agreed). The SRD §5 G1 row still said a time-late consistency failure makes G1 inconclusive, and did not carry rule 1; as the SRD wins conflicts, the pass needed it there. The row was amended to carry rule 1 as the user decided it on 12 October before the run. Rule 1 was committed at 05:29:44Z and published in PR #28 at 05:30:16Z, 17 s after the run was triggered (05:29:59Z) but before any continuation epoch finished (about 05:49Z) and about 8 h before the test split was scored (13:25Z). The round-1 explanation of the consistency failure (undertraining) is withdrawn for the continued model.
Minor findings, answered in the record: the caveat now states the gap rather than only "not shown
better"; the epoch-48 spike (above); a cross-process resume test for the bidirectional layer,
which a reviewer ran (byte-identical weights) and which iteration 2 adds to the suite; the per-job
record and both CPUs, now in the review; re-scored manifests lack cpu_model, which
evaluate-g1.yml will record in iteration 2. The re-invitation corrected the wording of the gap,
the margin and the spike. No reviewer rejects and no blocker or major finding is open; the
verdict's decided_by names round 3.
Outcome: G1 passes, with a caveat
G1 passes under pre-registered rule 1 (SRD §5 as amended 2026-10-12). Both ratio intervals lie wholly below 0.5, instantaneous 0.419 [0.407, 0.431] and time-late (continued to 50 epochs) 0.450 [0.438, 0.464], and the shuffled-label model sits at naive. A small network does extract range information from bearings alone, in real time and in reconstruction, though at the gate point it is still about 3.5× (instantaneous) and 3.8× (time-late) worse than the classical solver.
Caveat, recorded with the pass:
- Time-late not shown better than instantaneous. It is measurably worse: at zig+300s, 0.173 vs 0.161, a ratio of 1.075 [1.046, 1.108]. Its median is higher at 18 of the 20 gated points, by more than 5% (up to 9%) at 14 of them (120–480, 600 and 720–1020 s). It is lower only at zig+60s (0.984 [0.964, 1.006]) and zig+1200s (0.973 [0.864, 1.126], n 516); neither difference is significant.
- Not explained by undertraining. The continued model trained to early stopping.
- The BiLSTM under-uses the future. Its error profile is shaped like a causal estimator's. Masking and the backward pass are verified correct, and truncating the future does change outputs, but weakly.
- One training draw per variant. Training-seed variance is unquantified.
What G1 cannot tell us
- The targets move at constant velocity and the ownship zig is instantaneous; nothing here says how the models cope with manoeuvres or gradual turns.
- σ = 0.5° and no dropouts only; noise and gaps are iteration 2's question (G2).
- The time-late model's estimate at zig + 300 s uses bearings recorded after that time. It is a post-hoc reconstruction, not a real-time solution.
- The interval covers test sampling, not training variation: each model was trained once, and weights differ across runner CPU families.
- The time-late model was trained in three stretches on three CPU families with one optimizer restart; the instantaneous model in one. The protocol equalised the epoch budget, not the training path, and early stopping ended the time-late model at 50 epochs against 60.
- The shuffled-label test rules out leakage through labels and the training harness; truth hidden in the inputs is ruled out by a separate counterfactual test.
The full list, with mitigations, is in
specs/002-iteration-1-signal/g1-limits.md.
Next: iteration 2
Iteration 2 (spec 003) runs the same models on σ = 1–2° with 10% dropouts, judged at G2: median range error below 20% and course below 10° five minutes after the zig, speed below 1 kt ten minutes after, at σ = 1°. At σ = 0.5° the instantaneous model is at 16.1% and 9.4° (zig + 300 s) and 1.83 kt (zig + 600 s), against classical's 4.6%, 1.9° and 0.31 kt, so speed is furthest off. Leads from this gate:
- Pre-zig rows carry 64% of the time-late model's training loss but are never scored.
- Gradient clipping, against spikes like the one at epoch 48.
- Both models still follow the training prior by range band, and are held to the training prior on the held-out 20–30 km set; G2 is to include a range-extrapolation check.
- Why the bidirectional model under-uses the future, before time-late is relied on.
- Engineering: training budgets in epochs rather than wall-clock time; the runner CPU recorded in re-scored manifests; a cross-process bidirectional resume test; a git commit in model manifests.