Results: iteration 2 (G2)
Question: is the estimate useful on realistic data? Iteration 2 adds bearing noise (σ = 1° or 2°), dropouts of up to 10%, and a rate-limited ownship turn. G2 is judged on the σ = 1° slice and has three thresholds:
- median range error below 20% at zig + 300 s;
- median course error below 10° at zig + 300 s;
- median speed error below 1 kt at zig + 600 s.
"Zig" is the end of the turn. Only 59% of the σ = 1° scenarios run long enough to reach zig + 600 s, so the speed figure covers those longer windows only.
Each threshold applies to both the instantaneous and time-late models, and each is judged on a bootstrap confidence interval:
- pass if the whole interval is below the threshold;
- fail if the whole interval is at or above it;
- inconclusive otherwise.
The six G1 checks are repeated. The evaluation page defines every metric.
Training ran in two phases on validation data only:
- Phase A: the G1 configurations at 60 epochs. It stalled.
- Phase B: continued the same models to 150 epochs with patience 6. It was carried by a rule fixed before it ran.
The test split was then scored once, for phase B, with phase A reported beside it.
Speed fails decisively for both models. The instantaneous model is at 2.09 kt [2.00, 2.18] and the time-late model at 2.38 kt [2.28, 2.48], against 1 kt. Classical reaches 0.62 kt on the same scenarios, which is at the Cramér–Rao bound. So the speed information is in the bearings, and the models are not extracting it.
Range passes for both models, and course passes for the instantaneous model (9.46°, upper bound 9.90°).
The verdict holds for this architecture, these inputs and this training budget. It does not show that a network cannot reach 1 kt: training loss was still falling when patience stopped phase B.
Experiment 20261014-i2-g2: machine verdict G2 fail (agent review panel, round 2 (no rejections; no open blocker or major findings): G2 FAIL (SRD §5); speed fails for both variants (CI wholly above 1 kt), time-late course and g1-time-late also fail; see review.md).
| Check | Result | Detail |
|---|---|---|
g2-instantaneous-range | pass | instantaneous lstm-instantaneous median range error 0.1650 vs threshold 0.2 on sigma=1deg at zig+300s; 95% CI [0.1606, 0.1710] (bootstrap, 2000 resamples, n 4707); model n 4707, excluded 341 (93.2% of the slice reach zig+300s; decision 13); naive 0.3698 (n 4707, excluded 341), classical 0.0876 (n 4707, excluded 341) |
g2-instantaneous-course | pass | instantaneous lstm-instantaneous median course error 9.4574° vs threshold 10° on sigma=1deg at zig+300s; 95% CI [9.0199, 9.9017] (bootstrap, 2000 resamples, n 4707); model n 4707, excluded 341 (93.2% of the slice reach zig+300s; decision 13); naive 89.9179 (n 4707, excluded 341), classical 3.4203 (n 4707, excluded 341) |
g2-instantaneous-speed | fail | instantaneous lstm-instantaneous median speed error 2.0889 kt vs threshold 1 kt on sigma=1deg at zig+600s; 95% CI [1.9997, 2.1804] (bootstrap, 2000 resamples, n 2995); model n 2995, excluded 2053 (59.3% of the slice reach zig+600s; decision 13); naive 4.2452 (n 2995, excluded 2053), classical 0.6212 (n 2995, excluded 2053) |
g2-time-late-range | pass | time-late bilstm-time-late median range error 0.1848 vs threshold 0.2 on sigma=1deg at zig+300s; 95% CI [0.1773, 0.1906] (bootstrap, 2000 resamples, n 4707); model n 4707, excluded 341 (93.2% of the slice reach zig+300s; decision 13); naive 0.3698 (n 4707, excluded 341), classical 0.0876 (n 4707, excluded 341) |
g2-time-late-course | fail | time-late bilstm-time-late median course error 10.5748° vs threshold 10° on sigma=1deg at zig+300s; 95% CI [10.1646, 11.0408] (bootstrap, 2000 resamples, n 4707); model n 4707, excluded 341 (93.2% of the slice reach zig+300s; decision 13); naive 89.9179 (n 4707, excluded 341), classical 3.4203 (n 4707, excluded 341) |
g2-time-late-speed | fail | time-late bilstm-time-late median speed error 2.3809 kt vs threshold 1 kt on sigma=1deg at zig+600s; 95% CI [2.2760, 2.4764] (bootstrap, 2000 resamples, n 2995); model n 2995, excluded 2053 (59.3% of the slice reach zig+600s; decision 13); naive 4.2452 (n 2995, excluded 2053), classical 0.6212 (n 2995, excluded 2053) |
g1-instantaneous | pass | lstm-instantaneous median range error 0.1749 vs limit 0.1854 (0.5 × naive 0.3708) at zig+300s; n 9315 (naive 9315), excluded 685 (naive 685); model/naive 0.472, 95% CI [0.458, 0.484] (paired bootstrap, 2000 resamples, n 9315) |
g1-time-late | fail | bilstm-time-late median range error 0.1905 vs limit 0.1854 (0.5 × naive 0.3708) at zig+300s; n 9315 (naive 9315), excluded 685 (naive 685); model/naive 0.514, 95% CI [0.500, 0.529] (paired bootstrap, 2000 resamples, n 9315) |
time-late-not-worse | fail | 14/19 points worse: t=180s 0.2114 > 0.1967; t=240s 0.1991 > 0.1851; t=300s 0.1905 > 0.1749; t=360s 0.1860 > 0.1670; t=420s 0.1789 > 0.1662 (4 later points with n < 500 reported, not gated) |
instantaneous-beats-control | pass | lstm-instantaneous vs cnn-causal-control median range error — zig+300s: 0.1749 vs 0.2770, final: 0.1720 vs 0.2830 |
time-late-beats-control | pass | bilstm-time-late vs mlp-control median range error — zig+300s: 0.1905 vs 0.2608, final: 0.1944 vs 0.2768 |
shuffled-labels-at-naive | pass | lstm-instantaneous-shuffled median range error 0.3762 = 1.015 × naive 0.3708 (allowed 1 ± 0.1) at zig+300s; course error median 70.3° vs naive 90.3° (reported, not gated) |
Model / naive at the G1 point
Median range error of each model divided by naive's, 5 minutes after the zig, with a paired bootstrap 95% interval. G1 passes only if the whole interval is below 0.5; an interval that straddles 0.5 is inconclusive.
| Model | Ratio | 95% interval | n |
|---|---|---|---|
classical | 0.330 | [0.316, 0.344] | 9315 |
bilstm-time-late | 0.514 | [0.500, 0.529] | 9315 |
cnn-causal-control | 0.747 | [0.732, 0.766] | 9315 |
lstm-instantaneous | 0.472 | [0.458, 0.484] | 9315 |
lstm-instantaneous-shuffled | 1.015 | [1.001, 1.029] | 9315 |
mlp-control | 0.703 | [0.686, 0.720] | 9315 |
Error against time since the zig
Dashed lines are the naive and classical baselines; test split, all scenarios.
- ━ naive
- ━ classical
- ━ bilstm-time-late
- ━ cnn-causal-control
- ━ lstm-instantaneous
- ━ lstm-instantaneous-shuffled
- ━ mlp-control
Metrics by noise slice and point
Brackets: bootstrap 95% interval of the median (σ and overall slices at zig + 300 s and zig + 600 s).
| Estimator | n | Median range error | P90 range error | Median course error | Median speed error | Gross-error rate |
|---|---|---|---|---|---|---|
naive | 4707 | 37.0% [35.9%, 38.0%] | 147.5% | 89.9° [87.1°, 92.4°] | 4.22 kt [4.10 kt, 4.31 kt] | 33.7% |
classical | 4707 | 8.8% [8.4%, 9.2%] | 59.9% | 3.4° [3.2°, 3.7°] | 1.23 kt [1.16 kt, 1.29 kt] | 12.0% |
bilstm-time-late | 4707 | 18.5% [17.7%, 19.1%] | 46.3% | 10.6° [10.2°, 11.0°] | 2.52 kt [2.45 kt, 2.60 kt] | 8.2% |
cnn-causal-control | 4707 | 27.7% [26.8%, 28.5%] | 79.8% | 22.4° [21.7°, 23.2°] | 3.82 kt [3.70 kt, 3.93 kt] | 19.6% |
lstm-instantaneous | 4707 | 16.5% [16.1%, 17.1%] | 42.4% | 9.5° [9.0°, 9.9°] | 2.30 kt [2.24 kt, 2.41 kt] | 6.4% |
lstm-instantaneous-shuffled | 4707 | 37.5% [36.4%, 38.5%] | 120.6% | 70.4° [68.2°, 73.2°] | 4.23 kt [4.10 kt, 4.36 kt] | 33.9% |
mlp-control | 4707 | 25.7% [24.9%, 26.4%] | 65.7% | 15.8° [15.2°, 16.6°] | 3.28 kt [3.16 kt, 3.40 kt] | 16.9% |
By slice
Median range error by slice:
| Estimator | 0-5pct | 5-10pct |
|---|---|---|
naive | 37.4% n 4684 | 36.7% n 4631 |
classical | 12.2% n 4684 | 12.3% n 4631 |
bilstm-time-late | 19.3% n 4684 | 18.8% n 4631 |
cnn-causal-control | 28.0% n 4684 | 27.5% n 4631 |
lstm-instantaneous | 17.3% n 4684 | 17.6% n 4631 |
lstm-instantaneous-shuffled | 37.6% n 4684 | 37.6% n 4631 |
mlp-control | 25.7% n 4684 | 26.3% n 4631 |
Held-out geometry (reported, not gated)
| Estimator | n | Median range error | P90 range error | Median course error | Median speed error | Gross-error rate |
|---|---|---|---|---|---|---|
naive | 9303 | 59.6% | 69.9% | 90.0° | 4.24 kt | 76.2% |
classical | 9303 | 48.8% | ∞ | 50.9° | 9.21 kt | 49.3% |
bilstm-time-late | 9303 | 44.8% | 58.9% | 36.0° | 3.71 kt | 34.0% |
cnn-causal-control | 9303 | 52.3% | 63.8% | 48.8° | 4.09 kt | 58.2% |
lstm-instantaneous | 9303 | 43.8% | 59.9% | 36.6° | 3.62 kt | 32.9% |
lstm-instantaneous-shuffled | 9303 | 63.9% | 73.3% | 84.3° | 4.23 kt | 83.0% |
mlp-control | 9303 | 49.2% | 63.7% | 44.8° | 4.00 kt | 47.8% |