Results: phase 3 and 3b (manoeuvring targets, stopped at validation)
Question: after a target turn, can a network beat the best tuned classical tracker without getting worse on targets that hold course? This is gate G4 of SRD Annex B.
- The problem. Generator 0.4.0 gives the target one course change of 30–120° in 80% of scenarios (generator note).
- The reference. The better tuned-turn tracker on validation. That was the IMM: constant velocity plus coordinated turn, at 13.6% median range error five minutes after the turn, against 15.3% for the EKF (trackers note).
- The network. The iteration-4 LSTM configuration, trained from scratch on seed 208. It is the only model, and it was declared before training.
On seed 208's validation split (σ = 1°), the network beats the reference after the turn, but it fails the held-course control, so seed 209's test split was not scored (FR-007).
| Check | Result | |
|---|---|---|
| Range at manoeuvre + 5 min, network / IMM | 0.77 [0.74, 0.81] | better |
| Course at manoeuvre + 5 min, network / IMM | 0.60 [0.57, 0.64] | better |
| Gross errors at manoeuvre + 5 min, network − IMM | −5.7 pp | better |
| Held course: course at zig + 5 min, limit 10° | 11.1° [10.3, 11.9] | fail |
| Held course: speed at zig + 10 min, limit 1 kt | 1.60 kt [1.47, 1.72] | fail |
Records:
- the validation record;
- the decision;
- the research;
- the write-up.
Seed-208 validation split, σ = 1° (medians)
| Estimator | After a turn: range | course | speed | Held course: range | course | speed |
|---|---|---|---|---|---|---|
| Network | 10.5% | 14.4° | 1.57 kt | 7.6% | 11.1° | 1.60 kt |
| IMM (reference) | 13.6% | 24.1° | 2.34 kt | 5.2% | 5.0° | 0.71 kt |
| EKF | 15.3% | 24.4° | 3.04 kt | 7.7% | 6.5° | 1.82 kt |
| Batch least squares | 29.7% | 64.6° | 3.26 kt | 3.0% | 1.2° | 0.31 kt |
| Naive | 41.0% | 89.1° | 4.29 kt | 40.5% | 85.7° | 4.20 kt |
| Iteration-4 LSTM (never shown a turn) | 26.1% | 53.2° | 2.85 kt | 6.4% | 4.7° | 0.87 kt |
- When each column is measured. After a turn means five minutes after the turn ends (n = 4,074). Held course means zig + 5 min for range and course (n = 1,042) and zig + 10 min for speed (n = 913).
- The network's lead holds across turn timing. It leads whether the target turned before ownship's second zig (10.7% against 12.5%), during it (10.2% against 14.4%), or after it, when ownship makes no later manoeuvre (10.8% against 14.1%).
Why the control failed
On targets that hold course, the network:
- reports speeds that are too low: median signed error −1.16 kt, with 74% of its estimates slow;
- gets its course error worse, not better, as time passes: 6° at zig + 2 min, 11° from zig + 5 min on. The IMM's falls to 4°.
In this data, targets usually turn 2–8 minutes after the zig. A network trained to minimise squared velocity error reports an average when it cannot yet tell whether a turn has begun. An averaged velocity is shorter, and its direction is hedged. The network has learned the tracker's dilemma, trading straight-leg accuracy for turn response, rather than escaping it.
Phase 3b: two hypotheses and a turn flag
Phase 3b (spec 011, decision 27) kept everything else and changed only the output. The network reports two estimates, one for "held course" and one for "turned", plus P(turned), trained on the true turn label. It reports the more probable estimate, never an average.
| Check | Phase 3b | Phase 3a | |
|---|---|---|---|
| Range at manoeuvre + 5 min, network / IMM | 0.76 [0.72, 0.80] | 0.77 | better |
| Course at manoeuvre + 5 min, network / IMM | 0.60 [0.57, 0.64] | 0.60 | better |
| Gross errors at manoeuvre + 5 min, network − IMM | −6.4 pp | −5.7 pp | better |
| Held course: course at zig + 5 min, limit 10° | 6.1° [5.5, 6.9] | 11.1° | pass |
| Held course: speed at zig + 10 min, limit 1 kt | 1.15 kt [1.05, 1.25] | 1.60 kt | fail |
Seed 210's test split was not scored. This was the last phase-3 attempt under Annex B §B.8. Records:
- the validation record;
- the decision;
- the write-up.
The turn flag (σ = 1°, p > 0.5):
- Detection. It detects a turn a median 70 s after it starts (P90 140 s) and misses 0.5% by manoeuvre + 5 min.
- False alarms. On held course it calls "turned" on 29–31% of rows at zig + 5 and 10 min. The AUC at zig + 5 min is 0.79.
Why speed still failed. On held course, rows that report "held" have a median speed error of 1.00 kt with almost no bias. The 29% of false alarms report the turned estimate, at 1.80 kt and biased slow. So:
- false alarms push the median over the limit;
- even a perfect flag would leave this design at about 1 kt, against the IMM's 0.71 kt.
The cost of hard switching shows in the tail: held-course course P90 rose from 34° to 51°.