Phase 3: better than the tracker after a turn, worse when there is none
Phase 3 asked the question the earlier gates could not: does a learned estimator have an edge where the classical model breaks, when the target turns?
After a turn, yes. Five minutes after the turn ends, on the validation split:
| Measure | Network | Tuned IMM tracker | Ratio |
|---|---|---|---|
| Median range error | 10.5% | 13.6% | 0.77 |
| Median course error | 14.4° | 24.1° | 0.60 |
| Gross errors | 5.7 points fewer |
The IMM was the better of two tracking filters tuned on the same data.
On targets that never turn, no. Its course error is 11.1° and its speed error 1.60 kt, both past G2's limits (10°, 1 kt). The pre-registered rule therefore stopped the phase at validation. The test split was never scored.
Records: results and the validation record.
What was tested
- The data. The target makes one course change of 30–120° in 80% of scenarios (generator 0.4.0).
- The reference. We tuned two tracking filters on validation, a single Cartesian EKF and an IMM (constant velocity plus coordinated turn). We froze the better one: the IMM.
- The network. It is the iteration-4 LSTM design, unchanged, trained from scratch on the new data. Its decision record was committed before training. It forecast about a 30% chance of a pass, with range the likely failure.
What happened
- After a turn. The network beats the tracker on range, course, speed and gross errors. Its lead holds whether or not ownship manoeuvres after the turn. The network has learned something the tracker's model does not hold, for example that the target's speed is unchanged through a turn.
- On held course. It regresses. It reports speeds 1.2 kt too low on median, and its course error rises to 11° five minutes after the zig, which in this data is exactly when targets usually turn.
A network trained to minimise squared error reports an average when it cannot tell whether a turn has started. An averaged velocity is shorter, and its direction is hedged. This is the dilemma every tracking filter faces, between following a turn quickly and staying steady on a straight leg, now inside the network.
Reading it critically
- The forecast was wrong in an informative way. We expected range to be the obstacle. Range and course both cleared easily; the held-course control failed instead.
- The control is strict but fair. It asks the network not to regress below G2's absolute bar on the 20% of targets that hold course. A tracker that is tuned for turns pays a similar price: the IMM is 5.0° and 0.71 kt on those rows, against batch least squares' 1.2° and 0.31 kt.
- One validation look. This is not a test-split result, and the test split remains unseen. The margins on the failing checks are wide: both intervals lie wholly beyond the limits.
What next
SRD Annex B §B.5 fixed this branch in advance:
- publish the result;
- then decide separately whether a different design is a new project.
The new design could be an end-to-end tracker, a network that outputs "turned or not" rather than an average, or a hybrid that uses the network only after a detected turn. That decision is the owner's.