Skip to main content

Phase 7: realistic bearing error — the frozen fused tracker still beats the frozen IMM, but T2 alone is more robust

· 3 min read

So far every tracker in this project has seen only white bearing noise. Real bearings also carry correlated error and a slowly drifting bias. Phase 7 asks one question: does the fused tracker's G5 advantage over the reference IMM survive that error, with nothing re-trained or re-tuned?

The setup​

The owner chose three things:

  • frozen trackers;
  • an error model made of a correlated term (sd 1.0°, 30 s) and a drifting bias (sd 0.5°, 20 min). This setting is double the one first proposed. At σ = 1° the total bearing error goes from 1.0° to 1.5°;
  • G5's rules, unchanged.

The generator adds the error from its own random stream, so each scenario can be regenerated without it. The two versions have identical geometry and identical white noise, which makes a paired clean comparison possible.

Result: G5 passes (seed 218, 40,000 scenarios)​

Fused / IMMRatio (97.5% CI)Limit
Overall range0.766 [0.755, 0.778]< 1
Overall course0.907 [0.897, 0.918]< 1
Settled range0.774< 1.1
Settled course0.875< 1.1
Settled speed0.907< 1.1

Validation on seed 217 had already shown much the same: 0.777 for range and 0.922 for course. Five independent reviewers agreed the verdict is right, and one reproduced it byte for byte.

What the pass does not show​

The absolute errors are large: the fused tracker's 90th-percentile course error is 106°. The panel narrowed the claim.

Median error, all G5 rowsRangeCourseSpeed
Fused tracker13.2%27°2.6 kt
IMM (reference)17.3%30°2.9 kt
T2 alone12.1%26°2.9 kt
EKF16.8%31°2.4 kt
Batch least squares17.6%47°2.4 kt
Naive40%90°3.9 kt
  • T2 alone beats the fused tracker on range, by 1.10 × [1.08, 1.11] overall. Its error grows least under realistic error (1.47 × its clean value, against the IMM's 1.82 ×). The IMM's range ends up biased about 7% short under the drifting bias, while T2 stays nearly unbiased. The fusion setting, frozen from clean data, plausibly gives T2 too little weight; k was not re-swept. Fusing keeps one coherent track state, but under this error it is not the best range estimator.
  • On straight legs, classical methods still win. On settled rows the EKF beats the fused tracker on course (10.6° against 13.5°) and on speed, and batch least squares does too (8.1°, 1.8 kt). Before any manoeuvre, batch least squares is best of all: 7.3% range and 2.2° course.
  • The reference was frozen, not adapted. Neither filter models correlated or bias error. An IMM with an inflated measurement noise or a bias state might close the gap; this phase does not test that.
  • Every tracker is much worse. Median range error rises 1.5–1.8 × overall and about 2 × on settled rows.
  • Not measured. Filter consistency (NIS, NEES) and covariance health (non-PD steps) under correlated error.
  • Scope. The result covers one error model at one severity, σ = 1° rows and simulated data.

The one-line version​

Under one setting of realistic bearing error, the frozen fused tracker keeps its G5 advantage over the frozen IMM, which was not adapted to this error. The learned range (T2) is likely what makes it robust (a hypothesis), and on its own T2 is more robust still. The classical filters remain better on straight legs.

Record: experiments/20261108-p7-g5/ (verdict, baselines, clean comparison, panel review); validation experiments/20261108-p7-val/; decision experiments/20261108-p7-decision/; G0 experiments/20261108-g0-p7-r1/; spec 019.

Phase 6 re-score: G7 passes, with coverage within tolerance on average but not calibrated

· 3 min read

G7 on seed 215 was inconclusive: the fused tracker's 50% range interval covered 48.3%, and its confidence interval straddled the 48% floor. The rules allow one re-score on a larger fresh seed. The owner chose to run it. Decision 34 fixed the details before the data existed:

  • What stayed the same: the tracker, the reference, the frozen models, the multipliers and the rules.
  • What changed: a fresh seed, 216, and a test split of 110,000 scenarios instead of 40,000. That is large enough to resolve a coverage of about 48.3% against the floor.
  • The forecast: about an even chance of passing.

Result: pass​

Coverage on seed 216 (184,667 G5 rows)50%90%
Range (log)48.6% [48.3, 48.9]89.9%
Course50.5%90.3%
Speed49.4%89.9%

All nine gated checks pass. The fused tracker's 90% intervals are narrower than the IMM's: 0.73 × for range, 0.94 × for course and 0.89 × for speed. This was the one re-score, so the verdict is final.

What the pass does and does not show​

Five independent reviewers agreed that the pass is correct. One recomputed every value with code of its own, and another re-ran the gate. All five also agreed on how far the claim goes.

  • Range intervals run about 1.5 points short. At 50%, the two seeds together give about 48.5% [48.3, 48.8]. That is inside the ±2-point tolerance but not centred on 50%. The pass rests on the tolerance. The reference IMM, fitted on the same validation data, lands on 50%. So reusing validation data does not explain the fused tracker's shortfall.

  • The intervals cover within tolerance only on the pooled mix of rows. Range at 50% covers:

    RowsCoverage
    Settled, before any manoeuvre (quiet legs)67%
    Settled, after a manoeuvre39%
    At zig + 5 min61%
    At zig + 20 min40%

    Seed 215 showed the same pattern within a point.

  • Course and speed intervals are often uninformative.

    • 17% of course 90% intervals span the whole circle, and the median course 90% interval is 188° wide. The course width ratio of 0.94 therefore compares two mostly vague intervals.
    • 27% of speed 90% intervals reach below zero.
  • This was a second look. Over both seeds, the chance of a false pass is about 5% at most, not 2.5%. A smaller re-score would probably have been inconclusive again.

  • The pass judges these frozen multipliers. The 0.3-point margin is smaller than the multipliers' own sampling error. So the pass does not show that the recalibration procedure works in general.

Phase 6 in one line​

On average over the G5 rows, the fused tracker's recalibrated intervals cover within the tolerance, with range about 1.5 points short, and they are tighter than the IMM's. Within a regime they do not: the intervals are too wide on quiet legs and too narrow after a manoeuvre, and its course intervals are often vacuous. Conditional calibration, for example a multiplier that depends on time since the last manoeuvre, would be a new design. The owner decides on the follow-on.

Record: experiments/20261106-p6v-g7/ (verdict, regimes, panel review); decision experiments/20261106-p6v-rescore-decision/; spec 018.

Phase 6: the fused tracker's uncertainty, recalibrated, is inconclusive at G7

· 3 min read

C.6's second follow-on asks whether the tracker that passed G5 can say how sure it is. The fused tracker is the frozen IMM with T2's range fused in.

The owner chose two things:

  • Calibrate the fused tracker's stated uncertainty, without training.
  • The test: calibrated and sharper. Its 50% and 90% intervals for range, course and speed must cover within ±2 points of nominal, and be narrower than the reference IMM's after the same recalibration.

What was done​

  • Stated sds. Both trackers now write them, from their covariance by the delta method. They are badly miscalibrated as stated: typical errors are a fraction of the stated sd, with heavy tails.
  • Recalibration, post hoc. One pair of multipliers per output, fitted on validation, scales the sd into 50% and 90% intervals. A two-fold check on validation passed, and decision 33 froze the multipliers before seed 215 was generated for scoring.

Result: inconclusive​

Coverage on seed 215 (G5 rows)50%90%
Range (log)48.3% [47.8, 48.8]89.8%
Course50.5%90.2%
Speed49.3%89.6%

The fused tracker's intervals are narrower than the IMM's: 0.73 × for range, 0.95 × for course and 0.89 × for speed. Eight of the nine checks pass. Range at 50% falls just short: its interval straddles the 48% floor.

What the panel added​

Five independent reviewers agreed the verdict is right, and reproduced it byte for byte. They also narrowed what can be claimed:

  • The intervals are calibrated only on the pooled mix of rows. On quiet straight legs before any manoeuvre they are far too wide: range 50% covers 67%. After a manoeuvre they are too narrow: 38%. One multiplier cannot fix that shape.
  • Course and speed intervals are often uninformative. 17% of course 90% intervals span the whole circle, and 27% of speed 90% intervals reach below zero.
  • The shortfall looks systematic, not noise. The likely cause is that T2's training epoch, the fusion settings and the multipliers were all chosen on the same validation split.
  • The rules were the agent's. The owner did not get to review them before the decision record.

What next​

The rules allow one re-score on a larger fresh seed. It would probably land near 48.3% again, so a pass would mean the range coverage just reaches the floor, not that the tracker is well calibrated. The owner decides whether to run it.

Record: experiments/20261106-p6u-g7/; spec 018.

Phase 5: frozen models do not beat a CUSUM at detecting manoeuvres

· 3 min read

G5 passed in phases 4b and 4c. SRD Annex C §C.6 therefore lists incremental follow-ons, and the first is manoeuvre detection, judged against a CUSUM on the reference tracker's innovations.

The owner chose two things for this attempt:

  • No training. The detector is built from models that already exist: the frozen recurrent tracker T2, and the reference IMM, which now uses decision 32's guarded initialisation.
  • The test. Detection delay, with every detector's threshold set for the same false-alarm rate: one alarm per 2 h of quiet tracking.

What was compared​

  • The baseline. A CUSUM on the IMM's innovations, tuned on validation from 12 declared settings to favour it.
    • Clipping. The innovations are heavy-tailed: a few huge values otherwise hold an alarm for hours. So both the variance kind and the mean-shift kind are clipped.
    • The winner. The two-sided mean-shift CUSUM, since a turn biases the bearings to one side.
  • The learned detectors. Two candidates, declared in advance:
    • T2's course disagreement with the IMM;
    • T2's own course change over the last minute.

Result: it stopped at validation​

At 0.5 false alarms per hour, on 7,479 course changes in seed-211 validation:

Detected within 5 min
CUSUM (mean shift)28.2%
T2's course change over 1 min19.2%
T2 against the IMM5.9%
  • The learned detector is not carried. It is 9 points behind, with interval [−10.2, −7.7] pp, so it was not taken to a test split, and fresh seed 215 was never generated.
  • The delay check could not be judged. Every detector misses more than half the course changes at this rate, so the median delays are infinite. That rule was the agent's, written before the data; any later attempt should declare a delay statistic defined at its operating point.

Reported, not used: a crossover​

The same detectors were also calibrated at looser rates:

False alarms per hour0.5125
CUSUM28.2%38.0%47.8%62.6%
T2's course change19.2%39.0%58.1%71.7%

From about 1 false alarm per hour, T2's course change catches more manoeuvres in time, though about 20–40 s later among those both catch. Moving the operating point after seeing this would choose the rule from the result, so it stays a reported observation.

What next​

The owner decides. Annex C's other follow-ons remain:

  • calibration of the learned uncertainty;
  • realistic bearing error.

Record: experiments/20261105-p5-research/validation.md; spec 017.

Phase 4c: one track, a tracker that listens to a network's range

· 3 min read

Phase 4b's hybrid passed G5, but it was two estimators stitched together: T2's range beside the IMM's velocity. Phase 4c keeps one filter. The frozen IMM takes the frozen network T2's range as an extra measurement, weighted by T2's own stated uncertainty and inflated four-fold, once a minute. Nothing was trained.

It passed G5 on a fresh 40,000-scenario test split (seed 214), decided by the agent review panel:

Fused tracker / IMM, 97.5% intervalLimitResult
Range, all gate rows< 10.82 [0.81, 0.84]
Course, all gate rows< 10.95 [0.94, 0.96]
Range, settled rows< 1.10.86 [0.84, 0.89]
Course, settled rows< 1.10.95 [0.93, 0.97]
Speed, settled rows< 1.10.90 [0.87, 0.92]

Records: the gate record and the panel review.

Phase 4: an end-to-end learned tracker wins overall, loses on straight legs

· 4 min read

Phase 4 asked whether a learned tracker, trained end to end, could beat a tuned IMM tracker on targets that manoeuvre up to three times. Two trackers were trained from scratch and compared with the frozen reference on the same validation split:

  • T1, a learned-gain filter: Kalman-style physics with a gain computed by a small recurrent network;
  • T2, a pure recurrent tracker (2 × 128 LSTM) that also states its own uncertainty.

Neither reached the test. The pre-registered rule needs every gated check to pass on validation.

Median error, ratio to the IMMLimitT2T1
Range, all gate rows< 10.852.18
Course, all gate rows< 10.841.32
Range, settled rows< 1.10.972.80
Course, settled rows< 1.11.442.12
Speed, settled rows< 1.11.662.30

"Settled" rows are at least five minutes after the target's last manoeuvre. Both of T2's failures have their whole 97.5% interval above the limit, so the fresh test split (seed 212) was never scored.

Record: the validation record.

Phase 4b: a hybrid passes G5 — a learned range with a tuned tracker's velocity

· 4 min read

Phase 4's recurrent tracker T2 beat a tuned IMM tracker overall but lost on straight legs, so it did not reach the test. The owner chose a follow-on: a hybrid of the two frozen estimators, judged by G5 unchanged. The hybrid takes T2's range always, and T2's course and speed only where the two estimators' courses differ by more than 45°. Elsewhere it takes the IMM's.

It passed, on a fresh 40,000-scenario test split (seed 213), decided by the agent review panel in two rounds:

Hybrid / IMM, 97.5% intervalLimitResult
Range, all gate rows< 10.86 [0.84, 0.88]
Course, all gate rows< 10.90 [0.89, 0.92]
Range, settled rows< 1.11.02 [0.99, 1.06]
Course, settled rows< 1.10.98 [0.97, 0.99]
Speed, settled rows< 1.10.99 [0.99, 1.00]

It also makes 3.4 percentage points fewer gross range errors than the IMM.

Records: the gate record and the panel review.

Phase 3b: two hypotheses fix the course, not the speed

· 4 min read

Phase 3 found a network that beats a tuned IMM tracker after a target turns, but that averages "turned" and "not turned" on targets that hold course. Phase 3b gave the network a way out. It outputs two estimates and the probability that the target has turned, and reports the more probable estimate.

It half worked. On the same validation split:

MeasurePhase 3bPhase 3aIMM trackerLimit
After a turn: median range error10.3%10.5%13.6%below the IMM
After a turn: median course error14.5°14.4°24.1°below the IMM
Held course: course error6.1°11.1°5.0°10°
Held course: speed error1.15 kt1.60 kt0.71 kt1 kt

The speed interval, [1.05, 1.25] kt, lies wholly above the limit. The pre-registered rule therefore stopped phase 3b at validation, and its test split was never scored. Under SRD Annex B, this was the last phase-3 attempt.

Records: results and the validation record.

Phase 3: better than the tracker after a turn, worse when there is none

· 3 min read

Phase 3 asked the question the earlier gates could not: does a learned estimator have an edge where the classical model breaks, when the target turns?

After a turn, yes. Five minutes after the turn ends, on the validation split:

MeasureNetworkTuned IMM trackerRatio
Median range error10.5%13.6%0.77
Median course error14.4°24.1°0.60
Gross errors5.7 points fewer

The IMM was the better of two tracking filters tuned on the same data.

On targets that never turn, no. Its course error is 11.1° and its speed error 1.60 kt, both past G2's limits (10°, 1 kt). The pre-registered rule therefore stopped the phase at validation. The test split was never scored.

Records: results and the validation record.