Skip to main content

Phase 5: frozen models do not beat a CUSUM at detecting manoeuvres

· 3 min read

G5 passed in phases 4b and 4c. SRD Annex C §C.6 therefore lists incremental follow-ons, and the first is manoeuvre detection, judged against a CUSUM on the reference tracker's innovations.

The owner chose two things for this attempt:

  • No training. The detector is built from models that already exist: the frozen recurrent tracker T2, and the reference IMM, which now uses decision 32's guarded initialisation.
  • The test. Detection delay, with every detector's threshold set for the same false-alarm rate: one alarm per 2 h of quiet tracking.

What was compared​

  • The baseline. A CUSUM on the IMM's innovations, tuned on validation from 12 declared settings to favour it.
    • Clipping. The innovations are heavy-tailed: a few huge values otherwise hold an alarm for hours. So both the variance kind and the mean-shift kind are clipped.
    • The winner. The two-sided mean-shift CUSUM, since a turn biases the bearings to one side.
  • The learned detectors. Two candidates, declared in advance:
    • T2's course disagreement with the IMM;
    • T2's own course change over the last minute.

Result: it stopped at validation​

At 0.5 false alarms per hour, on 7,479 course changes in seed-211 validation:

Detected within 5 min
CUSUM (mean shift)28.2%
T2's course change over 1 min19.2%
T2 against the IMM5.9%
  • The learned detector is not carried. It is 9 points behind, with interval [−10.2, −7.7] pp, so it was not taken to a test split, and fresh seed 215 was never generated.
  • The delay check could not be judged. Every detector misses more than half the course changes at this rate, so the median delays are infinite. That rule was the agent's, written before the data; any later attempt should declare a delay statistic defined at its operating point.

Reported, not used: a crossover​

The same detectors were also calibrated at looser rates:

False alarms per hour0.5125
CUSUM28.2%38.0%47.8%62.6%
T2's course change19.2%39.0%58.1%71.7%

From about 1 false alarm per hour, T2's course change catches more manoeuvres in time, though about 20–40 s later among those both catch. Moving the operating point after seeing this would choose the rule from the result, so it stays a reported observation.

What next​

The owner decides. Annex C's other follow-ons remain:

  • calibration of the learned uncertainty;
  • realistic bearing error.

Record: experiments/20261105-p5-research/validation.md; spec 017.