Evaluating perception reliability

Example 1 ATE before and after alignment

A reference ellipse and an estimated trajectory rotated by 8°, shifted by (3, −2) m and with 0.3 m of noise (simulated data).

import math
import numpy as np

t = np.linspace(0, 2 * np.pi, 40)
gt = np.c_[20 * np.cos(t), 10 * np.sin(t)]                           # m
th = math.radians(8)
rot = np.array([[math.cos(th), -math.sin(th)], [math.sin(th), math.cos(th)]])
est = gt @ rot.T + np.array([3.0, -2.0]) + np.random.default_rng(2).normal(0, 0.3, gt.shape)

raw = np.sqrt(np.mean(np.sum((est - gt) ** 2, axis=1)))
mu_e, mu_g = est.mean(0), gt.mean(0)
U, _, Vt = np.linalg.svd((gt - mu_g).T @ (est - mu_e))               # Umeyama without scale
D = np.diag([1, np.sign(np.linalg.det(U @ Vt))])
R = U @ D @ Vt
aligned = (est - mu_e) @ R.T + mu_g
ate = np.sqrt(np.mean(np.sum((aligned - gt) ** 2, axis=1)))
print(f"error without alignment {raw:.2f} m")
print(f"ATE after alignment {ate:.2f} m (rotation found {math.degrees(math.atan2(R[1, 0], R[0, 0])):.1f} deg)")

error without alignment 4.22 m
ATE after alignment 0.42 m (rotation found -7.7 deg)

Before alignment, most of the several metres of error comes from the frame difference. After alignment, it falls to near the noise level, which is the real error in the shape of the path. Reporting ATE must state which alignment was used (rotation and translation, or scale as well), because scale alignment hides the single-camera scale problem from Module 4.

Example 2 Periods without output

The system outputs at 20 Hz throughout a 30-second flight; the team treats gaps over 0.2 seconds as lost tracking (simulated data).

import numpy as np

ts = np.arange(0, 30, 0.05)
ts = ts[~(((ts > 12.0) & (ts < 13.6)) | ((ts > 21.0) & (ts < 21.4)))]   # missing periods
gaps = np.diff(ts)
lost = gaps > 0.2
print(f"{ts.size} outputs, {lost.sum()} gaps over 0.2 s")
for start, g in zip(ts[:-1][lost], gaps[lost]):
    print(f"  lost at t = {start:.2f} s for {g:.2f} s")
print(f"time without tracking {gaps[lost].sum():.2f} s = {gaps[lost].sum() / 30:.1%} of the flight, longest {gaps.max():.2f} s")

562 outputs, 2 gaps over 0.2 s
  lost at t = 12.00 s for 1.60 s
  lost at t = 21.00 s for 0.40 s
time without tracking 2.00 s = 6.7% of the flight, longest 1.60 s

The longest gap, about 1.6 seconds, is a dangerous period if the drone relies on visual position alone. Look at the recorded images for that period to find the cause, and define what the system must do when tracking is lost for longer than acceptable, such as dropping to a mode that does not need position, or holding position and waiting.