Model evaluation and edge

Example 1 The effect of including warm-up

205 latency measurements on a companion computer; the first five are warm-up (simulated data in milliseconds).

import numpy as np

rng = np.random.default_rng(9)
lat = np.r_[rng.normal(180, 20, 5), rng.gamma(20, 1.6, 200) + 10]    # ms
WARMUP = 5
print("first runs:", np.round(lat[:WARMUP]).astype(int))
for name, x in (("including warm-up", lat), ("after warm-up", lat[WARMUP:])):
    p50, p95 = np.percentile(x, [50, 95])
    print(f"{name:<18} n={len(x)}: mean {x.mean():5.1f}, p50 {p50:5.1f}, p95 {p95:5.1f}, max {x.max():6.1f} ms")
p95 = np.percentile(lat[WARMUP:], 95)
print(f"frame rate the model can sustain for 95% of frames: {1000 / p95:.1f} fps")

first runs: [164 185 147 193 203]
including warm-up  n=205: mean  46.1, p50  43.0, p95  59.3, max  202.9 ms
after warm-up      n=200: mean  42.8, p50  42.8, p95  56.4, max   63.4 ms
frame rate the model can sustain for 95% of frames: 17.7 fps

The median barely changes and the mean and p95 rise slightly, but the maximum rises more than threefold because of warm-up. Reporting everything together makes the model look slower than it is, while reporting only the mean after warm-up can hide the slow cases that make the system fall behind. This is still only model time; image reading, resizing and sending results must also be measured to get the end-to-end time.

Example 2 One outlier and INT8 error

10,000 weights from a normal distribution plus one outlier of 0.6. Compare using the true maximum with clipping at ±0.2 before computing the scale.

import numpy as np

rng = np.random.default_rng(1)
w = rng.normal(0, 0.05, 10000)
w[0] = 0.6                                      # outlier
for name, src in (("scale from max |w|", w), ("clip at +/-0.2 first", np.clip(w, -0.2, 0.2))):
    s = np.abs(src).max() / 127
    q = np.clip(np.round(src / s), -127, 127)
    err = q * s - w
    print(f"{name:<22} scale {s:.5f}: mean |error| {np.abs(err[1:]).mean() * 1e3:.3f}e-3, "
          f"outlier error {abs(err[0]):.3f}")
print(f"size: float32 {w.size * 4 / 1024:.1f} KiB -> int8 {w.size / 1024:.1f} KiB")

scale from max |w|     scale 0.00472: mean |error| 1.186e-3, outlier error 0.000
clip at +/-0.2 first   scale 0.00157: mean |error| 0.392e-3, outlier error 0.400
size: float32 39.1 KiB -> int8 9.8 KiB

Using the true maximum makes the error of most weights about three times larger than clipping first, but clipping badly distorts the outlier. There is no single answer: always evaluate model accuracy on the validation set after quantization, and set an acceptance criterion such as AP falling by no more than an agreed amount.