Anomaly analysis

Why this matters

One solar farm inspection yields thousands of images or hours of video, far too many for people to view. AI can shortlist suspicious images, but every AI both misses and raises false alarms. The key question is not “is the AI accurate?” but “how many defects does it miss, how many false alarms does it raise, and what can we accept?”, which depends on how much more expensive one missed defect is than sending someone to check a spot with nothing wrong.

The confusion matrix

Comparing AI results with specialists’ confirmation gives four groups:

• TP (true positive): AI says defect, and it is a defect
• FP (false positive): AI says defect, but it is normal, a false alarm
• FN (false negative): AI says normal, but it is a defect, a miss
• TN (true negative): AI says normal, and it is normal

Following the definitions in the scikit-learn documentation, precision is the share of alarms that are correct, recall is the share of real defects that are found, and F1 is their harmonic mean. In asset inspection TN is usually huge, so accuracy always looks high and is of little use.

results = {  # threshold: (TP, FP, FN)
    0.3: (46, 30, 4),
    0.5: (42, 12, 8),
    0.7: (33, 4, 17),
}
for th, (tp, fp, fn) in results.items():
    p, r = tp / (tp + fp), tp / (tp + fn)
    f1 = 2 * p * r / (p + r)
    print(f"threshold {th}: precision {p:.3f}, recall {r:.3f}, F1 {f1:.3f}, "
          f"missed {fn}, false alarms {fp}")

threshold 0.3: precision 0.605, recall 0.920, F1 0.730, missed 4, false alarms 30
threshold 0.5: precision 0.778, recall 0.840, F1 0.808, missed 8, false alarms 12
threshold 0.7: precision 0.892, recall 0.660, F1 0.759, missed 17, false alarms 4

From video to analysable images

A 30 frames-per-second video contains many repeated images. Analysing every frame wastes time and counts one defect many times, so frames should be sampled to give just the overlap needed.

FOOT_ALONG, OVERLAP, SPEED, FPS = 20.0, 0.80, 5.0, 30

interval_s = FOOT_ALONG * (1 - OVERLAP) / SPEED
step = int(interval_s * FPS + 1e-9)
print(f"need one frame every {interval_s:.2f} s = one in every {step} frames at {FPS} fps")
print(f"a 10-minute video: {10 * 60 * FPS:,} frames -> {10 * 60 * FPS // step:,} frames to analyse")

need one frame every 0.80 s = one in every 24 frames at 30 fps
a 10-minute video: 18,000 frames -> 750 frames to analyse

Image limits

The knowledge unit on the limits of RGB and thermal images warns that colour on an image is not a measurement: a thermal image exported as colour without radiometric data cannot be read as temperature. Something hidden does not appear in the image, but that does not mean it is absent, and comparing two images with different palettes or scales cannot show a trend. Reports must therefore separate “what is seen”, “what is hidden” and “what is still unknown”.

Module lab

Common mistakes

Summary

• The confusion matrix separates TP, FP, FN and TN; precision measures correct alarms and recall measures defects found
• The threshold trades precision against recall and must be chosen by the cost of misses and false alarms
• Sample video frames for the required overlap and merge repeated detections
• Colour is not a measurement, and hidden does not mean absent

Check your understanding

1. TP = 30, FP = 10 and FN = 20. What are precision and recall?
2. Using question 1, what is F1?
3. If missing a defect is very costly, should the threshold be high or low?
4. With a 15 m footprint, 75% overlap, 5 m/s and 30 fps video, how often must a frame be used?
5. Why is accuracy misleading in asset inspection?