Safety and effectiveness evaluation

Example 1 From test flight hours to reliability

The team flew 42 test hours with 2 failures that aborted the mission. One canal inspection round takes about 20 minutes (hypothetical data).

import math

T_HOURS, FAILURES, MISSION_MIN = 42.0, 2, 20
mtbf = T_HOURS / FAILURES
r_mission = math.exp(-(MISSION_MIN / 60) / mtbf)
print(f"MTBF {mtbf:.1f} h, reliability of one {MISSION_MIN}-min mission {r_mission:.3f}")
print(f"expected aborts in 100 missions: {100 * (1 - r_mission):.1f}")

for r_target, conf in ((0.95, 0.90), (0.95, 0.95), (0.99, 0.95)):
    n = math.ceil(math.log(1 - conf) / math.log(r_target))
    print(f"to show R >= {r_target} with {conf:.0%} confidence: {n} missions with zero failures")
print(f"30 missions with no failure -> failure rate below about {3 / 30:.0%} (rule of three)")

MTBF 21.0 h, reliability of one 20-min mission 0.984
expected aborts in 100 missions: 1.6
to show R >= 0.95 with 90% confidence: 45 missions with zero failures
to show R >= 0.95 with 95% confidence: 59 missions with zero failures
to show R >= 0.99 with 95% confidence: 299 missions with zero failures
30 missions with no failure -> failure rate below about 10% (rule of three)

From the data so far, a 20-minute mission succeeds about 98% of the time. But demonstrating 95% with 90% confidence needs 45 missions without a single failure, which is beyond the project’s time, so the report must state the confidence level the data actually supports.

Example 2 SUS from five irrigation staff, and effectiveness

import statistics as st

responses = [                      # 10 items, scale 1–5
    [4, 2, 4, 2, 4, 1, 4, 2, 3, 2],
    [5, 1, 4, 2, 4, 2, 5, 1, 4, 3],
    [3, 2, 3, 3, 4, 2, 4, 2, 3, 2],
    [4, 2, 4, 1, 4, 2, 4, 2, 4, 2],
    [4, 3, 3, 2, 3, 2, 4, 3, 3, 2],
]

def sus(r):
    return 2.5 * sum((v - 1) if i % 2 == 0 else (5 - v) for i, v in enumerate(r))

scores = [sus(r) for r in responses]
print("SUS per user:", scores, f"mean {st.mean(scores):.1f}, lowest {min(scores)}")

manual = {"minutes per round": 180, "people": 2}
drone = {"minutes per round": 45, "people": 2}
saved = manual["minutes per round"] * manual["people"] - drone["minutes per round"] * drone["people"]
print(f"staff time saved per 2 km round: {saved} person-minutes ({saved / (manual['minutes per round'] * manual['people']):.0%})")

SUS per user: [75.0, 82.5, 65.0, 77.5, 62.5] mean 72.5, lowest 62.5
staff time saved per 2 km round: 270 person-minutes (75%)

The mean is slightly above the reference average, but two users scored below 68, so interview them to find where they struggled. Five users is still a small sample, and the time saving must state that it excludes preparing flight permission documents and reviewing reports.