Autonomous missions from SITL to field

Why this matters

An autonomous mission that works perfectly in simulation can fail on its first real flight because of wind, sunlight, vibrating sensors or a dropped link. Testing must therefore be a staircase, adding realism one step at a time and limiting the damage when something fails, with pass criteria set at every step before seeing results. This module brings the whole course together as one mission, from SITL to the field.

The test ladder

Every stage tests both normal and abnormal cases, such as a hung companion computer, GNSS loss, control link loss and low battery, checking that failsafes and geofences behave as configured. The PX4 documentation gathers these failsafe settings on one page.

How many runs are enough?

A success rate from a few runs is highly uncertain, so report it with a confidence interval, as in UAT 106.

import math
from scipy import stats

for stage, runs, ok in (("SITL", 50, 48), ("HITL", 20, 19), ("field", 12, 11)):
    ci = stats.binomtest(ok, runs).proportion_ci(confidence_level=0.95)
    print(f"{stage:<6} {ok}/{runs} = {ok / runs:.0%}  95% CI [{ci.low:.0%}, {ci.high:.0%}]")

reliability, confidence = 0.90, 0.95
n = math.ceil(math.log(1 - confidence) / math.log(reliability))
print(f"to show >= {reliability:.0%} success with {confidence:.0%} confidence and no failures: {n} runs in a row")

SITL   48/50 = 96%  95% CI [86%, 100%]
HITL   19/20 = 95%  95% CI [75%, 100%]
field  11/12 = 92%  95% CI [62%, 100%]
to show >= 90% success with 95% confidence and no failures: 29 runs in a row

Run-time assurance

AI and path planners are too complex to prove safe in every situation. The run-time assurance (RTA) approach of ASTM F3269-21 therefore adds a simple, verifiable monitor that checks whether the system is still within a safe envelope and, if it is about to leave it, switches to a safe controller such as braking, holding or returning home.

fence, margin, latency, brake, dt = 100.0, 5.0, 0.3, 2.5, 0.1
x, v, t, mode = 80.0, 4.0, 0.0, "complex"
while t < 10:
    stop = v * latency + v * v / (2 * brake)
    if mode == "complex" and x + stop >= fence - margin:
        mode = "safe"
        print(f"switch to safe controller at t = {t:.1f} s, x = {x:.1f} m, v = {v:.1f} m/s")
    v = min(v + 0.5 * dt, 8.0) if mode == "complex" else max(v - brake * dt, 0.0)
    x += v * dt
    t = round(t + dt, 1)
    if mode == "safe" and v == 0:
        break
print(f"stopped at x = {x:.1f} m, {fence - x:.1f} m before the fence")

switch to safe controller at t = 2.0 s, x = 89.0 m, v = 5.0 m/s
stopped at x = 93.8 m, 6.2 m before the fence

Before flying for real

• Law: beyond-visual-line-of-sight (BVLOS) flight in Thailand is covered by CAAT guidance CAAT-GM-UAS-PDRA101 (May 2025), which specifies a predefined risk assessment, operations manual, training and emergency response plan. Check the latest issue before every application
• Operating procedures under ISO 21384-3 and the organisation’s operations manual
• Systems using ML: EASA’s AI Concept Paper Issue 2 (2024) sets out guidance for Level 1 and 2 machine learning applications, a good reference frame for writing safety evidence
• A pilot ready to take over at every real-flight stage, with a tested mode switch or kill switch

Module lab

Common mistakes

Summary

• Test as a ladder, SITL → HITL → controlled area → small field test → full mission, including abnormal cases at every stage
• Success rates need confidence intervals; showing at least 90% success at 95% confidence takes 29 consecutive successes
• Run-time assurance uses a simple monitor to switch to a safe controller before leaving the envelope
• Before flying for real, check the law and operations manual, and always have a pilot ready to take over

Check your understanding

1. Which stage tests real firmware on the real board with a simulated world?
2. To show at least 95% success with 95% confidence, how many consecutive successes are needed?
3. A drone is at 90 m, flying 5 m/s with 0.2 s latency and 2.5 m/s² braking; the fence is at 100 m with a 3 m margin. Should the monitor switch yet?
4. Why should the run-time assurance monitor be simple?
5. Which CAAT guidance covers BVLOS flight using a predefined risk assessment?