How to test firmware changes without breaking an aircraft

The problem with the usual method

The usual method is: change something, go to the field, hope.

It is an understandable method. Firmware changes are free, batteries are charged, and the field is where the aircraft is anyway. But as an experiment it is close to worthless, and it is not cheap.

It is not a controlled comparison. Between last week's flight and today's, you changed a PID, and also the wind changed, and also the battery is a different one, and also you are flying a slightly different line because you always do. When it feels better, you cannot say what made it better. When it feels worse, you cannot say what made it worse. You have collected one data point with four uncontrolled variables in it, and you will now make a decision on it anyway.

One flight is one sample. The interesting failure modes — the ones that end aircraft — do not appear in a two-minute hover on a calm evening. They appear at full throttle, in a hard braking manoeuvre, at 3.4 V per cell, in a gust, when the link drops behind a tree. A flight where nothing happened is not evidence that nothing will.

And the failure mode destroys the evidence. This is the part that makes field-only testing structurally bad. The moment you actually need to know what went wrong, you are holding a broken aircraft, quite possibly without a Blackbox log, and the fault is now unreproducible because the hardware that produced it no longer exists. You paid the full price of the crash and got nothing back in exchange for it.

The alternative is not "test more carefully." It is to stop asking one expensive experiment to answer questions that cheaper experiments could have answered first.

The ladder

Order your tests so that each one is cheaper to fail than the next one up. Start at the bottom. Do not skip rungs — every rung you skip is a class of fault you have decided to discover in the air.

1. Know exactly what one thing you changed

Before anything else: can you state, in one sentence, what is different? Not "I updated things." "D on roll went from 34 to 40, nothing else."

If you flashed new firmware, read the release notes and diff your config. If you changed code, read your own diff. If you cannot say what changed, you are not testing — you are gambling, and you will not learn anything from the result either way.

2. Bench: props off

Props off. Battery in. Arm.

This rung costs three minutes and catches an embarrassing proportion of everything:

• Do the motors spin, in the right direction, on the right outputs? Motor 3 in the Configurator should spin the motor you think is motor 3. This is where a bad resource mapping or a swapped ESC signal wire dies quietly instead of loudly.
• Does the aircraft arm at all? If not, the flags tell you why — Betaflight's Configurator shows the arming disable flags directly, and every one of them is a specific, named reason.
• Tilt the aircraft by hand and watch the motor outputs. Roll it right; the left motors should command up. If the correction goes the wrong way, you have just found a sign error on a bench instead of at one metre of altitude.
• Check the modes, the rates, the gyro trace at idle, the receiver channel map. All of it is visible with the props in your other hand.

3. SITL: run the real firmware against a physics model

Now you have a running loop with no consequences. This is where you do the tests you would never do on hardware.

Because the firmware in a SITL build is the actual firmware — not a game's approximation of it — the control loop, the filters, the arming logic and the failsafe state machine all behave exactly as they will on the aircraft. So:

• Drive it into the failure on purpose. Wind D up until it oscillates. Wind P up until it wallows. See it in the gyro trace, so that you recognise it instantly when it appears in a real log.
• Test the extremes. Full throttle, hard braking, a sustained climb until the pack sags. These are the flight regimes where tunes fall apart, and in a simulator they cost nothing.
• Test the failsafe. Cut the RC link. Watch what the aircraft actually does, rather than what you assume it does.
• Fly the same manoeuvre twice with one value changed, and compare the two logs. No wind, no battery variance, no different line. That is a controlled comparison, and it is the thing the field cannot give you.

Be clear about what this rung does not do. A simulator will never find a cold solder joint, a chafed motor wire, an antenna in the wrong place, an unbalanced prop, a cracked arm, or an ESC that fails at 40 A. It knows nothing about your RF environment, your soldering, or your mechanical assembly, and it never will. What it does is remove a whole class of control-loop, configuration and logic faults before you get to the field — so that the test flight is testing the hardware, not the arithmetic.

4. HIL, if you have a hardware or timing suspicion

If your suspicion is about the board rather than the logic — a DMA conflict, a loop-time overrun, a sensor driver, a UART fighting for a timer — SITL cannot answer it, because SITL has no board. Hardware In The Loop runs the firmware on the real flight controller and feeds it simulated sensor data. It is more work to set up and it is closer to the truth. Most changes do not need it. Timing and driver suspicions do.

5. A tethered or low hover, in a safe place, with Blackbox on

The first real flight of a changed configuration is a test, not a flight. Treat it as one.

Open space, no people, no one downwind of you, aircraft pointed away. Hover at a metre or two. Logging on. Do not go and fly a line. You are asking one question — does it hold attitude and does it hold it cleanly — and the answer arrives in the first ten seconds.

On a heavy platform, a physical tether is not paranoia; it is the difference between a bad hover and a runaway. And still: one change since the last known-good flight. Not two.

6. A real flight, with the log running

Only now. Same manoeuvre you always use as a reference, so today's log is comparable with last month's. Land, pull the log, look at the trace. Then, and only then, change the next thing.

Change one thing at a time

This is the single most violated rule in the hobby, and violating it is what turns a tuning session into a superstition.

If you change three things and the aircraft flies better, you have learned nothing. You do not know which of the three helped, whether one of them hurt and the other two overcame it, or whether the wind simply dropped. You now carry all three forward, forever, including the one that is quietly making the aircraft worse — and you will defend it, because it was in the batch that worked.

One change. One flight. One log. It feels slow. It is the only speed that actually converges.

Always fly with logging on

You cannot debug a flight you did not record.

Blackbox costs you a little flash and a little loop headroom, and it is the difference between "it fell out of the sky, I think maybe desync?" and knowing, from the trace, that motor 2 was commanded to 100% for 180 ms while the gyro went nowhere.

Every flight where you changed something gets logged. Ideally every flight, full stop, because the crash you did not anticipate is precisely the one you did not arm the logging for. And keep the logs. A log of the last flight that worked properly is the reference against which every future log is read.

Have a rollback

Know the last configuration that flew well, and be able to get back to it in five minutes without remembering anything.

In the Betaflight CLI, diff all prints every setting that differs from firmware defaults — a text file, small enough to read, complete enough to restore. dump prints the full configuration if you want everything. Save the output every time the aircraft flies well, name the file with the date and the aircraft, and put it somewhere you will find it. iNAV's CLI offers the same idea; ArduPilot users pull a parameter file.

If you are serious — and certainly if the aircraft does a job for somebody — put those files in version control. Then "what did I change?" is a question with an actual answer instead of a memory test, and a rollback is a checkout.

Test the failure cases, not just the happy path

Nearly everyone tests whether the aircraft flies. Almost nobody tests what it does when something breaks. Then something breaks.

The four worth rehearsing deliberately, and the four that most often destroy aircraft:

• Failsafe. Turn the transmitter off and find out what the aircraft actually does. Do you know, from having seen it, rather than from having read the setting? Is the failsafe stage 2 procedure the one you want — drop, or land, or GPS Rescue? Has GPS Rescue ever been tested with a real fix?
• RC link loss in a bad place. Behind you, low, near a treeline, at range. The failsafe you tested in the open field is not the failsafe you get behind an obstacle.
• Low voltage. Fly a pack down. Find out what the aircraft does at sag, under load, at the bottom — not what the OSD said at hover.
• A motor stopping. Desync, a shed prop, an ESC cutting out. A quad cannot recover from this, but you can respond to it, and you can know in advance what it looks and sounds like when it starts.

Every one of these can be run in SITL, dozens of times, for free. There is no other place you can rehearse a failsafe forty times before lunch.

The economics

On a 250 g racer, empirical trial-and-error is a perfectly rational strategy. A crash costs a prop, sometimes an arm, occasionally a motor. Twenty euros and an evening. Against that, the cost of building a careful test procedure is higher than the cost of just going and finding out. Fly it, break it, fix it, fly it again — that is not sloppiness, it is a correct reading of the incentives. Anyone who tells a racer to run a simulator before every PID nudge is selling something.

The calculus flips, and it flips hard, when any of three things is true.

When the airframe is expensive. A 5 kg platform with good motors, a big pack and a payload is several thousand euros, and it does not bounce. It arrives with the kinetic energy of a small motorcycle. One destructive oscillation event and the bill is not twenty euros — it is a rebuild, a lead time on parts, and possibly a written-off airframe. At that price, a test procedure that costs you two hours and catches one crash a year has paid for itself several times over.

When the aircraft is doing a job. If somebody is waiting on the survey, the inspection, the delivery, then the aircraft being down is a cost on top of the repair. Downtime is usually the larger number, and nobody puts it in the spreadsheet.

When people are underneath it. This is the one that ends the argument. A 5 kg aircraft losing attitude control over anyone is not a hobby problem, and no amount of "I usually get away with it" is a safety case. The failure modes you have never tested are exactly the ones you cannot claim to have controlled.

None of that makes the field optional. You still fly the tethered hover, still run the log, because the simulator never knew about your soldering. A sim-clean tune is not a proven tune — it is a tune that has already survived every mistake worth making indoors.