The moment you ship a connected product, you take on a promise: this thing will keep working, and keep getting better. OTA firmware updates are how you deliver on that. But a bad OTA push can turn thousands of working devices into bricks. We've seen it happen — to clients, to competitors, and once, painfully, to ourselves. Here's everything we know about doing it right.

The $204,000 lesson

A client had 8,000 environmental sensors deployed across a city. Cellular connected. Working great. They pushed a firmware update, and 30% of devices downloaded corrupted binaries because a CDN cache expired mid-rollout. No A/B partitioning. No rollback. Each bricked device needed a physical site visit.

At $85 per truck roll, the recovery cost was $204,000. From a single update. The firmware change itself was a three-line bug fix.

We rebuilt their entire OTA pipeline after that. Here's the architecture we use now.

A/B partitioning is non-negotiable

Your flash has two slots. Slot A runs the current firmware. Slot B sits empty, waiting. When an update arrives, it downloads into Slot B. The device verifies the image — CRC32 for integrity, ECDSA signature for authenticity. Only then does the bootloader swap the active pointer.

If anything goes wrong — power loss, corrupt download, incompatible firmware — the device boots from the known-good slot. It's still running the old firmware, but it's running.

The piece most teams miss: boot confirmation. After swapping to new firmware, the application has a window — we use 60 seconds or 3 boot cycles — to signal "I'm alive and healthy." If it doesn't (crash loop, hang, fault), the bootloader rolls back automatically. No human needed.

We've seen devices survive power outages mid-update, cellular dropouts at 80% download, and even deliberately corrupted test images. The architecture holds.

Delta updates for cellular fleets

If your devices are on cellular, you're paying per megabyte. Sending a full 512 KB firmware binary for a 200-byte fix hurts. Delta updates send only the binary difference — typically 5-15% of the full image.

We use bsdiff for generating patches. The device applies the patch to its current firmware to reconstruct the new image in the staging slot. Sounds elegant. There's a catch.

Your fleet will have devices on different firmware versions. If you have 6 versions in the field (you will), you need delta patches for every possible upgrade path — or a stepping-stone strategy where devices first update to a common intermediate version.

Our rule: full image for major versions, delta for patches and minor releases. One client saved $180,000 per year in cellular costs across 50,000 devices just from this optimization.

Staged rollouts: the 1-10-50-100 rule

Never push to everyone at once. We do 1% first — a canary group. Monitor for 24-48 hours. Boot success rate, connectivity, heartbeat frequency, error logs. If everything looks clean, push to 10%. Then 50%. Then 100%.

The update server tracks every device: pending, downloading, verifying, confirmed, rolled-back. If we see a spike in rollbacks during any stage, the rollout halts automatically. No one has to be watching the dashboard at 2am.

This isn't being cautious for the sake of it. It's math. A 2% failure rate on 100,000 devices is 2,000 truck rolls. Catching it at the 1% canary stage means 10 truck rolls. That's the difference between an inconvenience and a catastrophe.

Key Takeaways

• A/B partitioning with boot confirmation and auto-rollback is the foundation. No exceptions.
• Delta updates can cut cellular data costs dramatically — but plan for multi-version fleet management
• Use staged rollouts (1% to 10% to 50% to 100%) with automated health monitoring
• A bad OTA is the most expensive bug in IoT. The prevention is cheap by comparison.