When developing an embedded Linux device, one of the first architectural decisions is selecting the operating system (OS) distribution. Broadly, there are two approaches. You can build a custom Linux distribution from source using tools such as Yocto or Buildroot, giving you complete control over the software stack and footprint. Alternatively, you can adopt a pre-built binary distribution such as Debian, Ubuntu, or Raspberry Pi OS (Raspbian), which accelerates development by providing a fully packaged operating system with extensive software repositories and long-term maintenance. Each approach involves different trade-offs in customization, maintenance, security, and long-term device management.

When it comes to updating software on embedded Linux devices, over-the-air (OTA) updates are the most efficient and scalable approach, particularly for large-scale deployments where manually updating devices in the field is impractical. To achieve this, a secure end-to-end system architecture is needed to receive, deploy, and manage software updates remotely (see image). To this end, the design splits cleanly into two domains that have almost opposite constraints: the server side, which manages trust, versioning, and rollout at scale, and the client side, which has to survive power loss, flaky networks, and years of unattended operation while still guaranteeing it never boots into a broken or malicious image.

It is important to understand that an OTA update system is designed to secure the software update process, not the physical device as a whole. Its primary responsibility is to ensure that software updates are delivered, verified, and installed securely, protecting against threats such as tampered update packages, unauthorized software, and man-in-the-middle attacks during deployment.

‍

What's actually being updated

Nearly every embedded Linux device contains a bootloader responsible for starting the system, a Linux kernel that manages hardware resources, a root filesystem that provides the operating system and core services, and one or more additional partitions that store applications, device configurations, persistent data, or firmware. These are the primary software components that an OTA update system must be capable of deploying, managing, and recovering should an update fail. System software updates or also interchangeably called firmware updates primarily refer to these components whereas application software updates refer primarily to any software component at the user space.

It is worth being precise about what gets updated, since the answer changes your architecture:

• A single file (kernel, app binary) — usually a custom in-house process.
• A package (apt/yum-style) — common for binary distributions; transfers only necessary binaries and dependencies, reducing bandwidth compared to full system images.
• A full filesystem image — the most common embedded approach; simple but coarse-grained.
• An atomic, differential filesystem update — same end state as a full image, but only the changed files are transferred, which matters a lot on metered or low-bandwidth links.
• A container — isolates an individual application from the OS, so it can be updated independently using its own atomic, recoverable mechanism.

Whatever you update, the same requirements apply: it must be secure (can't become an attack vector), atomic (all-or-nothing), fail-safe (recoverable), complete (covers every layer that needs it — bootloader, kernel, rootfs, apps), and tuned to your required frequency and speed.

Client-side: the part that has to survive anything

The device-side update agent is the riskiest piece of this whole system, because a bug here can physically brick hardware that's in the field. Two architectural decisions dominate: how you lay out partitions for fallback, and how you chain cryptographic trust from silicon to rootfs.

Partition layout: symmetric vs. asymmetric

Most fallback-capable designs come down to one of two storage strategies. Symmetric (A/B) duplicates the entire image across two partitions — you write the new version into the inactive slot, flip an "active" flag, and reboot. Asymmetric updates the active image in place (typically by booting into an initramfs and patching from there) and relies on a smaller, separate recovery image as the fallback path. The trade-off is storage versus complexity and recovery speed. A reference diagram for is shown below.

‍

The actual update sequence on a symmetric layout looks like this:

• Boot into active Image 1.
• Receive and write the new image into the inactive slot (Image 2).
• Toggle the active flag and reboot.
• If the reboot fails — caught via a watchdog or heartbeat check — the bootloader falls back to the last known-good image automatically, no physical intervention required.

Chain of trust: from silicon to rootfs

This is the part that actually makes the rest of the system trustworthy. The goal is that no single compromised layer, not even physical access to flash storage  can get unsigned code to execute.

A few details worth calling out from the secure OTA update flow diagram below:

• Server authentication utilizes standard TLS-based PKI to ensure that edge devices only communicate with verified infrastructure. By validating certificates before any deployment metadata or system images are accepted, the device mitigates the risk of fetching firmware from malicious sources or falling victim to spoofed update servers.
• Bidirectional authentication establishes a mutual trust layer where the server and the edge device verify identities in tandem. This mechanism ensures that hardware only retrieves artifacts from validated sources, while simultaneously preventing the unauthorized disclosure of sensitive system configurations or firmware images to unverified clients.
• Mutual TLS (mTLS) utilization executes bidirectional identity verification within a single cryptographic handshake, providing a robust transport for secure artifact delivery. While conventional TLS requires only server-side validation, mTLS mandates that the edge device likewise present a verifiable certificate, effectively isolating the infrastructure from unauthorized clients. This architectural stance is vital for OTA deployments: it prevents compromised hardware from retrieving sensitive firmware images and mitigates redirection to rogue update servers. By leveraging industry-standard libraries such as OpenSSL or mbedTLS, this mechanism integrates authentication directly into the encrypted data channel. This eliminates the overhead of separate authentication cycles, ensuring that identity proof and binary transmission occur over a unified, trusted path.
• Image signing uses a separate key pair from the server/device authentication keys. The bundle (kernel, device tree, rootfs, application) is signed on a build or signing server, and the device verifies that signature before flashing anything.
• The device Root of Trust is what makes offline attacks (someone pulling the flash chip and reflashing it directly) pointless. The public key hash is burned into One-Time Programmable (OTP) fuses on the System-on-Chip (SoC). Read-Only-Memory (ROM) code verifies the bootloader against that fuse-stored hash; the bootloader then verifies the kernel/Flattened Image Tree (FIT); the kernel can carry forward a digest check into the rootfs. Each stage only hands off control after verifying the next.
• Trusted Platform Module 2.0 utilization offers a superior alternative regarding electronic waste reduction. While burning OTP fuses effectively turns hardware into eWaste once official maintenance ceases, a TPM-based secure boot enables hardware reclamation through a chip reset. Although the industry often favors the simplicity of permanent fuses, it's a better mechanism if we care about responsible use of resources on this planet.
• Anti-rollback matters independently of signature checking — a validly signed old image can still reintroduce a patched vulnerability, so the device needs to track a monotonic version counter and refuse downgrades.

‍

‍

Containers as an update unit

For application-level updates specifically, containers (Docker, containerd, runC, cri-o, or lighter embedded options) decouple application lifecycle from OS lifecycle entirely. You get per-container resource limits (CPU, memory, network), the ability to run conflicting library versions side-by-side, and atomic, independently recoverable updates per container — without touching the base OS image at all. The trade-off is that you now need a container runtime baked into your OS image, which is its own maintenance surface.

Server-side: orchestration, trust issuance, and scale

If the client side is about surviving the update, the server side is about deciding who gets what, when, and proving it's legitimate. The client-side agent serves as the final arbiter of legitimacy within the trust chain. A robust architectural stance dictates that the device should treat the OTA server with the same level of inherent skepticism as an unverified physical peripheral. Consequently, the update client must perform rigorous cryptographic verification of the deployment bundle prior to execution, maintaining a consistent security posture whether the artifacts are delivered via network infrastructure or local storage media.The architectural concerns here are different in kind, not just degree.

• Device fleet visibility. You need a live model of hardware revision, currently-installed software version, and device status (online/offline/power state) for every unit in the field — otherwise you can't target a rollout correctly or know who silently failed.
• Update catalog management. Each available image needs metadata: software version and the hardware revisions it's compatible with, so the server never offers an incompatible build to a device.
• Key custody. The server side is where the private halves of your signing keys live — separate key pairs for server/TLS authentication and for image-bundle signing, as discussed above. This is also where you'd integrate hardware security modules or signing services if you're doing this at any real scale, rather than keeping private keys on a build machine.
• Rollout orchestration. This is the actual hard scaling problem: staged/canary rollouts, retry logic for devices that were offline during the push, and monitoring for which units failed and needed attention.

Selecting an OTA framework

Several mature open-source frameworks support embedded Linux deployments.

SWUpdate is well suited for full-image A/B deployments and integrates tightly with Yocto-based build systems. It provides robust support for signed update bundles, bootloader integration, and atomic installation. It is particularly well suited for:

• Full-image deployments
• A/B partition layouts
• Dual-copy recovery mechanisms
• Industrial devices with long product lifecycles
• Products requiring flexible update handlers

‍

RAUC is another mature framework and like SWUpdate, it emphasizes robustness, cryptographic verification, and fail-safe recovery. It installs signed update bundles into predefined boot slots and relies on the bootloader to activate the new software only after successful validation. It is built around several key design principles:

• Cryptographically signed update bundles
• Slot-based system updates
• Automatic rollback
• Bootloader integration
• Strict image compatibility validation

‍

OSTree excels when bandwidth optimization and immutable filesystem deployments are priorities. Its Git-inspired object model enables efficient binary delta distribution and rapid rollback. This architecture offers several advantages:

• Transactional deployments
• Efficient binary delta updates
• Immutable operating system images
• Fast rollback
• Significant bandwidth savings

‍

Container-based approaches complement, rather than replace, operating system OTA mechanisms by simplifying application deployment while leaving kernel and platform updates to dedicated OTA frameworks. Container-based deployments provide:

• Independent application release cycles
• Runtime isolation
• Simplified dependency management
• Application-level rollback
• Consistent deployments across hardware platforms

There is no universally "best" OTA framework—only the one that best fits your product architecture and operational requirements. See the comparison table below.

‍

Learn More: OTA Best Practices Checklist

Qbee's OTA Framework-Agnostic Approach  

While frameworks like RAUC or SWUpdate provide the mechanisms to handle the delivery of OS updates, Qbee functions as the orchestration and operations layer for the entire fleet. Rather than mandating a specific OTA standard, it serves as a unified control plane that integrates with existing open-source frameworks to manage system, container, and configuration updates across diverse IoT deployments. This framework-agnostic approach is vital for organizations maintaining multiple hardware platforms with varying update requirements. By decoupling the fleet-wide management from the low-level implementation, engineering teams can utilize the optimal update strategy for each device type while maintaining consistent automation and remote access. This flexibility is particularly critical for edge AI environments, where application logic and models iterate much faster than the base OS, allowing for continuous software delivery without the overhead of full system image redeployments.

Putting it together

The client and server only work as a system if they're treated as one trust chain end-to-end: a server with perfect TLS hygiene is irrelevant if the device blindly trusts whatever lands in its inactive partition, and a device with a flawless A/B fallback is still exploitable. All update artifacts must be signed in a trusted environment before being uploaded to any OTA server solution. The practical takeaway is to design the signing and verification path first — server authentication, device authentication, image signing, and the hardware root of trust — and only then pick the layer that fits your bandwidth, storage, and update-frequency constraints.

There is no universally "best" OTA framework. Instead of a one-size-fits-all solution, evaluate each framework based on your device architecture, reliability requirements, and operational needs: how often you update, how much bandwidth is available, how critical rollback is, and how much fleet management functionality you need out of the box.  

Avoid restructuring your entire embedded Linux device fleet to accommodate a purpose-built OTA platform. Rather than adopting a vendor-mandated approach, you should select the OTA software update strategy that aligns with your fleet requirements. Qbee integrates seamlessly with established open-source frameworks, containerized environments, and application-level update mechanisms. This allows for the unified management of heterogeneous fleets via a single control plane, effectively eliminating the need for expensive architectural migrations.

Designing OTA infrastructure requires balancing flexibility, reliability, security, storage efficiency, operational complexity, and deployment velocity. The most successful implementations treat OTA as an integral component of the software delivery pipeline rather than an isolated firmware feature.