Multi-tenant AWS IoT architecture: where to draw the isolation boundaries

Sell an IoT product through distributors for long enough and one of them will ask for their own version of it: their logo on the dashboard, their customers logging in, and their customers' data walled off from every other distributor's, in writing. Making good on that promise on AWS is an architecture problem with more than one right answer. This article walks through how we solved it on a platform we built and ran at safeINIT: which parts of the system every tenant shares, which parts each tenant gets alone, and the storage decision that came back for the analytics bill later.

The platform itself is easy to picture. Hardware devices sit on infrastructure out in the field and measure the conditions around it, everything from temperature and humidity to noise and pressure, plus a camera taking periodic photos. Each device sends its readings to AWS, where a web dashboard shows operators what every site is doing right now, raises alerts when a reading crosses a threshold, and keeps the full history for analysis. One company, one fleet, one dashboard, until the distributors started asking.

A distributor's request turns an IoT product into a multi-tenant product. Each distributor becomes a tenant, one customer organisation with its own users and its own data inside a shared product, and the wall between tenants is what architects call an isolation boundary: the place where the separation is actually enforced. That can be a separate AWS account, a separate database, or nothing more than an application filter everyone hopes is applied everywhere. A multi-tenant AWS IoT architecture has no single place to build the wall, because a sensor reading crosses four layers on its way from a device to a dashboard: ingestion, data, application and deployment. Each layer takes its own decision about what tenants share.

Our answers cut both ways: devices connect through one shared front door, every tenant's production data and application live in AWS accounts of their own, and everything below production is shared on purpose. The reasoning is the useful part, so that is what follows, layer by layer.

How the data flows, before any of it is multi-tenant

The whole multi-tenant question comes down to one pipeline: which pieces of it each tenant gets their own copy of, and which stay shared. Here is that pipeline.

Devices talk to AWS IoT Core over MQTT, a lightweight publish-subscribe messaging protocol built for small devices on unreliable networks. A device authenticates with an X.509 certificate installed on it, everything travels over TLS, and each sensor reading is published to a topic, which is a named channel, something like a folder path for messages. Fresh out of the box, a device knows nothing about who it belongs to. It powers on, subscribes to a command channel, and waits for the platform to tell it where to send data. That bootstrap gets its own section below.

From IoT Core, routing rules send every reading down two paths at once:

• The real-time path. Readings flow into a Kinesis data stream, a Lambda function processes them, and the results land in DynamoDB, the database behind the live dashboard. This path powers the current-status view and the threshold alerts.
• The archive path. Every reading is also written to S3, AWS's object storage, for permanent keeping. The archive feeds the analytics layer: Athena runs SQL queries over the stored readings, and QuickSight turns the results into dashboards and reports.

Photos take a separate road, because an image does not belong in an MQTT message. The device trades its certificate for short-lived AWS credentials through the IoT credentials provider, signs a PUT request with them, and uploads each image through an API Gateway endpoint integrated directly with S3, no server in the path. When an operator views a photo in the dashboard, it comes back out through CloudFront using pre-signed URLs.

One pipeline, four different kinds of thing inside it. Each one isolates differently.

The four isolation boundaries

The generic advice comes in two flavours. AWS's own guidance for multi-tenant IoT platforms leans toward a dedicated AWS account per tenant, each with its own IoT Core endpoint, and a claiming service that routes devices to the right one. The Stack Overflow answer says the opposite: pool everything and filter on a tenant_id column in the application.

Both are answering a coarser question than the one you actually have, because the four layers isolate differently:

• Ingestion is the front door: the IoT Core endpoint devices connect to, the certificates that authenticate them, and the rules that route their readings.
• Data is where readings and photos rest: the S3 archive and the DynamoDB tables.
• Application is everything a person touches: the dashboard, its API, and the Lambda functions that process readings.
• Deployment is how new versions of the application ship: the source repositories and the CI/CD pipelines.

Here is what we chose at each layer.

Non-production is missing from that table on purpose. It gets its own section, because pooling it is the decision that gets challenged in every architecture review.

Pool ingestion: one AWS IoT Core endpoint for every tenant

Every device in the fleet connects to a single AWS IoT Core endpoint in one shared AWS account. That account owns the device registry, the certificates, the policies and the routing rules. Tenant separation happens one hop later, in the rules engine, where each tenant's traffic is routed into a Kinesis data stream that belongs to that tenant alone. No device ever connects to a tenant's account directly.

The reason is physical. Devices get built in batches, sit in a warehouse, ship through distributors, and get installed by an engineer with a van and a schedule. Binding a certificate to a tenant's AWS account at manufacture turns one hardware SKU into as many SKUs as you have customers, and every resale, return and warehouse transfer becomes a re-provisioning job. Pooling also collapses real operational duplication: one certificate authority instead of one per tenant, one endpoint in the firmware instead of a config table the installer has to get right.

Routing tenants apart inside the message layer, rather than inside application code, is the same event-driven architecture discipline that applies anywhere on AWS. The alternative is a conditional in a Lambda handler that somebody maintains forever.

The device policy is the boundary, so scope it like one

Be precise about where pooled ingestion can leak. Once every tenant shares an endpoint, the AWS IoT policy attached to each device certificate is the only thing standing between a compromised device and another tenant's stream. The AWS IoT Core documentation on Basic Ingest describes two shapes of publish permission: a wildcard across all rules, or permission scoped to specific rules and topics. They are not remotely equivalent. A wildcard grant means any device in the fleet can publish as any tenant, and the only thing preventing it is that none of them currently do.

So scope each device's publish permission to the one tenant topic it has been assigned, and make updating that policy part of tenant assignment from day one. Retrofitting policy scope onto a fleet of already-issued certificates is the expensive version of the same task.

Bind the device to its tenant at runtime, not at manufacture

The part of this design we still like most is the bootstrap flow, because it deletes a whole class of logistics problem.

A device leaves the factory with a certificate installed and no idea who it works for. On first power-up it connects and subscribes to its command topic: a channel named after the device's own ID, which only the platform's admin application writes to. Then it waits. Publishing nothing. The admin application sends one message to that topic carrying two things, the tenant ID and the URL where the device should upload its photos. From that point the device sends its telemetry to that tenant's ingest topic and its images to the URL it was handed.

That single message is the tenant assignment. Not a firmware build, not a site visit. A distributor orders forty units and you ship forty units from the same undifferentiated pile, because the thing that makes a device theirs is a message. Reassignment is a message too; if you scope device policies per tenant, as you should, pair the message with the policy swap.

Why we removed just-in-time provisioning from scope

Just-in-time provisioning is the reflexive choice for device onboarding on AWS, and we had it in the design as an optional item. It came out, and the reasons are worth writing down because they are in the AWS documentation rather than in anyone's opinion.

JITP registers a device the first time it connects, by loading a provisioning template associated with the CA certificate that signed the device's certificate. Two documented properties decide whether that fits your fleet. First, template parameters "are limited to what JITP can extract from the subject field of the certificate", and provisioning fails outright if the certificate is missing any property the template references (AWS IoT Core Developer Guide, retrieved 2026-08-28). Real design decisions get pushed back into certificate issuance, where they are expensive to change. Second, the same page warns that during provisioning JITP calls other IoT control-plane API operations, and those calls "might exceed the AWS IoT Throttling Quotas set for your account".

A fleet that arrives in bulk deliveries is exactly the shape that trips the second property. Distributor hardware lands in batches, not in a steady trickle. So certificates are issued in bulk ahead of time and installed on the device during the hardware process instead. Fewer moving parts on the live path, at the cost of a manual step somebody owns. At some multiple of the fleet volume that answer flips back toward automation. Decide where that threshold sits for your fleet and write it down, so it does not have to be rediscovered under pressure.

Pool non-production, and price what that costs you

Below production, the design inverts. One shared pool of real development devices feeds every tenant's development and staging environment through a single shared Kinesis stream. One shared upload endpoint, one shared storage account for all non-production data. And there is no generic staging at all: every environment belongs to a specific tenant, because the whole point of a white-label product is that tenants diverge, and a generic staging environment tests a version of the application nobody actually runs.

Two things drove the pooling. Cost is the obvious one; duplicating a full data plane per tenant per stage means paying for idle delivery streams and idle buckets across environments nobody looks at on a Tuesday. Data realism is the less obvious one. Development environments fed by synthetic payloads drift from reality, and IoT payloads drift fast, because the interesting bugs live in the malformed packet and the sensor that reports in the wrong unit after a firmware update. A shared pool of real devices gives every tenant's developers live data without giving anyone live customer data.

Now the cost, stated plainly: you cannot test tenant isolation in staging, because staging has no tenant isolation to test. Any assurance that tenant A's data cannot reach tenant B has to be validated against production wiring, in a controlled window. If your compliance posture requires demonstrating that separation on demand, this trade is not available to you, and you should pay for isolated non-production environments. For a platform whose regulatory exposure sits in commercial contracts rather than in health or personal data, it can be a reasonable trade. Know which one you are.

Silo the data layer: one account per tenant, in production

Production is where the account boundary earns its cost. Each tenant gets a dedicated data-storage account holding their raw telemetry, their device images and their archive, separate again from the account running their application. An application bug can corrupt the working set. It cannot reach the archive.

This is also the boundary that survives a conversation with a customer's security team. "Your data sits in a separate AWS account with no cross-account path to any other tenant" is a sentence you can prove with a policy document. "Our application filters by tenant ID" is a sentence you can only prove with a code review, and you will be asked to prove it again after every release. Once tenant accounts multiply under AWS Organizations, service control policies are what keep a misconfigured role in one account from becoming an organisation-wide problem, and they are worth wiring in before the tenant count grows.

Account sprawl is real, and every new tenant multiplies the environments someone has to patch, monitor and pay for. We took that cost deliberately, because the alternative failure mode is unrecoverable. A leaked row between two competing distributors is the kind of incident you lose the account over.

The storage decision that caught up with the platform

Go back to the archive path. Every reading every device sends is written to S3, and the analytics layer reads those stored files directly whenever a dashboard refreshes. So the shape those files take decides what analytics costs and how fast it runs.

The platform's first layout was the obvious one: each sensor reading saved as its own small JSON file. Easy to reason about, fine in a prototype, invisible for a long time. It stayed invisible until the analytics layer was carrying real load.

By the time we scoped the fix, the BI refresh was querying over 15 million individual files a day, refresh runs were taking more than 30 minutes and pushing past standard Athena limits into quota-increase territory, and the S3 bill had grown roughly twentyfold in four months, from about $10 a month to over $200. Small numbers in absolute terms. The curve was the problem: every new device added objects, and cost tracked object count, not stored bytes.

The mechanics are documented. When Athena plans a query it lists every partition location and pays a handling overhead per file, so "loading a single bigger file from Amazon S3 is faster than loading the same records from many smaller files". Push far enough and you hit S3 itself, which "supports up to 5,500 requests per second to a single index partition" and answers excess with SlowDown: Please reduce your request rate (Amazon Athena User Guide, retrieved 2026-08-28). A partition holding more than 1,000 files also forces sequential list calls, because S3 returns at most 1,000 objects per list.

The remediation we designed replaces the write path rather than the query: Kinesis Data Firehose buffers readings and flushes them as partitioned Parquet, so millions of daily file reads become hundreds of queries against the same data. Failed-delivery records go to a backup destination with automated reprocessing, so a delivery failure becomes a queue instead of a silent hole; proving that path works belongs in the same family as testing whether your backups actually restore. The design put a central analytics account on top: cross-account roles to read each tenant's data without moving it out of the tenant's own account, row-level security to separate tenants at query time, and SageMaker to train predictive maintenance models across the aggregated fleet.

Batching pays off in exactly the dimension that was hurting. The number of files written per day stops tracking fleet size and starts tracking time buckets and partitions, so a fleet ten times larger still writes the same few hundred objects a day, Athena scans compact columnar files instead of listing millions of tiny JSON ones, and the request side of the S3 bill flattens with it. The lesson costs nothing to apply early: the prototype write path is the production bill, and nothing about it degrades gradually. The dashboards are fine, the dashboards are fine, then the refresh does not finish.

How to choose your own boundaries

Work through the layers in order, because each answer constrains the next.

1. Start with the contract, not the architecture. What you have promised customers about data separation determines the data-layer boundary, and nothing else does.
2. Then look at the supply chain. If devices are manufactured, warehoused or resold before a customer is known, pool ingestion and bind tenants at runtime, or you will re-provision hardware for the life of the product.
3. Make the production data boundary an AWS account. It is the only boundary you can hand a security reviewer without asking them to trust your code.
4. Split application and data accounts per tenant. It costs little beyond account management and keeps an application bug away from the archive.
5. Price non-production honestly. Pooling it is a genuine saving. Write down that you have given up isolation testing below production, and agree where that testing happens instead.
6. Decide the archive's file layout before the fleet is real. Every reading the fleet produces will be queried from S3 for the life of the platform. Batch readings into larger columnar files from the start, or book the migration now while the dataset is small.

Where to start on your own platform

Take the four layers and write down, for each one, whether your tenants share it today and whether that was a decision or an accident. The isolation you can demonstrate to a customer is the isolation somebody chose on purpose; the rest is convention, and convention holds right up until a device, a query or an IAM role does something nobody expected.