IoT Platform / Data Pipeline
The IoT data pipeline
from first byte to boardroom chart
Ingestion and decoding, time-series storage with a retention policy that matches the product, hot and cold paths that do different jobs, aggregation into events people talk about, alerting worth acting on, and export into whatever analytics stack your business already runs.
Six Stages
A message becomes
something worth looking at.
Most connected products die somewhere between the device and the dashboard. The device sends data, something writes it into a table, and eighteen months later nobody can answer a simple question about a fleet because every query scans years of raw rows and nothing is joined to anything.
The pipeline is designed backwards from the questions the business will ask. What has to be visible within seconds, what has to be answerable across three years, what has to reach an operator as an alert, and what has to land in the warehouse next to the billing data. Those answers set the storage model, the retention policy, and the aggregate layer.
Accept the raw message
Time-series, partitioned by time
Rollups and derived events
Live views and alerts
Ingest
Endpoints that accept whatever the fleet speaks: MQTT sessions, HTTP batches, or a raw TCP listener for a binary protocol. The ingest tier does as little as possible, because it has to stay up when everything downstream is having a bad day.
Decode
A versioned decoder turns the payload into named fields. Decoders are versioned because firmware is versioned, and a fleet always has more than one firmware version live at once.
Validate and Enrich
The message is checked against what the device is allowed to send, then joined to the registry so it carries product, variant, lot, site, and customer. Enrichment at write time is what makes later queries simple.
Store
Written to time-series storage partitioned by time and keyed by device. Retention and downsampling policy is applied per product, since a lift and a battery sensor do not need the same resolution kept for the same length of time.
Fork
The same record goes down a hot path for live views and alert evaluation and a cold path for history. The fork is deliberate, because low latency and cheap long-range scanning are opposing requirements.
Serve
Dashboards, an API for your own applications, alert delivery, and scheduled exports. This is the only stage the rest of the business sees, and everything upstream exists to make it correct.
Hot Path and Cold Path
Two paths, because
one store cannot do both jobs.
A live fleet screen needs the last reading from ten thousand devices in under a second. An annual failure analysis needs to scan three years of samples across a lot. Tuning one store for both means it does neither well, so the write path forks and each side is sized for its own query pattern.
Hot Path
Seconds to hoursLatest state per device, live dashboards, alert evaluation, and command feedback. Sized for read latency and for the number of devices, not the volume of history.
Latest reading per device held ready to read
Threshold and rule evaluation as records arrive
Live fleet views and site status boards
Connectivity state and last-seen tracking
Cold Path
Days to yearsFull history at whatever resolution the retention policy keeps. Sized for storage cost and for scanning long ranges, and it is where every question about trends and failures gets answered.
Downsampled rollups for long-range charts
Failure analysis across lots and firmware versions
Warranty, compliance, and contractual reporting
Feature extraction for models and predictive work
Tell us the questions your team needs answered. We will design the storage and retention model backwards from them.
Book a Technical Session →Aggregation
Raw samples answer nothing.
Aggregates answer everything.
Nobody asks how many volts a unit reported at 14:32. They ask which sites had a fault this week, whether the new hardware revision runs hotter, and how many sessions a charge point completed last month. That gap between samples and questions is closed by the aggregate layer, and closing it at write time is what keeps dashboards fast.
- Minute, hour and day rollups
- Fleet rollups by product and lot
- Raw samples turned into events
- State transitions stored discretely
- Continuous aggregates kept current
- Baselines for what normal looks like
Time Rollups
Minute, hour, and day summaries with min, max, mean, and count. A year of hourly rollups renders instantly where a year of raw samples would not render at all.
Fleet Rollups
The same metrics aggregated by product, variant, lot, site, region, and customer, so a question about a hardware revision does not require scanning every serial.
Derived Events
Raw samples turned into things people talk about: a charge session, a lift trip, a door cycle, a journey. Events are what dashboards and reports should be built on.
State Transitions
Online to offline, healthy to faulted, firmware moved. Stored as discrete transitions rather than inferred from raw data at query time.
Continuous Aggregates
Rollups maintained incrementally as data lands, and recomputed for the affected window when a backlog arrives late. No nightly batch that is already stale by morning.
Baselines
What normal looks like for this device, this site, and this variant. Baselines are what let an alert say something is unusual rather than only that a number crossed a line.
Dashboards and Alerting
An alert nobody acts on
is worse than no alert.
Every fleet goes through the same cycle. Alerting is switched on with generous thresholds, the team gets four hundred notifications in a week, and within a month everyone has muted the channel. From then on the platform is technically monitoring a fleet that nobody is watching.
The fix is treating alert design as product design. Each alert has an owner, a defined action, and a review. If nobody can say what they would do when it fires, it does not ship.
Thresholds per product, since normal differs by device type
Sustained conditions rather than single-sample triggers
Deduplication so one fault raises one alert
Suppression while a device is in a known outage or a rollout
Routing by site, customer, region, and severity
Escalation when nobody acknowledges inside the window
Every alert linked to the device record and its recent history
Alert effectiveness reviewed, and useless alerts deleted
For rule design, delivery channels, and escalation in more depth, see our alerts and notification engine page.
Export and Integration
Device data belongs
next to your business data.
Telemetry becomes valuable when it is joined to contracts, invoices, service records, and customers. Your finance team already has a warehouse and your analysts already have a BI tool. The pipeline is built to feed both rather than to compete with them. The routes out cover:
- A query API scoped by device, group, metric and time range
- Streaming into the warehouse you already run
- Scheduled exports in the shape a downstream team consumes
- Read access to the aggregate layer for your BI tool
- Webhooks so a fault opens a ticket without anyone watching
- Bulk history extracts, so your data stays portable
Query API
A read API over both paths, scoped by device, group, metric, and time range, so your own applications and customer portals build directly on the same data.
Warehouse Streaming
Records streamed into your warehouse so device telemetry joins CRM, billing, and service data. This is where operational data starts answering commercial questions.
Scheduled Exports
Recurring extracts in the shape a downstream team already consumes, delivered to object storage or an endpoint, with schema versioning so a change does not break their pipeline.
BI Connections
Read access to the aggregate layer for the BI tool your analysts already use, rather than asking them to learn another dashboard product.
Webhooks and Streams
Events pushed to your systems as they happen, so a fault opens a ticket in your service desk without anyone watching a screen.
Bulk History
Full history extracts for modelling, audits, and migrations. Your data stays yours and it stays portable, which matters more than it sounds when a fleet outlives a contract.
FAQ
Data questions
worth settling first.
Ingest the raw message, decode it into named fields, validate and enrich it against the device registry, write it to time-series storage, fork it into a hot path for live views and alerting and a cold path for history, roll it up into aggregates, and expose it through dashboards, an API, and scheduled exports.
Related
Go deeper.
Design the pipeline
before the data piles up.
Whether you are choosing a storage model for a new product or trying to make three years of existing telemetry useful, bring us the message shape and the questions. We will work the rest.