A device sends a telemetry packet. The server acknowledges it. A dashboard displays the device as online. Everything appears to be working.

Now suppose the gateway crashes between receiving the packet and writing it to durable storage. Or an older position arrives after a newer one and overwrites the latest state. Or a replay duplicates an event in the historical database.

These failures look different operationally, but they have a common source: message delivery, durable acceptance, historical persistence, and correct device state are not the same guarantee.

This article examines how to specify those guarantees separately and how to test them in an industrial IoT pipeline. The reference architecture comes from Sofia Parser, a Rootcastle engineering project built around a protocol-aware gateway, a Kafka-compatible event log, stream normalization, Redis, and ClickHouse. The proposed 10,000-to-50,000-device experiments are evaluation targets, not completed benchmark results.

The problem with calling a pipeline "reliable"

A telemetry event passes through several independently failing components:

Device / tracker
    |
    v
Protocol-aware TCP gateway
    |    Device ACK must follow durable publish
    v
Kafka-compatible durable event log (Redpanda)
    |
    v
At-least-once normalizer
    |---------------------------|
    v                           v
ClickHouse history           Redis latest state
    |                           |
    v                           v
Audit / reconciliation       Live dashboard

The gateway can accept a packet while the broker is unavailable. The broker can retain a packet while the normalizer is restarting. A normalizer can successfully update one sink and fail to update the other. A replay can be necessary for recovery but dangerous if consumers are not idempotent.

Consequently, I separate reliability into five testable properties:

1. Acknowledgment safety: an acknowledged event has crossed the declared durable-ingestion boundary.
2. Accountable disposition: each accepted event is eventually persisted, explicitly quarantined, or still reported as unresolved; it must not disappear silently.
3. Logical idempotence: processing the same event again does not create another logical history event or transition.
4. Non-regressing state: delayed or replayed events cannot move a stored state backward under the selected ordering rule.
5. Cross-sink convergence: after a recoverable fault, the history and live-state materializations can be checked and reconciled.

None of these properties follows from throughput alone. A system processing thousands of packets per second may still produce an incorrect dashboard.

1. A device ACK is a commitment, not a courtesy

Consider this sequence:

T0  Device sends event E42
T1  Gateway receives E42
T2  Gateway replies ACK
T3  Gateway process crashes
T4  E42 was never written to the event log

The device may delete E42 from its local queue because it has already received the acknowledgment. From the device's perspective, delivery succeeded. From the server's perspective, there is no durable record.

The minimum useful safety condition is:

ACK(event) => DurableLogCommit(event)

For a native gateway, the publish operation should complete successfully under its configured broker acknowledgment and replication policy before the gateway sends the protocol ACK:

// Illustrative control flow; not a complete device protocol implementation.
if err := publisher.Publish(ctx, rawEnvelope); err != nil {
    // No successful broker publish: do not acknowledge the device.
    return err
}

return sendProtocolACK(conn, packetID)

Sofia Parser's documented native gateway follows this ordering and requests broker acknowledgment with acks=all. However, acks=all must be interpreted with the actual replication factor, in-sync replica configuration, persistence policy, and failure model. A single-broker development stack does not establish multi-node fault tolerance.

There is also a non-obvious boundary: a legacy device-protocol runtime may send its device ACK independently of a downstream forwarding bridge. Such a path does not inherit the native gateway's broker-before-ACK guarantee. Protocol coverage and acknowledgment safety must be certified independently.

2. At-least-once processing requires stable event identity

After an uncertain network failure, a device may retransmit a message even if its first copy was durably stored. A consumer may also replay records after a crash. Both behaviors are normal in an at-least-once pipeline.

The resulting question is not "How do we prevent every duplicate transmission?" It is "How do we prevent duplicate transmissions from changing the logical result?"

A stable event identifier should satisfy two different conditions:

• A retransmission of the same observation produces the same identity.
• Two different observations do not collapse into one identity, even if their payload values are equal.

A hash of raw packet bytes can identify byte-identical retries. But if a tracker emits the same coordinates, sensor reading, and timestamp for two legitimate measurements, raw-byte identity alone may merge them incorrectly. Where device firmware supports it, a device sequence and boot/session epoch provide a stronger identity contract.

Sofia Parser's existing deterministic packet and event IDs support replay-aware processing. A device-epoch-plus-sequence identity is a proposed research extension, not a production capability being claimed here.

Downstream processing can then use idempotent writes and duplicate-aware queries. A replacing-history table does not, by itself, guarantee that every read sees a deduplicated view immediately. Logical uniqueness must be checked against an independent event ledger.

3. Monotonic state is not the same as physically correct state

Suppose a tracker sends these events:

A naive "last arrival wins" update overwrites Point B with Point A. The live map moves backward even though the replay contains no newer observation.

An atomic comparison of a declared event rank can prevent that kind of regression:

new_state = max(current_state, candidate_state, by = event_rank)

A rank may combine event timestamp, sequence presence/value, record index, receive timestamp, and deterministic event ID. The comparison must be atomic: concurrent consumers should not be able to overwrite a newer rank with an older one.

But there is an important counterexample. A device with a badly configured clock may send an event timestamped hours into the future. A monotonic ranking rule could accept that event and then reject legitimate subsequent observations for hours. The ranking invariant would hold, but the physical state would be stale.

Therefore, the experiment must measure two outcomes separately:

• Rank regression: Did the stored rank decrease?
• Physical recency error: Does the displayed event represent the latest valid observation according to an independent ground-truth sequence?

A system that passes the first test has not automatically passed the second. Clock skew, rebooted sequence counters, session epochs, and missing timestamps are part of the correctness contract rather than incidental data-quality problems.

4. Protocol counts do not equal certified protocol support

The GitHub-inspected Sofia Parser HW2 compatibility catalogue registers 272 protocol names. The registration classification is:

The 36 direct registrations correspond to 29 distinct concrete implementation types. The remaining alias entries must not be presented as independently implemented and validated vendor decoders.

This distinction matters for reliability. Supporting a protocol name in a parser registry does not prove that a specific device firmware sends correct acknowledgments, handles fragmented TCP frames, or retries safely after broker failure.

Figure to insert here: Sofia Parser HW2 protocol-registration distribution (source-inspected repository data). Upload the companion PNG in Hashnode after pasting this article.

The evidence needed for protocol certification includes captured and sanitized packet fixtures, CRC/checksum tests, frame fragmentation and coalescing, authentication where supported, specific ACK bytes, reconnect/replay behavior, and negative cases. An internal simulator can validate software contracts, but it cannot substitute for real-device protocol testing.

5. Scaling begins with an offered-load equation

Connection count alone is a misleading performance metric. A fleet of 50,000 connected devices reporting once a minute presents a different workload from the same fleet reporting every ten seconds.

For a simple periodic model:

events_per_second = active_devices / report_interval_seconds

These are arithmetic offered-load estimates, not measured Sofia throughput. Real fleets introduce bursty reconnects, synchronized reporting, differing payload sizes, TLS overhead, variable device behavior, retries, and uneven per-tenant traffic.

Figure to insert here: Calculated offered load under 10-second and 60-second reporting intervals. Upload the companion PNG in Hashnode after pasting this article.

A benchmark should therefore report connection concurrency and event throughput separately. The Sofia operations plan identifies 10,000, 25,000, and 50,000 connections, a sustained workload of 5,000 events/s, a 25,000-events/s burst scenario, and a 72-hour soak as test targets. These targets have not been established as achieved results.

6. Fault injection is the experiment, not an afterthought

A baseline test with no failures reveals little about data integrity. The study should compare controlled architectures under the same event ledger, hardware, payloads, and fault schedule.

The minimal comparisons are:

These variants isolate what each design mechanism contributes. They do not assume that the strongest design will be fastest.

Fault scenarios should include gateway death before/after publication, broker leader loss, insufficient in-sync replicas, timeout immediately before an ACK, consumer crash after updating one sink, Redis and ClickHouse outages, replay after offset rewind, clock skew, sequence reset, and identical-payload distinct observations.

Each run needs an independent oracle to compare four records: what the generator intended to send, what the device was told was accepted, what the raw log committed, and what the history and latest-state stores expose. Without this independent ground truth, the system can incorrectly grade its own results.

7. Metrics that actually answer the question

For each test condition, measure:

• ACK safety violations: acknowledged logical events absent from the committed raw log within the declared observation window.
• Unresolved accepted events: events with no persisted or quarantined outcome after the allowed recovery window.
• Logical history duplicates: extra visible logical effects after replay, not just repeated physical insert attempts.
• State-rank regressions: transitions to a lower declared rank.
• Physical recency errors: differences from the independent valid device sequence.
• Tail latency: ACK and end-to-live p95/p99, with explicit measurement points.
• Recovery time: time to resume processing and reconcile history and live state.
• Cost under load: CPU, memory, network, broker lag, open connections, and backpressure.

Zero observed violations would be meaningful only within the tested failure model and sample size. It would not prove that zero loss is possible under every real-world failure. Conversely, a counterexample would identify which assumption or component boundary must change.

The test report should preserve source revision, image digests, broker settings, workload seed, fault schedule, event IDs, and verification scripts. A run with missing evidence should be marked NOT VERIFIED, not PASS.

8. What would count as a useful result?

A positive finding would be evidence that postponing ACKs eliminates the tested ACK-before-commit failures, while stable identities and ordered updates reduce measured duplicate effects and state-rank regressions. A negative finding would be equally informative if it isolates a capacity bottleneck, an unfair retry policy, an incorrect clock assumption, or a compatibility protocol that acknowledges too early.

There are unavoidable trade-offs. Withholding acknowledgments during broker quorum loss protects an acceptance contract but reduces availability. Idempotent sinks may require extra state or reconciliation work. Atomic latest-state updates may increase latency under contention. These costs must be measured rather than hidden behind a single throughput figure.

The objective is not to claim "exactly once" for an entire heterogeneous pipeline. It is to specify which operation is performed once, which operation may be retried, what remains observable, and which failure assumptions are required.

Conclusion

Industrial IoT reliability is an end-to-end, cross-layer property. A broker acknowledgment does not prove that a database has persisted an event. A non-regressing state rank does not prove that a device's physical state is correct. A protocol catalogue does not prove that every listed tracker is production-ready.

The practical engineering approach is to make those distinctions explicit, introduce a durable acceptance boundary, preserve stable logical identities, prevent stale state transitions, and run controlled fault injection against independent event truth. Only then can connection count, event rate, and recovery time be interpreted as evidence of a system's reliability at scale.

For Rootcastle, the next step is a reproducible fault-injection study of Sofia Parser's native and compatibility ingestion paths, including multi-broker operation and real-device fixtures. The published design is a research protocol; system-wide 50,000-device capacity and zero-loss operation remain unverified until the planned tests are executed.

Further reading

• Saltzer, Reed & Clark (1984), End-to-End Arguments in System Design.
• Lamport (1978), Time, Clocks, and the Ordering of Events in a Distributed System.
• Akidau et al. (2015), The Dataflow Model.
• Fragkoulis et al. (2024), A Survey on the Evolution of Stream Processing Systems.
• Apache Kafka: KIP-98 Exactly Once Delivery and Transactional Messaging.
• Apache Flink: Fault Tolerance.

Research context: Rootcastle Engineering & Innovation. Source-inspected code and engineering documents: Sofia Parser, revision 4e7e667 (11 November 2026); the repository is private.

A valid coordinate is only one part of a telemetry event. A useful ingestion contract also states the authenticated device identity, the protocol message identity, event time, receive time, position quality and the version of the parser that produced the record. These fields answer different questions. Receive time describes the server; event time describes a device observation. Neither timestamp should silently replace the other.

This walkthrough describes an architecture pattern. It does not disclose or claim to demonstrate a private REI Track deployment. The public Rootcastle engineering portfolio and public protocol repositories provide separate entry points for inspecting available work.

Frame before parsing a business record

TCP is a byte stream. One socket read may contain half a frame, several frames or the end of one frame followed by another. Maintain a bounded receive buffer, extract complete protocol frames and validate their length and checksum before asking the domain parser to interpret coordinates. UDP preserves datagram boundaries, but a datagram still needs protocol validation.

Represent incomplete data separately from invalid data. An incomplete frame may become valid when more bytes arrive. An invalid frame needs a bounded resynchronization policy or rejection. A maximum frame length and an idle timeout keep a malformed connection from growing memory indefinitely. Use a synthetic frame corpus to test those paths without sending customer telemetry to a third-party service.

Make duplicate policy a domain decision

Retries and reconnects can repeat messages. A protocol may carry a sequence number, but counters can wrap or reset after a reboot. A deduplication key therefore needs documented scope: device, boot or session identity where available, message type and sequence. If the protocol lacks a stable message identity, describe the limitations of a time-and-payload fingerprint instead of promising exactly-once delivery.

The following toy example illustrates a device-scoped key. Its in-memory set has no durability or expiry and is not a production deduplication store.

seen = set()
def accept(event):
    key = (event['device'], event['session'], event['sequence'])
    if key in seen:
        return False
    seen.add(key)
    return True

a = {'device': 'synthetic-a', 'session': 'boot-1', 'sequence': 7}
b = {'device': 'synthetic-b', 'session': 'boot-1', 'sequence': 7}
assert accept(a)
assert not accept(a)
assert accept(b)

A real service must bound retention, handle concurrent workers and preserve the chosen identity across restarts. An acknowledgement should follow the protocol contract and the intended durability point. Acknowledging before durable acceptance can lose an event; acknowledging too late can provoke unnecessary retries.

Preserve time and quality instead of repairing silently

Keep event and receive timestamps, including an explicit UTC interpretation. Record invalid or absent GNSS fixes as quality states. Do not convert a missing fix into the coordinate zero, zero. Similarly, a late packet should not automatically move a live vehicle marker backward in time. Separate storage of late historical events from the policy that updates the latest-position view.

Validate latitude and longitude ranges, units, hemisphere signs, fix flags and speed semantics against the selected protocol. A parser version belongs with the normalized event so that an interpretation change can be traced later.

Replay is an acceptance test

A useful fixture set includes split frames, concatenated frames, invalid lengths, bad checksums, reconnect duplicates, sequence wrap, device reboot, late data and absent fixes. Save the expected normalized events and rejection reasons. Replay the same corpus through a parser change and compare semantic outputs, not only whether the process stayed alive.

Track bounded queue depth, validation failures, processing latency and duplicate rate without putting raw device identifiers or payloads into general application logs. Detailed evidence belongs in a permissioned, retention-controlled store.

Checklist and public starting points

• Define the event identity and the durability point before implementing acknowledgements.
• Test stream fragmentation independently from domain parsing.
• Keep event time, receive time and GNSS quality distinct.
• Bound buffers, retries, retention and diagnostic payload storage.
• Version fixtures and compare normalized outputs before rollout.

For inspectable protocol work, see the public JT808 parser repository. A repository's current README defines its actual scope; the presence of a parser does not establish a complete commercial fleet platform. For integration requirements, describe your system to Rootcastle.

Originally published at Rootcastle Engineering & Innovation.

Explore the engineering tools · Public profiles and publications