By Steve Brightfield, Chief Product Officer

Defense sensing, industrial monitoring, and always-on workloads in a heterogeneous cluster

Part 1 of this series got Akida hardware running in a managed pool under a workload scheduler. Part 2 asks where that pays off. One answer is anywhere a system must watch continuous streams under tight limits on power, size or connectivity: defense sensing, industrial monitoring, and many other always-on jobs, including finance.

What a cluster needs is an orchestrator that describes a pool of Akida devices as a resource group, dispatches each workload to the right group, and reports utilization. Moving work around a modern server cluster is no longer the slow part. On a tuned InfiniBand or RDMA-capable Ethernet fabric, node-to-node latency is roughly 1 to 2.5 microseconds, down from 15 to 50 or more a decade ago. A private on-premises cluster is also steadier, because no other tenants share the network.

Getting a model ready to run is the slow part. A GPU that already holds a small model in memory answers in roughly 1 to 5 milliseconds, and most of that is the inference itself. Load a different model onto an idle GPU and the wait grows to roughly 2 to 10 seconds, because the weights must come off storage and the framework must initialize. Start a service on a node that was never provisioned, and it takes anywhere from 30 seconds to a couple of minutes to pull containers and register drivers before the model load even begins. These are typical figures and vary with hardware and software stack, but demonstrate that dynamic loading requirements can impact latency and utilization.

Set those against the work itself. Take a small anomaly-detection or signal-classification model that finishes in fractions of milliseconds on-chip. Sending the request costs microseconds, which is negligible. A model swap costs thousands of times more than the inference, and a node start-up costs over a hundred thousand times more.

That is why model agility and GPU efficiency pull against each other. Every time a GPU changes models, it spends seconds loading instead of computing, so operators pin models in memory and keep replicas pre-warmed. A pinned GPU can be reserved for one small job but may sit mostly idle between requests. Moving small, steady models to a separate pool is one way out of that trade-off, and whether Akida’s quick model-load time helps your workload is another number to measure.

Real pipelines are already a mix of processors, so the clean way to add Akida is as a third engine, not a replacement for the other two.

• CPU: coordination, sequencing, control logic and records.
• GPU: dense floating-point work, such as model training, heavy simulation and sensor fusion.
• Akida: sparse, event-driven inference on continuous streams, at low power. It does work when an event arrives rather than on every cycle.

Eight workloads are good examples of a fit for an Akida pool complementing a GPU. Three match a workload class already demonstrated in the public repo, and the rest are candidates that need your own models and data.

Table 1. Candidate workloads by sector

BrainChip has demonstrations of radar processing algorithms on off-the-shelf Akida hardware, including decoding communication and radar waveforms from raw in-phase and quadrature data. The target is radar platforms with severe size, weight, power and cost limits, such as drones, robots, aircraft and satellites.

In a heterogeneous cluster, the pattern is simple. An Akida pool near the sensor, in a vehicle, ship or ground station, screens every stream continuously at low power. When it flags something, the orchestrator cues a GPU for the heavy work: tracking, multi-sensor fusion or retraining. If the link to a larger site is limited, the pool keeps screening locally and the GPU work waits.

One boundary is worth stating. A scheduled service call does not belong inside a hard real-time control loop. Use the pool to screen and cue, not to close a loop.

A plant may have thousands of vibration, temperature and flow channels, almost all of them uneventful. Scoring every channel on a GPU means pinned models drawing power for streams that rarely change. The AKD1500 is event-based and takes advantage of activation sparsity, so a pool can score every channel continuously and escalate only the anomalies.

There is also a governance catch. A model that changes itself in the field is a model change, and defense programs, industrial safety teams and other regulated settings validate and control those. One design that respects that: run learning in a separate shadow resource group, log every learned update, and promote it to the scoring pool only through your normal validation and change process.

Part 1 explains why. A small anomaly detection model runs 0.25 ms on-chip, so dispatch overhead dominates and the gain from fanning out is modest. A small model’s end-to-end time is set by the scheduler, the PCIe hop and the service call, not by the silicon.

The fit is results needed within milliseconds on a stream that never stops, such as alerting, screening and cueing. One claim worth testing rather than assuming is steadier latency under bursty load. Measure the 99th and 99.9th percentile, not the average, because that is where tail jitter shows up.

BrainChip also publishes energy per inference measured on an AKD1500 for models in its zoo. Table 2 picks the ones closest to the workloads above.

Table 2. Energy per inference measured on an AKD1500 (BrainChip model zoo)

At 10 inferences per second, that is about 0.7 mW for keyword spotting and about 146 mW for person and car detection, before host and board power. We quote no multiple against GPUs, because we have not measured one on these workloads. The case for offloading steady, always-on inference is still easy to test: put the same screening model on both pools, measure watts at the wall for a fixed number of events, and let the numbers decide.

Clone symphony-akida, bring up the batch-inference app on real Akida hardware, and put three numbers beside your current pipeline:

1. End-to-end 99th percentile latency, with dispatch included, as the pool grows from one chip to seven.
2. Throughput per chip, to see where scaling flattens for your event size.
3. Watts at the wall per thousand events, measured the same way on every pool you compare.

Hardware is on the BrainChip Shop. The AKD1500 uses a PCIe Gen2 interface, and the Part 1 repo also supports AKD1000 PCIe and M.2 cards. The next post in the series goes deeper on the model side: compiling your own models for Akida with MetaTF, and how on-chip learning behaves in practice.

Stay tuned to the next in the series: our vision of heterogeneous computing and its impact on today’s world, posted every Thursday.