Multiple neuromorphic chips, one grid, no new middleware

Author: Nikunj Kotecha

A working cluster that manages BrainChip Akida devices with IBM Spectrum Symphony, and what it suggests about where neuromorphic silicon fits in the enterprise.

A commodity workstation in our lab holds eight Akida AKD1500 devices on PCIe, each of them visible to the operating system as an ordinary device. The boards are available from our shop and the host is unremarkable.

The interesting part is the software above them. The fleet is managed by IBM Spectrum Symphony, a workload management solution that financial institutions have relied on for two decades to distribute millions of short calculations across shared compute. Symphony discovers the nodes, selects which of them receive work, dispatches inference tasks, recovers from failures, and returns the capacity when a job completes. Every device executes its model on silicon, with no simulator and no CPU fallback.

The whole system is published as a public repository. It can be cloned, built from public sources as a container image, and brought up on your own Akida hardware.

Figure 1: Where Akida sits in the stack. The grid’s own layers run above, each compute node carries one AKD1500 below, and the device is reported to Symphony through the same per-host agents that already publish conventional load.

The grid’s own layers run above, each compute node carries one AKD1500 below, and the device is reported to Symphony through the same per-host agents that already publish conventional load.

Akida is an edge processor and grids run in data centers, so the pairing deserves an explanation. The reason is that both are built around the same shape of work.

Symphony exists to serve one class of problem: very large numbers of small, independent calculations, each of which has to complete quickly. A bank revaluing a portfolio breaks the job into hundreds of thousands of separate pricings that each take milliseconds, and the result is available only once every one of them has returned. This is not a batch workload. It calls for a workload manager that can place an individual unit of work for considerably less than that unit costs to execute.

Event-based processors are strongest under exactly those conditions: a single input, answered immediately, with no wait for a batch to accumulate. A Symphony task carries one input, so there is no batching layer to accommodate.

The mechanism behind Akida’s efficiency is worth setting out, because it is more specific than the phrase “low power” conveys.

Trained networks contain a high proportion of zeros as an ordinary consequence of how they are built. ReLU, the activation function used almost universally, maps every negative value to exactly zero. In a typical network roughly half of all activations are zero on any given input, and training with an activity regularizer, which penalizes activation magnitude in the loss function, raises that proportion beyond seventy per cent. None of it costs accuracy. The question is whether the hardware can avoid spending energy on those zeros.

Conventional accelerators can, but only by adding work in order to do so. A dense array multiplies whatever occupies each cell, so a zero consumes the same cycle as any other value. Recovering the saving requires the zeros to be identified first, whether by testing operands, compressing them out of the data, or maintaining index structures alongside it, and that identification consumes resources of its own.

Akida performs convolution as an event-based operation. Rather than traversing every position in the output and multiplying whatever it finds there, the device traverses the non-zero inputs and projects each contribution forward. A zero is never reached, so no mechanism is needed to identify or bypass it.

Figure 2: Dense multiply-accumulate compared with event-based scatter. The zeros on the right are never reached.

The effect is that the arithmetic a layer performs tracks the number of events it generates, which is the count of activations that were not zero. Sparsity therefore translates into reduced computation and reduced energy, with no detection cost in between.

Bringing Akida under Symphony’s management required very little.

A workload manager can allocate only a resource it can observe, and for an Akida device three properties are sufficient: whether the device is present on the host, whether it has been claimed and is healthy, and which model the worker on that host currently holds. Those are enough for Symphony to separate the nodes that can accept work from those that cannot, and enough for an operator to route particular work to the nodes holding a particular model.

No modification to Symphony was necessary. The product has long provided a mechanism for describing resources it has no built-in knowledge of. An external load information manager publishes operator-defined metrics on a schedule into the same resource model that carries conventional host load, and resource groups then select hosts on those properties. The device becomes an input to placement decisions while the layers above it remain untouched.

Placing a model on the device is the work of BrainChip SDK MetaTF, which quantizes a trained Keras or PyTorch model, converts it to Akida’s event-domain representation, and loads it. If a network needs more room on the fabric than a single configuration provides, the Akida Engine Library steps it through successive passes, rearranging the fabric between them and asking nothing of the caller.

The repository ships three applications that run against the same cluster and the same container image. The first two show the difference an orchestrator makes. The third shows something an orchestrator makes possible.

Figure 3: The three applications in the repository.

The first two dispatch the same workload with and without the grid; the third divides a single high-resolution frame across six devices and merges the results.

Serial round-robin is the baseline, and it is deliberately unsophisticated. A dashboard addresses a plain HTTP inference server on each node, sending a single request at a time and rotating around the fleet. This is what a set of accelerators looks like without middleware: approximately one device is working at any instant, however many are installed.

Batch inference submits the same workload as tasks within a single Symphony SOAM session and lets the session manager distribute them. Every ready device works concurrently. The inference code is identical between the two applications. What changes is that placement, concurrency, failure handling and retry become the platform’s responsibility instead of the application’s.

Image sharding addresses a different question. Akida’s hardware input stage accepts images up to 256 pixels tall, so a 448 × 448 frame is expressed instead as six 224 × 224 tiles: the four quadrants, an overlapping center crop, and the whole frame resized. Three SOAM services carry the pipeline. The first segments the frame, the second runs the six tiles concurrently across six devices, and the third merges the detections into one result for the full frame. Large tensors travel through a shared directory, and only small references cross SOAM.

The merge is where the engineering sits. A detection straddling a tile boundary has to be reconciled rather than counted twice, and tile order forms part of the contract, because fusion needs to know which fragment came from which region. The sixth tile contributes a disproportionate share of the result, and it is the full set of six that lets the merge suppress the false positives individual tiles introduce.

Together the three applications make one point: an orchestrator keeps many devices busy on independent work, and it also lets several devices serve a single item of work.

Several directions look worth pursuing, and we would be glad to know which of them matter to you.

Akida alongside GPUs rather than instead of them. An event-based device can run a continuous and inexpensive screening stage, engaging a GPU only for the small proportion of inputs that warrant more expensive analysis. Published work has already demonstrated the pairing on a single machine. Expressing it as two resource classes within one grid is the natural next step.

Financial services. Symphony’s established territory. Classification on streaming market data, anomaly and fraud screening, and risk pipelines are all cases where the latency of one item counts for more than total throughput.

Edge and distributed autonomy. Software that manages a rack can also manage a dispersed fleet. Drone swarms, distributed sensing and defense applications share a common shape: many independent nodes making rapid local decisions within a constrained power budget, requiring coordination without a high-bandwidth link back to a data center.

Cost and energy per unit. For a workload composed of small independent inferences, the relevant question is not peak throughput but how much silicon and how many watts are needed to sustain a given service level. That is a question better answered against your workload than against ours.

The repository is public, and so is everything it depends on. Symphony comes from the container image IBM publishes and the Akida runtime from its standard package index: no private registry, no vendor credentials, no support ticket. The cluster runs on IBM Spectrum Symphony Community Edition, which is free and provides the full functionality of the licensed editions within a 64-core ceiling. The image verifies itself at the end of the build, so a build either completes correctly or reports precisely what is missing.

Repository: github.com/Brainchip-Inc/symphony-akida Technical paper: BrainChip Akida as a Managed Resource in IBM Spectrum Symphony

If you operate a Symphony grid and want to understand what an Akida tier would contribute to it, or if you have a workload of many small, latency-sensitive inferences to serve efficiently one item at a time, we would like to talk.

IBM and IBM Spectrum Symphony are trademarks of International Business Machines Corporation. Akida, BrainChip and MetaTF are trademarks of BrainChip Holdings Ltd. This work uses IBM Spectrum Symphony Community Edition.