TENNs: It’s about Time! Unlocking the Power of Efficient Processing for Sequential Data (Video and Time Series Data)
By Olivier Coenen, Kristofor Carlson, and Peter van der Made
Artificial Intelligence (AI) has made remarkable progress since the advent of Artificial Neural Networks (ANNs) over 50 years ago. However, as AI workflows increasingly rely on spatiotemporal data, the limitations of traditional approaches such as Convolutional Neural Networks (CNNs) have become evident. CNNs excel at processing spatial information in images but struggle with effectively utilizing temporal information. In this blog post, we explore a groundbreaking solution called Temporal Event-based Neural Networks (TENNs) developed by BrainChip, which efficiently combines spatial and temporal convolutions to process sequential data like never before.
The Challenge of Spatiotemporal Data
While CNNs have been the backbone of image classification for the past decade, they fall short when it comes to effectively encoding and processing spatiotemporal information. The two prevalent approaches to tackle this challenge are incorporating an internal state with temporal dynamics (as seen in recurrent neural networks) or computing a temporal convolution of a kernel over past inputs. However, each approach has its limitations, prompting the need for a more efficient solution.
Recurrent Neural Networks (RNNs) provide an internal state with temporal dynamics, allowing them to capture sequential dependencies. However, RNNs have their own challenges, such as vanishing gradients and difficulty in parallelization. Temporal Convolution involves computing a temporal convolution of a kernel over past inputs. While it can capture temporal correlations, it often requires extensive memory resources and can be power hungry at inference time for the Edge.
The Emergence of Transformers
Transformer models, which learn contextual relationships in sequential data, were initially considered a promising alternative to RNNs. Their ability to capture long-range dependencies and parallelize computations made them an efficient choice for many applications in the cloud, yet, typically too power-hungry and data-hungry for the Edge. However, recent developments have shown that a new class of RNNs has surpassed transformer networks in performance for certain tasks to be performed at
the Edge.
Introducing Temporal Event-based Neural Networks (TENNs)
BrainChip, with its expertise in Event-Based Processing and Spiking Neural Networks, has developed a groundbreaking solution to address the limitations of existing approaches. The 2nd Generation Akida platform includes support for Temporal Event-based Neural Networks (TENNs), which combine spatial and temporal convolutions to efficiently process sequential spatiotemporal data.
TENNs leverage the strengths of both CNNs and RNNs by incorporating temporal and spatial convolution layers throughout the network. Unlike traditional CNNs that focus solely on spatial dimensions, TENNs integrate the spatial and temporal characteristics of the data, enabling them to learn spatial and temporal correlations effectively. This distinguishes TENNs from state-space models that primarily deal with time series data lacking spatial components.
Flexibility and Configurability
TENNs are spatiotemporal networks that can be configured to operate either in temporal convolution mode or in a recurrent mode. This flexibility allows researchers and practitioners to adapt the network to the specific requirements of their applications. Furthermore, TENNs benefit from efficient training on parallel hardware, potentially in the cloud with GPUs and TPUs like convolutional networks, while maintaining the compactness of RNNs for inference at the Edge.
State-of-the-Art Performance and Versatility
TENNs have demonstrated state-of-the-art performance across various domains of sequential data, as highlighted in BrainChip’s recent white paper, “Temporal Event-based Neural Networks: A New Approach to Temporal Processing.” Notable achievements include Raw Audio Speech Classification on the 10-Class Speech Classification SC10 dataset, Vital Signs Prediction on the BIDMC dataset, 2D Object Detection on the KITTI Vision Benchmark Suite (frame-based camera video), and 2D Object Detection on the Event-Based Prophesee 1 Megapixel Automotive Detection Dataset.
The versatility of TENNs makes them suitable for a wide range of applications on sequential data, including speech recognition, medical equipment for patient monitoring, and streaming video object detection and tracking. Furthermore, TENNs offer superior performance with a fraction of the computational requirements and significantly fewer parameters compared to other network architectures. This efficiency makes them an elegant solution for highly accurate models that support
video and time series data at the Edge.
Enabling Intelligent Edge Solutions with TENNs on Brainchip 2nd Generation Akida
The groundbreaking capabilities of TENNs have far-reaching implications, particularly for intelligent Edge solutions. With their efficient training on parallel hardware like GPUs and TPUs, TENNs leverage the computational advantages of convolutional networks. Moreover, their compactness during inference enables efficient deployment at the Edge on BrainChip’s 2nd Generation Akida IP, catering to real-time and resource-constrained applications.
Conclusion
Temporal Event-based Neural Networks (TENNs) represent a significant advancement in processing sequential spatiotemporal data. By combining spatial and temporal convolutions, TENNs excel at capturing and exploiting the relationships within sequential data. With their impressive performance, low computational requirements, and wide-ranging applications, TENNs, running on BrainChip’s 2nd Generation Akida IP, are poised to unlock new possibilities in intelligent Edge solutions and empower various industries with highly accurate models for sequential data analysis.
Click here to read the White Paper for more details.
Olivier Coenen, Senior Research Scientist at Brainchip Inc., is a highly accomplished and pioneering researcher in the field of brain-inspired processing and modeling, with a focus on robotics and autonomous systems. He holds a Ph.D. in Physics/Biophysics from UC San Diego and The Salk Institute, as well as a B.Sc. in Honours Physics from McGill University. Olivier’s expertise spans neural networks, event-based processing, motor control systems, adaptive robotics, spiking neural networks and computational neuroscience. Throughout his career, Olivier has made significant contributions to various organizations and projects. He founded Qelzal Corporation (DBA Kelzal), where he served as CEO and CTO from 2014 to 2020. Under his leadership, Qelzal developed neuromorphic event-based vision and processing technologies for autonomous retail, drones, and autonomous vehicles. Olivier successfully raised millions in funding for the company. Olivier has demonstrated exceptional leadership and innovative thinking. He has successfully established research groups at Sony and Brain Corp, secured funding grants, and generated a series of patents. Notably, his work on developing artificial nervous systems for UAVs contributed to the closing of Series B funding with Qualcomm Ventures for Brain Corp. With his multidisciplinary knowledge, innovative thinking, and ability to anticipate challenges, Olivier has established himself as a leading figure in the field. His contributions have been widely recognized, earning him 16 academic awards and fellowships. Olivier is known for his ability to integrate knowledge from various disciplines, identify novel directions, and establish connections between seemingly unrelated areas, providing unique perspectives. With a strong background in physics, biophysics, neuroscience, and machine learning, Olivier continues to push the boundaries of brain-inspired processing and its applications in robotics and AI. Olivier is bilingual in English and French with a good basis in Spanish. He has a passion for activities like wing foiling, windsurfing, swimming, mountain biking, and photography.
Kristofor D. Carlson, PhD received his PhD in Physics from Purdue University. Kristofor spent four years as a postdoctoral scholar at UC Irvine where he studied spiking neural networks, evolutionary algorithms, and neuromorphic systems. Afterwards, he worked as a postdoctoral appointee at Sandia National Laboratories for two years where he studied uncertainty quantification and neuromorphic computing. Kristofor has worked at Brainchip as a research scientist for six years and has been Manager of Applied Research for the past two years. At BrainChip he focuses on developing and optimizing neuromorphic and machine learning algorithms for deployment on BrainChip’s latest neural network architectures.
Peter van der Made has been at the forefront of computer innovation for 50 years. Mr. van der Made is the inventor of a computer immune system. He founded vCIS Technology, serving as CTO and later Chief Scientist when Internet Security Systems and, subsequently, IBM acquired it. Previously, he founded PolyGraphic Systems and designed a high-resolution, high-speed color Graphics Accelerator board and subsequent chip for IBM PC graphics. Mr. van der Made published a book, Higher Intelligence, which describes the brain’s architecture from a computer science perspective. Mr. van der Made designed the first generations of digital neuromorphic devices on which the Akida chip is based between 2004 and 2008 when he applied for a patent on this technology which was subsequently granted. He is actively involved in designing the next generation of Akida chips and continues his research in advanced cortical neuromorphic architectures.
