[Blog] Sensor Fusion and FPGA Solutions for Real-Time Edge AI

The Growing Role of Distributed Sensors
As modern systems increase their reliance on data, dynamic artificial intelligence (AI), and machine learning (ML) models, the need for real-time processing at the edge grows more pressing by the day. Connectivity and capacity are required for all edge components and devices, including routers, gateways, and scanners. But they’re especially relevant for the various types of sensors that are deployed to measure and process data as close to the source as possible.

Many system developers use multiple sensors in tandem to combine lidar and camera capabilities in their operations, which requires a form of sensor fusion: the process of combining data from multiple sensors to output more complete and reliable information.

As sensors and edge AI models become more sophisticated, sensor fusion offers a way to elevate the precision and complexity of their insights. Whether used alone or in combination with one another, sensors are a foundational component of any operational distributed network. To implement sensors across these kinds of systems, developers need to understand their functions, the obstacles to adoption, and the role of Field Programmable Gate Arrays (FPGAs) in supporting operations.

Sensor Functions and Implementation Challenges
Sensors cover a range of functions in connected systems. They can be used to measure and monitor physical parameters—such as temperature, motion, and pressure—or data inputs from Internet of Things (IoT) devices and other system components. The real-time data these devices provide enable quality control initiatives, inventory management, technological automation, and other core functions.

Implementing sensors at the edge, however, is not without its challenges. Common obstacles to sensor applications include:

• High costs. Any new technology costs money, and a quarter of those surveyed (25%) cited cost as a primary challenge to sensor adoption. Given the diverse roles sensors play in distributed systems, their total price tag can add up quickly.
• Integration and interoperability issues. Be it in automotive design schematics or industrial production lines, adding new components to existing infrastructure is likely to involve integration issues. In fact, over a third (37.5%) of survey respondents cited integration difficulties as a significant obstacle. New sensors may not be interoperable with existing components right out of the box, and the extent of each system’s required upgrades is unique.
• Power and space limitations. Like any other device, sensors need to be slotted into the space available in existing infrastructure and driven by whatever power the system can spare. This can prove difficult when the space and power available are limited, like in a vehicle build, and requires that sensors and any supporting devices have small form factors and low power needs.
• Processing constraints. While processing at the edge has become increasingly sought-after, computing capacity remains a challenge. Can these sensors capture and analyze the data the system needs without drawing excessive amounts of power or overtaxing their hardware? Does the system have enough I/O to process without causing high latency?

Overcoming these challenges and integrating competent, reliable sensors requires developers to take an informed approach supported by reliable hardware.

FPGA Solutions for Sensor Fusion Applications
This is where FPGAs come into play. These flexible semiconductors can act as a “bridge” between sensors and other system components—like actuators and CPUs—taking on low-level and sensor-specific tasks to lighten the load on Edge components and support streamlined system operations.

Key FPGA capabilities that support sensor integration and sensor fusion include:

• Parallel processing. Rather than processing data in sequence, FPGAs are capable of completing processing tasks simultaneously. This is critical for sensor fusion applications, where data is generated and aggregated from multiple sensors at the same time. Parallel processing significantly increases task speeds and reduces latency, helping teams overcome common capacity and processing constraints.
• Low power consumption. Not only does parallel processing streamline computing capacity, it also helps to reduce overall system power consumption by processing data closer to the source.
• Customization and reprogrammability. Offering high I/O and significant opportunities for customization, FPGAs help teams ensure that sensors are integrated successfully and updated as necessary. This helps surmount interoperability challenges at the front end and keep their hardware up to date with the latest functionality through field-programmability.

Lattice offers semiconductors that are geared to support sensor fusion applications, including a range of FPGAs built on the Lattice Avant™, Lattice Nexus™ 2, and Lattice Nexus™ FPGA platforms. Lattice Avant based FPGAs are well-suited for efficient edge processing use cases, and the Lattice Nexus 2 and Nexus based FPGAs provide best-in-class processing for vision processing from cameras and similar sensors.

Sensor Fusion for Autonomous Robots and Vehicles
The Avant-E FPGA is a key component of Lattice’s Sensor Hub solution, which is specifically designed to support preprocessing and interfacing in sensor fusion applications. By combining a camera, rotating lidar sensor, solid state lidar sensor, and radar sensor, developers can gather a range of critical environmental data and combine them into a single, unified output. The fusion of this data is supported by the I/O and parallel processing capabilities of the Avant-E FPGA and Lattice developer board, which combine the various signals and provide a single stream of real-time output.

Sensor fusion can be combined with AI/ML human detection models and applied to the operation of autonomous vehicles as well as robotics applications. These applications need to have a complete real-time understanding of their environment—including pedestrians, factory workers, other vehicles, machines, and more—to operate independently. The system must understand the distance of each obstacle from the respective sensor, the depth of field, and where both are located on an x/y plane to react accordingly.

The Avant-E FPGA acts as the “bridge” between the various input sensors and the CPU, ingesting and preprocessing the raw radar, lidar, and camera data in tandem and outputting only the scaled-down data that the human detection model needs to identify potential obstacles. This reduces the strain on the CPU, the power required to process this data, and the potential latency between data ingestion and system response—all in a small form factor component that can easily slot into existing automotive infrastructure.

When delayed processing could be the difference between hitting or avoiding a pedestrian, speed and low latency are crucial. With Lattice’s Sensor Hub solution supported by the Avant-E FPGA, manufacturers can rest assured that their systems are operating reliably and at high capacity.

Continued Support for Sensor Fusion Applications
While sensor fusion has applications beyond automotive applications—including smart industrial robotics, automated surveillance and safety systems, and more—it will always require the support of capable, interoperable, and efficient hardware.

To learn more about Lattice’s Sensor Hub solution, read our Sensor Hub For Near-Sensor Low-Latency Data Fusion in AI Systems white paper. If you’d like to discuss potential FPGA-enabled sensor fusion applications, contact our team today.