As advancements in hardware and software empower the shift from digital artificial intelligence (AI) solutions to physical AI, we're starting to see model deployments that have a tangible impact on the real world. Whether it be humanoid robotics in manufacturing environments or automated drones in the aerospace and defense industries, AI is being deployed in increasingly mobile, autonomous environments.

The challenge is not so much finding new physical AI use cases but supporting them at scale. In a recent webinar, Lattice experts met with NVIDIA and Tauro Technologies to discuss how developers can enable deterministic, low latency sensor-to-compute streaming for more effective physical AI deployments.

Physical AI is outpacing device connectivity
The roots of physical AI's scale challenge are incredibly simple: bandwidth and speed. Most modern devices carry dozens of high-resolution sensors (cameras, lidar, ultrasound, radar, etc.) that, in turn, produce an unprecedented amount of data that needs to be processed. Edge AI models have gone from gigabit (Gb) scale to tens and even hundreds of gigabits per second (Gbps), all of which must be streamed, processed, and acted upon in real-time.

This requires systems to have as little latency as possible. Humanoid robots, for example, may have as many as 72 degrees of freedom that are sending and receiving control signals 5,000 times per second. To coordinate their movement effectively, these robots must be able to respond to sensor input in microseconds, not milliseconds or more.

Unfortunately, a lack of standard connectivity can make timely processing incredibly difficult. Existing hardware, such as separate PLCs, CPUs, GPUs, and more, is likely optimized to run different connectivity protocols. Whether they rely on EtherCAT, CAN, RS-485, MIPI, or GMSL, each of these protocols runs at its own rate that's unsynchronized from the rest of the system. This is simply not capable of supporting low latency, real-time processing.

Unifying sensing and computing capabilities
To address this challenge, developers are considering methods of streamlining sensor data and simplifying processing. Instead of maintaining a separate protocol for each sensor, developers can integrate a near-sensor bridge that translates all sensor data into a unified Ethernet signal. This signal can then be aggregated over a single fabric and connected to a compute platform that handles perception, control, and safety together.

The NVIDIA Holoscan solution provides one such means of employing this Ethernet-centric connectivity in physical AI deployments. It discovers sensors on power-up and enumerates them and their capabilities, writing drivers in Python or C++ without creating common pain points like kernel modules or device-tree edits. With Holoscan, developers can experience benefits like:

• More effective timing, as multi-sensor outputs can be time-correlated with sub-microsecond synchronization that adheres to the IEEE 1588 PTP timing foundation.
• Reusable integrations, as a consistent connection enables the swapping of sensors or compute over time without requiring a more total redesign.
• More flexible topology, allowing sensors to be aggregated near the compute or distributed across the robot using standard Ethernet switching.

Ultimately, the Holoscan Sensor Bridge can help streamline physical AI workflows from sensors to compute platforms. But to do so, it must be able to handle various sensor interfaces, sensor counts, and ethernet speeds – meaning it must be programmable. That's where the power of Field Programmable Gate Arrays (FPGAs) comes in.

Streamlining sensor connectivity with Lattice FPGAs
Lattice's Holoscan Sensor Bridge solution was built to support reprogrammable Holoscan deployments in environments like humanoids and autonomous robotics. The solution leverages the low-power, programmable capabilities of Lattice FPGAs to aggregate and preprocess data from cameras, lidar, radar, audio, IMU, and time-of-flight sensors, packetize it into time-stamped Ethernet frames, and deliver it to NVIDIA Jetson and/or Thor compute processors.

This solution is available in multiple configurations; each suited for specific sensor-based needs. Lattice collaborated with the Tauro Technologies and NVIDIA teams to build and deploy these solutions, which include:

1. A two-chip model that uses a Lattice CrossLink™-NX FPGA to unify sensor input, and a Lattice CertusPro-NX to handle the packetization and Ethernet transmission. This is best suited for hard D-PHY up to 2.5 Gbps per lane and is widely adopted.
2. A single-chip model that uses one CertusPro-NX FPGA to support about 1.5 Gbps per lane or below, including adapter cards for MIPI, GMSL, and SDI from partner Tauro Technologies.
3. A higher-end model that uses a Lattice Avant™ FPGA for 8 megapixel (MP) and 10+ MP sensors, with up to 4 x 25G ethernet and additional preprocessing resources.

Beyond their programmability, what makes these FPGA-based models so compelling for developers is their low power operation and small form factor. Power and space are incredibly constrained in physical AI designs, so the less physical space and energy it takes to transfer and process sensor data, the better the overall system will operate.

Making Physical AI more practical
Physical AI can only be practically deployed when device sensors are effectively bridged to their compute resources. By integrating low-power, flexible connectivity through Lattice's FPGA-enabled Holoscan Sensor Bridge, developers can better construct their systems to carry out real-time, deterministic actions.

To learn more about this collaborative physical AI solution, watch the full webinar. If you'd like to dig deeper into Lattice's Holoscan solution, contact our team today.