Robots Atlas>ROBOTS ATLAS
Deployment

Edge AI

2017ActivePublished: 19 November 2026Updated: 19 November 2026Published
Key innovation
Moving the execution of AI models from the cloud to edge devices (smartphones, cameras, IoT, robots, vehicles, MLSoC accelerators) so that inference happens close to the data source — delivering low latency, privacy, offline operation and energy efficiency.
Category
Deployment
Abstraction level
Paradigm
Operation level
DeploymentInferenceServing
Use cases
Robots and mobile roboticsAutonomous vehicles (ADAS)Smart cameras and machine visionIoT sensors and devicesSmartphones and wearablesPredictive maintenance in industryMedical devices and diagnosticsSmart home and voice assistantsDronesGenerative AI at the edge (local LLMs/LMMs)

How it works

1) Training in the cloud: the model is trained on large datasets where GPU/TPU compute is available. 2) Model compression: to fit the device’s limited memory and power budget, teams apply quantization (e.g. INT8, INT4), pruning (removing weights) and knowledge distillation (transferring knowledge into a smaller model); a related strand is TinyML — milliwatt-class models on microcontrollers. 3) Hardware compilation: the model is optimized and compiled for the target accelerator (NPU, Edge TPU, mobile GPU, MLSoC, FPGA) using runtimes such as LiteRT (TensorFlow Lite), ONNX Runtime, TensorRT or ExecuTorch. 4) Inference at the edge: the model runs locally as an inference engine, processing sensor data where it is generated and returning results with minimal latency. 5) Feedback loop (optional): hard or uncertain cases can be sent back to the cloud for retraining and model updates. Key edge accelerators are NPUs (e.g. Arm Ethos, Apple Neural Engine) and MLSoCs — for example SiMa.ai Modalix (50 TOPS, sub-10 W, supporting CNN/ViT/LLM/LMM), capable of running even generative models at the edge.

Problem solved

It solves the problems that arise when AI inference depends on the cloud: excessive latency for real-time use cases (robot control, autonomous vehicles), no operation under poor or zero connectivity, high data-transfer and storage costs, and privacy risk from sending sensitive data (video, voice, medical data) off-device. It also enables operation within tight power and energy budgets.

Components

Compressed modelThe intelligence payload executed locally

An AI model shrunk to the size and power budget of the edge device via quantization, pruning and distillation.

Official

Edge acceleratorHardware execution of inference

A dedicated chip for efficient low-power inference: NPUs (e.g. Arm Ethos, Apple Neural Engine), Edge TPUs or MLSoCs (e.g. SiMa.ai Modalix).

Official

Inference runtimeModel-to-hardware execution layer

A lightweight runtime that compiles and runs the model on the target hardware, e.g. LiteRT (TensorFlow Lite), ONNX Runtime, TensorRT, ExecuTorch.

Official

Cloud-edge pipelineModel lifecycle in edge deployment

A flow where training and updates happen in the cloud and inference at the edge; hard cases can return to the cloud for retraining.

Implementation

Implementation pitfalls
Accuracy loss under aggressive quantizationHigh

Dropping to INT8/INT4 without proper calibration or quantization-aware training (QAT) can significantly degrade model quality.

Fix:Use calibrated post-training quantization or QAT and measure accuracy on a representative set.
Hardware and runtime fragmentationMedium

Different NPUs/MLSoCs have distinct SDKs, supported-operator sets and formats — a model that runs on one device may not compile on another.

Fix:Design for supported operators, use portable formats (ONNX) and test on the target hardware.
Exceeding memory and thermal budgetMedium

A model that fits "on paper" may overflow RAM or overheat the device under sustained load, throttling performance.

Fix:Profile on-device memory and temperature; size the model and batch to real limits.

Evolution

2017
Efficient mobile models (MobileNets)

CNN architectures designed for constrained mobile resources open the door to practical machine vision on devices.

2017
On-device runtimes and accelerators (TensorFlow Lite, Apple Neural Engine)
Inflection point

The arrival of lightweight inference runtimes and dedicated NPUs in phones makes on-device inference mainstream.

2019
Emergence of TinyML

The TinyML community formalizes milliwatt-class machine learning on microcontrollers "at the very edge of the cloud".

2023
Generative AI reaches the edge
Inflection point

Local generative models on devices (e.g. on flagship smartphones) show that the edge can run not only classifiers but also LLMs.

2024
MLSoCs for generative edge AI

MLSoC chips (e.g. SiMa.ai Modalix: 50 TOPS, sub-10 W) run full LLM/LMM pipelines on a single, energy-efficient edge chip.

Hyperparameters (configurable axes)

Power budgetCritical

The edge device’s maximum power draw, critical for battery-powered devices.

< 1 mW (TinyML)Microcontrollers, always-on
< 10 W (MLSoC)E.g. SiMa.ai Modalix
Numerical precisionHigh

The representation format of weights and activations; lower precision cuts memory and energy at the cost of accuracy.

INT8Edge standard
INT4Aggressive compression
Target latencyHigh

The maximum acceptable inference time for a given use case.

< 50 msReal-time interaction
Compute throughput (TOPS)Medium

Accelerator throughput in tera-operations per second; often considered as TOPS/W.

50 TOPSSiMa.ai Modalix

Compute bottleneck

Limited device memory, bandwidth and energy budget

The bottleneck is not raw cloud compute but the constraints of the edge device: small RAM, limited memory bandwidth, a tight power budget (often under 10 W) and a real-time requirement. This forces aggressive model compression and hardware optimized for energy efficiency (TOPS/W).

Hardware requirements

Primary

As a paradigm, edge AI runs across the full spectrum of edge hardware; in practice the main targets are dedicated NPUs and MLSoCs (no separate option in the dictionary).

Good fit

Edge TPUs and similar tensor accelerators offer high performance-per-watt for edge inference.

Possible

CPUs (including low-power cores such as Arm Cortex-M) handle lightweight models and TinyML, though at lower throughput.

Possible

FPGAs are sometimes used for configurable, energy-efficient edge inference in industrial settings.