Edge AI
How it works
1) Training in the cloud: the model is trained on large datasets where GPU/TPU compute is available. 2) Model compression: to fit the device’s limited memory and power budget, teams apply quantization (e.g. INT8, INT4), pruning (removing weights) and knowledge distillation (transferring knowledge into a smaller model); a related strand is TinyML — milliwatt-class models on microcontrollers. 3) Hardware compilation: the model is optimized and compiled for the target accelerator (NPU, Edge TPU, mobile GPU, MLSoC, FPGA) using runtimes such as LiteRT (TensorFlow Lite), ONNX Runtime, TensorRT or ExecuTorch. 4) Inference at the edge: the model runs locally as an inference engine, processing sensor data where it is generated and returning results with minimal latency. 5) Feedback loop (optional): hard or uncertain cases can be sent back to the cloud for retraining and model updates. Key edge accelerators are NPUs (e.g. Arm Ethos, Apple Neural Engine) and MLSoCs — for example SiMa.ai Modalix (50 TOPS, sub-10 W, supporting CNN/ViT/LLM/LMM), capable of running even generative models at the edge.
Problem solved
It solves the problems that arise when AI inference depends on the cloud: excessive latency for real-time use cases (robot control, autonomous vehicles), no operation under poor or zero connectivity, high data-transfer and storage costs, and privacy risk from sending sensitive data (video, voice, medical data) off-device. It also enables operation within tight power and energy budgets.
Components
An AI model shrunk to the size and power budget of the edge device via quantization, pruning and distillation.
Official
A dedicated chip for efficient low-power inference: NPUs (e.g. Arm Ethos, Apple Neural Engine), Edge TPUs or MLSoCs (e.g. SiMa.ai Modalix).
Official
A lightweight runtime that compiles and runs the model on the target hardware, e.g. LiteRT (TensorFlow Lite), ONNX Runtime, TensorRT, ExecuTorch.
Official
A flow where training and updates happen in the cloud and inference at the edge; hard cases can return to the cloud for retraining.
Implementation
Dropping to INT8/INT4 without proper calibration or quantization-aware training (QAT) can significantly degrade model quality.
Different NPUs/MLSoCs have distinct SDKs, supported-operator sets and formats — a model that runs on one device may not compile on another.
A model that fits "on paper" may overflow RAM or overheat the device under sustained load, throttling performance.
Evolution
CNN architectures designed for constrained mobile resources open the door to practical machine vision on devices.
The arrival of lightweight inference runtimes and dedicated NPUs in phones makes on-device inference mainstream.
The TinyML community formalizes milliwatt-class machine learning on microcontrollers "at the very edge of the cloud".
Local generative models on devices (e.g. on flagship smartphones) show that the edge can run not only classifiers but also LLMs.
MLSoC chips (e.g. SiMa.ai Modalix: 50 TOPS, sub-10 W) run full LLM/LMM pipelines on a single, energy-efficient edge chip.
Hyperparameters (configurable axes)
The edge device’s maximum power draw, critical for battery-powered devices.
The representation format of weights and activations; lower precision cuts memory and energy at the cost of accuracy.
The maximum acceptable inference time for a given use case.
Accelerator throughput in tera-operations per second; often considered as TOPS/W.
Compute bottleneck
The bottleneck is not raw cloud compute but the constraints of the edge device: small RAM, limited memory bandwidth, a tight power budget (often under 10 W) and a real-time requirement. This forces aggressive model compression and hardware optimized for energy efficiency (TOPS/W).
Hardware requirements
As a paradigm, edge AI runs across the full spectrum of edge hardware; in practice the main targets are dedicated NPUs and MLSoCs (no separate option in the dictionary).
Edge TPUs and similar tensor accelerators offer high performance-per-watt for edge inference.
CPUs (including low-power cores such as Arm Cortex-M) handle lightweight models and TinyML, though at lower throughput.
FPGAs are sometimes used for configurable, energy-efficient edge inference in industrial settings.