Inference

On-device AI

2017ActiveUpdated: 14 July 2026Published

Key innovation

Running AI models directly on the user's device (phone, laptop, robot, IoT, XR headset) instead of sending data to a remote server. Simultaneously addresses privacy (data does not leave the device), latency (no round-trip to the cloud), inference cost (zero per-token fees), and offline capability (works without internet).

How it works

1. Cloud training: standard training of a full model (7B-70B+) on large GPU clusters. 2. Compression: the base model is compressed via techniques: (a) knowledge distillation - a smaller student model learns from the teacher, (b) quantization - weights converted from FP16 to INT8/INT4/INT2 (2-8x memory reduction), (c) pruning - removing near-zero weights (unstructured/structured), (d) low-rank decomposition - W ≈ AB where rank(A,B) << original. 3. Conversion to on-device format: export to Core ML (.mlpackage, Apple), LiteRT/.tflite (Google), GGUF/GGML (llama.cpp), ONNX (cross-platform), MLC (WebGPU). 4. Hardware-specific optimisation: model compiled for a specific NPU: Apple Neural Engine (16-core, ~35 TOPS INT8 on M4), Qualcomm Hexagon (Snapdragon 8 Elite), MediaTek APU, Google Tensor G4, NVIDIA Jetson Orin. Leverages tensor cores, SIMD, and dedicated matmul units. 5. On-device runtime: framework loading (Core ML, ONNX Runtime, TFLite, MLC LLM, llama.cpp), inference per user query without network calls. 6. Hybrid dispatch: when a task exceeds on-device capacity (long context, complex reasoning), delegate to the provider's private cloud (Apple Private Cloud Compute, Gemini Cloud fallback, etc.) - with privacy guarantees.

Problem solved

Cloud-based AI has four fundamental drawbacks: (1) all sensitive user data flows to the provider's servers (leak risk, subpoena, use for training); (2) latency 100-2000ms per query depending on distance to the datacentre; (3) growing inference cost for mass deployment (Anthropic/OpenAI per-token fees); (4) no operation without internet. On-device AI eliminates all four simultaneously at the cost of model quality (physically fewer parameters fit in phone RAM vs a GPU cluster).

Components

Compressed modelInference core

AI model reduced via distillation/quantization/pruning to a size fitting in device RAM (typically 0.5-8 GB for LLMs on a 2026 smartphone).

Official

NPU / AI acceleratorDedicated hardware

Specialised tensor operations processor: Apple Neural Engine, Qualcomm Hexagon, MediaTek APU, Google Tensor G-series, Samsung Exynos NPU, Intel NPU (Meteor Lake+), AMD XDNA, NVIDIA Jetson.

Official

On-device runtimeExecution layer

On-device model execution framework: Core ML (Apple), LiteRT/TFLite (Google), ONNX Runtime Mobile (Microsoft), MLC LLM (WebGPU), llama.cpp, MediaPipe, PyTorch Mobile.

Official

Model formatWeight container

Serialised weights + compute graph format: .mlpackage/.mlmodel (Core ML), .tflite/LiteRT (TFLite), GGUF/GGML (llama.cpp), ONNX (cross-platform), safetensors.

Official

Hybrid fallback layerCloud delegation

Optional mechanism for delegating queries beyond on-device capacity to the provider's private cloud: Apple Private Cloud Compute, Gemini Cloud, Qualcomm AI Hub.

Official

Implementation

Reference implementations

Apple Core ML

Swift/Objective-C · Apple

Official

LiteRT (Google TFLite)

C++/Java/Swift/Python · Google

Official

ONNX Runtime Mobile

C++/C#/Java/Swift · Microsoft

Official

llama.cpp

C++ · Georgi Gerganov / open source

MLC LLM (WebGPU + native)

Python/TypeScript/C++ · MLC.AI (CMU + community)

Official

MediaPipe (Google mobile ML)

C++/Python/JS · Google

Official

Implementation pitfalls

Unaware training of a model too large for target RAMCritical

A 7B FP16 model = ~14 GB and will not fit in 8 GB RAM. Common mistake: exporting without planning the memory budget.

Fix:Choose model size and quantization bits so that ~50% of device RAM remains free for OS/apps.

Battery drain via sustained inferenceHigh

Continuous queries (background assistant, real-time transcription) can drain the battery in an hour - NPUs draw 3-8W at full load.

Fix:Batch requests, low-duty-cycle scheduling, offload to a ~10 MB wake-word detection model before starting the main model.

Thermal throttlingHigh

Passively cooled smartphones and laptops throttle the NPU after 30-60s of intensive inference - inference rate drops 50%.

Fix:Cap peak requests/s, chunk long generations, monitor thermal state (thermalState API); consider hybrid cloud fallback for long workflows.

Format mismatch across devicesHigh

A model exported to Core ML does not run on Android; TFLite is not optimal on the Apple Neural Engine.

Fix:Cross-platform: export to ONNX + use ONNX Runtime Mobile or MLC LLM. Alternatively: maintain parallel pipelines for Core ML and TFLite.

Quantization quality loss > accepted thresholdMedium

Naive INT4 quantization can cause 5-15% accuracy drops on sensitive benchmarks (math, coding, medical).

Fix:Use GPTQ/AWQ/SqueezeLLM instead of round-to-nearest; per-tensor scaling; per-task evaluation before deployment.

No hybrid fallback strategyMedium

For queries exceeding on-device capacity (long context, complex reasoning) the user gets a poor answer or an error.

Fix:Implement a complexity heuristic deciding on-device vs cloud (e.g. Apple Private Cloud Compute, Gemini Cloud) with privacy guarantees.

Evolution

Original paper · 2017 · Google 2017 (arXiv) · Andrew G. Howard

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam

2012

AlexNet + GPU inference

Krizhevsky et al. prove deep learning requires specialised hardware (GPU) - foundation for the understanding that AI needs dedicated compute, precursor of NPU/edge AI.

2015

TensorFlow Lite (proto)

Google begins work on TFLite for mobile deployment - the first widely available on-device deep learning framework.

2017

MobileNets + Core ML + iPhone A11 Neural Engine

Inflection point

Three simultaneous breakthroughs: (1) Google's MobileNets paper - depthwise separable convolutions for mobile vision, (2) Apple's Core ML framework, (3) A11 Bionic with the first Neural Engine (600 GOPS). On-device AI goes mainstream.

MobileNets (paper)

2019

DistilBERT + mobile Transformer

Sanh et al. show KD delivers 97% BERT quality at 40% size - a turning point for on-device NLP. Google begins mobile BERT experiments.

2022

llama.cpp - LLMs on CPU

Inflection point

Georgi Gerganov releases llama.cpp - LLaMA on laptop/phone CPU with INT4 quantization. Opens the on-device LLM era for hobbyists.

2023

Gemini Nano + Pixel 8

Inflection point

Google unveils Gemini Nano (2B) - the first flagship LLM designed primarily for on-device deployment on Pixel 8 Pro (Tensor G3 NPU).

2024

Apple Intelligence + Copilot+ PC

Inflection point

Apple unveils Apple Intelligence (on-device foundation model ~3B) + Private Cloud Compute (hybrid). Microsoft announces Copilot+ PC with NPU >40 TOPS. Phi-3 (Microsoft) and Llama-3.2 (Meta) release flagship on-device models.

2025

DeepSeek-R1-Distill + hybrid production

DeepSeek releases R1-Distill (1.5B-70B) - on-device reasoning models via distillation. The hybrid approach (on-device + private cloud fallback) becomes the production standard.

2026

Siri AI + multi-app orchestration on-device

Inflection point

iOS 27 Siri AI (Google Gemini + App Intents) makes multi-app agentic workflows mostly on-device with minimal delegation to Private Cloud Compute. Next stage: agents on Apple Watch, AirPods, Vision Pro.

Hyperparameters (configurable axes)

Model size (parameters)Critical

Number of parameters in the on-device model. Trade-off: more parameters = better quality but larger memory + slower inference.

0.5B-1BSmartwatch, IoT, embedded. Very fast, limited quality.

3BSweet spot for smartphones - Apple Intelligence, Gemini Nano.

7-14BCopilot+ PC laptops - Phi-3/Phi-4.

Quantization bitsCritical

Bits per weight after quantization. Lower = less memory but quality risk.

FP16 (16b)Training baseline; rarely on-device.

INT8 (8b)Standard for NPUs. Minimal quality loss.

INT4 (4b)Popular for on-device LLMs. 1-3% quality drop on benchmarks.

INT2/1.58bExtreme quantization (BitNet); still experimental.

Target hardwareCritical

The kind of NPU/CPU/GPU on the target device. Determines frameworks, optimisations, and model formats.

Apple Neural EngineiPhone/iPad/Mac. Format: Core ML.

Qualcomm Hexagon NPUSnapdragon X Elite, Snapdragon 8 Gen 4. Format: QNN/LiteRT.

NVIDIA JetsonRobotics, embedded high-end. Format: TensorRT.

On-device context lengthHigh

Maximum context length supported on-device (limited by KV cache memory).

4-8KTypical for LLMs ~3B INT4 on smartphones.

32-128KCopilot+ PC with 40+ TOPS NPU.

Computational complexity

Time complexity: O(N_params × N_tokens) - ograniczone bandwidth pamięci i throughput NPU. Space complexity: O(|θ_model| × bits/weight) + O(KV_cache × context_length).