1. Cloud training: standard training of a full model (7B-70B+) on large GPU clusters. 2. Compression: the base model is compressed via techniques: (a) knowledge distillation - a smaller student model learns from the teacher, (b) quantization - weights converted from FP16 to INT8/INT4/INT2 (2-8x memory reduction), (c) pruning - removing near-zero weights (unstructured/structured), (d) low-rank decomposition - W ≈ AB where rank(A,B) << original. 3. Conversion to on-device format: export to Core ML (.mlpackage, Apple), LiteRT/.tflite (Google), GGUF/GGML (llama.cpp), ONNX (cross-platform), MLC (WebGPU). 4. Hardware-specific optimisation: model compiled for a specific NPU: Apple Neural Engine (16-core, ~35 TOPS INT8 on M4), Qualcomm Hexagon (Snapdragon 8 Elite), MediaTek APU, Google Tensor G4, NVIDIA Jetson Orin. Leverages tensor cores, SIMD, and dedicated matmul units. 5. On-device runtime: framework loading (Core ML, ONNX Runtime, TFLite, MLC LLM, llama.cpp), inference per user query without network calls. 6. Hybrid dispatch: when a task exceeds on-device capacity (long context, complex reasoning), delegate to the provider's private cloud (Apple Private Cloud Compute, Gemini Cloud fallback, etc.) - with privacy guarantees.
Cloud-based AI has four fundamental drawbacks: (1) all sensitive user data flows to the provider's servers (leak risk, subpoena, use for training); (2) latency 100-2000ms per query depending on distance to the datacentre; (3) growing inference cost for mass deployment (Anthropic/OpenAI per-token fees); (4) no operation without internet. On-device AI eliminates all four simultaneously at the cost of model quality (physically fewer parameters fit in phone RAM vs a GPU cluster).
AI model reduced via distillation/quantization/pruning to a size fitting in device RAM (typically 0.5-8 GB for LLMs on a 2026 smartphone).
Official
Specialised tensor operations processor: Apple Neural Engine, Qualcomm Hexagon, MediaTek APU, Google Tensor G-series, Samsung Exynos NPU, Intel NPU (Meteor Lake+), AMD XDNA, NVIDIA Jetson.
Official
On-device model execution framework: Core ML (Apple), LiteRT/TFLite (Google), ONNX Runtime Mobile (Microsoft), MLC LLM (WebGPU), llama.cpp, MediaPipe, PyTorch Mobile.
Official
Serialised weights + compute graph format: .mlpackage/.mlmodel (Core ML), .tflite/LiteRT (TFLite), GGUF/GGML (llama.cpp), ONNX (cross-platform), safetensors.
Official
Optional mechanism for delegating queries beyond on-device capacity to the provider's private cloud: Apple Private Cloud Compute, Gemini Cloud, Qualcomm AI Hub.
Official
A 7B FP16 model = ~14 GB and will not fit in 8 GB RAM. Common mistake: exporting without planning the memory budget.
Continuous queries (background assistant, real-time transcription) can drain the battery in an hour - NPUs draw 3-8W at full load.
Passively cooled smartphones and laptops throttle the NPU after 30-60s of intensive inference - inference rate drops 50%.
A model exported to Core ML does not run on Android; TFLite is not optimal on the Apple Neural Engine.
Naive INT4 quantization can cause 5-15% accuracy drops on sensitive benchmarks (math, coding, medical).
For queries exceeding on-device capacity (long context, complex reasoning) the user gets a poor answer or an error.
Krizhevsky et al. prove deep learning requires specialised hardware (GPU) - foundation for the understanding that AI needs dedicated compute, precursor of NPU/edge AI.
Google begins work on TFLite for mobile deployment - the first widely available on-device deep learning framework.
Three simultaneous breakthroughs: (1) Google's MobileNets paper - depthwise separable convolutions for mobile vision, (2) Apple's Core ML framework, (3) A11 Bionic with the first Neural Engine (600 GOPS). On-device AI goes mainstream.
Sanh et al. show KD delivers 97% BERT quality at 40% size - a turning point for on-device NLP. Google begins mobile BERT experiments.
Georgi Gerganov releases llama.cpp - LLaMA on laptop/phone CPU with INT4 quantization. Opens the on-device LLM era for hobbyists.
Google unveils Gemini Nano (2B) - the first flagship LLM designed primarily for on-device deployment on Pixel 8 Pro (Tensor G3 NPU).
Apple unveils Apple Intelligence (on-device foundation model ~3B) + Private Cloud Compute (hybrid). Microsoft announces Copilot+ PC with NPU >40 TOPS. Phi-3 (Microsoft) and Llama-3.2 (Meta) release flagship on-device models.
DeepSeek releases R1-Distill (1.5B-70B) - on-device reasoning models via distillation. The hybrid approach (on-device + private cloud fallback) becomes the production standard.
iOS 27 Siri AI (Google Gemini + App Intents) makes multi-app agentic workflows mostly on-device with minimal delegation to Private Cloud Compute. Next stage: agents on Apple Watch, AirPods, Vision Pro.
Time complexity: O(N_params × N_tokens) - ograniczone bandwidth pamięci i throughput NPU. Space complexity: O(|θ_model| × bits/weight) + O(KV_cache × context_length).
Number of parameters in the on-device model. Trade-off: more parameters = better quality but larger memory + slower inference.
Bits per weight after quantization. Lower = less memory but quality risk.
The kind of NPU/CPU/GPU on the target device. Determines frameworks, optimisations, and model formats.
Maximum context length supported on-device (limited by KV cache memory).
Classic on-device is dense; some newer models (Phi-3-MoE, Gemini Nano MoE) use conditional/sparse for higher efficiency.
Prefill is fully parallel (compute-bound); decode is sequential (memory-bound); some NPUs have dedicated compute paths for both phases.
Mobile GPUs (Apple GPU, Adreno, Mali) support matmul but are typically slower and less energy-efficient than dedicated NPUs.
CPUs can run models (llama.cpp) but far slower than NPUs - useful for desktop/laptop without an NPU or as a fallback.
Neuromorphic chips (Intel Loihi 2, IBM NorthPole) offer ultra-low-power inference but the software ecosystem is immature.
FPGAs (Xilinx, Altera) are used in niche industrial edge; high configuration power but a complex toolchain.
