The first time I shipped a TensorFlow Lite model inside a production app, I was proud of myself for about forty-eight hours. Then the crash reports started rolling in from a cluster of older Android devices, and my beautiful 12 MB model turned out to be the reason. That was the moment I understood that on-device machine learning for mobile apps is not a scaled-down version of what happens on a server. It's a different discipline entirely.
Here's the thing most articles on this topic skip: they talk about AI in mobile apps as if the model lives somewhere magical in the cloud and the phone just displays results. That's the cloud inference model. On-device is the opposite bet — the model runs on the phone itself, with no round trip to a server. The tradeoffs are real, and they shape everything from your architecture to your battery budget.
Key Takeaways
- On-device inference means the model runs locally, cutting latency to milliseconds and keeping sensitive data off the network entirely.
- Model size and memory are your hard ceiling — most production models stay under 20 MB after quantization.
- Frameworks differ by platform: Core ML on iOS, TensorFlow Lite or ML Kit on Android, ONNX Runtime Mobile for cross-platform work.
- Privacy compliance gets simpler when data never leaves the phone — a genuine argument under GDPR.
- Thermal throttling and device fragmentation are the two problems that will bite you in production, not in the demo.
Why on-device machine learning changes the equation for mobile apps
Ask yourself where the model actually runs. This single question determines your latency, your infrastructure bill, and your privacy posture. When inference happens in the cloud, every prediction costs a network round trip — 150 to 400 milliseconds on a decent connection, far worse on a train. On-device, you're looking at single-digit milliseconds for a small model on modern hardware.
But the real reason I keep pushing teams toward on-device isn't speed. It's that the network is the least reliable part of any mobile app.
The offline argument
Users open apps in elevators, on planes, in basements, in rural areas with one bar of signal. If your feature needs the cloud, it doesn't exist in those moments. I built a barcode-scanning feature that ran a small classification model locally; it worked in a warehouse with zero connectivity. The cloud version of the same feature would have been dead on arrival there. That's not a nice-to-have. That's the feature working at all.
The privacy argument
Anything sensitive — health signals, voice recordings, text the user is typing, biometric data — is dramatically safer when it never leaves the device. Under GDPR, local processing removes an entire category of transfer and retention obligations. You're not shipping personal data to a server, so there's no transfer to document, no storage to secure, no breach surface to worry about. Legal teams love this. So do users, when you tell them.
What on-device inference actually costs you
Nothing is free. Here's the honest ledger.
- Model size. Shipping a 200 MB model is a non-starter for most apps. Quantization (converting weights to 8-bit or even 4-bit) shrinks it dramatically, often by 4x, at some cost to accuracy.
- Memory. Android and iOS both kill apps that hog RAM. A model that needs 300 MB resident on a mid-range phone is a liability.
- Battery and heat. Continuous inference drains power and warms the device. After a few minutes of sustained work, thermal throttling kicks in and your model slows down — sometimes by half.
- Fragmentation. Android hardware varies wildly. A model that runs fine on a flagship flies off a cliff on a five-year-old budget phone.
- Updates. Pushing a new model version over the air means versioning, rollback plans, and testing across device tiers all over again.
That last one catches people. In my first iteration I hardcoded the model into the app bundle, which meant every model tweak required a full store release and review cycle. Two weeks of latency per experiment. Not sustainable. Moving to a downloaded-model architecture cut our iteration loop from days to hours.
Which framework should you pick?
The answer depends entirely on your platform strategy. If you're iOS-first, Core ML is the obvious call — it integrates cleanly with Apple silicon and the Neural Engine. If you're Android-first, TensorFlow Lite (now part of the LiteRT family) or ML Kit are the paths of least resistance. Cross-platform? ONNX Runtime Mobile runs the same model on both.
| Framework | Platform | Best for | Hardware acceleration |
|---|---|---|---|
| Core ML | iOS / macOS | Apple-only apps, Neural Engine access | Yes, native |
| TensorFlow Lite / LiteRT | Android, cross-platform | Broad Android reach, quantization tooling | GPU delegate, NNAPI |
| ML Kit | Android & iOS | Ready-made tasks (text, vision, translation) | Yes, handled for you |
| ONNX Runtime Mobile | Cross-platform | Reusing one model across iOS & Android | Yes, via execution providers |
My opinion, and I'll defend it: unless you have a specific reason to go cross-platform, use the native framework. Core ML's integration with Apple's silicon is genuinely hard to beat, and fighting that to save a bit of code sharing has cost me more time than it ever saved.
How do you actually ship a model that survives production?
Three things separate a demo from a shipped feature.
1. Quantize, then measure again
Quantization is the single highest-leverage step you can take. Converting float32 weights to int8 typically shrinks a model by around 4x and speeds up inference on most hardware. But always re-run your accuracy evaluation afterward — a small accuracy loss on paper can become a visible regression on real inputs, and you'll only catch it if you actually test on the quantized model, not the original.
2. Build a device test matrix
This is where I lost the most time. Testing only on your own phone tells you almost nothing about the fleet. Pull a list of the lowest-spec devices your users actually have, and test on those. A model that works on a modern flagship may fall over on a phone with 2 GB of RAM.
3. Plan for model updates from day one
Assume you'll want to swap the model without a store release. That means downloading it, versioning it, and handling the case where a download fails or a new model misbehaves. It adds complexity, but trust me — you'll want it within the first month.
What about running multiple models?
You can, but be careful. Each resident model competes for memory and compute. In practice, most apps I've built run one model at a time — load it when needed, release it when done. If you must run several, stagger them and watch your RAM budget closely.
The part nobody tells you
On-device machine learning sounds like the "clean" option: no servers, no data pipelines, no cloud bills. It is genuinely simpler in some ways. But it shifts complexity into the client, where every device is different and you can't fix a bug by pushing a server update. The teams that succeed at this treat the model as a product with a lifecycle, not as a downloadable artifact dropped into the app bundle.
So the real question isn't "cloud or on-device?" It's "what's the cost of being wrong on this specific device, and can I afford it?" Answer that honestly and the architecture usually picks itself. And if you get it wrong the first time — I did, more than once — at least now you'll know exactly why your phone is getting warm.