As small, powerful language models run locally on phones and Macs, startups and incumbents are racing to redefine AI tools around privacy, latency and new business models.
Illustration by IMF Alpha editorial · Reviewed by Pedro Marini
The set-up
This feels like a pivot people saw coming but somehow minimized. The last 18 months were dominated by giant models and cloud GPUs. The next chapter looks different: compact, private LLMs running on phones, laptops and edge boxes. That shift is already changing what qualifies as an AI tool.
What’s changing and why it matters
Concrete examples and trade-offs
You can already find local assistants for note summarization, code completion and clinical drafting. A few patterns recur:
What’s interesting is how these trade-offs show up differently by use case. For quick conversational help, speed and privacy beat a few percent of accuracy. For regulated clinical workflows, that extra accuracy might still belong in the cloud.
A competitive shake-up
Cloud providers are not standing still. Expect hybrid approaches: run inference locally for sensitive inputs, call the cloud for heavy lifting. That middle ground will be valuable — and complicated. Startups that mastered compression, quantization and smart caching suddenly look attractive because they solve the practical bottleneck: making models genuinely useful on-device.
There’s also a geopolitical side. Jurisdictions with strict data rules prefer on-device processing, which creates demand for privacy-first tools and gives domestic vendors an edge in those markets. Global cloud providers, in turn, must add compliance plumbing to stay relevant.
Who should care — and why
Open questions and pushbacks
The reality will be messy. Some workloads will go edge-first; others will stay in the cloud. Some companies will hybridize and prosper; others will stumble over distribution or support.
The upshot
We are not leaving the cloud behind, but we are entering a more plural era: cloud, edge and device-native AI coexisting. That fragmentation should spur competition and better privacy practices, and it forces incumbents to innovate rather than only scale. If you are building or buying AI tools, plan for hybrid deployments, prioritize latency and privacy, and test on real devices — not just benchmark servers.
Look for
This moment feels a bit like the quiet PC revolution: incremental, practical, and ultimately decisive in reshaping who owns the interface between human intent and machine output.
As privacy rules bite and data costs spike, synthetic data startups and cloud giants are racing to replace real-world training sets. Investors should be selective.
From privacy gains to battery headaches, on‑device large language models force chipmakers, apps and regulators to rethink mobile AI—and investors to reassess winners.
As chip makers and developers push compact LLMs and NPUs into handsets, expect faster answers, tighter privacy—and a shakeup in how apps make money.