A quiet hardware arms race is moving large language models onto handsets. Privacy, latency, and profit margins are shifting the balance between cloud giants and chipmakers.
Illustration by IMF Alpha editorial · Reviewed by Pedro Marini
For the past decade the default path for AI has been up: models in the cloud, racks of GPUs, pay-as-you-go inference. Now another logic is gaining ground — push models down onto devices. It’s less flashy than a billion-dollar datacenter, yes, but it might matter more for who actually captures value in the AI era.
What’s interesting is how fast the hardware side reacted. Chip designers, phone makers, and a cadre of startups are optimizing models, compilers, and accelerators for low-power environments. Qualcomm is tuning Snapdragon for quantized language models. Apple keeps embedding bigger neural engines in each silicon refresh. A few startups are already showing small LLMs can do genuinely useful work offline.
The cloud is not going away. The very largest models will sit in datacenters for the foreseeable future. Think instead of a layered approach:
That shift changes who holds the cards. Historically cloud providers owned both the compute and the billing relationship. On-device AI hands new bargaining chips to chipmakers and handset vendors — and to app developers who can now promise offline, privacy-sensitive features.
Squeezing capable models into a few gigabytes of flash and a couple watts of power takes craft: quantization, pruning, distillation, hardware-aware compilation. These engineering moves are often as important as novel model architectures.
A few caveats worth calling out:
In practice, though, the story is messier. Some teams underestimate runtime variability; others find clever ways to mask differences with client-side orchestration.
A practical example: a healthcare app that triages patient notes locally can avoid uploading sensitive records, give clinicians faster responses, and cut monthly cloud bills. That turns procurement conversations from cloud credits into device capability assessments.
On-device AI reshapes revenue and margin dynamics across the stack. Firms that nail hardware–software co-design can command premium pricing and build stickier ecosystems. Conversely, companies too exposed to commoditized cloud GPUs may see slower incremental growth as some inference shifts to the edge.
The upshot
On-device AI is less a single technology than a strategic shift: value moves from centralized compute economics toward a more distributed set of winners. The technology has limits, yes, but the competitive balance is already tilting. For investors and product leaders the question isn’t cloud versus edge in a binary sense — it’s which parts of both you back. My bet is on both, with a clear tilt toward the edge.
Pedro Marini
Both the Securities and Exchange Commission and the Commodity Futures Trading Commission are actively scrutinizing the accelerating integration of artificial intelligence into financial markets, focusing on risk management, market integrity, and transparency.
Strong demand for advanced AI accelerators, particularly from major cloud providers, continues to drive Nvidia's revenue growth, despite anticipated moderation in capex.
Banks and fintechs are racing to replace fragile real-world datasets with synthetic alternatives. That promises speed and privacy, but also new biases, regulatory headaches, and systemic risk.