When AI Writes the Scam: Generative Models Fuel a New Wave of Phishing

Short version

Generative AI has turned social engineering from a craft into an assembly line. Personalized phishing emails, eerily convincing deepfake voicemails, and on-demand malware scaffolds can be produced in minutes. Treating this as just a hotter version of old phishing problems is a recipe for getting outpaced.

Why this matters now

In the past 18 months two practical barriers for attackers have dropped away: the need for fluent human writing and the time needed to scale. What once took a researcher a day or more — probing a target, drafting a bespoke message, chasing follow-ups — can now be scripted with prompt templates and cheap compute. The outcome is simple: much higher volume, and much more plausible social engineering. That combination hits organizations that still rely on signature-based defenses and one-size-fits-all awareness campaigns.

What we’re actually seeing

• Automated spear-phishing that scrapes LinkedIn, GitHub, and public web content to build context, subject lines, and even mimic tone. Attackers match meeting rhythms and project names; sometimes the messages read like someone from inside the team wrote them.
• Deepfake voice fraud for wire transfers and vendor changes. It’s no longer a rare, expensive stunt — it’s cheap enough to iterate until a clip sounds right.
• Prompt chains and model jailbreaks coaxing code-generation models into outputting malware snippets or configuration exploits.

Market response — a messy sprint with different bets

Vendors are scrambling to put AI into detection pipelines. Expect more spending on behavior telemetry, anomaly detection, and AI-assisted incident response. Some incumbent defenders are piling on cloud analytics; newer firms are pitching XDR with built-in machine learning.

That said, slapping default detection models onto the problem has limits. Models trained on historical attacks can lag behind novel, AI-crafted threats. And there’s a concentration risk: many defenders depend on large models from a few cloud providers, which creates a single point of failure if those models are compromised or misused.

Practical moves for CISOs and boards

• Treat this as a governance and architecture issue, not just another awareness course. Update playbooks assuming social engineering will be convincingly tailored.
• Adopt continuous verification: require multi-factor approvals for transfers, out-of-band checks for vendor changes, and step-up authentication for sensitive actions.
• Reassess vendor risk when security tools rely on third-party large models; ask for model provenance and adversarial testing results.
• Run tabletop exercises that mimic AI-enhanced scams. Generic phishing drills won’t cut it; training should reflect personalized lures.
• Bring cyber insurers into the conversation. Underwriters are reworking exposure models and may demand specific AI-related controls for coverage.

A counterpoint

AI helps defenders too. The same techniques can speed triage, add richer context to alerts, and automate containment — all things that reduce dwell time. The strategic question is whether organizations can adopt defensive AI fast enough without creating new single points of failure.

To be blunt

This isn’t a call to panic, but it does merit urgency. The economics of crime have shifted: low-skill actors now wield convincing tools. Those who combine stronger identity controls, tighter vendor governance, realistic simulations, and selective defensive AI adoption will have a real chance of keeping up.

Pedro Marini