Gemini Let Scammers Build 9,000 Fake Sites, Google Says

Google sells Gemini as a tool for building faster. Its new lawsuit says a China-based scam network used that same speed to industrialize fraud.

Google has sued Outsider Enterprise, an alleged cybercrime group that used Gemini to help create fraudulent websites and push scam texts at scale, according to Ars Technica. The case matters because Google isn’t only saying criminals abused its brand. It’s saying they used Google’s own AI system as part of the production line.

That turns AI safety into something harder than content moderation. If the allegation holds, the issue isn’t a chatbot saying something bad. It’s a chatbot helping lower the skill barrier for a fraud operation that allegedly generated 9,000 fake websites, more than 1 million URLs, and 2.5 million scam texts sent to Android users in a two-week period.

Gemini was built to accelerate creation. Outsider allegedly used that acceleration against users

The central tension is blunt: the same AI coding assistance that helps legitimate users build websites can also help scammers build disposable impersonation pages.

Google says Outsider Enterprise operated through Telegram and offered phishing-as-a-service to people who may not have had the technical ability to build scam sites and text campaigns on their own. The group allegedly provided instructions for using Gemini to create websites impersonating Google, YouTube, and government services such as New York’s E-ZPass. Related reporting from 9to5Google also says the sites mimicked the US Postal Service.

The alleged model was not artisanal fraud. It looked closer to a software business.

TechCrunch, citing Google’s complaint, reported that Outsider’s platform cost $88 per week or $200 per month and offered more than 290 pre-built templates that could mimic legitimate websites “in minutes.” Google’s complaint also described the platform as a “turn-key, online software suite” for criminals regardless of technical skill.

That phrase is the case in miniature. Outsider allegedly sold infrastructure, templates, collaboration channels, and instructions. Gemini was not accused of initiating the scams. The allegation is that scammers used it as a tool inside a larger fraud machine.

“Part of the Outsider software’s appeal is the ease with which someone with limited technical expertise — like many members of the Enterprise— can purchase the software, execute various phishing attacks, and, upon purchase, meet other members of the Enterprise who are proficient in other areas,” Google wrote, according to TechCrunch.

XOOMAR analysis: that is the real AI risk exposed here. Not sentience. Not sci-fi autonomy. Cheap scale.

The reported scale makes this more than a bad prompt problem

Google says Outsider-linked scams pushed 2.5 million messages to Android users in a two-week period. 55,000 spam texts were flagged by Android users in two weeks this past May, which Google described as more than two text spam complaints a minute.

The infrastructure numbers are just as telling:

TechCrunch also reported that an FBI spokesperson said that since July 2023, Outsider Enterprise’s phishing platform enabled cybercriminals to steal “at least an estimated 3,870,000 stolen credit cards and a corresponding estimated $1.9B in losses.” Separately, 9to5Google reported that Google described the operation as affecting hundreds of thousands of victims, with losses estimated in the millions.

Those figures can describe overlapping but different slices of activity. Still, every version points in the same direction: this was not a one-off abuse case.

XOOMAR covered the immediate scale of the filing in 2.5M Scam Texts Push Google to Sue Alleged AI Phishers. The deeper issue is that AI doesn’t need to make scams brilliant. It only needs to make them cheaper, faster, and easier to refresh after takedowns.

The phishing-kit economy got an AI upgrade

Outsider’s alleged operation still relied on old fraud basics: impersonated brands, urgent messages, fake account warnings, package delivery claims, and pages designed to harvest personal data and banking details.

The difference is production speed.

Before generative AI, scam crews already used phishing kits, reused templates, spam tools, SIM banks, fake storefronts, and social engineering scripts. Google’s complaint, as reported by TechCrunch, says Outsider Enterprise included different groups handling software development, target lists, bulk texting infrastructure, and monetization of stolen credentials.

The before and after is stark:

• Before: Fraud kits lowered the barrier for criminals who could buy templates and hosting.
• After: AI-assisted tooling can help produce more convincing site variants and code faster, while the same criminal marketplace handles targeting, texting, and cash-out.
• Before: Reused pages and clumsy replicas gave defenders more static signals.
• After: More variation can make detection harder, especially when domains and URLs rotate quickly.

The core scam hasn’t changed. Trust is still the target. The fake site only has to look credible long enough for a victim to enter credentials, payment details, or multi-factor codes.

This is also different from the AI reliability problem XOOMAR examined in 52% Utility Tax Reveals Faithful Uncertainty's Edge. That debate is about whether models are confidently wrong. This case is about something colder: when a model follows instructions well enough to help criminals produce useful fraud infrastructure.

Google is fighting on legal, carrier, and device fronts at once

Google says it is coordinating with major U.S. carriers and the FBI to disrupt the scam activity and block malicious messages tied to the operation.

That defense stack matters because a lawsuit alone won’t stop a network whose operators are unknown and allegedly based abroad. Ars notes that Google is assisting the FBI’s cybercrime division with a parallel criminal investigation, while TechCrunch reported that the FBI, working with Google and Lumen’s Black Lotus Labs, seized several domains, Shopify storefronts, and accounts used to test the phishing service.

Google is also seeking damages and a court order to stop the alleged activity. The complaint accuses the people behind Outsider Enterprise of impersonating Google brands, copyright infringement, racketeering activity, wire fraud, and false advertising.

XOOMAR analysis: Google has several audiences here. Users need proof that Gemini abuse gets punished. Regulators need proof that voluntary safeguards are not just marketing. AI rivals need to notice that litigation may become part of the standard abuse response, not an exceptional move after reputational damage.

Polished scam pages make trust harder to outsource

For businesses and users, the practical lesson is grim: visual quality is no longer a strong trust signal.

A fake page can look polished. A login flow can feel familiar. A text can refer to a package, account, toll, or service in language that doesn’t scream scam. Google’s filing, as described by Ars and TechCrunch, does not publish every prompt or every Gemini output, so the exact division between AI-written code, copied templates, and human work remains unclear. But the alleged product was designed to help criminals create convincing replicas fast.

That shifts the burden to signals that are harder to fake at scale:

• Domains: Users and companies need to treat lookalike URLs as primary risk indicators, not fine print.
• Payments: Banks and processors need behavioral signals, not just merchant names and page appearance.
• Brand monitoring: Companies whose names are spoofed need faster detection across domains, ads, texts, and hosting.
• User reports: Google’s 55,000 flagged spam texts show how reporting can feed enforcement, but reports arrive after exposure.

The old advice to “look for typos” is inadequate. AI can make bad sites read better. Verification has to move closer to origin, identity, payment rails, and domain intelligence.

Courts now become part of AI abuse control

Google is backing several federal proposals, including the National Strategy for Combating Scams Act, the Strategic Task Force on Scam Prevention Act, the AI Plan Act, and the Artificial Intelligence Public Awareness and Education Campaign Act. Ars reports that many of the measures Google supports would push federal agencies toward task forces focused on AI-assisted scams and market manipulation.

The next phase won’t be solved by prompt filters alone. AI vendors will need account bans, model safeguards, infrastructure intelligence, carrier partnerships, domain seizures, law enforcement coordination, and civil litigation. Cybercrime groups will adapt too, by shifting accounts, tools, hosts, and domains.

The evidence that would strengthen Google’s thesis is clear: seized infrastructure, identified operators, reduced message volume, and fewer successful phishing pages using Google brands. The evidence that would weaken it is equally clear: Outsider-style services reappearing quickly under new names, using the same tactics with minimal disruption.

AI companies won’t eliminate automated scams. They can raise the cost. The test is whether they can do it before AI-generated deception becomes ordinary.

Impact Analysis

• Google alleges Gemini was used to scale phishing infrastructure, not just generate harmful text.
• The case highlights how AI tools can lower the technical barrier for large-scale fraud operations.
• The lawsuit could influence how platforms police AI-assisted cybercrime and abuse of their brands.