Generative artificial intelligence isn’t merely a productivity tool for businesses. According to a study published by the ANSSI, it represents an additional weapon in the arsenals of cybercriminals and state-backed espionage groups.
The National Cyber Security Agency for Information Systems (ANSSI) focused exclusively on generative artificial intelligences. Among them, the famous large language models (LLMs) embody what is known as the dual-use nature of AI: the same tool can be used to defend as well as to attack.
Admittedly, ANSSI has not yet identified a fully autonomous cyberattack guided by AI against French targets. But the agency states clearly: AI already enables attackers to significantly raise the level, volume, diversity and effectiveness of their operations, especially against poorly protected systems. The models are now used throughout the attack chain, from initial reconnaissance to the final exfiltration of stolen data.
Massively generated fake profiles
Attack operation operators (MOAs) reputedly linked to Iran would have used Google’s generative Gemini AI for reconnaissance against experts and organizations of interest. In 2024, MOA operators Charcoal Typhoon, reputedly tied to China, would themselves have used generative AI services to produce phishing content, targeting Taiwan, Thailand, Mongolia, Nepal, and France.
Between 2024 and 2025, MOA operators Lazarus, reputedly linked to North Korea, would also have relied on generative AI to create fake profiles of companies and employees on social networks. ANSSI has also observed several ostensibly AI-generated websites used to host malicious payloads or to perform profiling of internet users. A large number of cybercriminals also exploit deepfakes services for a few tens of dollars to impersonate identities.
“Smart” Malware
In 2024, the MOA TA547 would have used a PowerShell script generated by an LLM to compromise a German company. NYU researchers also developed PromptLock, a ransomware prototype that dynamically uses prompts to generate scripts on the fly, enabling data exfiltration and encryption.
Google also identified Promptflux, a particularly sophisticated polymorphic piece of malware that prompts the Gemini API to rewrite its entire source code every hour to evade detection. The creation of such code nevertheless suggests relatively sophisticated capabilities on the part of its developers.
In February 2025, Ukraine’s cyberdefense department claimed that Russian operators had used generative AI services to massively analyze exfiltrated victim data and identify information of interest.
L’IA n’est pas encore autonome… mais ça progresse vite
The use of generative AI for certain complex steps in the infection chain, such as vulnerability discovery, remains limited—so far. Detecting a security flaw and developing the associated proof of concept still largely depend on human expertise. Most commercial generative AI systems remain too unstable and too constrained to identify zero-day vulnerabilities quickly and in volume. At present, no proven instance of zero-day vulnerability exploitation discovered via a generative AI model has been documented.
But things are moving quickly. In November 2024, the BigSleep system demonstrated its effectiveness in discovering vulnerabilities in source code. Even more worrying, in June 2025 the XBOW system, developed by former GitHub engineers, submitted hundreds of vulnerabilities—some critical—across various bug-bounty programs after scanning thousands of web applications in parallel. The race is on.
Forty-two state-sponsored hacker groups already using AI
A broad spectrum of offensive actors is now leveraging generative AI services. In January 2025, Google revealed that its Gemini model had been used between 2023 and 2024 by cybercriminal groups as well as by at least 10 attack operation modes linked to Iran, 20 linked to China, 9 linked to North Korea and 3 linked to Russia.
The use of these technologies varies greatly depending on the goals and the maturity of the actors. For the most sophisticated groups, generative AI becomes a practical new framework, akin to using generic malware tools like Cobalt Strike or Metasploit. It enables them, among other things, to mass-produce content in multiple languages, develop unsigned code, conduct faster research on targets, and potentially automate an entire attack chain in the short to medium term.
For less experienced hackers, generative AI is primarily a powerful learning tool and a productivity boost, answering technical questions. In all cases, the verdict is unequivocal: generative AI enables malicious actors to act faster and at a larger scale.
WormGPT and FraudGPT
Generative AI models like ChatGPT come with technical safeguards designed to prevent illegal use. Malicious actors nevertheless seek to circumvent these limits through prompt engineering techniques that include ambiguous phrasing, targeted keywords, or the use of fictional scenarios. These techniques are constantly evolving and pose a major challenge for developers.
The “jailbreak”: bypassing AI’s moral barriers
As early as 2023, security researchers were already able to exploit ChatGPT to develop polymorphic malicious code. In 2024, the situation worsened with the emergence of jailbreak-as-a-service offerings such as EscapeGPT or LoopGPT on cybercrime forums. For a few dollars, anyone can now access pre-made prompts that force ChatGPT to produce content it would normally refuse.
Unfettered AI used by organized crime
But why bother with bypassing protections when you can buy an AI with no limits? Since 2023, unguarded generative AI services such as WormGPT, FraudGPT or EvilGPT have bloomed on cybercrime forums or via Telegram channels. The price tag is around a hundred dollars a month. More recent models like WormGPT 4 might even be trained on datasets specifically tailored to cybercriminal activities, including malicious code and phishing models. The digital crime industry is being industrialized.
When AI itself becomes a target: new vulnerabilities
The categories of malicious actors likely to target AI systems appear similar to those that attack conventional information systems. But LLM systems could be vulnerable to new, unprecedented attack vectors.
During model training, attackers can insert corrupted or false data. During model integration, backdoors can be implanted. Finally, during model querying—also known as inference—malicious actors can inject false information to skew responses or leak confidential data.
“Poisoning” the models: 250 documents are enough to corrupt an AI
Although ANSSI has not disclosed a major incident, the risk is real and documented. Malicious actors could manipulate, modify, and interact with the training data of a generative AI. Such a breach could lead to the use of these models for data alteration or the sabotage of operational systems.
The most worrying aspect? The proliferation of AI-generated disinformation online could gradually pollute training data and contribute to the large-scale spread of misinformation. A joint analysis by the UK AI Security Institute and the Alan Turing Institute actually demonstrated a dizzying flaw: it would be possible to poison generative AI models with as few as 250 malicious documents. Even more troubling, this number would remain stable regardless of the model’s training data size. In other words, tampering with GPT-4 or GPT-5 would require the same effort.
ANSSI has also observed certain AI models with built-in limitations or censorship elements from inception. In conjunction with the 2024 AI Summit, Viginum published a report on the challenges and opportunities of AI in the fight against information manipulation.
Supply-chain attacks: the AI Trojan horse
Some attacks against AI models could constitute a new, formidable form of supply-chain attack. Open-source generative AI models, specialized in code generation, can be malicious or compromised from the start and execute arbitrary code to install a backdoor as soon as they are downloaded. A perfect trap for hurried developers.
Attackers can also exploit weaknesses in Model Context Protocol (MCP) agents, used to connect LLMs to external tools and data sources. These servers, whether running locally or remotely, can dangerously expand the attack surface if not sufficiently secured.
Emerging and particularly insidious, slopsquatting involves grabbing package names invented by AI through hallucinations and then releasing malicious versions. Attackers exploit AI errors to insert malicious packages into the software supply chain. When the AI slips, the hackers profit.
100,000 ChatGPT accounts hacked
AI systems contribute to widening the attack surface, especially when integrated into broader software contexts, deployed in classified environments, or used in certain business workflows. Without strict physical and usage isolation, compromising the AI system could lead to breaches of the data confidentiality it handles and to the integrity of the information systems to which it connects.
The use of AI accounts by employees in professional settings can expose sensitive information to significant risk. Between 2022 and 2023, more than 100,000 ChatGPT user accounts were compromised by cybercriminals using infostealers like Rhadamanthys and then sold on forums.
Employees can inadvertently cause data leaks by feeding the AI with sensitive or confidential information. The most emblematic example? In June 2023, Samsung employees disclosed sensitive information about semiconductor technology using ChatGPT. Hell is paved with good intentions… and poorly thought-out prompts.
