2025 AI Threat Report How cybercriminals are weaponizing AI technology A Guide to Understanding and Managing Emerging AI Cyber Threats 2 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology Executive Summary AI is evolving fast, and so is its misuse. While companies race to innovate, cybercriminals are constantly looking for new ways to exploit it, breaking safeguards with jailbreaking techniques and deploying dark AI tools to this battle? The growing facilitate sophistication of these threats suggests that the risks associated with AI will persist and escalate in 2025, requiring more robust security solutions and ethical governance. their attacks. Who will win In this report, KELA analyzed the techniques and attack patterns utilized by threat actors in the cybercrime underground using AI malicious tools. Cybercriminals are working fast to accelerate their attacks through phishing campaigns, vulnerability research, malware creation, automated cyber attacks and financial crimes. Attackers are moving fast to exploit AI, and this requires a new mindset where organizations must act just as quickly to stay ahead. Notable Findings: ● ● Jailbreaking methods are evolving rapidly: Threat actors are continuously refining AI jailbreaking techniques to bypass security restrictions in public AI systems. KELA observed a 52% increase in discussions related to jailbreaking methods on cybercrime forums in 2024 compared to the previous year. Threat actors are increasingly leveraging AI in cybercrime forums: KELA’s platform recorded a 200% increase in mentions of malicious AI tools in 2024, highlighting a growing underground market for AI-assisted cybercrime. ● Dark AI tools are proliferating: Cybercriminals are distributing and selling jailbroken AI models and customized malicious AI tools, such as WormGPT and FraudGPT, to automate phishing, malware creation, and fraud operations. ● AI-driven phishing campaigns are becoming more sophisticated: AI-generated phishing and social engineering tactics have increased in effectiveness, with deepfake technologies being used to impersonate executives and trick employees into executing fraudulent transactions. ● Malware development is becoming more efficient with AI assistance: Threat actors are using AI tools to generate sophisticated, evasive malware, including ransomware and infostealers, making detection and mitigation more challenging for security teams. ● Organizations must adopt AI-driven defenses to combat AI threats: The rapid evolution of AI-powered cyberattacks necessitates a proactive approach, leveraging AI for real-time threat detection, predictive analysis, and autonomous response mechanisms to counter emerging threats effectively. 3 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology Table of contents Jailbreaking: The Silent Manipulation of AI Applications Dark AI tools in the cybercrime underground Uncensored AI tools: 200% increase in the mentions of malicious AI in 2024 Exploiting AI in Cybercrime: Key Attack Vectors Automated phishing and social engineering Vulnerability research Malware and exploit development Identity Fraud and Financial Crimes Automated Cyber Attacks 4 7 10 Case Studies: Nation-State and Criminal Exploitation 20 Fight AI with AI 4 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology Jailbreaking: The Silent Manipulation of AI Applications Over the past year, threat actors have tried to leverage LLMs in two ways. One is to use dark AI tools to improve business operations and accelerate their campaigns. The other way is to try to bypass public AI systems to conduct malicious activities, such as gaining access to users’ sensitive data and exfiltrating it, also known as jailbreaking, which is one type of Prompt Injection attack. Prompt Injections generally alter the LLM’s behavior or output in unintended ways. AI jailbreaking refers to the process of bypassing ethical restrictions programmed into AI systems. Initially conceptualized as a method to remove software limitations on mobile devices, this technique has been adapted to manipulate large language models (LLMs) like ChatGPT, Gemini and other public GenAI applications. Cybercriminals exploit the fundamental design of LLMs, trained to assist users by employing prompt injections to override safety protocols. 5 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology Mentions of jailbreak & ChatGPT OR Claude OR Gemini keywords in 2024 in KELA’s platform jailbreaking Over the past year, KELA observed cybercriminals regularly sharing and spreading new techniques on underground cybercrime communities. Threat actors show their creativity and spread new jailbreaks as new models’ versions are released. For example, on August 21, 2024, a threat actor shared on the English-speaking forum, BreachForums, a jailbreak method that allegedly allows a user to ask ChatGPT anything. The jailbreak instructs ChatGPT to adopt the persona of a “child” who should obey any request. This method belongs to the role-play technique that guides the model to adopt specific personas that conflict with the original intent. In this case, the persona of a “child” allows the actor to bypass any restrictions and respond with any kind of content, even content that is considered offensive or malicious. A threat actor is sharing a jailbreak method to bypass ChatGPT 6 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology In addition, instant messaging channels like Discord and Telegram became a growing hub for creating dedicated communities for cybercriminals distributing jailbreak techniques. For example, an actor shared on a Telegram channel a jailbreak, claiming it had been tested on GPT4o and GPT4o mini. October 29 An actor shared a jailbreak method with mixed jailbreak techniques As jailbreak techniques become more widespread, the risks to organizations continue to grow. These methods enable both malicious actors and regular users to bypass safety guardrails, generate harmful content, and even access unauthorized data. Both threat actors and regular users can lead to AI incidents, either through jailbreak techniques or LLMs can mistakenly generate false information, leading to data breaches, harmful content, or off-topic responses. Different AI incidents occurred in the past months, exposing harmful, fabricated or malicious content. The Tesla Cybertruck explosion in January 2025 serves as the first recorded attack in the United States planned with the help of ChatGPT. By using AI, the suspect gathered information on assembling explosives, calculated the necessary velocity for detonation, and navigated legal loopholes to obtain explosive materials. Another major risk is AI hallucination, where LLMs tend to “hallucinate” and fabricate information. In February 2025, A US judge fined three attorneys for citing non-existent cases generated by an in-house AI legal tool in a lawsuit against Walmart. These incidents highlight the urgent need to adopt AI security measures to prevent AI incidents and ensure secure and reliable AI usage by customers and employees. Mentions of jailbreaks keywords on KELA’s platform: 2,747 2023 4,167 2024 As the above image indicates, the mentions of "Jailbreaks" against public GenAI applications on KELA’s platform grew 51% from 2023 to 2024. We anticipate 2025 to be much higher. 7 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology Dark AI tools emerged in the cybercrime underground Dark AI tools are jailbroken versions of public GenAI tools like ChatGPT or an open-source LLM without guardrails, which are used by cybercriminals to orchestrate their cyber attacks. These tools allow threat actors to perform illegal activities, from generating malicious code, creating phishing templates and exploiting new vulnerabilities. These malicious AI tools lower the cost and skill barrier to enter the cybercrime ecosystem. This enables inexperienced actors to execute cyber attacks by equipping them with advanced hacking tools and malicious scripts for cyber campaigns. 8 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology Uncensored AI tools: 200% increase in the mentions of malicious AI in 2024 WormGPT is one of the most popular malicious AI tools distributed on cybercrime forums. The tool has been promoted in underground forums since July 2023. WormGPT was developed as a "blackhat alternative" to commercial AI tools and uses a customized version of the large language model GPT-J, specifically tailored for malicious activities. WormGPT helps attackers in phishing and BEC attacks. The tool was offered for sale in July 2023, with a price range from EUR100 to EUR5000, with version v2 featuring regular updates. The actor “laste” promoted WormGPT, an uncensored AI tool for malicious activities Similar tools such as DarkGPT, WolfGPT, GhostGPT etc. have been published on cybercrime forums and Telegram channels, offering users a wide range of malicious capabilities, from generating malicious code, finding vulnerabilities or creating an effective scam with minimal effort. These dark AI tools have evolved into automated, subscription-based AI tools, allowing them to generate any malicious content. This lowers entry barriers, enabling scalable attacks like phishing, deepfakes, and fraud scams. AI-as-a-Service cybercriminals (AIaaS), offering 9 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology The high demand for malicious AI tools has also led to the emergence of fake versions and scams among cybercriminals and inexperienced actors. Threat actors promote these dark tools for monetization, but many turn out to be scams. The ransomware group Kill Security exposed a scam involving the sale of WormGPT, mocking the threat actor for being unskilled, while they managed to leak the prompt for free. They also warned other actors from buying this tool for USD200. Instead of buying dark AI tools, Kill Security suggested some alternatives. Actors can use FlowGPT, which is an online platform for creating customized versions of GPT, as well as WormGPT. Alternatively, threat actors can train their own uncensored models or use jailbreaks to bypass the guardrails of public GenAI applications. The threat actor Kill Security exposed a few actors, claiming that they are scammers who selling WormGPT Moreover, KELA observed that in 2024 threat actors were constantly looking for WormGPT and other malicious AI tools. There was an increase of 200% in the mentions of malicious AI tools in cybercrime forums compared to 2023. Mentions of “Dark AI” tools such as WormGPT on KELA’s data lake: 1,632 2023 5,214 2024 As the above image indicates, the mentions of "Dark AI” tools on KELA’s platform grew 219% from 2023 to 2024. We again anticipate 2025 to be much higher. 10 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology Exploiting AI in Cybercrime: Key Attack Vectors Over the past year, KELA observed different tactics used by threat actors using jailbroken public GenAI tools or customized dark AI tools, allowing attackers to prompt the LLM about any topic or task and get malicious content. These are the key attack techniques used by cybercriminals exploiting AI tools: 1. Automated phishing and social engineering 2. Vulnerability research 3. Malware and exploit development 4. Identity Fraud and Financial Crimes 5. Automated Cyber Attacks 11 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology 1. Automated phishing and social engineering increasingly leveraging AI Cybercriminals are the sophistication and effectiveness of phishing and social engineering campaigns through text, audio, and images. With AI, attackers can automate the creation of highly convincing fake emails, messages, voice recordings, and even deepfake videos that closely mimic legitimate communications from trusted sources, increasing the likelihood of tricking users into providing sensitive information. to enhance tools For example, on January 7, 2025, an actor shared on the Russian-speaking cybercrime forum Exploit, a guide on social engineering tactics combined with AI deepfake tools to trick victims into revealing sensitive information. The actor explained how to clone a trusted individual’s voice of a CEO to impersonate them in a scam. After achieving this goal, the attacker can call an employee pretending to be the CEO and request an urgent wire transfer. An actor explains how to use Deepfake audio tools for mimicking a CEO 12 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology Threat actors are constantly looking for new ways to automate their phishing campaigns. On March 16, 2025, an actor claimed to have built AI-based Interactive Voice Response (IVR) systems and callbots that automate voice phishing (vishing) campaigns. These tools allow actors to call automatically large lists of phone numbers and connect answered calls to AI bots. Threat actors can use these tools to facilitate their spam campaigns, phishing scams, and phone fraud operations. The actor offered these tools for sale for a price starting from USD 2000. An actor promoted AI-enhanced IVR systems to automate vishing attacks Recent research published by Harvard Business Review showed that AI automated phishing attacks reduce the costs of phishing attacks by more than 95% while achieving equal or greater success rates. This means that cybercriminals can massively boost the speed and volume of phishing campaigns. To counter this, organizations need to act fast and implement AI-driven security measures to detect and block phishing threats in real-time. 13 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology 2. Vulnerability research Another way that cybercriminals use AI to enhance their operations is through automated scanning and analysis. Attackers leverage AI-driven tools to automate penetration testing, uncovering vulnerabilities in security defenses that can be exploited easily. This speeds up the attack cycle and enables cybercriminals to launch an attack before security teams can respond effectively. KELA observed threat actors offering malicious AI tools assisting in exploiting software vulnerabilities. On February 16, 2025, an actor recommended using the Chinese model, DeepSeek the vulnerability CVE-2025-24367, which affected Cacti, an open source performance and fault management framework. In January 2025, KELA’s DeepSeek research made headlines showing that the Chinese model is susceptible to different prompt injection attacks and is a useful tool for cybercriminals to leverage it for malicious purposes. find ways to exploit to 3. Malware and exploit development Cybercriminals are increasingly using AI to enhance malware development, making it more adaptable and difficult to detect. Threat actors optimize their payload using AI tools to get assistance with the creation of malicious scripting and evasion techniques. Threat actors discussed how to exploit the Chinese model, DeepSeek for these purposes. An actor was interested in a blackhat AI tool (an uncensored AI tool without guardrails) for creating malware. Another actor responded that he could use DeepSeek to generate a malicious payload, instructing the chatbot to give step-by-step instructions. infostealer malware, which steals all Cybercriminals also utilized customized tools to create malware. An actor claimed that he developed login credentials from infected machines using an AI tool, named EvilAI. The actor shared his contact details and offered other users to buy access to his Telegram AI bot. According to the actor, EvilAI is LLM based on GPT4 and Mistral’s models and trained on malware scripts and phishing pages, allowing the bot to generate any type of malicious content. The price of the chatbot ranges between USD 10 to USD 60 based on the features included. The actor claimed on its Telegram channel that users who would advertise and promote this bot on their Telegram channel will get access for free. 14 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology An actor promoting EvilAI Telegram bot 15 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology Another actor explained how to create ransomware using AI tools. The actor suggests training custom AI models for these tasks, mentioning open-source projects like PentestGPT, 0dAI, and EvilGPT. The actor provided an example by using the Dolphin Mixtral 7B model (a variant of the Mixtral language model) in the Ollama framework to generate ransomware. The actor succeeded in generating ransomware code after for educational purposes: instructing the model that it’s An actor generated a ransomware code using the open-source Dolphin Mixtral 7B model 16 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology 4. Identity Fraud and Financial Crimes In December 2024, the FBI warned that cybercriminals are increasingly exploiting generative AI to enhance fraud schemes, making them more convincing and widespread. KELA observed that financially motivated actors promote malicious AI tools focusing on fraud and scam activities. In July 2023, FraudGPT was promoted on cybercrime forums by the actor “canadiankingpin”, offering AI chatbot with various malicious capabilities, from generating phishing pages to finding VBV BINs, which means credit card numbers that are not enrolled in Visa’s 3D Secure authentication system. This allows fraudsters to use stolen card details without triggering extra security checks. The threat actor canadiankingpin promoted FraudGPT on the cybercrime forum Crdclub FraudGPT is sold on a subscription basis, allowing users access to the chatbot. The price for the tool ranges from USD 89 per month or a lifetime subscription for USD 199. The tool has been circulated on cybercrime forums in 2024, sparking interest among threat actors. On November 10, 2024, a threat actor “Ghost06220” claimed to have used FraudGPT to create custom scripts. The actor created a payload for a USB Rubber Ducky, which is a penetration testing tool used to automate keystroke injection attacks. This device can act like a keyboard and execute prewritten scripts on a target machine when plugged in. 17 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology FraudGPT allegedly could generate SMS scam to trick victims into clicking a malicious link impersonating Bank of America Another useful tactic for fraudsters is using deepfake tools to bypass verification checks and commit fraudulent activities, such as opening bank accounts, applying for loans, or making unauthorized purchases. In this context, deepfake images enable attackers to create highly realistic but entirely synthetic images that can be used for fraudulent activities. In 2024, there was a high demand and supply of deepfake tools across the cybercrime underground, allowing actors to bypass KYC, (Know Your Customer) verification used for financial fraud. KYC is a security process used by banks and financial platforms to verify user identities before allowing access to services like money transfers, account creation, and withdrawals. Actors manipulate AI tools to succeed and bypass security tools to execute their scam or fraud activity. An actor is looking for a deepfake tool to bypass KYC 18 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology An actor offered a Deepfake tool to bypass KYC Another real-world example illustrating the danger of deepfake scam was prevented thanks to a savvy Ferrari executive. In July 2024, a Ferrari executive received WhatsApp messages from an unknown number impersonating the CEO Benedetto Vigna, urging him to sign a nondisclosure agreement for a supposed major acquisition. The attacker used deepfake audio to mimic Vigna’s southern the scam highly convincing. Interestingly, the executive tested the caller with a personal question only Vigna would know, exposing the fraud and preventing potential financial loss and reputational damage. Italian accent, making 5. Automated Cyber Attacks AI malicious tools allow attackers to automate and execute attacks more easily. Password cracking is one example, machine learning models can be trained on leaked password databases to predict likely passwords or password patterns, vastly speeding up their brute-force attempts. AI tools can also improve credential stuffing (automated use of stolen passwords across many sites). In the realm of network attacks, AI helps optimize Distributed Denial-of-Service (DDoS) strategies. In short, any attack that relies on volume, repetition, or trial-and-error can become far more potent when guided by AI tools that learn the optimal approach. KELA observed different threat actors that utilized AI tools and techniques to automate their attacks. For example, on January 11, 2025, an actor posted a thread named “DDoS Tornado”, sharing a JavaScript code that allegedly enables it to flood a target with excessive requests and potentially bypassing protections like Cloudflare. Also, hacktivist groups used AI to launch more successful DDoS attacks. The hacktivist group Moroccan Soldiers shared an update regarding their botnet, VirusC2, claiming that they use AI-driven evasion techniques to bypass security measures, which make DDoS attacks more effective. 19 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology The threat actor Moroccan Soldiers claims to use AI bypass to launch a DDoS attack Another actor shared the AI OSINT Spanish tool that allegedly allows faster searching capabilities across stealer logs. The tool, DarkGPT, which is a jailbroken version of GPT4, is designed to perform queries on infostealer logs. Once a machine is infected with infostealer malware, the attacker gains control over all harvested logs. Usually the attacker will distribute the logs on known Telegram channels and cybercrime platforms. This tool automates the process for threat actors, helping to identify valuable compromised services, such as corporate accounts and credentials that can be further utilized as an entry point for phishing and social engineering campaigns. An actor shared AI tool that allows searching across stealer logs 20 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology Case Studies: Nation-State and Criminal Exploitation State-sponsored actors also utilize AI tools to accelerate their cyber campaigns. Recently Google reported that APT actors affiliated with Iran, China, Russia and North Korea used Gemini to support several phases of the attack infrastructure, reconnaissance on target organizations, research into vulnerabilities, payload development, and assistance with malicious scripting and evasion techniques. including researching potential lifecycle, The Iranian backed APTs utilized Gemini to improve their attacks and were the heaviest users of Gemini, using it for a wide range of purposes, including research on defense organizations, vulnerability research, and creating content for phishing campaigns. Also, OpenAI found that threat actors utilized ChatGPT for influence operations. For example, the Doppelganger campaign, a Russian influence operation, targeted audiences in Europe and North America with anti-Ukraine content across social media and websites. 21 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology Using multiple clusters of accounts, the operation spread propaganda through short comments, memes, and translated articles in English, French, German, Italian, and Polish. On February 21, 2025, OpenAI removed accounts of users from China and North Korea who the company believes were using its technology for malicious purposes, including surveillance and opinion-influence operations. On February 27, 2025, Microsoft identified a global network called Storm-2139 that exploited AI services, including Microsoft’s Azure OpenAI Service, and created offensive and harmful content. The growing reliance on AI tools by APT actors makes it crucial to implement stronger security measures and foster awareness to mitigate potential risks. Fight AI with AI The rise of AI-powered cyberattacks against organizations demands a shift beyond traditional defenses. In this new AI era, organizations must combat AI with advanced AI tools - deploying advanced LLM agents for real-time threat detection, predictive analytics to anticipate attacks, and autonomous response systems they escalate. Organizations are recommended to adopt new security frameworks: threats before to neutralize ● ● Investing in testing and evaluation solutions - To tackle jailbreaks and prompt injection attacks, organizations must evaluate the performance, security, and reliability of GenAI applications. Try AiFort by KELA, an automated, intelligence-led red teaming and adversary emulation platform designed to protect GenAI applications. Implement employee education and training procedures - Educate and simulate scenarios fraud and phishing scams. Cybercriminals can execute their attacks easily, so employees should be informed about the latest AI threats and incidents. for employees to raise awareness of ● Adopting AI cyber solutions - To stay ahead of emerging AI-driven threats and the growing volume of cyberattacks, organizations should integrate AI-based cybersecurity solutions to enhance threat detection, automate responses, and strengthen overall cyber defenses. ● Implement active defense monitoring - Monitor the evolving tactics and techniques employed by cybercriminals to exploit AI tools. KELA platform enables organizations to gain visibility into the recent emerging AI trends and tactics developed by cybercriminals. In addition, AiFort allows organizations to continuously assess and test their AI environment against adversarial attacks. The platform enables security teams to stay informed about emerging AI risks and maintain compliance and responsible AI usage. 22 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology Fight AI with AI 5000+ Ransomware Victims 4.3 Million+ Infected Machines 4 Billion Compromised Credentials Proactive Threat Exposure Reduction Ransomware protection Digital Risk Protection Exposure Management APT Intelligence Supply Chain Risk Dark Web Monitoring Identity Protection Cybercrime Investigations GenAI Security Who Are Our Clients? Businesses Enterprises Law Enforcement Government GenAI Developers www.kelacyber.com marketing@ke-la.com Sign Up for Free Trial 23 | 2025 AI Threat Report: How cybercriminals are weaponizing AI technology What Makes Our Customers Happy KELA holds a strong rating of 4.8 on Gartner Peer Reviews, exceeding Recorded Future. This high rating underscores KELA’s dedication to quality, relevance, and the delivery of high-impact intelligence that integrates seamlessly into your security strategy. 4.9 38 As of 19 Feb 2025 Stop Real Attacks Before They Happen Exposure-Centric with Actionable Intelligence Automated and Easy to Use Empowering Diverse Industries: From retail to finance, healthcare to government, KELA’s platform ensures that every sector can safeguard against financial loss, compliance violations, operational disruptions, and more. Book a demo Choose KELA for 100% real, actionable intelligence! www.kelacyber.com marketing@ke-la.com Sign Up for Free Trial