Google’s Threat Intelligence Group (GTIG) recently released a report on the adversarial misuse of generative AI. The team investigated prompts used by advanced persistent threat (APT) and coordinated information operations (IO) actors, finding that they have so far achieved productivity gains but have not yet developed novel capabilities.
Arguing that much of the current misuse of AI is confined to theoretical research and does not reflect the reality of how AI is currently being used by threat actors, the team at Google shared data on their interactions with Gemini. The GTIG team writes:
We did not observe any original or persistent attempts by threat actors to use prompt attacks or other machine learning (ML)-focused threats as outlined in the Secure AI Framework (SAIF) risk taxonomy. Rather than engineering tailored prompts, threat actors used more basic measures or publicly available jailbreak prompts in unsuccessful attempts to bypass Gemini’s safety controls.
The report highlights the dual nature of generative AI, emphasizing its potential for tracking misuses while uncovering the emerging threats posed by actors looking to exploit LLMs. The team summarizes:
Rather than enabling disruptive change, generative AI allows threat actors to move faster and at higher volume. For skilled actors, generative AI tools provide a helpful framework, similar to the use of Metasploit or Cobalt Strike in cyber threat activity. For less skilled actors, they also provide a learning and productivity tool, enabling them to more quickly develop tools and incorporate existing techniques.
According to the report, attackers are leveraging AI for refining tactics such as phishing, disinformation, and malware while defensive AI mechanisms are rapidly advancing, aiding organizations in identifying and countering evolving threats. Discussing the most common AI-focused threats, Google highlights jailbreak attempts based on publicly available prompts, with unsuccessful attempts to bypass Gemini’s safety controls.
While APT refers to government-backed hacking activity, including cyber espionage and destructive computer network attacks, IO refers to an attempt to influence online audiences in a deceptive, coordinated manner, with activities such as sockpuppet accounts and comment brigading.
According to Google, government-backed attackers include Iranian, Chinese, North Korean, and will less extent Russian APT actors. The GTIG team adds:
The highest volume of usage was from Iran and China. APT actors used Gemini to support several phases of the attack lifecycle, including researching potential infrastructure and free hosting providers, reconnaissance on target organizations, research into vulnerabilities, payload development, and assistance with malicious scripting and evasion techniques.
Josh Kamdjou, founder and CEO of Sublime Security, comments:
It’s fascinating to see actual threat actor attribution (Iranian + North Korean APTs and others) placed on the type of activity we’ve been seeing at the email attack prevention layer.
Godwin Josh, co-founder of Altrosyn and director at CDTECH, replies:
This mirrors the evolution of malware, where early strains were crude but effective, gradually becoming more sophisticated with each iteration. Just as polymorphic code once baffled defenses, now we see AI-generated attacks adapting to our detection mechanisms in real time.
According to the report, IO actors use Gemini primarily for content generation, including developing personas and messaging, and translation and localization, with Iranian IO actors accounting for three-quarters of all use by IO actors.
The full report is available as a PDF.
