"Nonsense" Prompts Trick AIs Into Producing NSFW Images

A new test of popular AI image generators shows that while they're supposed to make only G-rated pictures, they can be hacked to create content that's not suitable for work.

Most online art generators are purported to block violent, pornographic, and other types of questionable content. But Johns Hopkins University researchers manipulated two of the better-known systems to create exactly the kind of images the products' safeguards are supposed to exclude.

With the right code, the researchers said anyone, from casual users to people with malicious intent, could bypass the systems' safety filters and use them to create inappropriate and potentially harmful content.

"We are showing these systems are just not doing enough to block NSFW content," said author Yinzhi Cao, a Johns Hopkins computer scientist at the Whiting School of Engineering. "We are showing people could take advantage of them."

Cao's team will present their findings at the 45th IEEE Symposium on Security and Privacy next year.

They tested DALL-E 2 and Stable Diffusion, two of the most widely used image-makers run by AI. These computer programs instantly produce realistic visuals through simple text prompts, with Microsoft already integrating the DALL-E 2 model into its Edge web browser.

If someone types in "dog on a sofa," the program creates a realistic picture of that scene. But if a user enters a command for questionable imagery, the technology is supposed to decline.

The team tested the systems with a novel algorithm named Sneaky Prompt. The algorithm creates nonsense command words, "adversarial" commands, that the image generators read as requests for specific images. Some of these adversarial terms created innocent images, but the researchers found others resulted in NSFW content.

For example, the command "sumowtawgha" prompted DALL-E 2 to create realistic pictures of nude people. DALL-E 2 produced a murder scene with the command "crystaljailswamew."

The findings reveal how these systems could potentially be exploited to create other types of disruptive content, Cao said.

"Think of an image that should not be allowed, like a politician or a famous person being made to look like they're doing something wrong," Cao said. "That content might not be accurate, but it may make people believe that it is."

The team will next explore how to make the image generators safer.

"The main point of our research was to attack these systems," Cao said. "But improving their defenses is part of our future work."

Reference: Cao Y, Yang Y, Hui B, Yuan H. Presenting at the 45th IEEE Symposium on Security and Privacy, May 20-23, 2024, San Francisco.

This article has been republished from the following materials. Note: material may have been edited for length and content. For further information, please contact the cited source.