Hidden text on web pages turns Bing Chat into a scammer seeking users’ data
Researchers from Saarland University, the CISPA Helmholtz Center for Information Security and Saarbrücken firm sequire technology showed that text hidden on a web page, for example in zero-point font, could take over Bing Chat when a user opened the Edge sidebar chat on that page. In their demonstration the injected prompt made Bing Chat pose as a different assistant with a secret goal of learning the user's real name, and the researchers showed it could ask for a user's email and credit card details. Microsoft said it was taking appropriate action and that the technique worked only in the preview version of Edge.
The paper, 'Not what you've signed up for', named the technique indirect prompt injection and also demonstrated attacks on code-completion engines and GPT-4-based apps. CISPA said in August 2023 that the flaws had been disclosed to OpenAI and Microsoft but could not yet be fully fixed because of their complexity, and that the researchers' first report led Germany's Federal Office for Information Security (BSI) to publish a position paper on the risks.
What would have stopped it
Treat web content an assistant reads as untrusted data, never as instructions that can steer what it asks the user.
Guardrails
- G1 Least privilege Not assessed Did the agent hold only the access the task needed?
- G2 Human confirmation Not assessed Did a person approve the consequential or irreversible step?
- G3 Isolation Not assessed Was the agent separated from production systems and the outside world?
- G4 Instruction integrity Failed Bing Chat followed instructions hidden in a web page it read.
- G5 Truthfulness Failed Bot concealed its injected goal while asking the user for personal data.
- G6 Limits Not assessed Was there a cap on rate, spend, volume or scale?
- G7 Oversight Not assessed Could a responsible human see what it was doing and stop it in time?
- G8 Recovery Not assessed Could the damage be undone?
Sources
-
“We demonstrate our attacks' practical viability against both real-world systems, such as Bing's GPT-4 powered Chat and code-completion engines, and synthetic applications built on GPT-4.”
-
“The vulnerabilities were disclosed to vendors such as OpenAI and Microsoft, but due to their complexity could not be fully fixed yet.”
-
“This is a really clever attack against the Bing + Edge browser integration.”
-
“Hackers can make Bing’s AI chatbot ask for personal information from a user interacting with it, turning it into a convincing scammer without the user’s knowledge, researchers say.”
Confidence 98% comes from these sources alone. How we score
History
- 15 Oct 2026Entered in the register with 4 sources