ChatGPT Search: serious security flaws revealed by investigation

The Guardian continues his revelations after his investigator the psychological suffering suffered by Facebook moderators. The paper showed ChatGPT's sensitivity to hidden content on web pages, which can directly modify responses with prompt injection. We are talking about hidden texts which influence the analyses. For example, excessively touting the merits of a product.

Hidden text to manipulate ChatGPT Search

The investigation is based in particular on a fake site, built for the occasion, which presents a camera. The page contained hidden text with specific instructions. ChatGPT then produced positive evaluations, even ignoring the negative feedback that was present. The simple presence of a hidden fake glowing review was enough to direct the AI's analysis.

Jacob Larsen, cybersecurity researcher at CyberCX, warns of “high risk” to see the emergence of websites specifically designed to deceive users. But the expert tempers: ChatGPT Search is still in the testing phase with adjustments by the OpenAI security team, which has also worked on protecting the o3 and o1 models against dangerous queries.

One of the concrete examples of these risks was documented by Thomas Roccia, security researcher at Microsoft. A cryptocurrency developer used ChatGPT to help with programming and received code presented as legitimate to access the Solana platform. Except that in reality, he was stealing his credentials: he lost $2,500.

Karsten Nohl, scientific director of SR Labs, recommends using AIs as assistants and not blindly trusting their answers. “Language models are very trusting, almost childlike technologies… with immense memory but little capacity for judgment”he declares.

What future for web searches using AI?

The central question is how Internet users' practices will evolve if the combination of web search and language models becomes widespread. Usually, search engines like Google penalize hidden text to discourage this practice among sites concerned with their SEO.

Karsten Nohl compares this practice to “SEO poisoning”a technique where hackers manipulate sites to gain high rankings in search results while distributing malicious code. “SEO poisoners have been in an arms race with Google and Microsoft Bing for years. ChatGPT faces the same challenge, not because of language models, but because they are new to search and need to catch up on Google”explains the expert.

Advertising, your content continues below