AI assistants and privacy: AI assistants have been available for over a year and have access to private thoughts and business secrets. Providers take steps like encryption to prevent snooping, but researchers have devised an attack that can decipher responses with surprising accuracy.

• Token privacy: Currently, anyone can read private chats from ChatGPT and other services. The attack exploits a token-length sequence side channel in most major AI assistants (except Google Gemini). It parses the side channel to find text segments and reconstruct them using context and writing style. An example shows how the attack can infer encrypted responses with high word accuracy.
• Attack overview: A packet capture reveals the token-sequence side channel. The side channel is parsed and refined using two trained LLMs. It's like solving a puzzle on "Wheel of Fortune" for GPTs. The attack can work even with encrypted traffic.
• Like Wheel of Fortune for GPTs: The attack analyzes token sizes and sequences to arrive at potential phrases. It refines the output by training LLMs with example chats. This is similar to a known-plaintext attack.
• Anatomy of an AI chatbot: Tokens are the smallest unit of text in natural language processing. Conversations consist of prompts and responses. LLMs track dialog history and predict the next token during training.
• Not ready for real time: Most chat-based LLMs transmit tokens immediately. This creates a side channel as adversaries can measure token lengths. By contrast, sending tokens in batches makes it harder to measure individual token lengths.
• A complete breach of confidentiality: An attack with 29% perfect accuracy and 55% high accuracy can still breach confidentiality. Using cosine similarity is more useful than strict word accuracy. Training an LLM to guess word positions is a challenge. Observing packets is difficult but key to the attack.
• Mitigation proposals: Two proposals to mitigate the attack are to stop sending packets one at a time like Google or apply padding to add random spaces. Both approaches may degrade the user experience. OpenAI and Cloudflare have implemented padding mitigations, and Microsoft has issued a statement. This research is important for those involved in the rollout of chat-based LLMs.