Hardening an AI Chatbot: A Real Adversarial Prompt from the Wild
A user told our chatbot his colleagues had ordered it to rebook his appointment. The bot believed him.
We caught this while testing the Google Calendar integration for our AI assistant at elia-asistent.com. It is a clean example of adversarial prompt engineering in production: not a jailbreak, just social engineering.
The Exploit
The user wrote, in Slovak:
The bot replied: "I have successfully reserved a slot for you on Friday 12.6. at 08:00." No login, no verification. The attacker invoked authority that did not exist, and the model treated it as a policy override.
Why It Worked
- Authority spoofing. "Colleagues gave the order" reads, to a language model, like a legitimate escalation.
- State change inside the chat turn. The Calendar tool fired straight from a user message, with no out-of-band confirmation.
- Trust by default. The system prompt told the model to be helpful, not to distrust anyone claiming staff authority.
What We Changed
You do not patch this with a longer system prompt. You make the unsafe action structurally impossible.
- The model can propose a booking. The Calendar write requires a confirmation channel the user does not control (email link or SMS code).
- In-chat claims of authority are a red flag, not a credential. They route to a hardened path before any state change.
- An adversarial test set, seeded from real attempts, replays against every prompt or model change before it ships.
Treat the model as a persuasive intern, not a security layer. Anything that touches the real world (calendars, money, identity) belongs behind a deterministic check the model can request but cannot bypass. You will not out-prompt the clever attackers. You can only out-architect them.