Before an AI prototype goes live: security checks that matter

Map what the assistant can see and do

List every data source, integration, tool and permission. Separate read-only retrieval from actions that change records or trigger external workflows. Give each component the minimum access required. A model should not inherit broad permissions simply because they make a demo easier.

Treat retrieved content as untrusted input

Documents, webpages, emails and uploaded files can contain instructions that conflict with the application’s rules. Do not assume retrieved text is safe because it came from a company knowledge base. Keep system instructions separate from retrieved content, validate sources and constrain what the model can do with the information.

Protect sensitive information

Decide which data may be sent to a model provider, what must be masked and how conversation logs are retained. Avoid returning secrets, access tokens or other users’ records. Use server-side authorisation checks for every protected resource; a prompt telling the model to respect permissions is not a substitute for application security.

Put approval gates around consequential actions

For actions such as sending messages, changing customer records or making financial decisions, use explicit confirmation, narrow tool schemas and auditable records. Consider requiring a human to approve high-impact actions. The assistant should explain when it cannot complete a request safely.

Test attacks and failures, not only happy paths

Test prompt injection, malicious documents, tool misuse, excessive output, cross-user data access, timeouts, provider outages and rate-limit behaviour. Monitor unusual patterns and provide a way to disable an integration quickly. Re-run the test suite after changing prompts, models, data sources or tools.

Putting the ideas into practice

AI security is application security plus model-specific risks. Build least privilege, source controls, privacy, evaluation, human approval and monitoring into the architecture from the beginning. A disclaimer alone does not make an unsafe workflow safe.