Generative AI makes privacy assessment deceptively difficult. A user may type personal data into a prompt, retrieve it from an internal knowledge base, receive it in an output, copy the output into another system and leave traces in logs that are controlled by several suppliers.
A conventional checklist that asks only where data are stored can miss most of that journey.
Where processing is likely to result in a high risk to individuals, a data protection impact assessment (DPIA) is not paperwork to complete after procurement. It is a way to identify the purpose, necessity, data flows, risks and controls before the processing is embedded. The UK Information Commissioner’s Office guidance on AI and data protection makes DPIAs a central part of accountable AI risk management.
The following nine questions are a practical starting point for a GenAI DPIA. They are not legal advice and should be adapted to the law, jurisdiction and use case that actually apply.
1. What decision or task is the AI supporting?
Start with the business purpose, not the model. “Use a large language model” is a technology description, not a purpose.
Write a precise statement: for example, summarising customer-service notes for an adviser, drafting replies that a human approves, or searching internal policy documents. Then ask whether personal data are genuinely necessary to achieve it and whether a less intrusive design could work.
Purpose clarity prevents scope creep. A tool approved for drafting should not quietly evolve into scoring people without a fresh assessment.
2. Which personal data enter the workflow?
Inventory the obvious and the unexpected. Prompts may contain names, contact details, health information, employee concerns or customer histories. Uploaded documents may contain metadata or information about people who never interact with the AI directly.
Map data by source, sensitivity, individual and stage of the workflow. Include prompts, attachments, retrieval sources, outputs, feedback, logs and support records.
Professionals responsible for this work can develop AI-specific privacy capability through the Certified AI Data Protection Officer course.
3. What is the lawful basis and what do people reasonably expect?
Identify the lawful basis for each material processing purpose and test whether the proposed use remains compatible with what individuals were told. Special-category or criminal-offence data may require additional conditions.
Do not treat a vendor’s privacy statement as the organisation’s analysis. The controller remains responsible for its own purpose, lawful basis, transparency and risk decisions where those duties apply.
4. Where does the data actually travel?
Draw the end-to-end data flow. Include the user interface, orchestration layer, model provider, retrieval system, vector store, plug-ins, monitoring tools, logs, backups and human support access.
For each step record who controls it, where processing occurs, what is retained, for how long, and whether data can be used to improve a service or model. This is where a simple “we do not train on your data” claim often proves incomplete: training is only one possible use.
5. Can the model reveal more personal data than the user should see?
Retrieval-augmented generation can make internal knowledge easier to use, but it can also make weak permissions easier to exploit. Test whether a user can retrieve information through the AI that they could not lawfully or appropriately access through the source system.
Use role-based scenarios, prompt variation and deliberately adversarial requests. Access control must survive the conversational interface.
6. How reliable is the output when it concerns a person?
An incorrect product description is inconvenient. An incorrect statement about an employee, patient, applicant or customer can affect a person directly.
Define which outputs require source verification, how users see supporting evidence, how errors can be corrected and whether disputed information propagates into other records. The NIST Generative AI Profile is useful for structuring risks created by generative systems, including confabulation and information integrity.
7. What meaningful human oversight exists?
“Human in the loop” is not enough. State what the reviewer can see, what they must check, whether they can override the output and whether automation bias is being managed.
For higher-impact decisions, test the actual workflow under time pressure. If staff routinely click approve because the queue is too large, the formal control may not be meaningful.
This operational ownership also links privacy to broader governance. The Certified AI Business Steward is relevant where business teams need to translate central AI rules into day-to-day decisions.
8. How will attacks and unintended disclosure be controlled?
Privacy and security meet at the prompt. Prompt injection, malicious documents, insecure plug-ins and excessive tool permissions can expose personal or confidential information even when the model behaves as designed.
Record threat scenarios in the DPIA and connect them to technical testing and incident response. Where organisations need specialised assurance, the Certified AI Cyber Risk Assessor provides a complementary risk perspective.
9. What change triggers a new review?
GenAI systems evolve. A new model, retrieval source, plug-in, automated action, vendor subprocesser or user group can alter the privacy risk materially.
Define review triggers before launch. The DPIA should behave like a living risk record, not a static approval certificate.
Use a simple residual-risk gate
At the end of the assessment, force an explicit decision for each material risk:
- What harm could occur to an individual?
- How likely and severe is it before controls?
- Which control changes that risk, and who owns it?
- What evidence shows the control works?
- What residual risk remains after the control?
- Is that residual risk acceptable under the organisation’s governance and applicable law?
If the answer depends on an assumption—such as a vendor retaining no prompts—turn the assumption into evidence or a contract requirement.
A practical example: an HR policy assistant
Imagine an internal GenAI assistant that helps employees interpret HR policies. The initial scope looks low risk because it retrieves approved documents.
During the DPIA, however, the team discovers that employees can type details of grievances into the prompt, conversation logs are retained by a support provider, and the assistant can retrieve restricted manager guidance because the retrieval index does not enforce the source permissions.
The DPIA changes the design: minimise logging, apply source-level permissions, add user guidance, separate restricted collections and define an escalation path for personal cases. The useful outcome is not the completed form. It is the safer system.
Final takeaway
A strong GenAI DPIA follows the real workflow from purpose to retirement. It asks what personal data enter, where they travel, who can access them, how outputs affect people, what human review means in practice and what happens when the system changes.
That makes privacy part of AI design rather than a late-stage approval exercise. Teams building a broader capability can explore The Case HQ’s certified artificial intelligence courses alongside specialist privacy, governance and cyber-risk learning.
Responses