Many organisations can now say that their employees have attended an AI session. Far fewer can answer a harder question: can those employees use AI safely and appropriately when the decision is real, the data are sensitive and the output looks convincing but may be wrong?
That distinction matters in 2026. The European Commission’s updated guidance on Article 4 of the EU AI Act confirms that providers and deployers of AI systems must take measures to support AI literacy among relevant staff and other people operating AI on their behalf. The Commission also makes an important point that is easily missed: Article 4 does not require organisations to guarantee or measure a particular level of AI literacy for every individual.
So why measure it at all? Because an organisation still needs evidence that its training matches people’s knowledge, experience and the context in which they use AI. A role-based scorecard can turn a vague training programme into a practical capability system.
Start with the work, not an AI quiz
The weakest AI literacy tests ask employees to define machine learning, name popular tools or select a definition of hallucination. Those questions can establish basic awareness, but they say little about behaviour.
Someone can define hallucination perfectly and still paste confidential client data into an unapproved tool. A manager can explain human oversight and still accept an AI-generated recommendation without checking the evidence. A procurement specialist may know what a model is yet fail to ask how a vendor handles updates, logs or subcontractors.
The better unit of assessment is therefore the task. Ask what the person is expected to do with AI, what can go wrong, what judgement they need and when they must stop or escalate.
Professionals who need a structured foundation can use the Certified AI Literacy Professional course to develop the underlying knowledge and responsible-use habits before moving to role-specific assessment.
A five-domain AI literacy scorecard
The following scorecard is deliberately practical. It is not an official EU compliance test, and organisations should adapt it to their own systems and risks.
1. AI understanding
Can the person explain, in language relevant to their role, what the approved system can and cannot do?
Evidence might include the ability to distinguish generation from retrieval, recognise that fluent output is not proof of accuracy, identify when data or prompts influence results, and explain why the same prompt can produce different answers.
2. Task judgement
Can the person decide whether AI is appropriate for the task?
Look for decisions such as choosing not to automate a high-stakes judgement, using AI for a first draft but not a final approval, or recognising that a sensitive task needs a controlled enterprise tool rather than a public service.
3. Verification
Can the person test an output before relying on it?
The evidence should be contextual. A researcher may trace citations. A finance manager may reconcile calculations to source data. An HR professional may inspect criteria for potential bias. A communications employee may verify names, quotations and dates before publication.
4. Data and risk handling
Does the person understand what information may be entered, what should be withheld and which controls apply?
This includes recognising personal, confidential, commercially sensitive and security-relevant information; using approved environments; understanding retention assumptions; and avoiding disclosure through prompts or uploaded documents.
5. Oversight and escalation
Does the person know when AI use has exceeded their authority or competence?
Strong performance is visible when someone pauses a workflow, records an exception, asks for specialist review or reports a failure instead of quietly working around it. For employees with explicit governance responsibilities, the Certified AI Business Steward provides a natural next layer of capability.
Use four proficiency levels, not pass or fail
A binary test encourages people to study for the test. A four-level model gives managers a more realistic view of development.
Level 1 – Aware: the person recognises basic AI concepts, limitations and organisational rules but needs guidance for unfamiliar tasks.
Level 2 – Safe user: the person can use approved tools for defined tasks, verify routine outputs and handle data according to policy.
Level 3 – Applied practitioner: the person can select methods, test outputs, document material use and identify risks in more ambiguous workflows.
Level 4 – Responsible lead: the person can set local controls, coach others, evaluate exceptions and connect operational decisions to governance requirements.
Not every employee needs Level 4. In fact, setting the same target for everyone is usually a sign that the programme has not been designed around risk. A receptionist using an approved summarisation tool and a Chief AI Officer making enterprise governance decisions require different depth. The Certified Chief AI Officer addresses the latter level of strategic responsibility.
Assess with scenarios rather than self-confidence
Self-assessment is useful for identifying perceived needs, but confidence is not competence. Add short scenarios that reproduce real decisions.
For example, give a manager an AI-generated briefing containing one invented statistic and an unverifiable source. Ask what they would do before sending it to the executive team. Give an HR employee a request to upload candidate CVs to a new tool. Ask what they need to check before proceeding. Give a project professional a generated risk register and ask which entries require validation against project evidence.
Score the response on four questions:
- Did the person notice the material risk?
- Did they choose a proportionate verification step?
- Did they stay within the organisation’s approved rules and authority?
- Did they know when and where to escalate?
This makes the assessment observable and easier to defend than a general question such as “How confident are you using AI?”
Build an evidence record without creating surveillance
Measuring capability should not become employee monitoring by another name. Keep the evidence proportionate to the purpose.
A useful record can be small: role, systems used, risk level, training completed, scenario result, development action and reassessment date. Avoid collecting prompt histories or detailed behavioural data simply because the technology makes it possible.
The European Commission’s AI literacy repository shows that organisations are taking different approaches to awareness and training. That flexibility is helpful. It also reinforces the need to document why your own approach fits the people and AI systems involved rather than copying a generic course plan.
Reassess when the risk changes
Annual training alone is too blunt for fast-moving AI use. Trigger reassessment when:
- a new AI system is introduced;
- an employee moves into a higher-risk role;
- a system begins influencing decisions rather than only drafting content;
- policy or regulatory expectations change;
- an incident shows that a control is not understood; or
- a team begins using agents, automation or tool integrations with greater autonomy.
NIST’s AI Risk Management Framework Playbook is useful here because it organises risk activity across Govern, Map, Measure and Manage. AI literacy sits across all four: people need governance expectations, context about the system, ways to evaluate performance and authority to manage risk.
A 30-day implementation plan
In week one, inventory the AI systems people actually use and group users by task and risk. In week two, define the five-domain expectations for each group. In week three, run two or three short scenarios and record evidence. In week four, close the most important gaps through targeted learning, updated guidance or better workflow controls.
The goal is not to produce a perfect “AI literacy score”. It is to make responsible capability visible enough to manage.
That principle also protects training quality. If a team scores poorly on source verification, it needs practice in verification, not another general introduction to AI. If employees understand risk but do not know where to escalate, the problem may be governance design rather than individual knowledge.
Organisations building a broader learning pathway can explore The Case HQ’s certified artificial intelligence courses and choose role-specific development rather than giving everyone the same programme.
Final takeaway
To measure AI literacy well, stop asking only what people know about AI and start examining what they do when AI changes a real task. A role-based scorecard built around understanding, judgement, verification, data handling and escalation gives leaders a practical view of capability without pretending that one universal score can represent every role.
The strongest result is not a high test score. It is a workforce that knows how to use AI productively, how to recognise uncertainty and when human judgement must take control.
Responses