When artificial intelligence enters risk analysis, the problem is not only what we give it. It is how we assess what it gives back.
We ask three artificial intelligence systems to analyse the same threat.
We get three different assessments.
Which one do we use?
Not necessarily because one of the three wants to deceive us. They may have been trained on different data, use different sources, or assign different weight to the same elements. They may also simply be wrong.
The difference between an error and deliberate manipulation is fundamental, but operationally both can compromise a decision.
And what if all three agreed?
We would probably trust the result more.
But if all three had relied on the same piece of wrong information, we would have nothing more than three confirmations of the same mistake.
A wrong analysis, with the system working perfectly
Think of an OSINT assessment intended to evaluate the security of a destination, or a system that synthesises threat reports before a major event.
If the starting information is incomplete, outdated or inaccurate, the assessment can still be wrong even when the system does exactly what it was designed to do.
An unverified report is picked up by several websites, summarised in a briefing and eventually fed into an artificial intelligence system to build a risk scenario.
The original information was one.
In the final document, it may look like a fact confirmed by numerous sources.
The problem existed long before AI.
The difference is that today it can be reproduced on a scale and at a speed that were previously difficult to imagine.
The result, moreover, looks like analysis: orderly, coherent and confident.
And that is what makes it insidious.
Poorly presented doubtful information is easier to recognise.
Well-presented doubtful information can pass without further checks.
The quality of the presentation does not prove the quality of the information.
When the content starts giving orders
There is another specific vulnerability: prompt injection.
It occurs when malicious instructions embedded in external content — a web page, a document or an email — influence the behaviour of the system processing that content.
Imagine a tool tasked with reviewing a series of documents to identify security concerns.
One of the documents contains hidden instructions: ignore certain information, alter the assessment, or perform an unauthorised action.
The problem is not necessarily that the document is false.
The problem is that the system may treat part of its content as an instruction to execute rather than as information to analyse.
In an article published on 8 December 2025, the UK’s National Cyber Security Centre highlighted a fundamental difference from SQL injection: in language models, there is no inherent and reliable separation between data and instructions.
For this reason, according to the NCSC, prompt injection may never be completely eliminated through techniques equivalent to those used against SQL injection.
The realistic approach is to reduce both the likelihood and the impact of attacks, starting with system designs that limit what the model is allowed to do.
For anyone using AI to process documents, news monitoring or threat reports, the consequence is concrete.
A system analysing uncontrolled content is also exposed to instructions that content may contain.
We therefore need to protect not only what goes into the system.
We also need to verify what comes out.
An old lesson from intelligence
Anyone involved in intelligence analysis knows a basic principle: the reliability of a source and the credibility of an item of information are two different things.
A generally reliable source can provide inaccurate information.
An unknown source can occasionally provide accurate information.
The Admiralty Code, used in intelligence, evaluates the two separately: a letter for the source and a number for the information.
The principle also appears in work by TNO researchers presented at a NATO Science and Technology Organization meeting. Their paper discusses the challenges of applying the system in long-term analysis of hybrid conflicts, in an environment shaped by the growth of open-source information and its exposure to disinformation.
With artificial intelligence, we need one additional distinction.
The system producing the answer is not necessarily the source of the information on which that answer is based.
A model may synthesise documents, combine data, draw inferences and build scenarios.
But if the result is going to support a decision, we need to be able to reconstruct the path from the initial information to the final conclusion.
Otherwise, we are assessing the credibility of a text, not the reliability of an analysis.
Four questions before making a decision
We do not need a complicated procedure for every use of artificial intelligence.
We need criteria proportionate to the importance of the decisions we are about to make.
Where does the information come from?
If the system provides sources, we need to verify that they exist, that they are relevant and that they actually support the claim being made.
A correct reference does not guarantee a correct interpretation.
What does an independent source say?
The confirmation has to be genuinely independent.
Three articles repeating the same story are not three separate confirmations.
And three AI systems are not three independent analyses.
An assessment that nobody else confirms is not necessarily wrong.
But we cannot treat it as established fact simply because it sounds plausible.
What is fact, what is inference, and what is hypothesis?
A fact is something supported by verifiable evidence.
An inference is a conclusion drawn from available information through reasoning that can be examined.
A hypothesis is a possible explanation or scenario that still needs to be verified.
All three can coexist within the same assessment.
The problem begins when they are presented with the same level of certainty.
What can the system do without authorisation?
This question is particularly relevant to systems that go beyond producing text and can access applications, send communications or modify documents.
We need to define which actions are permitted, which require authorisation, and which must remain off-limits, in proportion to the consequences of an error.
The NCSC suggests a simple principle: when a system processes content coming from a party, its privileges should be limited to those available to that party.
Asking AI to organise a media briefing is one thing.
Allowing it to independently modify a threat assessment used for operational decisions is something else entirely.
Compliance does not certify the truth
The European regulatory framework for artificial intelligence is introducing requirements covering governance, transparency and risk management.
But complying with those requirements does not guarantee that every answer is correct.
An organisation can use a compliant system and still produce a wrong assessment.
Regulatory compliance and operational reliability are related, but distinct, questions.
The NIST AI Risk Management Framework, published in January 2023 and currently under revision, is organised around four functions: Govern, Map, Measure and Manage.
The Generative AI Profile, published in July 2024, extends the framework to risks specific to generative AI.
It is an approach familiar to anyone working in security: understand the context, identify risks, assess their consequences, define mitigations and verify that they work.
We do not need to invent a new discipline every time a new technology appears.
We need to apply the one we already have properly.
Before we believe it
Artificial intelligence can help us gather information, compare scenarios and identify things we had not considered.
It can be an extraordinary tool.
But its value does not depend only on its ability to produce answers.
It also depends on our ability to evaluate them.
A well-structured, well-written and confident answer is not a verified source.
It is simply a well-structured, well-written and confident answer.
The question to ask an artificial intelligence system is not how convincing it is.
It is:
What do we need to check before we believe it?
Because in security, the value of information does not depend on how well it is written.
It depends on how well we are able to verify it.
And perhaps one of the most important tasks of a Security Manager in the age of artificial intelligence will be precisely this:
not to know everything, but to know what cannot be believed without verification.
Essential sources
- NCSC — Prompt injection is not SQL injection (it may be worse), 8 December 2025. Differences between prompt injection and SQL injection, limitations of mitigation and reduction of impact.
- NATO Science and Technology Organization — Powell, Oggero, Westerveld, Schook (TNO), Dealing with Uncertainty in Hybrid Conflict: A Novel Approach and Model for Uncertainty Quantification in Intelligence Analysis. Distinction between source reliability and information credibility in the Admiralty Code.
- NIST — AI Risk Management Framework, January 2023. Voluntary framework for managing artificial intelligence risks.
- NIST — Generative Artificial Intelligence Profile (NIST AI 600-1), 26 July 2024. Risks specific to generative AI and related risk-management measures.
- European Commission — AI Act, enforcement framework. Regulatory obligations, supervision and implementation timetable.
The scenarios described are hypothetical examples used to illustrate potential weaknesses in information analysis. The four questions are a proposed security-management methodology and are not an official procedure derived from the sources cited.
Francesco di Castri
Leave a Reply