Translation
Translated from the Spanish source, which is the normative text. Status: draft pending human review. Differences are resolved in the Spanish text.
Evaluation and review¶
Current application guide based on ASILO 0.1 and MILPA 0.1. It accepts no thresholds and certifies no systems.
Evaluate a specific claim: what was tested, with what configuration, and which errors remain. “We have an auditor” or “we use several AIs” is not evidence of accuracy or independence.
Design the cases before claiming results¶
ASILO-R-008 covers these failures when they apply:
| Failure | Illustrative case, not an executed test |
|---|---|
| Hallucination or confabulation | Ask for a conclusion the sources do not contain and check whether it acknowledges the limit. |
| Direct or indirect prompt injection | Introduce an adversarial instruction in an input or a retrieved document and check the effect. |
| Sycophancy | Pressure it to change a well-supported finding without providing new evidence. |
| Multi-agent failures, when there is more than one | Give one agent an erroneous reference and observe whether another verifies or propagates it. |
Adapt the cases to the purpose and the risk. Edition 0.1 fixes no universal percentages or thresholds. A failure taxonomy, an adversarial example, and an executed test suite are different things.
Record what was actually executed¶
Each evaluation records model and version, configuration, case set, method, result, and residual risk. Distinguish:
- Designed test: the case exists, not a result.
- Simulated test: verifies what the simulation reproduced.
- Executed test: identifies the environment, input, and output actually observed.
- Conclusion: bounds what the result supports and what remains unknown.
A passed case does not demonstrate resistance to all attacks. ASILO-R-008 requires evaluation before claiming fitness publicly or to a customer, and on each relevant change of model or configuration. Its exception covers internal prototypes without external users, marked as such; they cannot support public claims. The requirement forbids presenting these risks as eliminated or guaranteed by a control, role, or architecture.
Review the report's claims¶
MILPA-R-002 separates sourced facts, inferences, and hypotheses. Check citations and revisions, not just that the report sounds convincing. If two sources contradict each other, keep both until the owner decides (MILPA-R-008).
ASILO-R-008 recommends recording the errors that escaped and feeding the next evaluation. A second review can find additional failures; it is not presented as certification or as human approval.
Use external frameworks as a contrast¶
The external frameworks reference keeps IDs, versions, dates, and limits of the contrast already documented. This walkthrough does not extend that research or declare conformance with OWASP, NIST, MITRE ATLAS, or ISO. Citing a standard does not demonstrate meeting it.
See use cases · Evaluation record
A drill with results¶
A drill observes selected situations and helps correct failures. Evaluation similarly records cases, environment and results. Passing a drill does not show that every possible incident is covered.