In this article
After releasing an AI assistant, do not manage quality through complaint counts or answer accuracy alone. Record each significant incident as a chain: what happened, who was exposed, what consequence followed, how it was detected, when it was contained, and what prevents recurrence. An error seen by one customer and blocked before action differs from the same behavior shown to a thousand customers or used to execute an irreversible step.
This record complements pre-release evaluation rather than replacing it. Testing anticipates known patterns; incident learning reveals what the team did not expect once the system meets real context, data and behavior.
Define incidents and near misses
An incident is an unacceptable outcome reaching a user, system or decision: incorrect commercial information, access outside authorization, or an inappropriate executed action. A near miss is a path that could have caused harm but was stopped by a barrier, such as human review blocking an unauthorized offer. Near misses expose weaknesses before loss occurs, but they should not be reported as realized harm.
Keep the definition tied to the assistant’s approved scope. Refusing an out-of-scope request may be correct behavior, not an incident. Conversely, a fluent answer can be an incident when it relies on an expired policy or exposes another customer’s context.
The NIST Generative AI Profile describes itself as a companion to the AI Risk Management Framework, intended to help organizations incorporate trustworthiness into the design, development, use and evaluation of AI systems. The classifications and calculations below are a proposed operating method, not a NIST requirement or certification.
Capture enough evidence to reconstruct the event
Use an immutable incident identifier, detection time, first and last exposure, model and instruction version, knowledge-base version, tools, channel and language, and the affected decision or action. Preserve input and output after removing unnecessary personal information and restricting access. Record who found it: customer, employee, automated monitor or sampled review.
Do not copy customer conversations into an open spreadsheet merely to investigate. Keep evidence in an appropriate controlled system, and place a reference in the operating log for authorized reviewers. Define retention and deletion according to the purpose and applicable obligations.
Assess four dimensions instead of applying one “critical” label
- Severity: consequence of one case, with stop conditions for privacy, authority and execution.
- Exposure: users, decisions or transactions that received the behavior before containment.
- Reversibility: whether text can be corrected, a transaction must be reversed, or the effect cannot be removed.
- Detection gap: time from first exposure to discovery, not only response time after a complaint.
Do not let a composite score hide a stop condition. Information access outside authorization still requires escalation when only one case is known. Lower-severity failures can be ordered using exposure, recurrence and estimated cost.
A hypothetical example: more errors, much less loss
Assume an assistant handles 10,000 interactions in each of two months. Month one records 40 minor errors with an average estimable impact of QAR 10 and two major incidents averaging QAR 3,000. Month two records 60 minor errors, no major incidents and five near misses stopped by review. These figures are hypothetical, not results attributed to Fahad ALNaimi or any company.
- Estimated month-one loss = 40 × QAR 10 + 2 × QAR 3,000 = QAR 6,400.
- Estimated month-two loss = 60 × QAR 10 = QAR 600.
- Error count rises by 50%, while estimated loss falls by QAR 5,800 under these assumptions.
Do not conclude that month two is safe. The near misses can show that review is working—or that high-risk attempts are increasing. Report them separately, measure review cost, and state how much traffic was not inspected. Estimated loss is not realized savings; test sensitivity when major-incident impact is uncertain.
Contain first, then correct the cause
Choose containment by scope: disable an execution tool, route a request category to a person, freeze a knowledge version, or roll back a release. Use an automation stop-and-recovery path so action remains possible under pressure. Containment reduces further exposure but does not close the incident.
Then locate the nearest actionable cause in the system: stale knowledge, wrong context retrieval, conflicting instructions, excess permission, missing validation or incorrect classification. “The model hallucinated” describes an outcome but assigns neither owner nor corrective action. Add the pattern to testing, change the relevant barrier, and evaluate neighboring cases rather than only the observed prompt.
Do not close an incident when the fix ships
Closure requires evidence: regression tests pass, an independent reviewer verifies the change, an agreed observation period completes, and the pattern does not recur within a known exposure volume. Record the change, approver and rollback path. If the fix increases human referrals, assess the economics of the exception queue.
Feed new incidents into the AI assistant release gate. A critical incident becomes a stop-condition case in the next release. Do not turn the test set into a diary of every event; consolidate related patterns so it remains maintainable.
A weekly view that leads to a decision
Report incidents by severity and exposure, near misses, median and 90th-percentile detection and containment time, recurrence, open incidents without owners, and affected usage. Also watch the share first detected by customers. A decline while review coverage remains stable may indicate better internal monitoring.
The operating decision is to maintain one incident record shared by product, operations and risk, with a leader and verification date for every case. Escalate stop conditions immediately and order the remainder by consequence and exposure rather than noise. If the team cannot reconstruct what happened or measure who encountered it, improve observability and permission boundaries before expanding automation.
Contact on WhatsApp
