AI that a person can check, correct and overrule

01 · Directive: Algorithmic Impact Assessment

Impact assessment before design

Every engagement starts by naming who is affected by a wrong output and how badly. That decides how much the AI is allowed to do on its own.

• Health content (Crohn's and Colitis Canada): a wrong answer reaches patients, so the AI only drafts, and only from the charity's own approved sources. Nothing is published without staff review.
• Lending decisions (Forward Funding): the lender sets the decision criteria; the system applies them and explains the result; underwriters review the outcome.
• Grant applications (Play Forever): the charity's name is on every submission, so no application leaves the system without staff approval.
• For federal work we complete the Government of Canada Algorithmic Impact Assessment with the client department before build, and size the controls below to the impact level it returns.

02 · Directive: human intervention, recourse

Human review and intervention

• Approval gates: Play Forever staff approve or send revision notes on every draft; staff can edit any answer inline, and edits are allowed only while a grant is in review, so an edit cannot race a redraft. Our end to end test of the revision flow (May 22, 2026) confirmed the AI followed the staff instruction (the revised answer came back at 177 words against a 200 word limit).
• Uncertainty goes to a person: facts the system cannot verify are flagged NEEDS_VERIFICATION for staff (for example a budget amount, or new partner suggestions in a support letter package) instead of being guessed.
• Client led correction: Forward Funding's underwriters tested the system on live submissions before relying on it. Each error they found (a skipped month, NSFs not weighted, a low deposit month still approved) became a fixed, retested rule.

03 · Directive: quality assurance, data quality, testing and monitoring outcomes

Data quality and testing of outcomes

We test the data before we tune the model, and we measure outcomes on the live system, not only on our own output.

• Source coverage and quality (Crohn's and Colitis Canada): when answers were weak, we measured rather than guessed. A self similarity test (0.9999) ruled out an embedding mismatch; a coverage audit found only 15 of 282 knowledge pages indexed; about 91% of the index was raw webinar speech. We backfilled 141 website pages with 0 failures, added a guard that refuses near empty pages, and stripped boilerplate repeated across the site so it could not skew results. Edited written material now supplies 53% of top answers, against 33% for transcripts.
• Testing for wrong outcomes: we proved that no similarity threshold could separate answerable from unanswerable questions (measured separation 0.000), so we replaced the threshold with an answerability check. Results: 11 of 11 live queries correct, 15 of 15 failure path tests (every error results in “do not generate”), 13 of 13 chunking tests, and 5 of 5 end to end tests through the production workflow.
• Factual accuracy guardrails (Play Forever): after the client flagged an incorrect service region, we added a hard rule against claiming regions the charity does not serve and tested it: 0 mentions of the off list regions in the guarded test run.
• Data integrity checks (a US credit technology client): when we encrypted Social Security numbers at rest, every one of the 58 encrypted records was decrypted and compared with the original before sign off: 0 mismatches. Their credit grading engine ships with 17 of 17 unit tests passing.
• Fairness: rules are set by the client, written down, and applied the same way to every file, and we test them on the client's real cases. Where personal data allows it, we test outcomes across groups, including a Gender Based Analysis Plus review, as the Directive requires for federal systems.

04 · Directive: notice, explanation

Transparency and explanation

• Every Crohn's and Colitis Canada draft cites the sources it was written from, so a reviewer can check each claim.
• When the content engine declines a request, it writes the reason and the missing content into the request record. Staff see why, and the content team gets a list of articles to write.
• Forward Funding receives a written analysis with each approve or decline result, not just a label.
• Systems log why no action was taken: for example the credit platform records “No Report Pulled” with the reason when a lead cannot be processed.

05 · Directive: security, IT and business continuity

Privacy, security and continuity

• Personal data is minimized and protected: Social Security numbers are encrypted at rest and masked in logs and stored copies; the content engine's index holds only the charity's own published material, no patient or member data, and its retrieval service uses shared secret authentication across every workflow that calls it.
• We plan for failure and fix the cause: in August 2026 one of our own test calls failed and the automation platform switched a client workflow off, a 14 hour outage. We traced it the next day and added retries, a dead letter queue and a visible “Failed” status so a stalled request can never look like a pending one.
• For public sector work we keep data in Canadian hosted databases where the contract requires it, and choose model providers and settings that do not train on client data.

06 · On a public sector contract

How we apply this on a government contract

1. Complete the Algorithmic Impact Assessment with the department and agree the human review points.
2. Audit the training or source data for coverage, quality and representation before building.
3. Agree written acceptance tests with the department, including failure path tests, and run them on real cases.
4. Ship with explanations, logging and a named human owner for every decision the system supports.
5. Monitor outcomes after launch and report any drift or incident with its root cause and fix.