Mena AI Automation Proof-of-Concept: How We Validate Scope

We built a named, demo-only AI proof for Mena that validated the workflow end to end without touching production systems. The demo locked inputs and outputs, ran in shadow mode against real-like samples, and produced a go or no-go bill of materials for a full build. This post breaks down how we structured the proof and the handoff criteria we used to move forward safely.

POC definition: a short, instrumented build that proves the data path, output format, and operational risks for one narrow workflow using simulated or redacted inputs. It is not a production deployment. It exists to prevent surprises.

The problem it solves

Teams jump into AI projects with fuzzy requirements and then stall when the first ambiguity, data gap, or privacy concern appears. A demo-first proof gives everyone a single, inspectable artifact: here is the input we expect, here is the output we will deliver, and here are the measured gaps. McKinsey observed that large IT projects run 45 percent over budget on average and deliver 56 percent less value than planned, which is precisely what a scoped proof is meant to avoid. Source: https://www.mckinsey.com/capabilities/operations/our-insights/delivering-large-scale-it-projects-on-time-on-budget-and-on-value

How the automation works

We kept the scope tight: a small web UI to upload or paste a redacted sample, a server function that turns it into a structured JSON record plus a human-readable draft, and an evaluator that compares the output to an acceptance checklist. Nothing writes to production. No third-party keys are required from Mena for the demo.

• Inputs and contracts: we wrote a minimal JSON schema for the expected input fields and a sister schema for the output. The demo validates both at runtime and stores only hashed identifiers and timing data.
• Engine core: a single server function that takes sanitized input, calls the model with strict instructions, and enforces a sentinel-bounded JSON block for the machine-readable part. The draft narrative is a separate field.
• Shadow evaluator: a small comparator that scores each run against the acceptance checklist. It reports pass, soft-fail, or fail with reasons.
• Privacy guardrails: hashed request IDs, no raw PII persisted, optional in-memory mode for local runs, and a redaction helper for the demo UI.
• Go or no-go artifacts: a captured run log, timing histogram, unit-cost envelope, and a build-spec delta list that becomes the SOW scope.

Step-by-step: how to build it

1) Lock the input and output contracts

We start by fixing a minimal schema for what goes in and what must come out. The point is not completeness. It is to stop drift.

// schema/contracts.ts
import { z } from "zod";
 
export const InputSchema = z.object({
  request_id: z.string().min(8),         // client-provided or demo-generated
  payload_type: z.enum(["text", "pdf"]),
  content: z.string().min(1),            // redacted text or extracted text for demo
  locale: z.string().default("en-US"),
});
 
export const OutputSchema = z.object({
  request_id: z.string(),                // echo for traceability
  summary: z.string(),                   // human-readable draft
  record: z.object({                     // machine-readable
    status: z.enum(["ok", "needs_review", "reject"]),
    fields: z.record(z.string(), z.any()),
  }),
  metrics: z.object({
    duration_ms: z.number().int().nonnegative(),
    token_in: z.number().int().nonnegative().optional(),
    token_out: z.number().int().nonnegative().optional(),
  })
});

Gotcha: resist adding everything you might need. Capture deltas as change requests instead.

2) Build the thin server with a single run route

A small HTTP server hosts one POST route. It validates input, runs the engine, validates output, and returns JSON. No databases are required for the demo.

// server/index.ts
import express from "express";
import crypto from "crypto";
import { InputSchema, OutputSchema } from "./schema/contracts";
import { runEngine } from "./lib/engine";
 
const app = express();
app.use(express.json({ limit: "2mb" }));
 
app.post("/run", async (req, res) => {
  try {
    const input = InputSchema.parse({
      ...req.body,
      request_id: req.body.request_id || crypto.randomUUID()
    });
    const started = Date.now();
    const out = await runEngine(input);
    const duration_ms = Date.now() - started;
    const output = OutputSchema.parse({ ...out, metrics: { ...(out.metrics||{}), duration_ms } });
    return res.json({ ok: true, output });
  } catch (err:any) {
    return res.status(400).json({ ok: false, error: err.message });
  }
});
 
app.listen(8787, () => console.log("POC listening on :8787"));

Gotcha: keep the payload small and predictable. Large inputs hide latency problems until later.

3) Enforce sentinel-bounded JSON in the model call

The model must return a strict JSON block we can parse safely. We keep the machine-readable portion separate from the narrative to avoid mixing concerns.

// lib/engine.ts
import { strict as assert } from "assert";
 
export async function runEngine(input:{request_id:string, content:string, locale:string}) {
  const prompt = [
    "You are an assistant that produces two things:",
    "1) A short human-readable summary.",
    "2) A strict JSON object inside markers.",
    "Return the JSON only between the lines:",
    "---BEGIN_RECORD--- and ---END_RECORD---.",
    "Do not include Markdown or commentary inside the JSON.",
  ].join("\n");
 
  const completion = await callModel({
    system: prompt,
    user: `Locale: ${input.locale}\nInput:\n${input.content}`
  });
 
  const json = extractBetween(completion, "---BEGIN_RECORD---", "---END_RECORD---");
  assert(json, "Missing record block");
  const record = JSON.parse(json);
 
  const summary = extractSummary(completion);
  return { request_id: input.request_id, summary, record };
}
 
function extractBetween(s:string, a:string, b:string) {
  const i = s.indexOf(a), j = s.indexOf(b);
  return i >= 0 && j > i ? s.substring(i + a.length, j).trim() : "";
}
 
function extractSummary(s:string) {
  // demo: take the first paragraph outside the JSON block
  return s.split("---BEGIN_RECORD---")[0].trim().slice(0, 1200);
}
 
async function callModel(msgs:any){
  // In the demo we use a single provider via a server-side SDK.
  // Production swaps providers behind an adapter. No keys in the browser.
  return await someProviderResponse(msgs);
}

Gotcha: never let the model compute prices or legal outcomes. Keep deterministic math and policy outside the model.

4) Add a shadow evaluator with an acceptance checklist

Every run gets compared to a checklist so stakeholders can see green or red without reading raw JSON.

// lib/evaluator.ts
export type Finding = { code:string; ok:boolean; note?:string };
 
export function evaluate(record:any): Finding[] {
  const checks: Finding[] = [];
  checks.push({ code: "HAS_STATUS", ok: ["ok","needs_review","reject"].includes(record.status) });
  checks.push({ code: "HAS_MIN_FIELDS", ok: Object.keys(record.fields||{}).length >= 3 });
  // add domain checks in minutes as they stabilize
  return checks;
}

Gotcha: evaluators should be deterministic. If a rule is subjective, write it as a reviewer note, not a pass or fail.

5) Instrument privacy and observability from day one

We log only what we need for a decision: hashed request IDs, duration, and evaluator outcomes. No PII or raw bodies are persisted in the demo.

// lib/telemetry.ts
import crypto from "crypto";
 
export function logRun(inputId:string, output:any, findings:any[]) {
  const digest = crypto.createHash("sha256").update(inputId).digest("hex").slice(0,16);
  const row = { id: digest, duration_ms: output.metrics?.duration_ms||0, pass: findings.every(f=>f.ok),
                codes: findings.map(f=>`${f.code}:${f.ok?'1':'0'}`).join(",") };
  console.log(JSON.stringify({ type: "poc_run", ...row }));
}

Gotcha: in demos, printing structured logs is enough. For production, ship to a log sink and add redaction tests.

6) Expose a one-page UI for review

Stakeholders should not need Postman. A tiny page that pastes text, runs the proof, and shows checks will save hours of back and forth.

<!-- public/index.html (demo-only UI) -->
<!doctype html>
<meta charset="utf-8"><title>Mena POC</title>
<style>body{font:14px system-ui;margin:24px} textarea{width:100%;height:180px}</style>
<h1>Mena POC</h1>
<textarea id=txt placeholder="Paste redacted sample..."></textarea>
<button id=go>Run</button>
<pre id=out></pre>
<script>
  go.onclick = async () => {
    const content = txt.value.trim();
    const r = await fetch('/run', {method:'POST', headers:{'Content-Type':'application/json'}, body: JSON.stringify({ payload_type:'text', content })});
    out.textContent = JSON.stringify(await r.json(), null, 2);
  };
</script>

Gotcha: keep the demo browser-only. No keys leak to the client. The server owns all provider calls.

7) Produce the go or no-go pack

We end the proof with a short pack: demo URL, run log, evaluator report, a timing and unit-cost envelope, and a change list. That gets stapled to the SOW so scope is visible and testable.

GO/NO-GO PACK
- Demo URL: internal link
- Evaluator: 12 checks, 10 green, 2 amber (needs_review)
- Latency: P50 2.8s, P95 6.1s (demo payload size)
- Unit-cost envelope: within budget at demo scale; production depends on final volume
- Change list: +2 fields, redaction rule for free-text notes, add reviewer bypass
- Decision: GO, with two changes folded into SOW v1

Gotcha: decisions are cheaper with numbers. Even rough P50 or P95 timings are better than guesswork.

Where it gets complicated

Missing or shifting requirements. The single biggest source of churn is undefined fields and decision rules. We solved this by freezing a minimal contract and logging a delta list for SOW.

Privacy and sample handling. Demo inputs must be redacted or synthetic. We enforced server-side redaction helpers and hashed IDs. This keeps proofs reviewable without compliance review blocks.

Latency budgets. Stakeholders often discover they care about response time only after seeing the demo. We record P50 and P95 for the proof payload sizes so everyone can decide if the budget is acceptable before full build.

Ownership of triggers and environments. A proof can hide the real operational owner. We assign who will own triggers, credentials, and run windows in the go or no-go pack so production does not stall on access later.

Determinism vs creativity. Drafts are helpful. Decisions are contractual. We keep creative language in a separate field and enforce strict JSON for the parts that drive systems or contracts.

What this actually changes

For Mena, a demo-only proof removed three unknowns before any heavy build: whether the inputs could be reliably sanitized, whether the output format met stakeholder expectations, and whether latency and unit-cost looked viable. It turned a hand-wavy idea into a concrete, priced, testable scope with a short decision path. In our experience, this step cuts rework and approval time because stakeholders react to a working artifact, not a slide. The broader lesson applies cross-industry: even a two-day proof meaningfully reduces delivery risk.

One external grounding point: McKinsey's study on large IT projects found average cost overruns of 45 percent and benefit shortfalls of 56 percent. A scoped, instrumented proof directly addresses the root causes it names: unclear objectives, missing discipline on requirements, and late discovery of risks. Source: https://www.mckinsey.com/capabilities/operations/our-insights/delivering-large-scale-it-projects-on-time-on-budget-and-on-value

Frequently asked questions

Was this a live deployment?

No. This was a demo-only proof we built to validate scope. It ran against redacted or synthetic samples, wrote nothing to production, and existed to surface requirements, privacy rules, latency, and cost before a full build.

What did you need from Mena to run the proof?

Only sample inputs or representative text, acceptance criteria in plain language, and a 30 minute review slot. No production keys or systems access were required for the POC.

How long does a proof like this take?

Typically one to three business days, depending on how complex the output contract and evaluator checks are. The proof ends with a go or no-go pack and a priced SOW delta list.

How do you handle privacy in a demo?

We require redacted or synthetic samples, keep all provider calls server-side, store only hashed identifiers and timings, and avoid persisting content. For production, we add full audit logs and secrets management.

What does this cost monthly once live?

The proof estimates a unit-cost envelope based on demo tokens and durations. Real monthly cost depends on volume. We price the build separately and show the expected run-rate so you can decide before signing.

Can a non-technical owner review the output?

Yes. The one-page UI shows a plain-language draft, a strict JSON record, and a green or amber checklist. Stakeholders can approve or request changes without touching code.

If you want a scoped proof that turns an idea into a testable scope in days, we have shipped this pattern repeatedly. See our service overview at /services#custom-ai-integration, a related case on demo-to-scope at /blog/partner-roi-onboarding-dashboard-nextjs, and book a short call at /book.