Automation Case Study: How We Scoped and Shipped for Elaine

We delivered a full automation engagement for Elaine: scoped the workflow, designed the system, shipped a production deployment with idempotent processing, human approval gates, and an auditable trail, then handed over a runbook owners could operate confidently. This post shows exactly how we did it and where the hard parts appeared.

Automation engagement, defined: a scoped program that turns a manual multi-step process into a reliable system with clear inputs, deterministic logic, auditability, and safe human touchpoints.

The problem it solves

A single coordinator at Elaine re-keyed data between tools, chased status by email, and stitched results together in spreadsheets. Failures hid in inboxes. Backfills were scary because a rerun could double send. Owners needed a system that ran on its own, showed its work, and never processed the same item twice.

How the automation works

The architecture is simple on purpose: a small orchestrator pulls new work, runs adapters with idempotency guards, emits audit rows, and pauses for human approval on sensitive branches. A dashboard page reads the audit log so operations can see what ran and why.

• Orchestration engine: runs the workflow on a schedule or webhook, enforces idempotency, retries with backoff, and routes to a dead-letter queue when needed.
• Adapters: thin mappers that translate in and out of each system with deterministic field transforms. They never contain business rules.
• State store: append-only audit table plus a compact ledger to remember processed keys for idempotency.
• Human approval queue: sensitive actions write a pending row. Operators approve in the UI, which releases the step.
• Backfill mode: same codepath, just with a bounded date or ID range and dry-run toggles visible in the UI.

Step-by-step: how to build it

1) Lock acceptance criteria and guardrails

Capture inputs, outputs, failure handling, and human gates before you write code. We store these in the repo so product and ops see the same source.

# acceptance.yml
inputs:
  - name: intake_row
    unique_key: elaine_ref
    constraints:
      - must_have: email
      - must_have: created_at
outputs:
  - name: confirmation
    delivery: email_or_queue
  - name: audit_row
    delivery: postgres.append_only
rules:
  idempotency:
    strategy: unique_key_ledger
    on_duplicate: skip_with_audit
  approvals:
    - step: send_sensitive_message
      gate: human_required
  retries:
    max: 5
    backoff: exponential_200ms_to_30s
  backfill:
    allow: true
    dry_run_default: true

Key gotcha: define on_duplicate behavior up front so reruns are safe.

2) Stand up environments and config hygiene

Keep secrets out of code. Make the run mode explicit so dry runs cannot leak to production by accident.

# .env.example
NODE_ENV=production
RUN_MODE=DRY_RUN   # DRY_RUN or LIVE
DATABASE_URL=postgres://...
LEDGER_TABLE=processed_keys
AUDIT_TABLE=audit_log
APPROVAL_WEB_URL=https://ops.example.com/approvals

Key gotcha: default to DRY_RUN in non-CI shells. Flip to LIVE only in controlled deploys.

3) Implement idempotency and retries in the engine

Idempotency is a first-class feature, not an afterthought. We use a ledger table keyed by a stable business identifier.

// engine/idempotency.ts
export async function withIdempotency<T>(
  uniqueKey: string,
  doWork: () => Promise<T>,
  ledger: { has: (k: string) => Promise<boolean>; add: (k: string) => Promise<void> },
  audit: (e: Record<string, unknown>) => Promise<void>
): Promise<T | null> {
  if (await ledger.has(uniqueKey)) {
    await audit({ at: new Date().toISOString(), key: uniqueKey, status: "skipped", reason: "duplicate" });
    return null;
  }
  try {
    const res = await doWork();
    await ledger.add(uniqueKey);
    await audit({ at: new Date().toISOString(), key: uniqueKey, status: "ok" });
    return res;
  } catch (err: any) {
    await audit({ at: new Date().toISOString(), key: uniqueKey, status: "error", error: String(err?.message || err) });
    throw err;
  }
}

Key gotcha: the unique key must be stable across backfills. Do not use auto-increment IDs that change when data is re-exported.

4) Keep adapters thin and deterministic

Adapters translate fields and nothing else. Business rules live in the engine so you can test them in one place.

// adapters/crm.ts
export interface CrmLead {
  id: string
  email: string
  firstName?: string
  createdAt: string
}
 
export function mapIntakeToCrm(intake: any): CrmLead {
  return {
    id: String(intake.elaine_ref),
    email: String(intake.email).trim().toLowerCase(),
    firstName: intake.first_name?.trim() || undefined,
    createdAt: new Date(intake.created_at).toISOString()
  }
}

Key gotcha: make transforms pure. No network calls inside mappers so they are easy to unit test.

5) Add an append-only audit log and a compact ledger

Auditability unblocks operations and compliance. We use an append-only audit table and a separate small ledger for idempotency keys.

-- db/schema.sql
create table if not exists audit_log (
  id bigserial primary key,
  occurred_at timestamptz not null default now(),
  workflow text not null,
  key text not null,
  status text not null check (status in ('ok','skipped','error','pending','approved','sent')),
  detail jsonb not null default '{}'
);
 
create index if not exists audit_log_key_idx on audit_log(key);
 
create table if not exists processed_keys (
  key text primary key,
  first_seen timestamptz not null default now()
);

Key gotcha: never update audit rows in place. Append a new row for each state change so history remains trustworthy.

6) Wire the human approval queue

Sensitive steps pause until an operator approves. Approval writes a new audit row and releases the work.

// engine/approvals.ts
export async function requestApproval(key: string, info: Record<string, unknown>, audit: Function) {
  await audit({ key, status: "pending", detail: info });
  // UI reads pending rows and presents Approve/Reject
}
 
export async function approve(key: string, audit: Function, release: Function) {
  await audit({ key, status: "approved" });
  await release(key); // push back to the work queue
}

Key gotcha: approvals must round-trip through storage. Do not hold pending state only in memory, or a redeploy will lose it.

7) Ship the runbook and safe backfill path

We include one-page instructions for operators and a dry-run backfill command that prints intent before sending anything.

# Backfill dry run, last 7 days
RUN_MODE=DRY_RUN node cli/backfill.js --from 2026-09-01 --to 2026-09-07 --print-plan
 
# Promote to live only after reviewing the plan
RUN_MODE=LIVE node cli/backfill.js --from 2026-09-01 --to 2026-09-07 --confirm yes

Key gotcha: require an explicit confirm flag in LIVE mode so a missed env var cannot send live traffic.

Where it gets complicated

• Credentials ownership: production credentials must live in the client's vault and be rotated on handoff. Do not ship with agency-owned keys.
• Idempotency key choice: choose a business key that survives exports and sandbox moves. Email plus a client reference is safer than a row number.
• Backfills vs dupes: backfills use the same codepath as live runs with the ledger on. Dry-run first, then confirm.
• Human gates: approvals are great until they clog the lane. Cap the queue, add auto-expire rules, and alert on growing pendings.
• Partial failures: a retry for one record should not block the batch. Use per-record retries with exponential backoff and a dead-letter queue.
• Observability: emit a heartbeat and success rate to the dashboard. Silence is failure when owners expect a daily run.

What this actually changes

Elaine's team stopped re-keying between tools and gained a visible, append-only audit trail. Reruns no longer risk double sends because idempotency keys and a ledger enforce skip-on-duplicate. Sensitive messages moved behind a visible approval queue so leaders can review before send. The structural gains were reliability, fewer silent errors, and a workflow operators can run without a developer on call.

One reason this class of project pays back: McKinsey estimated that about 60 percent of occupations have at least 30 percent of activities that could be automated with current technology (https://www.mckinsey.com/capabilities/quantumblack/our-insights/where-machines-could-replace-humans-and-where-they-cant). Asana's Anatomy of Work report found knowledge workers spend roughly 58 percent of their time on work about work: status checks, handoffs, and coordination (https://asana.com/resources/anatomy-of-work-2022). The system we shipped cut directly into those categories for Elaine.

Frequently asked questions

What did you automate for Elaine, specifically?

We turned a multi-step intake and follow-up process into an orchestrated workflow with adapters, idempotency keys, an audit log, and a human approval queue for sensitive actions. The same engine handles daily runs and bounded backfills with a dry-run default.

How long did it take to go live?

We shipped a working skeleton in days, then layered adapters, approvals, and the dashboard over the next iteration. The critical path was locking acceptance criteria and choosing the idempotency key, not writing the first line of code.

What happens if the same record shows up twice?

The ledger catches it. Each item carries a business-stable key. If the key already exists, the engine logs a skipped row and moves on. Reruns and backfills are safe by design.

Can non-technical operators run it?

Yes. The dashboard surfaces status and approvals. The runbook documents backfills and dry-run promotion. Operators do not need to touch code or servers to use the system day to day.

How do you prevent silent failures?

We treat silence as failure. The engine writes every step to the audit table, emits a heartbeat, and raises alerts on growing pending approvals or repeated dead-letter entries. Partial errors do not stop good records from finishing.

What if we need to change a vendor tool later?

Adapters are thin and deterministic on purpose. Swapping a tool means changing only that adapter. Business rules and idempotency live in the engine, so the rest of the workflow stays intact.

If you want this level of reliability in your own operation, we can scope the first workflow on a short call and tell you whether your process maps to this pattern. See our custom integration approach at /services#custom-ai-integration, read why most automation projects fail and how to avoid it in /blog/why-90-percent-of-automation-projects-fail, and when you are ready to move, /book.