Crohn's and Colitis Canada RAG: How We Fixed Coverage

We diagnosed Crohn's and Colitis Canada's RAG content-creator and built a fix set that restored content coverage, removed duplication, and added an answerability gate. The deployment was paused org wide, so we shipped the full remediation package and verified it end to end in tests.

RAG automation is a retrieval plus generation system: it fetches the right source passages first, then asks an LLM to answer using those passages.

The problem it solves

A RAG assistant only works when the corpus is complete and the retriever serves the right passages. At CCC, the index was dominated by raw YouTube transcripts and missed many core site pages, so answers drifted off topic or sounded like rambling transcripts.

We found three root causes that explain the symptoms:

• Index composition: ~91 percent of stored chunks were raw YouTube ASR transcripts while only a small fraction of site pages were in the index. Only 15 of ~282 site pages were present, and only 51 tracked pages existed at audit time.
• Query pollution: boilerplate prepended to user queries reduced retrieval relevance.
• Implementation defects: duplicate IDs from truncation, no delete-before-upsert, and a tracked-URL hash that compared content length rather than a true hash.

How the automation works

We rebuilt the pipeline around coverage and hygiene. Airtable remains the control plane, Make.com coordinates ingestion triggers, a Cloudflare Worker handles canonical ingest and retrieve, Pinecone stores vectors, and an answerability gate protects the chat surface.

• Airtable tracked URLs: One table is the single source of truth for what belongs in the index. It stores URL, last ingested content hash, and status.
• Make.com orchestration: Push-only triggers call the worker with a token and payload. Imports never write secrets into blueprints.
• Cloudflare Worker ingest: Fetches the source, normalizes text, splits into chunks, computes a stable ID, deletes any prior chunks for that document, then upserts the new set.
• Vector index: Stores embeddings and minimal metadata. Title and section headers are folded into the text before embedding so retrievers can use them.
• Retrieve plus answerability: A retrieve route runs top-k search, then a gate checks if the retrieved passages are sufficient. If not, the assistant declines and suggests covered topics.

Step-by-step: how to build it

1) Make tracked URLs the source of truth

Define exactly what you want indexed and give each row a durable content hash. That lets you detect deltas and avoid re-ingesting unchanged pages.

// tracked-urls.ts
export type Tracked = { url: string; title?: string; lastHash?: string };
export const tracked: Tracked[] = [
  { url: "https://www.example.org/conditions/crohns" },
  { url: "https://www.example.org/conditions/ulcerative-colitis" },
  { url: "https://www.example.org/treatment/biologics" }
];

Gotcha: do not use content length as the hash. Use a real digest of normalized text so minor template changes do not trigger thrash.

2) Secure the ingest path and normalize inputs

A single worker endpoint handles ingest. It verifies a shared token, fetches content, strips boilerplate, and computes a content hash for change detection.

// worker-ingest.mjs
export default {
  async fetch(req, env) {
    const token = req.headers.get("x-worker-token");
    if (token !== env.INGEST_TOKEN) return new Response("unauthorized", { status: 401 });
    const { url } = await req.json();
    const html = await fetch(url).then(r => r.text());
    const text = normalizeHtmlToText(html); // remove nav, footers, boilerplate
    const hash = await sha256(text);
    return new Response(JSON.stringify({ ok: true, hash }));
  }
};

Gotcha: keep the worker public only during a phased rollout. Add the token gate first and rotate it after tests.

3) Delete before upsert to prevent orphans

Before writing new chunks for a document, remove the old ones. This keeps the index tidy when pages change.

async function reindexDocument(idx, docId, chunks) {
  await idx.delete({ filter: { doc_id: docId } });
  const vectors = chunks.map((c, i) => ({
    id: `${docId}:${i}`,
    values: embed(c.text),
    metadata: { doc_id: docId, url: c.url, section: c.section }
  }));
  await idx.upsert(vectors);
}

Gotcha: a truncated or case sensitive ID scheme creates dupes. Use a full digest of the canonical URL as the docId.

4) Fold titles and headers into the embedding text

If you only embed body text, the retriever cannot leverage headings. Concatenate title and section headers before embedding.

function makeEmbedText(title, section, body) {
  const parts = [title?.trim(), section?.trim(), body?.trim()].filter(Boolean);
  return parts.join("\n\n");
}

Gotcha: storing titles only as metadata does not affect similarity. Put salient headers into the embedded text.

5) Add an answerability gate before the LLM

A cosine threshold cannot reliably separate covered from uncovered. Use a lightweight gate that inspects retrieved snippets and decides if they actually answer the question.

async function answerabilityGate(question, snippets) {
  const prompt = `You are strict. If the snippets do not clearly answer the question, say NO and explain which content is missing. Otherwise say YES.\n\nQuestion: ${question}\n\nSnippets:\n${snippets.map(s => `- ${s}`).join("\n")}`;
  const out = await smallModel(prompt);
  return /\bYES\b/i.test(out);
}

Gotcha: gate with a small, cheap model to control latency and cost. Only call the larger model when the gate says yes.

6) Treat orchestration imports as new webhooks

When you import updated scenarios, many orchestration tools mint fresh webhook URLs. Update any Airtable buttons or scripts that point at the old ones.

Deployment checklist:
- Import blueprint -> copy new webhook URLs
- Update Airtable button formulas
- Toggle test mode ON in the worker until first green run

Gotcha: importing over the top without updating callers silently breaks buttons.

7) Verify retrieval health with a self-similarity check

Store a chunk and query the index with the exact same text. If you do not get a near perfect match, your embedding and storage pipeline is misaligned.

# smoke_self_similarity.py
q = "sample stored chunk text"
res = index.query(vector=embed(q), top_k=1)
assert cosine(q, res[0].vector) > 0.99

Gotcha: this test rules out the common but wrong conclusion that embeddings are the problem when coverage is the actual root cause.

Where it gets complicated

• Coverage, not embeddings: Our self-similarity check returned ~0.9999, which ruled out an embedding mismatch. The real issue was missing source pages.
• Transcript dominance: ~91 percent of indexed chunks were raw YouTube ASR. That crowding effect degraded retrieval quality by overwhelming the index with long, off-structure text.
• Query boilerplate: Prepending instructions to user queries polluted retrieval. Clean the query before embedding.
• ID hygiene: An ID generator that truncated to 100 characters created duplicates. We replaced it with a digest of the canonical URL.
• No delete-before-upsert: Without removal first, stale chunks accumulated as orphans and confused retrieval.
• Bad change detection: A length based tracked-URL hash failed to notice real content edits and over-triggered on template shifts. We switched to a real content digest.
• Orchestration imports: Importing blueprints minted new webhook URLs. Airtable buttons still pointed at the old ones until we updated them.

What this actually changes

After we shipped the fix set, tests showed a clean corpus and healthier retrieval behavior:

• Index composition moved from ~91 percent ASR dominated to a balanced set with foundational site pages added first.
• Only 15 of ~282 site pages were present at audit time; we added a tracked-URL list and a deterministic ingest so the assistant references canonical pages.
• Self-similarity checks at ~0.9999 confirmed embeddings were wired correctly, so fixes focused on coverage and hygiene rather than tuning.

Why this matters: when employees spend less time hunting for answers, they get more done. McKinsey estimates knowledge workers spend about 19 percent of their time searching and gathering information. Reducing failed lookups is a direct lift. Source

Frequently asked questions

Does fixing embeddings improve answers if coverage is poor?

No. If the right sources are not in the index, a better embedding model will not recover missing content. Start with a tracked URL list, clean ingest, and delete-before-upsert. Use a self-similarity test to confirm embeddings are wired correctly, then focus on coverage.

How do you prevent duplicate or orphaned chunks?

Use stable document IDs derived from canonical URLs, delete existing chunks for that ID before writing, and avoid truncation or case sensitivity in IDs. Keep a content digest per URL and only re-ingest when the digest changes.

Can this run with Airtable and Make.com, or do we need a full rebuild?

You can keep Airtable as the control plane and Make.com for orchestration. The key is to move content normalization and index hygiene into a dedicated worker and treat imported blueprints as new webhooks that require updater steps.

How long does a coverage-first RAG remediation take?

Our build pattern is one to two weeks: day 1, 2 audit and plan, days 3, 5 worker and index hygiene, days 6, 7 answerability gate and end-to-end tests. Larger corpora or heavy media sources add time for normalization and chunking policy.

Do we need Pinecone, or can we use another vector store?

Any reliable vector database with delete and filtered search works. The fixes here are store agnostic: tracked URLs, delete-before-upsert, stable IDs, and an answerability gate matter more than the specific vendor.

What does this cost monthly to run?

The worker and vector store are the primary infrastructure costs. The gate uses a small model for pennies per call. Spend scales with ingest volume and query traffic. The larger cost is the one-time remediation and test pass to get coverage and hygiene right.

If you are running a RAG assistant and answers feel transcript-like or off topic, the issue is almost always coverage and hygiene, not embeddings. We built and verified the complete fix set for CCC's stack. See our earlier diagnosis write-up in CCC RAG content-creator: diagnosis and demo, read more about our AI search optimization service, or book a 15-minute call to scope your remediation.