Putting an LLM where it cannot block the work

A case study — Teams Knowledge Capture

Summary

| | |
|---|---|
| **Problem** | Institutional knowledge lives in Teams threads and Outlook mail, where it is unfindable a month later and lost when the author leaves. |
| **Constraint** | Capture has to take seconds, inside the conversation. Any friction and people stop doing it — including friction the AI introduces when it is slow, wrong, or down. |
| **Solution** | A Teams message action and Outlook add-in that capture a discussion into a governed knowledge base through an Adaptive Card wizard, with Azure OpenAI drafting the metadata and detecting near-duplicates — every AI path optional and fail-open. |
| **Outcome** | Built and tested at unit level; AI features ship disabled by default and require configuration plus a licence entitlement. The Copilot connector is specified, not built. |
| **Role** | Architecture, bot, Outlook add-in, SPFx web parts, AI integration, licensing design, documentation. |
| **Stack** | Node 18+ / TypeScript · Bot Framework 4.23 · restify · Azure OpenAI (gpt-5 family) · Azure Blob · Application Insights · SPFx · Adaptive Cards |

1. The problem

The answer to "why did we do it that way" is usually in a Teams thread from eight months ago. It was written by someone who has since moved teams, in a channel nobody searches, and it will be rediscovered — expensively — by the next person to ask.

Knowledge bases exist to solve this and mostly fail for one reason: capture is a separate act. Someone has to leave the conversation, open a different tool, re-type what was just said, classify it, and publish it. The people best placed to capture are the busiest, and they never do.

2. The constraint

Capture has to happen inside the conversation, in seconds. That single requirement rules out most of the design space and dictates everything below — including the AI decisions, which are the interesting part.

It also creates an unusual constraint on the AI itself. If an LLM sits on the capture path and is slow, wrong or unavailable, it does not degrade the feature — it destroys it, because the user abandons the capture and never comes back. The AI has to be genuinely optional, at runtime, on every call.

3. Architecture

A Teams message action opens an Adaptive Card wizard over a 45-field submission schema. Drafts persist in Azure Blob. On submit, the item goes into a governed SharePoint knowledge hub through an approval flow, and is searchable through three SPFx web parts. An Outlook add-in does the same for mail.

The piece worth naming is src/shared/ — a deliberately runtime-agnostic capture core, free of Node, Office and browser dependencies, shared by the bot, the Outlook task pane and the SPFx web parts. submission.ts is the single source of truth for the field set, the wizard steps and the validation.

Three surfaces, one definition of what a submission is. Without that, the Teams card, the Outlook pane and the web part drift into three subtly different schemas, and the knowledge base fills with items that cannot be compared.

4. The AI decisions

This is the substance, and each decision is about containing the model rather than showcasing it.

4.1 Auto-fill that can only return valid answers

Azure OpenAI extracts a title, a summary and five classification values from the captured message. The interesting part is the constraint: the model is given the active choice sets and can only return options from them — it cannot invent a department that does not exist in the taxonomy.

The prompt forces json_object output and instructs the model to omit a key when unsure rather than guess: *"OMIT the key if unsure. Never invent a value."*

And the failure path: every error returns an empty object. Capture is never blocked. The user gets an unfilled form instead of a filled one, which is exactly what they would have had without the feature — the AI can only ever add.

That is the correct shape for AI on a critical path: constrained output, silence over invention, and a failure mode identical to absence.

4.2 Duplicate detection as a two-stage pipeline, not a prompt

Semantic near-duplicate detection is where a naive implementation sends the whole knowledge base to the model and hopes. This one is a real retrieval pipeline:

1. Fetch the approved-article list (cached).
2. Shortlist candidates with a cheap lexical overlap score — no model call.
3. Let the model semantically confirm the true near-duplicates from that shortlist, and say why.

Falling back to the lexical shortlist alone if the model call fails. The expensive, non-deterministic step runs last, over a small candidate set, and its absence degrades quality rather than breaking the feature.

A later revision added a SharePoint Search mode that takes precedence: server-side ranked candidates, which scales to any knowledge-base size instead of pulling a list client-side. The trade-off is written down rather than glossed: *fresh items lag until they are indexed.* Faster and unbounded, at the cost of a window where a brand-new article is invisible to duplicate detection.

4.3 A spend cap in the right layer

The daily spend cap sits in front of the chat call itself, not in the HTTP layer. The reason is documented, and it is the kind of thing only discovered by tracing the real call graph:

The Teams bot calls the suggest and similar functions in-process, never through the HTTP endpoints. A cap on the HTTP endpoints would miss the busiest path entirely.

A cap on the API routes would have looked complete on an architecture diagram and protected nothing. The same comment refuses to overclaim what the cap is: *"defence-in-depth against a leaked add-in secret or a runaway loop, NOT a billing guarantee."*

4.4 A model migration written down as it was hit

The Azure chat wrapper documents a real constraint of the gpt-5 family: those models reject `max_tokens` and non-default temperature, and they spend part of the token budget on internal reasoning before producing output — *"so keep it generous or the visible JSON can come back empty."*

That is a failure that presents as a bug in your own parsing code. Writing it in the wrapper, next to the parameter it explains, is worth more than a paragraph in a wiki nobody opens.

The same comment records a timeout defect found in review: the duplicate-detection judge call had no timeout, so the endpoint could wait on Azure OpenAI indefinitely.

4.5 The Copilot connector, specified and not built

A Microsoft Graph connector would ingest approved-only items into the Microsoft Search and Copilot index, mapping confidentiality and audience fields onto connector ACLs.

It is designed, documented, and marked planning stage. The design document is unusually honest about why it might not be worth building:

M365 Copilot already reasons over SharePoint content a user can access, so published pages get some coverage for free. The connector adds curation, rich metadata for ranking, clean citations, and governance control.

That is an argument that the feature's value is *incremental*, written by the person proposing it. It is the opposite of how AI features are usually pitched.

5. Licensing

Entitlement is a self-issued RS256 JWT verified offline, chosen over a marketplace transactable offer, with the model stated compactly: *the licence is the entitlement ceiling; the admin toggle is on/off within it.*

Offline verification means the product works in a tenant with no outbound path to a licensing service, and there is no licence server to run or keep up. The trade-off — revocation is not immediate, and the signing key becomes critical — is the accepted cost.

6. Outcome, and what is not real yet

Built, with 274 unit tests across 27 files. Three SPFx web parts, the bot, and the Outlook add-in, over a 45-field schema.

Stated plainly, because it matters for how this is read:

• Every AI feature defaults to off, and requires both Azure OpenAI configuration and a licence entitlement before it will run.
• The Copilot / Graph connector is planning-stage only. Its own decisions log is headed *"Status: planning / proposed."*
• The freshness-nudge and weekly-digest flows are documented to be built in Power Automate, not built.
• Outlook attachment content-copy is unresolved — flagged as a governance change to confirm with the client rather than a technical gap.
• Tests are unit-level only. No end-to-end run against a live tenant.

7. What I'd take from this

Put the model where its absence is survivable. Every AI path here fails open: auto-fill returns nothing and the form is blank, duplicate detection falls back to lexical matching, the spend cap trips and capture continues. Nothing the model does is load-bearing. That is what makes it safe to ship an LLM feature into a workflow where friction kills adoption.

Constrain the output space rather than trusting the prompt. Handing the model the valid choice sets and telling it to omit rather than guess is worth more than any amount of instruction about accuracy. It cannot return an invalid department because the invalid department is not on the menu.

Cheap filter, expensive judge. The two-stage duplicate pipeline is the general pattern: use a deterministic, free scoring pass to reduce the candidate set, then spend a model call on the small set that survives. It bounds both cost and latency without bounding quality much.

Trace the real call graph before placing a control. The spend cap in the HTTP layer would have been invisible to the busiest caller. Controls belong where the calls actually are, not where the architecture diagram suggests they should be.