The situation
The client runs an established online membership for classroom educators, with a library of teaching resources, planning materials and professional development hosted on Kajabi, with a subscriber base built over several years. The library is large and, by the client's own admission, under-used: members joined for it, then struggled to find the specific thing relevant to the situation in front of them on a given Tuesday. The hypothesis was that an AI coach that knew both the library and the individual educator could close that gap. The question was whether members would actually use such a thing, and nobody knew.
What made this engagement unusual is that the client arrived with the right instinct already formed. They did not want a SaaS platform. They wanted the smallest functional version capable of testing engagement with ten to twenty existing members, before committing to anything larger. They had written down what they explicitly were not building (payments, mobile apps, analytics, single sign-on with the existing membership, real account structures) and were comfortable with manual access provisioning for the beta.
Why off-the-shelf didn't fit
The obvious answer here is a general-purpose assistant with the library uploaded to it, and the client had tried exactly that with an off-the-shelf custom GPT before calling us. It failed on two counts that turned out to define the real requirement.
Memory was the first. A teacher's context is stable and specific (year group, subject, class size, the constraints of their particular school) and re-establishing it at the start of every conversation is precisely the friction that stops people coming back. The off-the-shelf version began each session as a stranger. The second was the citation problem: asked something the library did not cover, it answered anyway, in a confident and generically useful way, without any link back to the membership. That is worse than unhelpful for this client, because the entire commercial logic of the tool is driving members into resources they are already paying for. An assistant that answers well from general knowledge actively undermines the product it is supposed to support.
Building it properly as a bespoke application was the remaining option, and we argued against it. At this stage the client did not have a product question, they had an engagement question, and engagement questions are answered by the cheapest artefact that can generate a real answer. Quoting a full build here would have been more revenue and worse advice.
Scoping the real workflow
Most of the scope call was spent removing things, which is unusual and was the most valuable part of it. The client's initial list included a fairly complete admin panel, per-member usage analytics, and branded theming to match the membership. We quoted a version with a minimal admin view, no analytics beyond raw conversation logs, and enough branding to not feel like a prototype, on the argument that with a cohort of twenty, the client would learn more from reading twenty conversations directly than from any dashboard built on top of them, and that a dashboard over a sample that small would mostly manufacture false confidence.
We spent the reclaimed time on what would actually determine whether the beta succeeded: what makes a coaching response good. Proof-of-concept work usually skips this question, and it decides whether the engagement data means anything. We sat with the client and two of their most engaged members and wrote a short rubric. A good response is specific to this educator's stated context, points to a named resource in the library when one exists, says plainly when one does not, and gives something actionable this week rather than a restatement of best practice.
That rubric became the thing we built against and evaluated against. The scope was five items: authentication and manual provisioning, an onboarding questionnaire producing a persistent member profile, the chat interface, retrieval over the client's proprietary material with resource linking, and cumulative context that survives between sessions.
What we built
Onboarding is a short structured questionnaire, not a conversation, because the facts it captures are ones the system needs reliably and a chat interface makes them optional. It produces the member profile: year group, subject, school context, class size, what they are currently finding hardest. That profile is a structured record the agent reads on every turn.
The memory design is the part worth describing, because the naive approach fails quickly. Appending the full conversation history to context degrades noticeably within a few sessions and gets expensive at the same time. Instead there are two separate things. Conversation history is stored and retrieved selectively, by relevance to the current turn. Alongside it sits the structured profile, which the agent updates deliberately: when a member says something durable about their situation, such as moving year groups or starting to mentor an NQT, the agent writes that to the profile as a discrete fact rather than hoping it survives in a transcript. What persists across sessions is a small, legible set of facts about that educator, not an ever-growing log.
Retrieval runs over the client's own material, chunked with its source resource preserved so every answer can name and link the thing it came from. The library is around four hundred resources, which is small as corpora go and creates a specific hazard covered below.
The admin view is deliberately thin: a list of members, their profiles, and their full conversations, readable end to end. For a cohort of twenty, that is the analytics layer.
Where it got hard
The dominant problem was retrieval on a small corpus. With four hundred resources, a vector search returns the nearest neighbours to any query, and on a corpus that size the nearest neighbour to a question the library does not address is still something, often with a similarity score that looks respectable. The first version therefore confabulated: asked about a topic the membership genuinely did not cover, it would retrieve a tangentially related resource and construct an answer that cited it, which is the exact failure mode the client had already rejected in the off-the-shelf tool, reproduced by us with more steps.
Fixing it took two changes. A relevance floor, calibrated against a set of questions the client confirmed the library does not answer, below which retrieval returns nothing rather than its best guess. And an explicit no-support path in the agent, so that when nothing clears the floor it says the membership does not cover this, answers from general knowledge if it usefully can while labelling that it is doing so, and flags the gap. The log of questions the library could not answer is now a content roadmap the client did not previously have.
Evaluation was the other hard part, and the rubric written in scoping was what answered it. The client's success metric was engagement, and engagement is a metric an agent can win dishonestly. A warm, agreeable coach that validates whatever the member says will produce return visits and teach nobody anything. We scored a sample of conversations against the rubric weekly through the beta, specifically watching for responses that were pleasant and empty. Two rounds of prompt and retrieval changes came out of that, both aimed at making the agent more willing to be specific and to disagree.
One smaller problem remained. The first profile-update logic was too eager and wrote conversational asides into the profile as durable facts, so a member mentioning a one-off cover lesson ended up permanently profiled as teaching a different year group. Updates are now narrower and the member can see and correct their own profile, which is both a fix and the right default.
Rollout & results
The beta launched at the end of week three with twenty members provisioned by hand, as scoped. At thirty days, seventeen were still active, which is well above the threshold the client had set in advance as the signal to continue. The threshold was set before the beta ran, not after.
Roughly 64% of sessions surfaced at least one membership resource, which is the number that matters commercially, since the tool exists to make an existing library findable rather than to replace it. The gap log has so far produced a list of eleven topics members asked about repeatedly that the library does not cover; the client has begun producing material against it.
The engagement did what a proof of concept is supposed to do, which is end. The client has a decision, made in six weeks total rather than after a year of building, and a specific brief for the larger version informed by twenty real conversations rather than by assumption.
"I came in with a list of about thirty things and left with a build of five. The five were the right five, and I'd have spent four months finding that out on my own." Founder, client engagement
What we'd do differently
We measured whether members came back and not whether anything changed in their classroom. Return visits were the right primary signal for a first beta and we would use it again, but a single question at the end of a session, asking whether they used it and whether it helped, would have cost almost nothing to add and would have made the go/no-go conversation considerably better. We are now including one lightweight outcome question by default in engagements of this shape.
We would also calibrate the retrieval relevance floor before wiring the agent rather than after seeing it confabulate. Assembling the set of questions a corpus cannot answer takes an afternoon with the client and is the fastest way to find the edge of what a knowledge base actually contains, and on a small corpus that edge is what breaks the product.