Testing Your AI Colleague
Every colleague has a Tests tab — a regression suite that replays real conversations against your current prompt and checks the colleague still answers the way it should. The point is simple: when the colleague gets something wrong, you capture that exact moment as a test, fix it until the test passes, and it never regresses again. This is how a colleague gets reliably better over time instead of drifting.
What a test actually is
A test is a saved conversation plus an expected outcome. When you run it, the colleague re-answers the same messages using your current prompt and the real reservation context, and the suite checks the answer against your expectation — both with simple rules (must say / must not say) and an AI judge that reads the reply the way a person would.
- Pass — the current prompt handles this situation correctly.
- Fail — the colleague no longer answers the way you want. Something in the prompt needs fixing.
How to add a new test
You don't write tests from scratch — you promote them from real calls and chats. The flow is the same whether it started as a phone call or a text conversation:
When you spot a bad reply — in the Inbox (a chat) or a Call transcript (a voice call) — mark that message as failed. That flag is the signal that this moment is worth turning into a test.
Open the colleague's Tests tab. Flagged conversations appear at the bottom under Candidates — a shortlist of real failures waiting to become tests. Each candidate carries its own conversation and the reservation it belongs to.
Promote a candidate to turn it into a saved test case. The conversation and its reservation context are captured with it, so the colleague re-answers under exactly the same conditions the real guest saw.
Open the case and describe what a good answer looks like — what the colleague must say, what it must not say, and whether it should escalate to a human. This is what the run checks against. You can trim the conversation to the turns that matter, and edit it later as your expectations sharpen.
Run the case (or Run all). The colleague answers live and you get a pass / fail with the actual reply and which checks passed. A freshly promoted failure will usually fail — that's the point; it reproduces the bug.
On a failing case, use Fix with AI. It iterates on the prompt until the case passes without breaking the others, and files the change as a proposal for you to review — it never edits your live prompt behind your back.
Approve the proposed prompt change, then Publish from the header. Publishing is what makes the new prompt live for real guests — until you publish, everything stays a draft.
Running vs. publishing
Two separate ideas that are easy to mix up:
- Running a test only checks the current draft against your expectation. It changes nothing your guests see — run as often as you like.
- Publishing (from the header) is what promotes your edited instructions to the live prompt. The test suite is the gate you run before publishing to make sure a prompt change didn't quietly break something that used to work.
What makes a good test
- One clear expectation per case. "Offers the entrance code only to a matched guest" is testable; "answers well" is not.
- Real edge cases beat happy paths. The unmatched caller, the expired link, the request the colleague should refuse — those are the ones worth locking down.
- Be explicit about escalation. If the colleague should hand off to a human, say so; if it should not, say that too — a false escalation is a real failure.
- Keep the conversation tight. Trim to the turns that lead up to the moment you care about.
Where to go next
- Prompting Your AI Colleague — the tests tell you what to fix; this tells you how to fix it.
- Call Transcripts — where you flag a bad reply on a voice call.
- How Guest Message Automation Works — where chat replies come from, and where you flag them.