What people are actually asking

Most people who search for a symptom are not trying to diagnose themselves. They are trying to decide something: is this an emergency, is this worth a clinic visit, or can I wait and see?

Search results answer that question badly. A list of pages ranked by keyword density is a list of documents, not a decision. So a symptom assistant looks like an obvious improvement - and the improvement is real, until the moment it is asked something it should not answer.

The obvious design, and why it fails

The first version of every symptom assistant is: ask the model, stream the answer, and if the model is not confident enough, show a message telling the user to see a doctor.

That message is correct. It is also close to worthless, because it is the answer to a question the user did not ask. The user asked do I need to go now. The system replies go, always - which is unfalsifiable advice, and therefore not advice at all.

Worse, it is a dead end. The conversation ends at the moment of maximum anxiety, and the user leaves with nothing to do, no idea what to watch for, and no sense of whether the situation is getting worse.

Refusal is a feature, not an error state

So the refusal has to be designed as carefully as the answer, because in a health product the refusal is the part that matters most.

The change was to treat "I am not going to answer this" as a first-class response with its own content, not as an error. It carries:

• A clear reason. Not "I can't help with that", but this sounds like it needs someone who can see you.
• A triage shape. What would make this an emergency. What would make it a same-week appointment. What would make it something to watch. The boundaries are stated in the interface, not hidden in a model prompt.
• A next action that is not "go to a doctor". A specific question to answer, a symptom to track, a clock to start. Something the person can do in the next ten minutes.
• A way to bring the information with them. A short summary of what was said, so the appointment is not a blank slate.

The escalation behaviour is not decoration around the product. In MedAssist it is the product: the assistant is designed to refuse, and the measure of success is that it refuses the right things, clearly, at the right moment.

Making the boundaries real

An assistant that promises to escalate and then does not is worse than one that never promises. So the escalation rules are not only in a prompt - they are in code, in a layer that can be tested.

The pattern that worked:

1. Classify the request into a small set of buckets - emergency, urgent, routine, out of scope - with explicit, written criteria.
2. Refuse before generating. The bucket is decided first. If the request lands in emergency or out of scope, the model is never asked to produce medical advice, which means there is no answer to leak and no wording to get wrong.
3. Say what was decided and why. The interface shows the reason for the escalation, in plain language, so the person can disagree with it.

Step three is what turns a refusal from a wall into a conversation. "This looks like it needs a doctor" invites "why do you think so?" - and the answer can be specific: because of a symptom and a duration, not because of any instruction.

What the model is actually for

Once the boundaries are handled in code, the model's job becomes much smaller and much more valuable: the open-ended part. The questions that are not triage questions at all.

• "What is this rash usually called?" - naming something is not diagnosing it.
• "What should I write in the message to my landlord?"
• "What does this blood test result usually mean in general terms?" - followed by a clear note that a number without a reference range means little, and a reference range is personal.

These are the questions where a language model is genuinely useful, and none of them require it to decide whether someone should go to an emergency department tonight.

The part that is easy to get wrong

Tone. A refusal delivered in a flat, hedged, bureaucratic register reads as an automated non-answer, and people conclude the system has not understood them. A refusal delivered in a warm, specific, practical register reads as a system that is taking them seriously - even when the outcome is "not here".

The test I use: read the refusal aloud. If it sounds like a policy document, it is not finished. If it sounds like a competent person who has run into a limit and is telling you what happens next, it is.

What I would keep from this

• Decide the triage before you generate the answer. After that, the dangerous path does not exist.
• A refusal with a next action is still a good experience. A refusal without one is a dead end.
• Say the reason. It is the difference between a wall and a conversation.
• Keep the model's job narrow. The more it is asked to decide, the more it will eventually decide something it should not.

The uncomfortable truth underneath all of it: in a health product, the quality of the system is mostly the quality of its refusals. The answers are the part everybody demos.