Test a Claude skill with five inputs before client work

Anthropic's skill guide says to build evaluations first, and its checklist asks for at least three test scenarios (source). Five covers the cases people forget.

Here is why it works. A skill is a set of written instructions, and Claude follows instructions literally. If your instructions leave a gap, Claude fills it with a guess, and you only see the guess when the input is different from the one you built the skill around. Most people build a skill while looking at one good example, so that example always works. The other four inputs show you what happens when the job arrives in a form you did not plan for. That is what real clients send.

First check that the skill is switched on: open Customize, then Skills, and look for your skill in the list. Then open a new chat. A fresh chat means Claude has only the skill and nothing else, so the test shows how the skill behaves for someone who just uses it. The prompt runs all five inputs in this one chat. If one result looks odd, rerun that input on its own in a new chat to confirm it.

1. 1

   Normal. A typical job you do every week.

2. 2

   Empty. Almost no information to work with.

3. 3

   Messy. Typos, missing details, pasted straight from an email.

4. 4

   Off-topic. A request the skill should not handle. A good skill says it is not its job, and it does not force an answer.

5. 5

   Edge case. The odd client, the rush job, the strange file.

Most people skip numbers four and five. They are where skills break.

Before each run, write down what a good answer looks like in two or three lines. Do this before you see the output, so you score against your standard and not against what Claude produced.

For the off-topic input, a pass means the skill declined or asked a question.

Be specific. "A good summary" is too vague to score. "Three bullet points, each under 20 words, with the client's name in the first line and no invented dates" is something you can check in ten seconds. If you cannot picture a clear fail, the line is not specific enough yet.

Paste this into a fresh chat where your skill is available.

Read the table. For each FAIL, change the skill's instructions using the suggested fix, or write your own.

Any fail means you fix the skill and run all five again. A fix for one input can break another, so rerun the whole set.

A worked example

Say you run a small bookkeeping firm with six people. You built a skill called "Client email reply" that turns a client question into a short, friendly reply in your firm's voice. You open a fresh chat and use the prompt above.

Claude writes five inputs. For the normal one, it drafts a client asking when their quarterly paperwork is due. The "good answer looks like" lines say: two short paragraphs, a clear date or a clear "I will confirm the date", a friendly sign-off, and no tax advice. The messy one reads like a real email: a client writes in a rush, mixes two questions, and forgets to say which month they mean.

Then Claude runs each input and scores it. A slice of the table might look like this:

You add the sentence to the skill's instructions for the empty case. Then you paste the test prompt into a new chat and run all five again. This time the empty input passes, but the messy one now fails, because the new rule made the skill ask too many questions. You soften the rule: ask only when no question at all can be found. Round three passes. The whole thing took about 30 minutes, and you found two gaps before a client did.

Common mistakes

1. 1

   Testing in a long, used chat. Earlier messages shape later answers, so a pass can be an accident. Fix: start every round in a new chat. If one result looks odd, rerun that input on its own in another new chat to confirm it.

2. 2

   Writing the inputs in your own tidy style. A messy test that is secretly clean tests nothing. Fix: copy a real email you received, typos and all.

3. 3

   Fixing a fail and running only that input again. One change can quietly break another input. Fix: always rerun all five.

4. 4

   Letting Claude score its own work without the quote. A bare PASS or FAIL gives you nothing to check. Fix: keep the rule that every FAIL quotes the part that failed, so you can check the verdict yourself.

5. 5

   Skipping the off-topic input because it feels pointless. This is the test that shows whether your skill invents answers when it should stop. Fix: always include one request that is clearly not the skill's job.

If a skill keeps failing after three rounds, step back and look at the job itself. A skill that tries to cover too many different jobs rarely passes all five. Split it into two smaller skills and test each one on its own.