Claude Prompting and Evaluation

Table of Contents

1. Challenge

   Introduction

   In this lab, you'll design, compare, and evaluate Claude prompts for common business analysis work at Globomantics. You'll structure prompts with Role-Task-Context-Format (RTCF) and compare prompt variations, break work into chained steps, compare zero-shot and few-shot prompting, refine output across several turns, evaluate output quality and document its issues, and analyze unstructured customer feedback with several prompting styles.

   You send prompts to Claude from the Terminal with run.py, a small script in the lab folder. The globomantics-prompting folder holds everything you need:

   • data/: meeting notes, support tickets, release notes, a customer call transcript, and customer feedback
   • prompts/: prompt files you run and edit
   • evaluation/: a summary to review and a scorecard to fill in
   • outputs/: where run.py saves Claude's responses
   • run.py: sends a prompt to Claude and saves the response

   Most tasks follow the same pattern: edit a prompt file in the editor, run it with run.py, and attach the files Claude should read. run.py saves the response to outputs/. Claude's responses vary from run to run, so yours won't match anyone else's word for word. The task checks look at structure and at accuracy against the source data, not at exact wording.

   Before you start, set up your Terminal:

   1. In the Terminal, move into the lab folder:

      cd globomantics-prompting
      

   2. Copy the API Key from the top of the lab page. Paste it between the quotes in this command, then run it:

      export LAB_API_KEY="paste-your-api-key-here"
      

      Note: run.py uses this key for every prompt you run in this Terminal. If you open a new Terminal, run the export command again.

   Each run.py command starts a new conversation with Claude, so earlier tasks don't influence the next result. Claude reads only the prompt and the files you attach with -f.

2. Challenge

   Structure prompts with RTCF

   A vague prompt leaves Claude to guess who the reader is, what matters, and what shape the answer should take. Role-Task-Context-Format (RTCF) gives every prompt the same four parts:

   • Role: the perspective Claude takes, such as a senior business analyst. Giving Claude a role, called role-based prompting, shapes its vocabulary, priorities, and tone.
   • Task: the specific action, with a precise verb such as summarize, extract, or classify
   • Context: the audience, purpose, and background that Claude can't infer
   • Format: the structure, length, and sections of the answer

   Wording, ordering, and formatting instructions all change the result. Precise verbs reduce guesswork, and explicit format instructions keep output consistent from run to run. A reliable prompt produces the same structure every time. To find the most reliable version, run the same prompt more than once and compare the structure of the outputs, not the wording. prompts/02-variation.md makes the same request as your RTCF prompt, but its Format section is a single line.

3. Challenge

   Decompose tasks and scaffold reasoning

   Complex requests produce better results when you split them into smaller prompts and chain them so that the output of one prompt becomes the input to the next. Each smaller prompt is easier to write, check, and fix.

   The first step in the chain is structured extraction: ask for a fixed JSON structure and specify how to handle missing values so the output is easy to reuse. The next link reasons over the extracted items. This is where chain-of-thought scaffolding helps: ask for the reason behind each verdict. Say what Claude should check, and ask for a one-sentence reason next to each result, so Claude works through each check where you can see it. Each reason is short enough to check against the source and uses relatively few output tokens. Placing the final list inside <answer> tags makes it easy to find and reuse.

4. Challenge

   Compare zero-shot and few-shot prompting

   • Zero-shot prompting gives Claude instructions with no examples. It works well when the task and the labels are unambiguous.
   • Few-shot prompting adds a few labeled examples. Examples show conventions that are hard to state as rules, such as how to label a ticket that reads like a complaint but asks for a feature the product doesn't have.

   Good examples are short, cover different labels, and don't come from the data you're classifying, so they teach a pattern instead of giving away answers.

5. Challenge

   Refine output with roles and follow-up turns

   In this step, the role is a communications lead, so the announcement takes a customer-facing perspective and tone. Style or persona prompting focuses more on setting the voice, such as plain and direct, rather than assigning a specific job role.

   In a multi-turn conversation, Claude uses earlier turns as context, so follow-up requests can be short, such as Make it shorter or Add a line for admins. Refining over several turns is often faster than trying to write a perfect prompt in one go. With run.py, add --continue to each follow-up to preserve the conversation context. Without it, Claude starts a new conversation and no longer has the previous draft to work from. After you check Task 5.1, see how much the role contributes. Send the same request without its first sentence, as a new conversation:

   python3 run.py -m "Using data/release-notes.md, write a release announcement for Globomantics Insights customers." -f data/release-notes.md -o outputs/05-no-role.md
   

   Compare the tone, structure, and emphasis of outputs/05-no-role.md with outputs/05-draft-1.md.

6. Challenge

   Evaluate outputs with human and model review

   Before you rely on Claude's output, check it against its source and against the prompt that produced it. Score these criteria:

   • Accuracy: every fact matches the source.
   • Completeness: the output covers everything the prompt asked for.
   • Alignment: the output follows the prompt's instructions and format.
   • Relevance: the output leaves out details the reader doesn't need.

   When a score is low, document each issue behind it with its type. Issues fall into five types. A hallucination is a claim the source doesn't support, an incorrect answer contradicts the source, and an irrelevant detail adds nothing for the reader. Omission and format issues cover missing content and ignored instructions. Claude can apply the same rubric in seconds, which makes it a useful second reviewer. It can also miss errors or score generously, especially when it reviews its own work, so treat model evaluation as a check on human review rather than a replacement.

7. Challenge

   Analyze unstructured customer feedback

   Unstructured text such as survey comments, support chats, and transcripts is where prompting saves analysts the most time, and where unchecked output does the most damage. In this task, you apply four prompting styles to the same customer feedback. A broad zero-shot prompt is a fast way to explore. For results you can report, combine the other three in one prompt: a role that sets the analyst's perspective, constraints such as a fixed range of themes, every comment assigned to a theme by ID, and quotes copied word for word, and few-shot examples that show the exact shape and level of detail you expect. Constraints like these make every claim traceable to the source. Grounded themes are a solid base for recommendations. Try a role-based follow-up in the same session:

   You are a product strategy advisor at Globomantics. Using @outputs/07-themes.json, write a five-bullet brief for the VP of Product with your top three recommendations. Cite the comment IDs that support each one.
   

8. Challenge

   Next steps

   Congratulations! You've now structured, compared, refined, and evaluated Claude prompts against real Globomantics data.

   Ideas to try in this workspace:

   • Adapt your RTCF prompt to summarize data/call-transcript.md for a different reader, such as the Finance team.

   • Turn your themes into recommendations with a different role:

     python3 run.py -m "You are a product strategy advisor at Globomantics. Using outputs/07-themes.json, write a five-bullet brief for the VP of Product with your top three recommendations. Cite the comment IDs that support each one." -f outputs/07-themes.json -o outputs/08-recommendations.md
     

   • Ask Claude to evaluate outputs/07-themes.json with the scorecard criteria, then verify its claims yourself.

   • Move the Format section of prompts/02-rtcf.md to the top, run it twice, and check whether the order changes the output.

   To keep learning, explore these resources:

   • Advanced Prompt Engineering for Claude on Pluralsight
   • More courses and labs in the Anthropic Claude skill path on Pluralsight
   • Prompt engineering overview in the Claude documentation