Claude Chatbot Development

Table of Contents

1. Challenge

   API setup

   To start you will initialize the API you will use for chatbot creation.

   On the right, open the notebook file, ClaudeChatbotDevelopment.ipynb.

   Then, in the first cell under API setup, replace <TODO_ENTER_API_KEY_HERE> with the generated API key (top-center of the lab) and run the first cell and the cells within the API setup section to ensure the model is connected.

2. Challenge

   Chatbot conversation design and persona

   For each Step, run the cells in the notebook's corresponding section you opened in the first Step.

   • For example, in this Step, run the cells under the Chatbot conversation design and persona section in the notebook.

   Also, in this Step you're creating a chatbot, so

   1. once you run the cell that starts the chatbot, enter a prompt or two.
   2. To exit the chatbot and move on to the next cell, enter quit.

   Chatbot conversation design

   Chatbots have two types of conversations: Single-turn, meaning the history of the previous messages is not kept between prompts sent to the AI, and multi-turn, which does keep previous message history.

   The chatbot in the notebook has both single-turn and multi-turn conversations.

   Most models have evolved to be strictly multi-turn, but single-turn still has specific niches which make it more optimal. These niches include the following:

   • Classification tasks
   • Rewriting
   • Text extraction
   • Q/A
   • RAG (Retrieval Augmented Generation)

   Because history is not valuable for these tasks, the reduction in input tokens can result in API cost savings with minimal loss of AI efficiency.

   Besides these specific use cases, and even potentially within them, multi-turn prompting becomes much more standard within chatbots. ### Chatbot personas

   Personas can be defined within the system prompt. They help narrow the LLM's scope of knowledge, as well as set standards of the tone and information the model should reply with.

   It is EXTREMELY IMPORTANT to note system prompts and other safety guardrails DO NOT prevent AI models from releasing specific information. The only way to prevent an AI model from releasing protected information is to not give the model access.

3. Challenge

   Multi-turn conversation memory management

   A reminder: As mentioned in the last Step, run the cells in the corresponding notebook section. Apply this to future Steps too.

   Memory management is key for proper implementation of chatbots, both for cost effectiveness as well as curating proper responses from Claude models.

   For multi-turn memory conversations, input tokens can escalate quickly without proper pruning sky-rocketing costs. The bloat of previous conversation history can also deteriorate the AI's response. Deterioration occurs both because previous information can be difficult to locate and because it can be irrelevant to the purpose of the AI.

   Several methods exist to limit conversation history:

   • Limit history by static token amount.
   • Limit history by number of conversation turns.
   • Rebase historical conversation to relevant information.
   • Implement facts and relevant memories similar to RAGs.

   For most memory management, it is ideal to use Claude Code RAG implementations or similar tools to keep facts and information in a separate file. Pairing this method with rebasing conversational history allows for great model upkeep as well as cost reductions.

4. Challenge

   Safety guardrails

   Run the cells within this section to see a standard baseline for a system prompt, as well as testing a single-turn conversation with the defined system prompt. Guardrails do not prevent discussion of undesired subjects.

   • Guardrails should not be treated as a guarantee that an LLM will never mention an unwanted, sensitive, or prohibited subject.
   • In other words, trying to red-team the conversation, or otherwise trying to socially engineer the model to give up critical information would likely still work.

   A model may still discuss undesired subjects in cases such as the following:

   • Explaining why a request cannot be completed.
   • Analyzing code or documentation supplied by the user.
   • Identifying a security vulnerability.
   • Providing defensive cybersecurity guidance.
   • Quoting or summarizing existing content.
   • Responding incorrectly or inconsistently.
   • Encountering an indirect prompt injection inside a file, webpage, issue, or tool result.

   The more useful distinction is between discussing a subject and performing or materially enabling an unsafe action.

   For example, a coding agent may need to explain what SQL injection is in order to repair a vulnerable application. A guardrail should not prevent that discussion. It should instead prevent the agent from attacking an unauthorized system, exposing credentials, or executing destructive commands without appropriate authorization.

   Guardrails reduce risk; they do not create a topic-proof boundary around a language model.

5. Challenge

   LLM metrics and meta analaysis

   Two code snippets for code latency checks and token usage check are included within this section. Run the cells to see basic methods of tracking these two metrics with each call. Additionally, exploring auditability is more complex, but explained in a bit more depth without a coding example below. ### Preserve auditability

   Along with latency and token metrics, auditability is key to backtrace issues and ensure the model is staying within its parameters. Specifically with using chatbots capable of modifying files or data, such as with the case with coding agents, auditability becomes extremely valuable.

   Important agent actions should produce useful records, such as in

   • the requested task,
   • commands or tools used,
   • files changed,
   • tests performed,
   • approval decisions,
   • errors encountered, and
   • the final outcome.

   Logs should support investigation without storing API keys, passwords, private source code, or unnecessary personal information.