What you'll learn

Generative AI applications require evaluation, testing, and monitoring to ensure they perform reliably in production. In this course, AIP-C01: Testing, Validation, and Troubleshooting, you'll gain the ability to evaluate, troubleshoot, and validate generative AI workloads on AWS. First, you'll explore how to evaluate foundation model outputs using Bedrock's LLM-as-a-judge evaluations, human feedback workflows, and RAG and agent evaluation frameworks. Next, you'll discover how to build automated quality gates with Step Functions, validate deployments using synthetic user workflows, and report on model performance with CloudWatch and Quick Sight. Finally, you'll learn how to troubleshoot common issues, including FM API integration errors, prompt engineering problems, and RAG retrieval quality using CloudWatch GenAI Observability and the Bedrock Agent trace. When you're finished with this course, you'll have the skills and knowledge of generative AI testing, validation, and troubleshooting needed to evaluate and maintain production-grade GenAI applications on AWS.