Evaluate generative AI applications

By the end of this module, you're able to:

• Apply best practices for choosing evaluation data
• Understood the purpose of and types of synthetic data for evaluation
• Comprehend the scope of the built-in metrics
• Choose the appropriate metrics based on your AI system use case
• Understand how to interpret evaluation results

By the end of this module, you're able to:

• Assess a generative AI app response using performance and quality metrics
• Assess a generative AI app response using risk and safety metrics
• Run an evaluation and track the results in Azure AI Studio
• Create a custom evaluator with Prompty
• Send queries to an endpoint and run evaluators on the resulting query and response
• Generate a synthetic dataset using conversation starters

• Introduction
• Evaluate generative AI
• The role of data in evaluations
• Choose and utilize metrics
• Custom prompt-based evaluators
• Interpret evaluation results
• Common pitfalls in result interpretation
• Knowledge check
• Summary

• Introduction
• Prepare
• Exercise - Performance and quality metrics
• Exercise - Risk and safety metrics
• Exercise - Track evaluation results in Azure AI Studio
• Exercise - Custom evaluator with Prompty
• Exercise - Evaluate an endpoint
• Exercise - Generate a synthetic dataset
• Knowledge check
• Summary