AWS Flash - Operationalize Generative AI Applications (FMOps/LLMOps)

This course provides an overview of challenges in productionizing LLMs and a set of tools available to solve them. The course will provide an overview of the reference architecture for developing, deploying, and operationalizing LLMs, as well as expand on each stage of the process.

• Course level: Intermediate
• Duration: 90min

Activities

This course includes presentations, real-world examples and case studies.

Course Objectives

In this course you will learn to:

• Differentiate between MLOps and LLMOps and define core challenges in operationalizing LLMs
• Learn how to select the optimal LLM for a given use-case
• Understand how to evaluate LLMs and the difference between evaluation and benchmarking
• Define core components of Retrieval-Augmented Generation (RAG) and how it can be managed
• Differentiate continual pre-training from fine-tuning
• Understand fine-tuning techniques available out-of-the-box on AWS
• Learn about what to monitor in LLMs and how to do it on AWS
• Understand governance and security best practices
• Illustrate reference architecture for LLMOps on AWS

Intended Audience

This course is intended for:

• Data Scientists and ML Engineers looking to automate the build and deployment of LLMs
• Solution Architects and DevOps engineers looking to understand the overall architecture of an LLMOps platform

Prerequisites

We recommend that attendees of this course have:

• Completion of Generative AI Learning Plan for Developers (digital)
• A technical background and programming experience is helpful

Course Outline

Module 1: Introduction to LLMOps

• Introduction to LLMOps
• LLMOps Roles
• Challenges in operationalizing LLMs

Module 2: LLM Selection

• Use-case benchmarking of LLMs
• Priority-based decision making

Module 3: LLM Evaluation

• Evaluation methods
• Evaluation prompt catalog
• Evaluation framework and metrics
• Benchmarking framework and metrics

Module 4: Retrieval Augmented Generation (RAG)

• LLM customization
• Embedding models
• Vector databases
• RAG workflows
• Advanced RAG techniques

Module 5: LLM Fine-tuning

• Continual pre-training vs. fine-tuning
• Parameter-efficient fine-tuning (PEFT)
• Fine-tuning architecture

Module 6: LLM Monitoring

• LLM monitoring
• LLM guardrails

Module 7: LLM Governance and Security

• Security and governance best practices
• Security and governance tools

Module 8: LLMOps Architecture

• LLMOps lifecycle

Demos

• Text embedding and semantic similarity
• LLM fine-tuning and evaluation at scale
• Inference safeguards

Keywords

• Gen AI
• Generative AI