Incorporating LLMs in High-stake Use Cases - Best Practices and Considerations

Explore best practices for incorporating Large Language Models (LLMs) in high-stakes applications through this insightful conference talk by Yada Pruksachatkun. Learn how to build applications with non-determinism in mind, consider human-in-the-loop systems, and address challenges in deploying LLMs for critical use cases. Discover techniques for breaking tasks into smaller components, utilizing in-context examples, and evaluating models rigorously. Gain valuable insights on when to use LLMs, the benefits of fine-tuning custom models, and open questions surrounding black-box LLMs in high-risk scenarios. Drawing from her experience as an NLP scientist and engineer, Pruksachatkun provides practical guidance for leveraging LLMs responsibly in production environments.

Intro
Let's start with a therapy bot
Best practices from Dialogue Systems 1.0
Best Practice: Breaking up your tasks into smaller tasks
Best Practice: In context Examples
Best Practice: Don't use LLMs if you don't have to
Best Practice: Finetune your own LLM
Evaluate your model rigorously
Open Questions in using Black Box LLMs for high-stakes use cases