Creating Representative Query Sets for Offline Evaluation

Discover a powerful method for constructing representative query sets for offline evaluation in this insightful conference talk from Haystack US 2023. Explore how Getty Images tackles the challenge of scaling AI/ML experimentation and reduces A/B test candidates through offline testing. Learn about the importance of creating query sets that accurately represent customer activity across various attributes. Gain valuable insights into a simple yet effective approach for building minimal, randomly sampled query sets that maintain representativeness across multiple dimensions. Delve into the speaker's expertise in ranking systems, algorithm development, and the integration of Natural Language Search at Getty Images. Understand how offline testing can provide crucial insights into the impact of sort algorithms on guardrail metrics and estimate effects on high-level customer metrics like conversion and interaction.

Haystack US 2023 - Karel Bergmann: Creating Representative Query Sets for Offline Evaluation