Evals and Data Curation

Objective: Master methods for evaluating LLM performance and curating high-quality training data. Students will learn how to create effective evaluation metrics and manage data for LLM applications.

Case Study

LLM Evals and Data Curation by the course's Data/Prompt Engineers

Required Readings

• How to Write Data Labeling/Annotation Guidelines
• Data Flywheels for LLM Applications
• Task-Specific LLM Evals that Do & Don't Work

Optional Reading

• Evaluating the Effectiveness of LLM-Evaluators