Maria Fernanda Rojas Castro

The Fellowship of Agentic Evaluations: How to evaluate an agent?

Autonomous agents are redefining what we can build with LLMs, allowing AI not only to respond but to reason and execute actions across complex flows. However, as we give them more autonomy with tools like LangGraph, a critical engineering problem arises: How do we know if the agent is truly reliable? In this workshop, we will address the challenge of agent evaluation. We do not want to limit ourselves to checking whether the final response sounds good, but to implement different ways of evaluating these agents, ranging from deterministic validations to metrics based on other LLMs (LLM-as-a-judge). For this exercise, we will apply these concepts to the universe of The Lord of the Rings mixed with real medical standards to validate mainly: - Data Integrity: Validate that the agent correctly extracts and formats structured medical codes. - Fidelity: evaluate that the agent does not hallucinate diagnoses. - Observability: Trace and cost monitoring with Langfuse. Upon completion, attendees will have a clear idea of how to answer the question: How to evaluate an agent?