Artificial Intelligence Machine Learning

The Fellowship of Agentic Evaluations: How to evaluate an agent?

FORMAT: WorkshopLEVEL: Intermediate LANGUAGE: Spanish

Autonomous agents are redefining what we can build with LLMs, allowing AI not only to respond but to reason and execute actions across complex flows. However, as we give them more autonomy with tools like LangGraph, a critical engineering problem arises: How do we know if the agent is truly reliable? In this workshop, we will address the challenge of agent evaluation. We do not want to limit ourselves to checking whether the final response sounds good, but to implement different ways of evaluating these agents, ranging from deterministic validations to metrics based on other LLMs (LLM-as-a-judge). For this exercise, we will apply these concepts to the universe of The Lord of the Rings mixed with real medical standards to validate mainly: - Data Integrity: Validate that the agent correctly extracts and formats structured medical codes. - Fidelity: evaluate that the agent does not hallucinate diagnoses. - Observability: Trace and cost monitoring with Langfuse. Upon completion, attendees will have a clear idea of how to answer the question: How to evaluate an agent?

Speakers

Maria Fernanda Rojas Castro

Advanced Data Engineer @ Loka INC

I'm a Data & ML Engineer with a background in Biomedical Engineering. I work in tech consulting, where I design and build data workflows and experiment with applying machine learning and GenAI in practical ways. I’m especially interested in LLMs, knowledge graphs, and making complex ideas easier to understand, sometimes with the help of pop culture. Outside of work, I love dancing, reading fiction, watching movies, and obsessing over Formula 1. I like finding creative ways to learn and teach, and I bring that spirit into every project or talk I’m part of.

View speaker

Nicolas Roldan Fajardo

Advanced Machine Learning Engineer @ Loka INC

I’m a ML Engineer with a background in Biomedical Engineering. Back in my later years of college, I started focusing on Biomedical Signal and Image Processing and its different applications. This, in turn, led me to develop a strong interest in machine learning, whose roots have been entangled with my dear Signal Processing from the beginning. Nowadays I spend my days shipping (or at least trying) ML models to production for diverse use cases, going from Computer Vision applications in traffic analysis, Drug Discovery using Deep learning based protein embeddings, and automating medical scribe applications using agentic AI

View speaker

Want to know more?

Join PyCon Colombia newsletter and get a complete overview of our events, speakers and community participation.