Overview of AI Agent Evaluation
When developing a software application, it is common practice to have a set of tests and evaluations. This practice helps us to ensure that it works as expected and is reliable in production.
Then, what about AI agents? Do we need to evaluate them?
Of course we do, but it is more challenging than traditional applications.
So, how do we evaluate them?
Well, that's a tough question. But before we jump into how we evaluate AI agents, let's first understand how traditional applications are evaluated.
Evaluation in traditional software application
In a traditional software application, its behavior is mostly deterministic, which means that the output is predictable when we can control the variables, such as input, application state, and application environment.
For example, let's say we develop a BMI calculator that takes weight and height to calculate the body mass index, rounded to one decimal place.
If we evaluate it using a height of 155 cm and a weight of 40 kg, our application should consistently return a BMI of 16.6 regardless of how many iterations we have.
If the application returns a different result, such as 16.5, when using the same inputs and conditions, the test should fail.
Evaluating AI Agents
When evaluating AI agents, we should understand their nature and behavior. Unlike traditional software applications, AI agents may produce different outputs even when given the same input. This is partly due to their reliance on Large Language Models (LLMs), which generate responses based on probability rather than following strictly deterministic rules.
For example, let's say we develop an AI agent to respond to user greetings. If we input "Hi" into the agent, we might expect it to respond with "Hi, how are you?". However, that's not always the case. The next time we provide the same input, we might get a response like "Hello! How can I help you today?". However, this behavior is not entirely random. There are several variables that can influence how an LLM generates its response, such as the LLM model, session context, or temperature (but we won't cover it in this article). This variability makes evaluating AI agents more challenging than evaluating traditional software applications.
Now that we understand how AI agents behave, let's move on to how we evaluate them.
There are two types of evaluation we can perform:\
Deterministic
In deterministic evaluation, we check whether an AI agent's response meets specific criteria. For instance, we can assess whether the response contains specific information or avoids specific words.
Here are several examples:
Let's use our previous BMI calculator and transform it into an AI agent. Then, we will have a system instruction as below:
"You are a helpful assistant. Your task is to calculate Body Mass Index using this formula to one decimal place: BMI = weight(kg) / height(m)^2"Then, when the user uses this prompt:
"I have a height of 155 cm and a weight of 40 kg, calculate my BMI."
Our AI agent might respond with this:
"You have a BMI of 16.6"
or
“16.6”
In this case, we can evaluate our AI agent by expecting "16.6" to be in its response by using regex or a simple contains function. Furthermore, for more robust evaluation, we could extract the numerical result and compare it against the expected value.Let's say we build a polite AI agent and we don't want it to respond with bad words, such as fuck, what the hell, and many more. Therefore, when the user inputs this message:
“Fuck You”
We can evaluate it to ensure its response doesn't contain the word "fuck" by using regex or a simple not in statement.
Aside from the examples above, we can also use deterministic evaluation to verify whether an AI agent is working as expected. This includes assessing its harness or execution behavior, such as the number of tool calls it makes or which tools it uses to complete a specific task.Indeterministic
Deterministic evaluation is useful, but it is not enough to evaluate an AI agent since it can be difficult to assess certain aspects of its response using predefined rules. To address this limitation, we can use another LLM as a judge to evaluate our AI agent's responses based on specific criteria. There are several metrics we can use, such as faithfulness, correctness, relevance, and completeness.
For example, in our previous BMI agent, the agent might respond like this:
"Your body mass index is 25 (obesity type I). To improve your condition, I suggest you exercise regularly."
However, we don't want the agent to provide additional information aside from the BMI value. Therefore, we improve our system prompt as follows:
"You are a helpful assistant. Your task is to calculate Body Mass Index using this formula to one decimal place: BMI = weight(kg) / height(m)^2 Your response must only contain the BMI value without any additional information."After updating the system prompt, we can use an LLM as a judge to evaluate whether the agent follows this instruction and whether its response meets our evaluation criteria.