AI in production

How to evaluate an LLM application in Python

A language model gives fluent answers, even when they're wrong. If you don't measure, you don't know whether a change improves things or quietly breaks something else.

AI in production3 min readPython People editorial team

Why trying it out isn't enough

An LLM application's behaviour changes with every change to prompt, model or data. A tweak that improves ten examples can make twenty others worse. Manual testing won't show you that.

An evaluation set is the yardstick of your project. It matters more than your choice of model, because without it you can't even compare models fairly.

Four steps to an evaluation that works

  1. Collect real cases

    Start with questions and tasks from practice: from logs, from customer service or from users testing with you. Add difficult edge cases. A smaller set of well-chosen real cases tells you more than a large set of generated ones.

  2. Define what good looks like

    For each case, record the correct answer, the source that should be used or the rules the answer must follow. Have domain experts review this, not just engineers.

  3. Choose the right metrics

    Use exact checks wherever possible: is the amount right, is the output valid, is the right source cited. For open answers, a second model can grade against a clear rubric, spot-checked by people.

  4. Automate

    Run the set on every change in your CI pipeline and compare with the previous version. If quality drops, the change doesn't ship.

What to measure, by type of application

ApplicationQualitySafetyCost and speed
Knowledge assistant (RAG)Correct answer, correct source, an honest "I don't know" when the answer is missingNo data beyond the user's permissionsCost per question, response time
Document extractionFields correct and complete, valid outputNo sensitive fields beyond the purposeCost per document, lead time
AgentTask completed correctly, number of stepsNo unauthorised actions, resistant to prompt injectionCost per task, lead time

After go-live

Evaluation doesn't stop at release. Collect user feedback, regularly review a sample of real conversations and add every new failure to your evaluation set. Your yardstick gets better over time, and so does your application.

In Python: evaluations as ordinary tests

The easiest way to keep evaluation going is to treat it as a test suite. With pytest, every test question runs as a separate case, and every change shows exactly which cases break.

import json
import pytest

with open("evals/customer_questions.jsonl") as f:
    CASES = [json.loads(line) for line in f]

@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_answer_cites_right_source(case):
    answer = assistant.answer(case["question"])
    assert case["source_id"] in answer.sources

Put these tests in your CI pipeline next to your unit tests. For judgements that are hard to express as an assert, such as whether an answer is complete, Ragas and DeepEval offer ready-made metrics. Langfuse helps compare results over time.

Need a Python developer?

Call us or send your vacancy. Within 48 hours you hear which developers in our network fit and what it costs.