AI in production

From AI pilot to production with Python: where it goes wrong

With today's models, a convincing demo is quick to build. The gap between a demo and an application hundreds of people rely on every day is wider than most organisations expect.

AI in production2 min readPython People editorial team

The pattern

In 2024 Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept, citing poor data quality, inadequate risk controls, escalating costs or unclear business value. Source: Gartner, 29 July 2024 (opens in a new tab)

Notice that none of these causes is the model itself. They are all about everything around it. That's good news, because you can do something about them.

Five things to get right from day one

1. A measurable goal

Pick one process and agree upfront what success looks like: minutes per case, percentage of questions handled correctly, cost per transaction. Without that yardstick, every discussion about value stays a matter of opinion.

2. An evaluation set

Collect real questions or cases, with the correct answer. Every change to prompt, model or data is tested against it. This is the biggest difference between teams that scale and teams that stall.

3. Data in order

Know which sources you use, how current they are and who may see what. Start small, with sources you trust.

4. Risk in the design

What happens when the model gets it wrong? Build in checks where errors hurt, and let a human decide on actions that can't be undone.

5. Cost per outcome

Don't just track the provider's bill; track the cost per resolved question or task. Then you'll see straight away whether a smaller model, caching or a smarter approach pays off.

Is your pilot at this point?

We're happy to think through what it takes to get to production, and who you need for it.

What that looks like in Python

Most of the gap between pilot and production is ordinary software engineering. One example: never trust model output blindly. Validate it with Pydantic, like any other external input.

from pydantic import BaseModel, ValidationError

class Classification(BaseModel):
    category: str
    confidence: float
    source_id: str | None = None

def process(raw_output: str) -> Classification | None:
    try:
        return Classification.model_validate_json(raw_output)
    except ValidationError:
        # invalid output goes to a person, not to the customer
        send_to_review(raw_output)
        return None

Around it sits a familiar Python stack: FastAPI for the API, pytest for tests and evaluations in the CI pipeline, OpenTelemetry or Langfuse to trace every call with cost and latency, and uv to pin dependencies reproducibly.