Expertise · Data engineering

Data that is ready for AI

Poor data quality is one of the most frequently cited reasons AI projects stall. Our data engineers lay the foundation: reliable pipelines, clear agreements and data you can trust.

What AI demands from your data

  • Unstructured data. PDFs, emails and scans become usable, searchable data, with structure and metadata preserved.
  • Data contracts. Clear agreements with upstream teams, so a change at the source doesn't silently break your AI application.
  • Quality and lineage. Tests at every step and visibility into where data comes from. That helps with accountability too.
  • Privacy. Personal data is detected and masked before it ends up in an index or training set.

Gartner lists poor data quality as one of the reasons generative AI projects are abandoned after proof of concept. Source: Gartner, 29 July 2024 (opens in a new tab)

What that looks like in code
pipeline/orders.pyPython 3.12
from collections.abc import AsyncIterator
from decimal import Decimal
from typing import Literal
from pydantic import BaseModel, ValidationError

class Order(BaseModel):
    order_id: str
    total: Decimal
    currency: Literal["EUR"]

async def load_orders(rows: AsyncIterator[dict]) -> RunStats:
    stats = RunStats()
    async for row in rows:
        try:
            order = Order.model_validate(row)
        except ValidationError as err:
            # set deviations aside, never drop them silently
            await quarantine.put(row, reason=str(err))
            stats.rejected += 1
            continue
        # idempotent: re-running after a failure is safe
        await warehouse.upsert("orders", order, key="order_id")
        stats.loaded += 1
    return stats

Request profiles

Profiles of suitable data engineers within 48 hours

  • No obligation: you only pay for hours worked
  • Dutch-speaking, from our own network and assessed by us
  • On-site, hybrid or remote

Indicative € 106–132 per hour, all-in, excl. VAT

Rather call? +31 35 7111 373

What we select for

  1. 01

    Strong in SQL and modelling

    Python is the tool; a good data model is what makes the difference.

  2. 02

    Thinks about consumers

    Builds data that models, analysts and applications can actually work with.

  3. 03

    Reliable

    Makes sure a failed run gets noticed, can be recovered and leaves no silent errors.

Technology we often see

Tools change fast. We select on principles, but this is what the developers in our network use in practice.

AirflowDagsterdbtPolarsSparkSnowflakeDatabricksBigQuery

Profiles we are looking for

Need a Python developer for this?

Send us your vacancy or request. Within 48 hours you hear which developers in our network fit and what it costs.