Workshops: Agentic Coding by Tech Stack

Data Engineering with AI Agents: dbt, Airflow & Spark Workshop

Data pipelines are code, and coding agents write dbt models, Airflow DAGs and PySpark jobs as quickly as application code. But a pipeline error rarely shows up as a failing build; it appears weeks later as a wrong number in a dashboard. This advanced workshop teaches data engineering teams to use agents with tests, lineage and cost guardrails that catch problems before they reach data consumers.

Who is this workshop for?

  • Data and analytics engineers working with dbt
  • Platform teams running Airflow, Dagster or similar orchestrators
  • Engineers building Spark and PySpark jobs on Databricks or comparable platforms
  • Data architects responsible for modelling standards and data contracts
  • Teams on Snowflake, BigQuery, Databricks or PostgreSQL

What you will take away

  • Structure dbt projects so agents follow your layering, naming and materialisation conventions.
  • Have agents add generic and custom data tests alongside every model change.
  • Use lineage and state-based selection so the agent only builds and tests what a change affects.
  • Write and refactor Airflow DAGs and PySpark jobs with the agent, including idempotency and retries.
  • Limit warehouse cost and data access for agents working against real platforms.

Agenda

  1. Module 1

    Agents in the data stack

    • Claude Code, OpenAI Codex, Cursor and GitHub Copilot on a dbt and Python repository
    • Context: sources, model layers, macros and project conventions
    • Access: development schemas, samples, no production writes
  2. Module 2

    dbt with agents

    • Exercise: new staging and mart models from a source definition
    • Incremental models and snapshots: where agents get it wrong
    • Documentation and column descriptions generated from code
  3. Module 3

    Data quality

    • Generic tests, unit tests and custom checks with the agent
    • Data contracts and schema change detection
    • Test failures and query results as agent feedback
  4. Module 4

    Orchestration and processing

    • Airflow DAGs: idempotent tasks, retries, backfills
    • PySpark transformations and performance traps such as skew and shuffles
    • Local testing with sample data
  5. Module 5

    CI and cost guardrails

    • Slim CI with state comparison and deferral
    • Query budgets, timeouts and warehouse sizes for agents
    • Review checklist for pipeline pull requests

Why pipelines need a different safety net

In application code, a mistake usually breaks a test or a build. In data engineering, a join that duplicates rows or a filter in the wrong layer produces plausible but wrong figures. Agents do not make these mistakes more often than people, but they ship changes faster, so the safety net has to scale with them.

We therefore combine three kinds of feedback: data tests on every model, comparison of results before and after a change, and lineage to see which downstream reports a change affects.

Data protection and platform access

Agents belong in development schemas with sampled or masked data. Personal data from production tables has no place in prompts or logs, both under GDPR and under your internal data classification. We show how service accounts, warehouse roles and query limits keep agents within clear boundaries.

  • Separate development targets per person and agent
  • Masked or synthetic samples for exploration
  • Cost limits and timeouts on agent queries
  • Production runs only through the orchestrator

How this differs from our analytics workshop

Our data science and analytics workshop covers exploration, notebooks and analysis. This one is about production pipelines: modelling standards, testing, orchestration and operations. We work with your repository or a reference project, remotely or on-site, in English or German, with neooutput AI Code Engineers.

Frequently asked questions

Does this work with dbt Core and the managed dbt platform?

Yes. Agents work on the project files either way. We adapt commands and CI integration to your setup; dbt Labs also offers an MCP server, which we include where it fits.

May the agent query our warehouse?

With its own role, development data and cost limits, yes, and it becomes far more useful. Without those guardrails we recommend working only with local samples and compiled SQL.

Which orchestrators and platforms do you cover?

Airflow is our default example; Dagster and Prefect work similarly. For processing we cover Spark and PySpark, with warehouses such as Snowflake, BigQuery, Databricks or PostgreSQL. We follow your stack.

Workshop

Get in touch

Want agents to build pipelines whose numbers your business can trust?

Tell us roughly how many people will join and pick dates that work – we will get back to you within 24 hours with a proposal for content, format and schedule.

Prefer to talk first?

15 minutes, free and without obligation – with our AI Trainers and AI Code Engineers.

Book a call for free (opens in new tab) +49 761 519 751 50 (24/7)

Request a training / workshop

Workshop: Data Engineering with AI Agents: dbt, Airflow & Spark Workshop · 1 day

Dates that work for you (optional)

Suggestions about 2 weeks ahead – pick as many as you like

Reply usually within 24 hours · jocapps is ISO 27001 certified · Privacy