Workshops: Agentic Coding by Tech Stack
Data Engineering with AI Agents: dbt, Airflow & Spark Workshop
Data pipelines are code, and coding agents write dbt models, Airflow DAGs and PySpark jobs as quickly as application code. But a pipeline error rarely shows up as a failing build; it appears weeks later as a wrong number in a dashboard. This advanced workshop teaches data engineering teams to use agents with tests, lineage and cost guardrails that catch problems before they reach data consumers.
Who is this workshop for?
- Data and analytics engineers working with dbt
- Platform teams running Airflow, Dagster or similar orchestrators
- Engineers building Spark and PySpark jobs on Databricks or comparable platforms
- Data architects responsible for modelling standards and data contracts
- Teams on Snowflake, BigQuery, Databricks or PostgreSQL
What you will take away
- Structure dbt projects so agents follow your layering, naming and materialisation conventions.
- Have agents add generic and custom data tests alongside every model change.
- Use lineage and state-based selection so the agent only builds and tests what a change affects.
- Write and refactor Airflow DAGs and PySpark jobs with the agent, including idempotency and retries.
- Limit warehouse cost and data access for agents working against real platforms.
Agenda
Module 1
Agents in the data stack
- Claude Code, OpenAI Codex, Cursor and GitHub Copilot on a dbt and Python repository
- Context: sources, model layers, macros and project conventions
- Access: development schemas, samples, no production writes
Module 2
dbt with agents
- Exercise: new staging and mart models from a source definition
- Incremental models and snapshots: where agents get it wrong
- Documentation and column descriptions generated from code
Module 3
Data quality
- Generic tests, unit tests and custom checks with the agent
- Data contracts and schema change detection
- Test failures and query results as agent feedback
Module 4
Orchestration and processing
- Airflow DAGs: idempotent tasks, retries, backfills
- PySpark transformations and performance traps such as skew and shuffles
- Local testing with sample data
Module 5
CI and cost guardrails
- Slim CI with state comparison and deferral
- Query budgets, timeouts and warehouse sizes for agents
- Review checklist for pipeline pull requests
Why pipelines need a different safety net
In application code, a mistake usually breaks a test or a build. In data engineering, a join that duplicates rows or a filter in the wrong layer produces plausible but wrong figures. Agents do not make these mistakes more often than people, but they ship changes faster, so the safety net has to scale with them.
We therefore combine three kinds of feedback: data tests on every model, comparison of results before and after a change, and lineage to see which downstream reports a change affects.
Data protection and platform access
Agents belong in development schemas with sampled or masked data. Personal data from production tables has no place in prompts or logs, both under GDPR and under your internal data classification. We show how service accounts, warehouse roles and query limits keep agents within clear boundaries.
- Separate development targets per person and agent
- Masked or synthetic samples for exploration
- Cost limits and timeouts on agent queries
- Production runs only through the orchestrator
How this differs from our analytics workshop
Our data science and analytics workshop covers exploration, notebooks and analysis. This one is about production pipelines: modelling standards, testing, orchestration and operations. We work with your repository or a reference project, remotely or on-site, in English or German, with neooutput AI Code Engineers.
Frequently asked questions
Does this work with dbt Core and the managed dbt platform?
Yes. Agents work on the project files either way. We adapt commands and CI integration to your setup; dbt Labs also offers an MCP server, which we include where it fits.
May the agent query our warehouse?
With its own role, development data and cost limits, yes, and it becomes far more useful. Without those guardrails we recommend working only with local samples and compiled SQL.
Which orchestrators and platforms do you cover?
Airflow is our default example; Dagster and Prefect work similarly. For processing we cover Spark and PySpark, with warehouses such as Snowflake, BigQuery, Databricks or PostgreSQL. We follow your stack.
Workshop
Get in touch
Want agents to build pipelines whose numbers your business can trust?
Tell us roughly how many people will join and pick dates that work – we will get back to you within 24 hours with a proposal for content, format and schedule.
Prefer to talk first?
15 minutes, free and without obligation – with our AI Trainers and AI Code Engineers.
Book a call for free (opens in new tab) +49 761 519 751 50 (24/7)Request a training / workshop
Workshop: Data Engineering with AI Agents: dbt, Airflow & Spark Workshop · 1 day
More workshops
- Workshops: Agentic Coding by Tech StackLLM App Engineering with Coding Agents: 2-Day Training
- Workshops: Agentic Coding by Tech StackEmbedded C/C++ & Firmware with AI Agents: Workshop
- Workshops: Agentic Coding by Tech StackCOBOL & Mainframe Modernisation with AI Agents: Workshop
- Workshops: Agentic Coding by Tech StackWordPress Plugin & Theme Development with AI Agents: Workshop
- Workshops: Agentic Coding by Tech StackShopify Themes & Apps with AI Coding Agents: Workshop
- Workshops: AI Automation & AgentsWorkshop: How Do You Measure the ROI of AI?
