Apache Airflow is the right tool if you're running 100+ interdependent DAGs with complex retry logic and a platform team to babysit them. For the other 90% of data teams — the ones with 5-20 scheduled queries, a couple of CDC triggers, and a Slack alert when something breaks — Airflow adds weeks of setup time to a problem that doesn't need a distributed scheduler.
I've deployed Airflow twice. Once at a company that needed it (200+ DAGs, real dependency chains, a data platform team). Once at a 5-person startup that didn't (12 scheduled jobs, no dependencies between most of them, one engineer who also did everything else). The second deployment was a mistake, and it's a mistake I've watched other small teams make repeatedly since.
The Airflow tax (an honest assessment)
None of this is a knock on Airflow as software. It's genuinely well-built for what it does. But "well-built" and "right-sized for your team" are different questions, and the second one is the one that matters when you're deciding what to run.
Setup: 2-4 weeks for a production-ready deployment. Even with a managed option like Astronomer or AWS MWAA, you're not done after helm install. You need to configure connections, set up secrets management, wire up logging and monitoring, figure out your executor (Celery? Kubernetes?), size your workers, and write your first real DAGs — not the tutorial ones. Self-hosting adds Kubernetes or Docker Compose expertise on top of all that.
Learning curve: DAGs, operators, XCom, connections, pools. Airflow has its own vocabulary and its own mental model. A PythonOperator isn't just a Python function — it runs in a specific execution context with its own quirks around templating, Jinja, and task instance state. Passing data between tasks means XCom, which has size limits and its own serialization behavior. None of this is hard for someone who's done it before. All of it is non-trivial for a team without a dedicated platform engineer.
Maintenance: upgrade cycles, worker scaling, metadata DB management, log rotation. Airflow's metadata database grows forever unless you actively prune it. Major version upgrades (2.x had several breaking changes) require testing every DAG. Worker autoscaling needs tuning so you're not either paying for idle capacity or queuing jobs during peak load. This is ongoing work, not a one-time setup cost.
Cost: $300-500+/month minimum for managed options. AWS MWAA starts around $300/month for the smallest environment, and that's before you're running much through it. Astronomer's managed offering starts around $500/month. Self-hosting removes the subscription fee and replaces it with engineering time — which is rarely actually cheaper once you count it honestly.
When it's worth it: complex dependency graphs where task B genuinely can't start until tasks A1-A5 finish, 50+ daily runs across many pipelines, and a team with platform engineering capacity to own it. If that's your situation, use Airflow — it's the right tool and this article isn't for you.
Signs you don't need Airflow
You're probably over-buying orchestration tooling if most of these are true:
- You have fewer than 20 scheduled data jobs
- Your jobs are mostly independent — no complex DAG where one failure cascades through five downstream tasks
- Your team doesn't have a dedicated platform or data engineer
- Your triggers are simple: cron, a table update, a webhook, a threshold crossing
- You'd rather spend your time on analysis than on pipeline infrastructure
If you nodded along to three or more of those, keep reading — there are simpler tools that will get you running in an afternoon.
Five alternatives for simpler pipelines
1. dbt Cloud (~$100+/mo) — if your pipeline is purely SQL transforms
If everything you're doing is SQL-based transformation — raw tables in, modeled tables out — dbt Cloud gives you scheduling, CI/CD, and observability without touching Airflow. It's purpose-built for the transform layer and does it well.
The limitation is right there in the description: it's for dbt jobs. You can't run arbitrary Python, you can't do CDC-style triggers off an external event, and you can't call out to a third-party API mid-pipeline. If your pipeline is "SQL in, SQL out," this is a great, low-effort fit. If it's anything more than that, you'll hit the wall fast.
2. Prefect Cloud (~$500+/mo) — if you want Python DAGs without managing infrastructure
Prefect uses roughly the same mental model as Airflow — Python-defined flows and tasks — but as a managed service, so you're not running workers, a metadata DB, or a scheduler yourself. It removes the ops burden, not the authoring burden.
That's the tradeoff to understand clearly: you still write Python DAGs (Prefect calls them flows). You still need someone comfortable writing orchestration code. What you get back is not having to run the infrastructure underneath it. For a team that's Python-heavy and wants managed hosting but doesn't want to abandon the DAG-as-code model, this is a reasonable middle ground — at a price point closer to Airflow than to the DIY options below.
3. GitHub Actions / cron — if you need 3-5 simple schedules
For a genuinely small number of jobs, a scheduled GitHub Actions workflow or a cron entry on a box you already have is free (for public repos) or nearly free, and takes minutes to set up.
The honest limitation: no observability beyond "did the Action pass or fail," no retries without writing that logic yourself, and no event triggers — cron and Actions schedules are time-based only, not "when this table changes" based. This is fine for a nightly dbt run or a weekly report generation script. It stops being fine the moment you need to react to something happening in your data rather than the clock.
4. Event-driven triggers (Fastero, Kestra) — if your pipeline is "when X happens, do Y"
This is a different shape of tool entirely. No DAGs, no Python files to deploy, no CI/CD pipeline for your orchestration layer. You define triggers — a cron schedule, a Postgres NOTIFY, a BigQuery table update, a Kafka message, an inbound webhook — and attach actions to them: run a SQL query, execute a notebook, send a notification, call an API. You can chain these into multi-step workflows with conditions in between.
This is the category most 5-20-job teams are actually reaching for when they say they want "something like Airflow but simpler." It trades DAG-level dependency modeling for a much shorter path from "I have an idea for a pipeline" to "it's running in production."
5. Temporal — if you need durable execution for microservice workflows
Temporal is worth mentioning because people occasionally reach for it as an Airflow alternative and it's usually the wrong fit. It's built for durable execution of long-running business logic in microservice architectures — think "this multi-day approval workflow needs to survive service restarts" — with SDKs in Go, Java, TypeScript, and Python. It's excellent at that. It's overkill for data pipelines, where you're mostly moving and transforming data on a schedule or in response to an event, not coordinating distributed service state over days.
The event-driven approach, in more detail
Traditional orchestration thinks in terms of schedules: run job A, then B, then C, every day at 6am. Event-driven orchestration thinks in terms of triggers: when new rows land in the orders table, run the revenue calculation, check whether it crosses a threshold, and alert if it does.
This is a real mental model shift, not just different syntax for the same thing. Instead of scheduling everything to run whether or not there's anything new to process, you react to data actually changing. A few consequences fall out of that:
Advantages:
- Lower latency — the pipeline runs when the data changes, not at the next scheduled slot
- Simpler reasoning — a trigger and its actions are a self-contained unit; you're not tracing a DAG to understand what depends on what
- No wasted runs — nothing executes when nothing has changed
Disadvantages:
- Harder to see full pipeline state at a glance — there's no single visual DAG showing you every step across every pipeline
- Dependency chains that do exist (this trigger's output feeds that trigger's input) are less explicit than in a DAG-based tool, so you have to be more deliberate about naming and documentation to keep it legible
For teams whose pipelines are mostly independent — which is most teams below the 20-job threshold — the disadvantages rarely matter in practice, because there isn't much cross-pipeline dependency to lose visibility into.
How Fastero handles this
Fastero is one of the event-driven tools in the list above, purpose-built for the use case where most pipelines live: react to data changes, run queries, send alerts.
Trigger types: cron schedules, PostgreSQL LISTEN/NOTIFY, BigQuery table updates, Kafka messages, Snowflake streams, and inbound webhooks.
Actions: execute SQL queries, run Python notebooks, send Slack or email notifications.
Multi-step workflows: chain actions together with SQL-based conditions and wait steps — "run this query, if the result crosses a threshold then run this notebook, then notify Slack."
Best for: event-driven pipelines where you're reacting to data changes rather than modeling deep dependency graphs. If your pipelines are under 20 jobs and event-driven (which covers most teams), this replaces the orchestrator entirely.
More on how the trigger and action model works is on the triggers product page.
Decision matrix
| Your situation | Use this |
|---|---|
| 100+ daily DAGs, dedicated platform team | Airflow / Dagster |
| SQL-only transforms, dbt-based | dbt Cloud |
| Python-heavy, want managed hosting | Prefect |
| 5-20 jobs, event-driven, no platform engineer | Fastero / Kestra |
| 3 cron jobs, budget = $0 | GitHub Actions |
If you want a broader shortlist beyond these five, our roundup of data orchestration tools and workflow automation tools covers more ground, and if your jobs lean more toward general app automation than data pipelines specifically, n8n is worth a look too. If what you actually need is alerting on top of whatever pipeline you land on, see how to set up SQL alerts without Datadog.
Closing
The data industry's default recommendation is always "use Airflow." For the top 10% of data teams — the ones running real dependency graphs at real scale with a platform team to support it — that's genuinely good advice. For everyone else, there are simpler tools that get the job done in an afternoon instead of a month. Most teams over-buy orchestration tooling because Airflow is the name everyone knows, not because it's the tool their actual pipeline count and complexity call for. Match the tool to the 12 jobs you actually have, not the 200 you might have someday.
Try Fastero free — automate your data workflows with triggers, scheduling, and monitoring — connect your sources and start building in minutes. No credit card required.