I ran SQL-based data quality checks for about a year before I ever looked at a data observability platform. Cron jobs, a handful of queries, a Slack webhook. It caught most of what actually went wrong — a broken ETL job that silently stopped loading rows, a source system that started sending null emails, a table that hadn't updated in two days because a credential expired. Then I finally sat through a Monte Carlo demo, and the honest reaction wasn't "we need this" — it was "this automates the parts of what we're doing that were getting tedious to maintain by hand."
That's the real distinction, and it's worth being precise about before you evaluate a $30-50k/year platform. Monte Carlo, Bigeye, and Anomalo aren't solving a problem SQL can't solve. They're solving the problem of doing it at scale, automatically, with lineage, without a human writing and maintaining every threshold. If you don't have that scale problem yet, you're paying for automation you don't need.
This post is the DIY path: what data observability actually catches, why the enterprise tools cost what they do, and the SQL checks you can have running by end of day.
What data observability actually catches
Strip away the marketing and "data observability" is five categories of problem, all of which are really just "did something change that shouldn't have":
Schema changes. A column got renamed, dropped, or changed type upstream, and every downstream query that references it either breaks loudly or — worse — silently returns wrong results because a type coercion papered over the change.
Row count anomalies. A load job that should insert ~50,000 rows a day inserts 200, or 0, or 5 million. Almost never a business event. Almost always a pipeline that partially failed and didn't error.
Freshness / staleness. A table that's supposed to update hourly hasn't updated in 14 hours. This is the single most common data incident, and it's the easiest to catch, which is why it's the first check most teams reach for.
Distribution drift. The shape of the data changed even though the row count and schema look fine — average order value dropped 40% overnight, a categorical column that used to have 12 values now has 400, a numeric field's range shifted. This is the hardest category to catch without a baseline to compare against.
Null spikes. A required field — email, customer_id, order_total — starts coming back null at a rate it never used to. Usually means an upstream form validation broke, an API contract changed, or a join started failing silently.
Every enterprise data observability platform is built around detecting these five things automatically, across every table in your warehouse, without you writing a check for each one.
Why the enterprise tools cost $30-50k/year
It's worth understanding what you're actually paying for, because it clarifies what you're giving up by going DIY.
ML-based anomaly detection. Instead of you setting "alert if row count drops below 40,000," the platform learns your table's normal row count distribution over time — including day-of-week seasonality, monthly patterns, and slow trend growth — and flags outliers against that learned baseline. This is the single biggest value-add over a manual threshold, and it's genuinely hard to replicate yourself without investing real time in it.
Automated lineage. The platform crawls your warehouse metadata, dbt manifests, and BI tool connections to build a map of which tables feed which dashboards and which downstream tables depend on which upstream ones. When something breaks, you get "this affects 14 downstream tables and 3 exec dashboards" instead of having to trace it by hand.
Incident management. Deduplication, severity scoring, ownership assignment, an audit trail of past incidents, integration with PagerDuty/Jira/Slack — the operational layer around "something's wrong" that turns a raw alert into a workflow a team can actually run.
Coverage at scale, with no manual threshold-setting. Point it at your warehouse and it starts monitoring hundreds of tables without you writing a check for each one. This is where the ROI math actually works — if you have 300 tables, hand-writing and maintaining SQL checks for all of them is a worse use of an engineer's time than paying for the platform.
That last point is the crux of the buy-vs-build decision, and it's genuinely about table count and team size, not about whether the underlying problem is important.
The DIY approach: SQL checks you can write today
If you're running fewer than 50 tables that actually matter and you don't have a dedicated data platform team, you can cover the same five categories above with SQL you write once and schedule. None of this requires anything beyond a database connection and somewhere to run a query on a timer.
1. Freshness check
The highest-value, lowest-effort check. Catches the most common failure mode: a pipeline stopped running and nobody noticed.
SELECT
EXTRACT(EPOCH FROM (NOW() - MAX(updated_at))) / 3600 AS hours_stale
FROM orders;Alert if hours_stale exceeds whatever your expected update cadence is, with headroom — if orders updates every 15 minutes, alert at 2 hours, not 16 minutes, or you'll page yourself over routine latency.
2. Row count check
Catches partial failures and silently-empty loads. Run this against your staging or landing table right after a load job finishes.
SELECT COUNT(*) AS rows_last_hour
FROM events
WHERE created_at > NOW() - INTERVAL '1 hour';Alert if rows_last_hour falls below a floor you set from historical volume — for a table that typically ingests 8,000-12,000 rows/hour, alert under 2,000. Set the floor loose at first; tighten it once you've watched it run for a couple of weeks and know your real variance.
3. Null rate check
Catches upstream contract breaks — a form validation removed, an API field going optional, a join silently dropping matches.
SELECT
COUNT(*) FILTER (WHERE email IS NULL) * 100.0 / COUNT(*) AS null_rate_pct
FROM users
WHERE created_at > NOW() - INTERVAL '1 day';Alert if null_rate_pct exceeds a small tolerance — for a required field like email, anything above 1-2% is almost always a bug, not real-world messiness.
4. Distribution drift check
The hardest one to get right by hand, but a simple version — comparing today against a trailing baseline — catches the majority of real incidents without any ML.
WITH today AS (
SELECT AVG(order_total) AS avg_today
FROM orders
WHERE created_at > NOW() - INTERVAL '1 day'
),
baseline AS (
SELECT AVG(order_total) AS avg_7day
FROM orders
WHERE created_at BETWEEN NOW() - INTERVAL '8 days' AND NOW() - INTERVAL '1 day'
)
SELECT
today.avg_today,
baseline.avg_7day,
(today.avg_today - baseline.avg_7day) / NULLIF(baseline.avg_7day, 0) * 100.0 AS pct_change
FROM today, baseline;Alert if pct_change moves beyond, say, ±25%. That threshold is a starting guess, not a law — widen it for genuinely volatile metrics, tighten it for stable ones. This is the tradeoff you're accepting versus a learned ML baseline: you set the number, and you're wrong about it until you've watched a few false positives and false negatives go by.
5. Schema check
Catches column drops, renames, and type changes before they break a downstream query.
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = 'orders'
ORDER BY column_name;Snapshot the result on a schedule, diff it against the previous snapshot, and alert on any added, removed, or type-changed column. This is crude compared to automated lineage-aware schema tracking, but it catches the failure mode that actually matters: something changed under you and you found out from a broken dashboard instead of from a check.
Scheduling these checks
Writing the SQL is the easy 20%. The other 80% is making sure these run reliably, on a schedule, with alerting wired up — and staying wired up when you're not the one watching.
Cron + a script. The simplest path: a scheduled script that runs each query, compares the result to a threshold, and posts to a Slack webhook on failure. Works fine for a handful of checks. Gets tedious to maintain once you're past 10-15 of them — you're now maintaining a small fleet of near-identical scripts, each with its own credential handling and error logic.
dbt tests. If your checks are already expressible as dbt tests (not_null, accepted_values, custom singular tests), this folds data quality into your existing transform pipeline and CI. The limitation: dbt tests run when your dbt jobs run, not on an independent schedule, and they're scoped to what dbt already models — they're not the tool for freshness checks against raw source tables outside your dbt project.
Fastero triggers. This is the case this kind of check is a natural fit for: set up each SQL query above as a Fastero trigger on a cron schedule (hourly, every 15 minutes, nightly — whatever matches the table), define the threshold as a condition, and route failures to Slack. No script to deploy, no cron box to maintain, no credential management beyond the initial database connection. It's the same DIY logic above, just running somewhere that isn't a laptop or a fragile box someone will forget exists.
Whichever way you schedule them, the discipline that matters more than the tool is: write down why each threshold is set where it is, and revisit it after the first month once you've seen real variance. A check nobody remembers the reasoning behind is a check nobody trusts enough to act on when it fires.
The honest limitations of DIY
This approach genuinely gets you 80% of the value for a fraction of the cost, but it's worth being straight about the 20% you're giving up, because it will eventually matter:
- No automatic anomaly detection. Every threshold is one you set by hand, based on your own read of "normal." You will get this wrong in both directions — alerts that fire on routine variance, and real incidents that stay under a threshold set too loose.
- No lineage. When a check fires, you're tracing "what does this table feed into" yourself, from memory or documentation, not from an automatically generated dependency graph.
- No incident management layer. No deduplication, no severity scoring, no audit trail of past incidents. A Slack channel filling up with alerts is your incident history.
- Coverage is manual and it doesn't scale linearly. Every new table you care about is another check to write, tune, and maintain. This is fine at 20 tables. It's a real time sink at 150.
- Thresholds drift and nobody notices. A threshold set correctly for last year's volume can be silently wrong for this year's, and there's no system nudging you to revisit it — that's on you, on a calendar, if you remember.
None of these are fatal for a small team. All of them are exactly what you're paying $30-50k/year to have handled automatically once they are.
When to upgrade to a real platform
The DIY approach has a ceiling, and it's worth naming rather than pretending it doesn't exist:
- 100+ tables you actually need monitored. Past this point, writing and maintaining individual checks stops being a reasonable use of engineering time.
- You need ML-based anomaly detection, because your data has enough seasonality and volume that static thresholds produce too many false positives or miss too much.
- Your team is past 15 people, with data consumers spread across enough teams that "someone remembers why this threshold is set" stops being a safe assumption.
- You have SOX, HIPAA, or other compliance requirements that demand an audit trail, formal incident management, and documented data lineage — not a Slack channel history.
If none of those describe you yet, the SQL checks above will catch the incidents that actually happen to a team your size. Buy the platform when the problem you're solving changes shape — from "did this specific table break" to "monitor everything, automatically, at a scale no person can hand-tune" — not before.
If you're already looking at Fastero for the scheduling and alerting layer, how to set up automated SQL alerts without Datadog covers the alerting side in more depth, and how to monitor schema drift in BigQuery goes deeper on the schema-check pattern specifically.
Try Fastero free — connect your data sources, ask questions in plain English, and automate the reports nobody has time to build. No credit card required.