There's a specific flavor of dread you feel when a number looks too good. We felt it on a Tuesday, staring at an activation rate hovering near 100%.
For a moment it was exhilarating. Almost every user who signed up was "activating." Then the dread arrived, because anyone who's run a product knows: your activation rate is not 100%. It's never 100%. If a metric says it is, the metric is lying to you — and you need to find out how before you make a decision on it.
We did find out. It's a small, embarrassing, deeply instructive story, and it's the reason we now think about metric definitions the way we do. Here's exactly what happened.
The setup: what "activation" is supposed to mean
Fastero fires events as users move through the product. Some are lightweight signals; some are meaningful. One of them is signup_created.
The name is the trap. signup_created sounds like it belongs in an activation metric — a user showed up, an account exists, surely that's the top of the funnel. And in our event system, it fires for every signup. Every single one. Including every click from a paid ad that creates an account and then does absolutely nothing.
Here's the thing we hadn't been careful about: signup_created is a learning signal, not an activation signal. Real product activation is a strong event — someone built a dashboard from a file, activated a connection, created a query or report, deployed an app. Those are the moments where a user actually got value. A signup is just a door opening.
How the definition went wrong
Someone (it doesn't matter who — the point is the system let it happen) built an "activation rate" on top of the wrong event. Activation numerator = users with an activation event. But the event set quietly included signup_created.
Since signup_created fires for essentially everyone who signs up, and the denominator was... users who signed up... the metric was mathematically doomed to approach 100%. It wasn't measuring activation. It was measuring "did a signup create a signup event," which is a tautology dressed up as a KPI.
The dashboard was correct. The query ran fine. The chart was beautiful. Every layer of the stack did its job. The definition underneath was wrong, and nothing in the tooling was going to catch that, because tools don't know what your metrics are supposed to mean. Only the definition does.
The second trap, for good measure
While we were untangling this, we hit the other classic. When you query the activation events, the field that links an event to a user is user — an ObjectId. Not userId. If you write the intuitive query and filter on userId, you don't get an error. You get an empty result. Silently. Zero rows, no warning, a perfectly clean "no data."
So the failure modes were stacked: use the wrong event and activation looks like 100%; use the wrong field name and it looks like 0%. Two different wrong definitions, two opposite lies, both fully plausible, neither one throwing so much as a warning.
That's the whole problem with metrics in one anecdote. The dangerous errors don't crash. They compute.
What we actually changed
We fixed the number in an afternoon. Fixing the class of problem took longer and mattered more. Three changes:
1. We wrote the definition down, explicitly. Activation is now a named, documented metric with an exact event set — only the strong events count (dashboard_created_from_file, connection_activated, query_or_report_created, and the rest) — and an explicit note that signup_created is a lead signal, not activation. The definition lives where both a human and an AI agent can read it before touching the number. If we'd had that written contract earlier, the bad dashboard would have been caught the moment it disagreed with it.
2. We made the edge cases part of the definition. The user-not-userId gotcha is now written into the spec, next to the metric, because that's exactly the kind of tribal knowledge that produces a silent zero six months later when a new person writes the "obvious" query.
3. We started treating drift as a monitoring problem, not a hope. The reason this bug survived as long as it did is that nothing was watching. A definition that depends on an event set and a field name should tell you when either changes. That belief is now baked into how we build — event-driven triggers and schema-drift detection exist so that "the number quietly went wrong" becomes "we got an alert," instead of "we noticed in a leadership review."
The lesson we can't un-learn
Every analytics failure we've had traces back to the same root, and it's almost never the part people worry about. It's not the SQL. It's not the chart library. It's not even, usually, the raw data. It's the meaning — the definition of what the number is supposed to represent.
This is why we're a little allergic to the "just point an AI at your data and ask it anything" pitch. The AI will happily compute your fake 100% activation rate too, and phrase it beautifully. Generating SQL is the easy, solved part. Knowing that "activation" excludes signup_created — that's the whole ballgame, and no model infers it from a schema. Someone has to define it, and something has to notice when the definition rots.
We wrote up the practical version of what we learned as a step-by-step guide: How to Define Your Metrics Once (and End the "Whose Number Is Right?" Fight). And if you're shopping for where to put those definitions, we compared the options honestly in the best semantic layer tools of 2026.
How this shaped the product
Full disclosure, since this is our blog: the experience is a big part of why Fastero looks the way it does. We built it so you can define your key metrics once on a connected warehouse, let your team ask questions in plain English against those definitions through the NL→SQL engine, and get alerted when the data underneath a definition drifts — the "keeps itself honest" behavior we wish we'd had that Tuesday. It is deliberately not a Looker-grade governance suite; it's the lightweight version for teams who'd rather catch the fake-100% number early than build a compliance program. More on the metrics & semantic layer capability page.
The near-miss was a good teacher. A metric that looks perfect is not a trophy. It's a question: what exactly did we just measure?
Want a metric layer that tells you when a number starts lying to you?
👉 Start your free 30-day trial (no credit card required)
A true story, lightly anonymized. Last updated: July 2026.