Team Rebase
,
Every AIOps vendor claims "AI-powered root cause analysis." The pitch is seductive: feed alerts into an ML model, get root causes out. No more 4-hour war rooms. No more manual correlation across tools.
Here's the problem: most of these tools don't find root causes. They find correlations.
That distinction matters. Correlation tells you what happened together. Causation tells you what caused what. When you're deciding whether to roll back a deployment or page the database team at 3am, you need causation. Correlation gives you a list of suspects. Causation gives you the culprit.
The gap between these two is where most incident investigations go wrong, and why "AI-powered RCA" often makes things worse, not better.
Correlation Is Easy. Causation Is Hard.
Finding correlated events is computationally straightforward. You look for:
Events that happened around the same time
Alerts from the same service or topology
Patterns that appeared together historically
Any decent ML model can cluster alerts by time window and service topology. That's table stakes. Most AIOps and event correlation platforms do this reasonably well.
But correlation ≠ causation. Classic example:
Observation: Ice cream sales and drowning deaths are highly correlated. Conclusion from correlation: Ice cream causes drowning. Actual cause: Summer. Hot weather increases both ice cream consumption and swimming.
In incident response, this manifests constantly:
Observation: Payment latency spiked at 14:23. A deployment happened at 14:15. CPU usage increased at 14:20. Correlation-based RCA: "Root cause: deployment at 14:15 (high confidence based on temporal proximity)" Actual cause: A batch job that runs every Tuesday at 14:20 hit a new code path that exhausted the connection pool. The deployment was unrelated.
Correlation-based tools would flag the deployment because it's temporally close and deployments are a common cause of incidents. An engineer following this signal would waste time investigating the wrong change while the batch job continues to cause problems.
The Three Ways Correlation Fails
1. Temporal Proximity ≠ Causation
Most RCA tools weight events by how close they are in time to the incident. Reasonable heuristic, wrong conclusion.
Consider a typical Tuesday afternoon:
14:00 - Marketing campaign goes live (traffic +40%)
14:10 - Auto-scaler adds 3 nodes
14:15 - Deploy of auth-service (unrelated feature flag change)
14:20 - Batch job starts processing backlog
14:23 - Payment API latency spikes
14:25 - Alert fires, on-call paged
A time-based correlation engine sees four events before the incident. It might weight the 14:15 deployment highest because "deployments often cause problems" (true historically) and it's close in time.
But the actual causal chain was:
Batch job (14:20) → new code path → connection pool exhaustion → payment latency
The deployment is a red herring. The marketing campaign contributed (more traffic = more pressure) but wasn't the root cause. The batch job was the trigger.
Time correlation can't distinguish between:
Events that caused the incident
Events that coincided with the incident
Events that are symptoms of the same underlying cause
2. Symptom Clustering ≠ Root Cause
When something breaks, you often get a cascade of alerts:
Database: connection pool exhausted
API: latency SLA breached
Frontend: error rate spike
Kubernetes: pod health checks failing
PagerDuty: 4 alerts in 2 minutes
Correlation-based tools excel at clustering these: "These 4 alerts are part of the same incident." Helpful for noise reduction. But which one is the root cause?
Most tools pick the "first" alert or the "most severe" alert. Neither is necessarily the root cause:
The first alert might be a symptom that manifests fastest
The most severe alert is based on your alerting thresholds, not system causality
The database connection pool alert fired first, but why is the pool exhausted? That requires understanding that a code change introduced a new query pattern that holds connections 10x longer than before. The database alert is a symptom. The code change is the cause.
Correlation clusters symptoms. It doesn't trace causation.
3. Historical Patterns ≠ Current Causation
"This alert pattern has been associated with deployment issues 73% of the time."
Great. What about the other 27%? And does the historical pattern account for:
Architecture changes since those incidents?
New services that share infrastructure?
Team changes that affect code quality?
Configuration drift in the underlying systems?
Historical correlation assumes the future resembles the past. In complex systems, that assumption breaks constantly. The deployment that caused an incident 6 months ago touched different code than today's deployment. The "similar alert pattern" might have a completely different root cause.
This is why experienced SREs are skeptical of "AI confidence scores" in RCA tools. A model trained on historical incidents can miss novel failure modes entirely, and novel failures are precisely the ones that cause the worst incidents.
What Causal Reasoning Actually Requires
Causation requires more than temporal and topological correlation. It requires understanding mechanisms: how changes propagate through a system.
Correlation asks: "What happened around the same time?" Causation asks: "What could have produced this effect, given how the system works?"
To answer causal questions, you need:
1. A Model of System Behavior
Not just "service A calls service B" but:
Which code paths in A call B, and under what conditions?
What resources do A and B share (databases, connection pools, message queues)?
How do failures in B manifest in A (timeouts, errors, degraded responses)?
What are the capacity limits and current utilization?
This is why observability alone isn't enough. Traces show you what happened. They don't model what could happen or what would happen if a specific change occurred.
2. Temporal Ordering That Respects Causality
Causes must precede effects. Obvious, but most correlation engines don't enforce this rigorously.
If the latency spike started at 14:23:00 and the deployment completed at 14:23:30, the deployment cannot be the root cause of the initial spike. It might have contributed to making things worse, but it didn't cause the incident.
Proper causal reasoning requires:
Precise timestamps (not "around 14:23")
Understanding of propagation delays (how long does it take for a deployment to affect production traffic?)
Distinguishing between "completed" and "started affecting users"
3. Counterfactual Reasoning
The gold standard for causation: "Would the incident have happened if X hadn't occurred?"
For the batch job example:
Would payment latency have spiked if the batch job hadn't run? Probably not. The batch job was the trigger.
Would payment latency have spiked if the deployment hadn't happened? Yes. The deployment was unrelated to the database code path.
Would payment latency have spiked if the marketing campaign hadn't happened? Maybe later. The extra traffic accelerated the problem but wasn't the root cause.
Counterfactual reasoning requires simulating alternative histories. That's computationally expensive and requires a detailed model of system behavior. Most RCA tools don't even attempt it.
4. Mechanism Tracing
The best RCA doesn't just identify the cause, it explains the mechanism:
This is fundamentally different from "deployment at 14:15 correlates with latency spike at 14:23." It explains how the incident happened, which tells you:
What to fix (batch the queries)
What to monitor (connection pool utilization during batch jobs)
What was a red herring (the 14:15 deployment)
Why Observability Platforms Can't Solve This
Some observability vendors are starting to distinguish correlation from causation in their marketing. They'll use dependency models and trace anomalies through system topology to identify root causes. That's more sophisticated than time-based clustering.
But it's still limited by what observability platforms can see. They ingest traces, metrics, and logs: runtime behavior. They don't see:
What code changed in the deployment
The git history of the affected files
Whether similar code changes caused past incidents
Who wrote the code and what their historical patterns are
Infrastructure capacity trends that preceded the incident
An observability platform can tell you a deployment caused an incident. It can't tell you which code change in that deployment, whether the author has a history of similar issues, or why this deployment broke things when the last 50 didn't. That requires seeing the code, not just the deployment event.
This isn't a vendor problem. It's structural to how observability tools are built. They model runtime behavior. They don't model the full causal chain from code change → deployment → infrastructure state → runtime behavior → incident.
The Cross-Domain Problem Revisited
We wrote previously about cross-domain correlation, connecting signals across code, infrastructure, and operations. Causal reasoning is why cross-domain matters.
Consider the full causal chain for an incident:
Most RCA tools only see steps 5-7. They correlate infrastructure metrics with application symptoms with operational alerts. They might correctly identify "connection pool exhaustion" as the proximate cause.
But the root cause is the code change from 3 days ago. Understanding that requires:
Code domain: What changed in the batch processor?
Review domain: Was this pattern flagged during review?
Deploy domain: When did this code actually reach production?
Infrastructure domain: What were connection pool levels before/after?
Operations domain: Have similar patterns caused incidents before?
Causal reasoning across these domains is what separates "root cause" from "proximate cause." Proximate cause: connection pool exhaustion. Root cause: unbatched query pattern in batch processor, not caught in review, deployed last week, triggered today.
What This Means for Tooling
If you're evaluating RCA tools, ask:
1. What data does it actually use for causation? If it only ingests alerts and metrics, it can only do correlation. True causal reasoning requires code changes, deployment history, and infrastructure state.
2. Does it model system behavior or just cluster events? "These alerts are related" is clustering. "This deployment caused this failure because the code change affected this code path which interacts with this resource" is causal modeling.
3. Can it explain the mechanism, not just identify the cause? Confidence scores without explanations are correlation. "73% confidence this deployment caused the incident" doesn't tell you anything actionable. The mechanism (how the change propagated to the failure) is what you need.
4. Does it distinguish temporal proximity from causal precedence? Events that happen "around the same time" are not equally likely to be causal. The tool should weight events by whether they could have caused the effect, not just whether they coincided with it.
5. Can it rule things out, not just flag things in? Good causal reasoning eliminates hypotheses: "Deployment X couldn't have caused this because it completed after the incident started" or "Service Y isn't involved because it doesn't share resources with the affected system." Correlation flags everything. Causation focuses investigation.
The Honest Answer
True causal reasoning in complex systems is hard. Really hard. We're not going to pretend we've solved it completely.
What we've built at Rebase is a system that:
Ingests data across code, infrastructure, and operations domains
Builds a model of system relationships (not just topology but dependencies, shared resources, code ownership)
Applies causal constraints (temporal ordering, mechanism plausibility, counterfactual reasoning where possible)
Explains its reasoning, not just its conclusions
We're not claiming 100% accuracy. We're claiming a fundamentally different approach than correlation-based clustering.
When we surface a root cause, we show the evidence chain:
What changed, when, and who changed it
How that change interacts with the affected system
Why other concurrent events are less likely to be causal
What the propagation mechanism was
Here's what that looks like in practice:
Correlation-based RCA:
Causal RCA:
The first gives you a confidence score. The second gives you a mechanism you can act on, hypotheses you can verify, and alternatives you can dismiss.
You can disagree with our reasoning. You can see why we ruled out the 14:15 deployment and prioritized the batch job. That transparency is the difference between a tool that helps you investigate and a black box that gives you a confidence score.
The RCA market is full of "AI-powered" tools that are really correlation-powered tools with ML-generated confidence scores. They're useful for noise reduction and alert clustering. They're not doing root cause analysis in any rigorous sense.
Causation requires modeling how systems work, not just what events occurred together. It requires cross-domain data (code, infrastructure, and operations). It requires mechanism tracing, not just pattern matching.
That's a higher bar. It's also the bar that matters when you're trying to actually fix problems instead of just finding them faster.
Other Articles
How to Evaluate Engineering Intelligence Platforms - EIP (Guide for Engineering Leaders)
A framework for evaluating AI-powered engineering intelligence platforms before vendors define success for you.
Team Rebase
Nov 24, 2025
/
Engineering
Cross-Domain Correlation: Why Your Observability Stack Can't Find Root Cause
Engineers waste hours manually correlating data across Datadog, GitHub, PagerDuty, and AWS during incidents. Cross-domain correlation automates this by understanding causal relationships across your entire engineering system, turning 4-hour investigations into instant root cause analysis.
Team Rebase
Nov 24, 2025
/
Engineering
What Is Engineering Intelligence? And Why We Built Rebase
After interviewing 200+ engineering leaders, we kept hearing the same problem: teams have all the monitoring data but none of the intelligence to understand why incidents keep happening. Rebase introduces Engineering Intelligence—a new category that correlates signals across your entire SDLC to predict risks, prevent incidents, and transform reactive troubleshooting into proactive decision-making.
Team Rebase
Nov 24, 2025
/
Engineering