When a true concurrent A/B test isn’t possible, for instance a site-wide change with no realistic way to hold back a control group, valid before/after inference requires a causal-inference methodology that explicitly models what would have happened without the change, rather than a naive percentage comparison of the before period against the after period. The established approaches for this are counterfactual time-series methods such as CausalImpact (Bayesian structural time series) or synthetic control, which use a set of correlated but unaffected reference metrics or sites to construct an expected counterfactual trajectory, against which the actual post-change trajectory is compared.
Why naive before/after comparison fails
A simple before/after comparison, measuring the percentage change in traffic or rankings from the pre-change period to the post-change period, implicitly assumes that nothing else affecting the outcome changed between those two periods except the thing being tested. That assumption almost never holds for organic search. Seasonality affects most query categories to some degree, Google rolls out core algorithm updates and smaller ranking-system changes on an ongoing basis independent of any single site’s actions, and general market/demand trends shift over time regardless of what a site does. A before/after comparison that doesn’t account for these factors conflates the actual treatment effect with all of this background noise, and there’s no way to separate them after the fact from the raw before/after numbers alone.
This is a particularly acute problem for SEO specifically because algorithm updates are frequent, often unannounced in their exact timing or scope, and can produce swings in organic traffic that are easily large enough to be mistaken for, or to mask, the effect of a genuine site-side change made around the same time.
How counterfactual methods solve this
The core idea behind CausalImpact and similar Bayesian structural time series approaches is to build a statistical model of the outcome metric using a set of control time series that weren’t affected by the change but are correlated with the treated metric for reasons unrelated to the treatment, other pages on the same site that didn’t receive the change, a comparable competitor or industry benchmark series, or other unaffected metrics that move with the treated one due to shared external factors like seasonality or algorithm updates. The model learns the relationship between the treated series and these controls during the pre-change period, then uses that learned relationship to predict what the treated series would have looked like during the post-change period had the change not occurred.
The actual observed post-change values are then compared against this modeled counterfactual, not against the simple pre-change baseline. If Google rolls out an algorithm update that affects the whole reference set (the treated pages and the control pages together) during the test window, that shows up as movement in both the treated series and the model’s counterfactual prediction, and because the model incorporates the controls’ behavior, the update’s effect gets absorbed into the counterfactual baseline rather than being misattributed to the tested change. This is the mechanism that specifically separates true treatment effect from concurrent external shocks that a naive before/after comparison cannot separate.
Synthetic control methods work on a similar underlying logic but construct the counterfactual as a weighted combination of multiple control units chosen to closely match the treated unit’s pre-treatment trajectory, which is particularly useful when there isn’t one obvious single control series but a pool of candidate comparison series (e.g., a set of similar but untreated pages or sites) to draw from.
What a valid control set actually requires
The methodology only works if the control series are genuinely unaffected by the treatment (no spillover from the tested pages to the control pages) and are exposed to the same external shocks the treated series is exposed to (the same algorithm updates, the same seasonal patterns, ideally the same general market conditions). A control set that differs meaningfully from the treatment group on baseline characteristics, different topic area, very different traffic volume, different historical crawl or ranking behavior, weakens the model’s ability to construct an accurate counterfactual, since the learned pre-period relationship may not hold as cleanly. Building this control set thoughtfully, checking that it tracks the treated series closely during the pre-period before trusting the post-period comparison, is as important to validity as the choice of statistical method itself.
Practical implication
Don’t run a time-based SEO test relying on a simple before/after percentage change as the sole evidence of effect, particularly for any change being evaluated over a period long enough to plausibly overlap with an algorithm update or seasonal shift. Identify a set of correlated, unaffected reference series (comparable unchanged pages, industry benchmark data, or other internal metrics not touched by the change) before the test begins, verify they track the treated series reasonably well in the pre-period, and apply a counterfactual method like CausalImpact to separate the modeled expected trajectory from the actual observed outcome. Treat the confidence interval the model produces around the estimated effect as part of the result, not just the point estimate, since a counterfactual approach that returns a wide interval spanning near-zero or negative effect is telling you the naive percentage change you’d otherwise have reported wasn’t a reliable signal in the first place.
