Google’s CausalImpact documentation reports that Bayesian structural time-series models require a minimum of three to six months of daily pre-intervention data and at least one correlated control series to construct a valid counterfactual for time-based experiments. SEO split tests frequently cannot run treatment and control simultaneously because site-wide changes (robots.txt modifications, schema deployments, CMS migrations) affect all pages at once. Naive before-after comparisons are statistically invalid because algorithm updates, seasonality, and competitive shifts during the intervention period become alternative explanations for observed effects. BSTS modeling solves this by constructing a synthetic prediction of what would have happened without the intervention, then measuring the difference between actual and predicted performance with posterior probability intervals that quantify confidence in the causal estimate.
Bayesian Structural Time Series Constructs a Synthetic Control From Pre-Intervention Data
The core innovation of BSTS for SEO testing is constructing a counterfactual: a statistical prediction of what would have happened to the target metric if the intervention had not occurred. The model does not compare before versus after directly. It compares actual post-intervention performance against a statistically generated baseline that accounts for all patterns present in the pre-intervention data.
The model uses three types of information to build the counterfactual. Pre-intervention time-series data captures the target metric’s historical patterns, including trends, level shifts, and day-of-week effects. Seasonality patterns model recurring cycles (weekly, monthly, annual) so that seasonal effects are projected into the counterfactual rather than appearing as treatment effects. Correlated control series, such as traffic to unaffected site sections or competitor visibility data, provide real-time adjustment signals that capture environmental changes the historical patterns alone cannot predict.
The counterfactual estimate represents what the metric would have looked like in the absence of the intervention. The difference between actual post-intervention performance and the counterfactual estimate is the estimated causal effect. If actual organic traffic is 15% above the counterfactual prediction with high confidence, the intervention likely caused approximately a 15% lift.
The control series selection is critical. The ideal control series correlates strongly with the target metric during the pre-intervention period (demonstrating predictive power) but is not affected by the intervention (ensuring independence). Traffic to a different site section that targets different keywords but experiences the same seasonal and algorithmic influences often serves this purpose well.
The Model Requires Sufficient Pre-Intervention History and Stable Control Variables
BSTS validity depends on having enough pre-intervention data to learn the target metric’s patterns and stable covariates that predict the metric well before the intervention and remain unaffected by it.
The minimum pre-intervention window is typically three to six months of daily data. Shorter windows may not capture seasonal patterns, monthly cycles, or sufficient variance to train a reliable model. For sites with strong quarterly seasonality, six months of pre-intervention data is the minimum. For sites with annual seasonal cycles, twelve months of pre-intervention data produces more reliable models.
Covariate selection follows specific criteria. Each candidate control series must correlate with the target metric during the pre-intervention period (testable through correlation analysis), must not be affected by the intervention (verifiable through domain knowledge), and must not contain its own structural breaks during the analysis window that would confuse the model.
Diagnostic checks determine whether the model fits the pre-intervention period accurately enough to generate reliable counterfactual predictions. The primary diagnostic is the one-step-ahead prediction error during the pre-intervention period. If the model cannot predict the pre-intervention data within acceptable error bounds, its post-intervention counterfactual predictions are unreliable. Visual inspection of the model’s fit to historical data is the first diagnostic step. Statistical measures like the model’s posterior predictive density during the pre-test period quantify the fit quality.
If the diagnostics indicate poor fit, the response is not to use the model anyway but to either collect more pre-intervention data, find better control series, or conclude that time-based testing is not feasible for this specific intervention.
Posterior Probability Distributions Replace Binary Significance With Calibrated Uncertainty
Unlike frequentist hypothesis testing that produces p-values and binary significant/not-significant conclusions, BSTS produces a posterior probability distribution of the causal effect. This probabilistic output provides richer decision-making information.
The posterior distribution shows the full range of plausible effect sizes and their associated probabilities. From this distribution, the analyst extracts the posterior probability that the effect is positive (or negative), the expected effect size (the distribution’s mean or median), and credible intervals that contain the true effect with a specified probability (typically 95%).
A result stating “there is a 92% probability that the intervention caused a 5-10% traffic lift, with a 95% credible interval of 2-15%” communicates both the confidence in the direction and the uncertainty about magnitude. This is more useful for decision-making than a binary “significant at p < 0.05" because it allows the team to assess whether the range of plausible outcomes justifies the implementation cost.
The Bayesian framework also handles the multiple-comparison problem more naturally than frequentist methods. When testing multiple metrics from the same intervention (traffic, impressions, CTR, conversions), Bayesian posterior probabilities maintain their interpretability without the corrections (Bonferroni, Holm) that frequentist multiple testing requires.
For practical communication, translate the posterior distribution into decision-relevant statements. “Based on the analysis, we are 92% confident the title tag change increased organic traffic by between 2% and 15%, with the most likely effect being approximately 8%. The probability that the change had no effect or a negative effect is 8%.” This framing enables informed decision-making about rollout.
External Events During the Post-Intervention Period Can Invalidate the Counterfactual
If a major algorithm update occurs during the post-intervention window, the counterfactual prediction may no longer represent what would have happened without the SEO change. The control series partially address this by adjusting the counterfactual for environmental changes, but not all external events are captured by available control series.
Detect counterfactual invalidation by monitoring control variables for unexpected shifts during the post-intervention period. If the control series themselves show structural breaks (sudden level changes or trend shifts), the model’s predictions become unreliable because the covariates are no longer behaving as they did during the training period.
Check whether the residuals in the post-intervention period show structural breaks. If residuals are randomly distributed around zero with consistent variance, the model remains valid. If residuals show systematic patterns (trending, level shifts, or increasing variance), the model may be failing to capture a post-intervention environmental change.
Apply sensitivity analysis to determine whether conclusions hold under different assumptions about external event impact. Remove the period around a known algorithm update and see if the effect estimate changes substantially. If the conclusion is robust to excluding the update window, the result is more trustworthy. If the conclusion depends entirely on the data during the update window, the external event may be driving the observed effect.
When external events invalidate the counterfactual, the honest conclusion is that the test is inconclusive, not that the treatment had no effect. Inconclusive results require either extending the test past the disruption period or redesigning the test with a simultaneous control group.
The Methodology Cannot Prove Causation When Confounds Are Unavoidable
Even with BSTS, time-based testing has weaker causal claims than simultaneous split testing because time itself is a confound that cannot be fully controlled. The confidence hierarchy for SEO testing reflects this reality.
Simultaneous page-level splits provide the strongest causal evidence. Because treatment and control pages experience the same time period, algorithm updates, seasonal effects, and competitive changes, the only systematic difference is the treatment itself. Confounds are controlled by design rather than by statistical modeling.
BSTS-based time-series analysis provides moderate causal evidence. The statistical model accounts for known patterns and measurable covariates, but unmeasured confounds that happen to coincide with the intervention remain alternative explanations. The confidence level depends on the quality of the model fit, the availability of good control series, and the absence of major external events during the test window.
Before-after comparisons without statistical modeling provide the weakest evidence. Without a formal counterfactual, any change in the metric could be attributed to the intervention, seasonality, algorithmic shifts, or random variation. This approach should be avoided for any change where the causal claim matters for resource allocation decisions.
Teams should use simultaneous splits when possible, reserve BSTS for changes that cannot be split-tested, and avoid naive before-after comparisons as the basis for strategic decisions.
How many months of pre-intervention data does BSTS need for reliable SEO test results?
Three to six months of daily data is the minimum for most sites. Sites with strong quarterly seasonality require six months, and sites with annual seasonal cycles need twelve months to capture recurring patterns. Shorter windows fail to model monthly cycles and produce unreliable counterfactual predictions that undermine the entire analysis.
Can BSTS detect small SEO effects like a 2-3% traffic lift?
Detection of small effects depends on control series quality and traffic volume. BSTS can detect 2-3% lifts on high-traffic sites with strongly correlated control variables, but most configurations require effects of 5% or larger for confident detection. Weak control series or high traffic variance increases the minimum detectable effect, making small lifts indistinguishable from noise.
What happens if an algorithm update occurs during a BSTS post-intervention window?
The counterfactual prediction may become invalid if the algorithm update is not captured by the control series. Apply sensitivity analysis by removing the update window and checking whether the effect estimate changes. If conclusions depend entirely on data during the update period, the test is inconclusive. Extending the test past the disruption or redesigning with a simultaneous control group are the appropriate responses.
