Conducting interrupted time-series analysis for

single- and multiple-group comparisons

Ariel Linden, DrPH

Department of Medicine
Medical School
University of California, San Francisco
 Agenda
• Provide definitions and illustrate possible patterns
• Briefly describe modelling approaches (advantages/disadvantages)
• Detail OLS model parameters
• Present some examples
• Describe methods to improve causal inference in ITSA
 Definition of interrupted time series
analysis
An outcome variable that is observed:
• over multiple, equally-spaced time periods
• before and after the introduction of an intervention
• which is expected to interrupt its level and/or trend
 Definition of interrupted time series
analysis
• Single group time-series design (Campbell and Stanley 1966):
0 0 0 0 X0 0 0 0

• Multiple-group time-series design (Campbell and Stanley 1966):

0 0 0 0 X0 0 0 0
0 0 0 0 0 0 0 0
 When is ITSA useful
Single-group ITSA:
• For N-of-1 studies
• When the only data available are summarized at the population-level
(e.g. mortality or morbidity rates)
• When observations are available for multiple evenly-spaced time
points

Multiple-group ITSA:
• Same as above but when a control group is available for comparison
Some possible single-group ITSA patterns*

E
A
F

C G

* Reproduced (more or less) from Campbell and Stanley (1966)

 Statistical modeling approaches for ITSA
• Statistical models used for ITSA account for autocorrelation (the
relationship between a variable's current value and its past values)
• The two general approaches historically used in ITSA are:

• Autoregressive Integrated Moving-Average (ARIMA) models w/transfer function

(Box and Tiao 1975)
• Y = Y-1,-2,-3... X, (#p,#d,#q) , p is autoregressive, d is differencing, and q is moving-
average
• Y = Y-1,-2,-3... X, (#p,#d,#q) (#P,#D,#Q,#s), where second part relates to seasonality
(can be implemented in Stata using –tstf-)
• OLS regression models designed to adjust for autocorrelation (see Simonton
[1977])
 Advantages of ARIMA
• Designed to handle autocorrelation, seasonality and cyclical trends
• Non-linear models fit the data better than linear models
 Disadvantages of ARIMA
• At least 50, and preferably more than 100 observations are needed to
stabilize the estimates (Box and Jenkins 1976)
• ARIMA models are inherently designed for univariate time-series with
a single intervention, not sequential interventions. Comparison to a
control group is possible, but complicated
• The procedural complexity of the methodology requires a great deal
of expertise. Velicer and Harrop (1983) demonstrated that even highly
trained researchers classified only 28 percent of computer-generated
time series data correctly
 Advantages of OLS-ITSA models
• OLS may require as few as four observations (Simonton 1977)
• OLS models are sufficiently flexible to accommodate multiple treatment
periods and comparisons with one or more control groups
• Post-estimation tests are available to assess whether the model
correctly adjusted for autocorrelation
• -lincom- can be used post-estimation to produce other estimates
• OLS models can be used in conjunction with weighting or matching to
improve causal inference
• OLS models avoid the problems of model identification that encumbers
the ARIMA methodology
 Disadvantages of OLS-ITSA models
• They produce linear estimates that may not fit the data well
• Adjusting for seasonality and cyclical trends adds complexity to the
modeling process
 The single-group OLS-ITSA model
Yt = β0 + β1Tt + β2Xt + β3XtTt + ϵt

• Yt is the summarized outcome variable measured at each equally spaced time point t,
• Tt is the time since the start of the study,
• Xt is a dummy (indicator) variable representing the intervention (pre-intervention
periods 0, otherwise 1), and
• XtTt is an interaction term
 The single-group OLS-ITSA model

β3(XT)
Outcome (Y)

β2(X)

β0
β1(T)

Pre-intervention Post-intervention
 The multiple-group OLS-ITSA model
Yt = β0 + β1Tt + β2Xt + β3XtTt + β4Z + β5ZTt + β6ZXt + β7ZXtTt + ϵt

• β0 to β3 represent the control group, and β4 to β7 represent the treatment group

• Z is a dummy variable to denote the cohort assignment (treatment or control)
• ZTt ZXt, and ZXtTt are all interaction terms among previously described variables
 The multiple-group OLS-ITSA model

β7(ZXT)
Outcome (Y)

β3(XT) β6(ZX)

β5(ZT)

β4(Z)

Pre-intervention Post-intervention
 Cigarette sales p
The multiple-group OLS-ITSA model

140

120

100

80

60

40
California and average of controls
Intervention starts: 1982 1989
_[_t] _[_t] + _[x_t1982]

_[_t] + _[x_t1982] + _[x_t1989]

_[t] + _[z_t]

_[_t] + _[_z_t] + _[x_t1982] +

_[_t] + _[z_t] + _[_z_x_t1982] + _[x_t1989] +
_[x_t1982] + _[_z_x_t1989]
_[z_x_t1982]

1970 1980 1990 2000

Year

California: Actual Predicted

Controls average: Actual Predicted
Regression with Newey-West standard errors - lag(1)
 Threats to internal validity of ITSA models
Single-group ITSA:
• History -- some event other than the intervention produces the shift
in the time-series (Campbell and Stanley 1966)
• Seasonality

Multiple-group ITSA:
• Unmeasured confounding that affects the treatment group’s series
differently than the control group’s series
Examples of where SG-ITSA results
can be misinterpreted
(math free)
Florida’s 2000 repeal of the helmet law on

Motorcycle deaths as a percent of total

motorcycle deaths

Raw motorcycle deaths

600

500

400

300

200

100

.15

.05
.2

.1
1975 1985 1995 2005 2015 1975 1985 1995 2005 2015
(a) (b)

Motorcycle deaths in Florida before and after repeal of the helmet law in July 2000
 Motorcycle deaths vs registrations (std)
Florida’s 2000 repeal of the helmet law on

Motorcycle registratons (in thousands)

motorcycle deaths
9,000

7,000

5,000

3,000

-1

-2
2

0
Registrations

Deaths

1975 1985 1995 2005 2015 1975 1985 1995 2005 2015
(c) (d)

Motorcycle registrations in Florida before and after repeal of the helmet law in July 2000
 Motorcycle deaths - Florida vs all states (std)

Motorcycle deaths - FL vs NV and NC (std)

Florida’s 2000 repeal of the helmet law on
motorcycle deaths

-1

-2

-1

-2
2

0
Controls

Controls

Florida
Florida

1975 1985 1995 2005 2015 1975 1985 1995 2005 2015
(e) (f)

Florida’s motorcycle deaths vs those of all other States, and matched control States
 California cigarette sales per-capita (in packs)
California’s Proposition 99 (1988)

Cigarette sales per-capita (in packs)

on per capita cigarette sales
140

120

100

120

100
80

60

40

80

60

40
1970 1975 1980 1985 1990 1995 2000 1970 1975 1980 1985 1990 1995 2000
(a) (b)

(a) Cigarette sales in California before and after Proposition 99. (b) Structural break in 1983
 California vs Colorado, Idaho and Montana
California’s Proposition 99 (1988)

California vs all other states

on per capita cigarette sales
140

120

100

140

120

100
80

60

40

80

60

40
All other states

Control states

California California

1970 1975 1980 1985 1990 1995 2000 1970 1975 1980 1985 1990 1995 2000
(c) (d)

Comparing California’s cigarette sales to all other States, and to matched control States
 Motorcycle deaths vs registrations (std)
Louisiana’s repeals and reinstatements

Raw motorcycle deaths - LA

of the helmet law
150

100

50

-1

-2
0

0
Registrations

Partial Repeal Reinstated Amended Reinstated Deaths

1975 1985 1995 2005 2015 1975 1985 1995 2005 2015
(A) (B)

Louisiana’s multiple repeals/amendments/reinstatements of helmet law on motorcycle deaths

 Motorcycle registrations ; LA vs all states (std)
Motorcycle deaths - LA vs all states (std)
Louisiana’s repeals and reinstatements
of the helmet law

-1

-2
-1

-2

0
3

Controls

All States

Louisiana Louisiana

1975 1985 1995 2005 2015 1975 1985 1995 2005 2015
(C) (D)

Louisiana’s motorcycle deaths and registrations compared to all other States

 Take-aways about SG-ITSA?
• A single-group ITSA is unlikely to be more valid than a simple pre-post
test
• Perform structural break analysis to further support/refute a
treatment effect
• Adding one or more cross-overs may not improve validity
• Identifying confounders, history, secular trends, may support/refute a
treatment effect
• A multiple-group ITSA is ALWAYS preferred to a single-group ITSA
• A matched (or weighted) multiple-group ITSA is EVEN better!
 Improving causal inference in MG-ITSA
• Use weights derived from the “synthetic controls” method (Abadie
and Diamond 2010) in conjunction with ITSA
• Match on covariates to identify controls using “ITSAMATCH” then
estimate treatment effects using ITSA
• Run permutation tests (which includes matching and estimation)
using “ITSAPERM”
 Cigarette sales per-capita
Synthetic controls and ITSA
140

120

100

80

60

40
1970 1975 1980 1985 1990 1995 2000
Year

SYNTH assigned weights to Colorado (.159), Connecticut (.068), Montana (.203), Nevada (.235) and Utah
(.335)
 Cigarette sales per-capita (i
ITSAMATCH and ITSA

140

120

100

80

60

40
1970 1975 1980 1985 1990 1995 2000
Year

ITSAMATCH found Colorado, Idaho, and Montana as best matches to California

 ITSAPERM

State
AL
AR
CA
CO
CT
DE
GA
ID
IL
IN
IA
KS
KY
LA
ME
MN
MS
MO
MT
NE
NV
NH
NM
NC
ND
OH
OK
PA
RI
SC
SD
TN
TX
UT
VT
VA
WV
WI
WY

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6
Benefits Treatment Benefits Controls

Difference-in-differences of trends (95% CIs)

After iteratively matching the treatment unit (CA) and pseudo-treatment units (all other States) to controls,
only CA appears to have a statistical (and directionally correct) effect
 Conclusions
• Interrupted time series analysis is an observational (natural) study
design that capitalizes on having many data-points for determining
treatment effects (both visually and statistically)
• A single-group ITSA may be no more valid than the simple pre-post
design if some (non-observed) event other than the intervention
produced the shift in the time-series
• A multigroup ITSA that compares the treated unit to one or more
comparable controls (via weighting or matching) is the most valid
approach with observational data
• Adding permutation tests provides an additional robustness check
 ITSA related packages for Stata
1. ITSA – Performs interrupted time series analysis for single and multiple group comparisons
2. ITSAMATCH – Performs matching in multiple group interrupted time series analysis
3. ITSAPERM – Performs permutation tests for matched multiple group interrupted time series analysis
4. ITSARAND – Performs randomization tests for single-case and multiple-baseline AB phase designs
5. XTITSA - Performs interrupted time series analysis with panel data
 References
1. Abadie A, Diamond A. 2010. Synthetic control methods for comparative case studies: Estimating the effect of California's tobacco control program. J Am Stat
Assoc. 105: 493‐505.
2. Box, G. E. P., and G. M. Jenkins. 1976. Time Series Analysis: Forecasting and Control. San Francisco, CA: Holden Day.
3. Box, G. E. P., and G. C. Tiao. 1975. Intervention analysis with applications to economic and environmental problems. Journal of the American Statistical
Association 70: 70–79.
4. Campbell, D. T., and J. C. Stanley. 1966. Experimental and Quasi-Experimental Designs for Research. Chicago, IL: Rand McNally.
5. Linden A. 2015. Conducting interrupted time series analysis for single and multiple group comparisons. Stata Journal 15: 480-500.
6. Linden A. 2016. Challenges to validity in single-group interrupted time series analysis. Journal of Evaluation in Clinical Practice 23: 413–418.
7. Linden A. 2017. A comprehensive set of post-estimation measures to enrich interrupted time series analysis. Stata Journal 17: 73-88.
8. Linden A. 2017. Persistent threats to validity in single-group interrupted time series analysis with a crossover design. Journal of Evaluation in Clinical Practice 23:
419–425.
9. Linden A. 2018. A matching framework to improve causal inference in interrupted time series analysis. Journal of Evaluation in Clinical Practice 24: 408-415.
10. Linden A. 2018. Combining synthetic controls and interrupted time series analysis to improve causal inference in program evaluation. Journal of Evaluation in
Clinical Practice 24: 447-453.
11. Linden A. 2018. Using permutation tests to enhance causal inference in interrupted time series analysis. Journal of Evaluation in Clinical Practice 24:496-501.
12. Linden A. 2022. Erratum: A comprehensive set of post-estimation measures to enrich interrupted time series analysis. Stata Journal 22:231-233.
13. Simonton, D. K. 1977. Cross-sectional time-series experiments: Some suggested statistical analyses. Psychological Bulletin 84: 489–502.
14. Velicer, W. F., and J. Harrop. 1983. The reliability and accuracy of time series model identification. Evaluation Review 7: 551–560.
Thank You!