Assignment II — Streaming-Recommender A/B Test

Overview

A music-streaming company is running a 3-arm online A/B test on its home-screen recommender. The three arms are:

  • control — current top-of-funnel ranker.
  • variant_A — new ranker with an added collaborative-filtering signal.
  • variant_B — variant_A plus a one-time onboarding nudge banner.

Users were assigned at first login during a 3-week ramp-up. The company tracked two outcomes over the 28 days following assignment:

  • minutes_28d — average daily minutes listened, continuous.
  • engaged_28d — 1 if the user listened ≥ 30 min on ≥ 10 of the 28 days, 0 otherwise.

Your job is to take this experiment from raw counts to a shippable product recommendation. The product team wants to know:

  1. Is the experiment safe to interpret? (Sample Ratio Mismatch, covariate balance.)
  2. What can we detect? (Power and minimum detectable effect.)
  3. What is the average treatment effect? (With appropriate inference: HC-robust standard errors, bootstrap, randomization inference. And a deliberate bad-control comparison to reinforce the collider-bias trap from Assignment I.)
  4. Can we tighten the SE without enrolling more users? (CUPED and CUPAC variance reduction.)
  5. Can we peek at the data weekly without inflating the false- positive rate? (Sequential / anytime-valid inference.)
  6. Which arms should we ship? (Bayesian posterior + expected-loss decisioning.)

The dataset is simulated from a Structural Causal Model that has been calibrated to look realistic; this is disclosed so that you focus on methodology rather than on real-world data quirks. Students may submit in R or Python.

Dataset & Codebook

The dataset contains ~6,000 user rows with 11 columns.

Download streaming_ab_assignment.csv (≈ 300 KB) — save this file into the same folder as your source file (.qmd or .ipynb). Your code should then read it with a plain relative path: read_csv("streaming_ab_assignment.csv") (R) or pd.read_csv("streaming_ab_assignment.csv") (Python).

Column Type Description
user_id integer Unique user identifier
arm factor Treatment. control, variant_A, variant_B.
assignment_week integer (1–3) Week of first login during the 3-week ramp-up
country factor DEU, GBR, FRA, ESP, Other
device_type factor mobile, desktop, tablet
tenure_days integer Days since account creation
prior_minutes continuous (min/day) 28-day daily-avg listening time before the experiment
prior_skip_rate continuous ∈ [0, 1] Fraction of tracks skipped, pre-experiment window
premium_signup binary (0/1) 1 = the user signed up for Premium after assignment but during the 28-day window. The Premium-funnel team’s algorithm uses both the recommender signal AND the user’s emerging engagement to decide whom to nudge — i.e., this is a post-treatment variable.
minutes_28d continuous Continuous outcome. Daily-avg listening time over the 28-day window after assignment.
engaged_28d binary (0/1) Binary outcome. 1 = listened ≥ 30 min on ≥ 10 of the 28 days.
WarningImportant — what the dataset does NOT directly tell you

The data do not flag which users were affected by the rumoured mobile-SDK issue during the ramp-up. Part of Step 1 is to find this yourself with a stratified Sample-Ratio-Mismatch diagnostic.

The data do not indicate that premium_signup is post-treatment. Read the column description carefully and decide for yourself whether it is safe to include in your ATE regression (it is not — but you should derive that conclusion explicitly).

library(tidyverse)
df <- readr::read_csv("streaming_ab_assignment.csv")
glimpse(df)
import pandas as pd
df = pd.read_csv("streaming_ab_assignment.csv")
print(df.shape); print(df.head())

Step 1 — Sample-Ratio-Mismatch & Randomization Check (15 %)

Required:

  1. Overall SRM. Compute the counts in each arm and run a chi-squared test against the intended 1:1:1 split. Apply Kohavi’s \(p < 0.001\) threshold and report whether the overall ratio passes.

  2. Stratified SRM diagnostic. Repeat the chi-squared by device_type × assignment_week. Identify which cell (if any) carries the mismatch. Hypothesise in one or two sentences what the operational cause might be (redirect failure, bot filtering, opt-out flow), citing your stratified table.

  3. Mitigation. Drop the offending cell (or — if you prefer — reweight using inverse propensity for the assignment; either is accepted, but justify the choice). Re-run the overall chi-squared on the mitigated dataset and decide: is the experiment now safe to interpret?

  4. Covariate balance. On the mitigated dataset, compare prior_minutes, prior_skip_rate, and tenure_days across the three arms (pairwise t-tests with a Bonferroni correction, or a one-way ANOVA). Report a balance table and interpret it in a paragraph.

TipHint

The mismatch is small enough that the overall chi-squared may not breach \(p < 0.001\). The stratified diagnostic is the diagnostic that matters. Most SRM cases in production are localised — one device, one country, one browser version where the assignment pipeline silently drops users. See S05 §4 (“When A/B breaks”) and Ex 5 for the canonical pattern.

Step 2 — Power & Sample Size (10 %)

Required:

  1. Minimum Detectable Effect (MDE) at the realised per-arm sample size (post-mitigation), \(\alpha = 0.05\), power \(1 - \beta = 0.80\). Compute for both outcomes:

    • minutes_28d — use pwr::pwr.t.test() (R) or statsmodels.stats.power.tt_ind_solve_power (Python).
    • engaged_28d — use pwr::pwr.2p.test() (R) or statsmodels.stats.power.NormalIndPower (Python). Convert Cohen’s d / Cohen’s h back to the raw outcome scale (min, probability points) so the result is interpretable.

  2. Required sample size to detect a \(\delta = 5\) min uplift on minutes_28d and a \(\delta = 2\) pp uplift on engaged_28d.

  3. Write a one-paragraph “are we powered?” verdict for each outcome. State explicitly whether the experiment is well-powered to detect the kind of effects the product team plausibly cares about.

Step 3 — ATE Estimation & Inference (20 %)

Estimate the ATE on minutes_28d four ways and put the numbers in one comparison table:

Estimator What it computes
Naïve diff-in-means \(\bar Y_{\text{arm}} - \bar Y_{\text{control}}\) for each arm
OLS without covariates lm(minutes_28d ~ arm) with HC1-robust SE
OLS with covariates (Lin 2013 interaction) lm(minutes_28d ~ arm * (prior_minutes_c + prior_skip_rate_c)) where _c indicates demeaning
Bad-control variant (deliberately wrong) OLS adding premium_signup to the previous spec

For the OLS-with-covariates row, additionally report:

  • The HC1-robust standard error (sandwich::vcovHC in R / cov_type="HC1" in statsmodels).
  • The bootstrap SE with \(B = 1{,}000\) resamples.
  • The randomization (Fisher) inference p-value with 1,000 permutations of arm within country × device_type strata.

For engaged_28d, fit a logistic regression with and without the two covariates and report the Average Marginal Effect of arm (use marginaleffects::avg_slopes in R or compute the AME manually in Python from the fitted probabilities).

Required discussion (one page):

  • Which estimator do you trust most for minutes_28d? Why?
  • Do the HC1 SE, the bootstrap SE, and the randomization-inference p-value agree? If they disagree, what does that tell you about the assumptions of each procedure?
  • The bad-control variant is included on purpose. Compare its estimate to the OLS-with-covariates estimate. Is the bad-control number larger or smaller? Why? Cite the collider-bias logic from Assignment I (in two or three sentences), and identify the collider in this dataset by the structural-causal mechanism described in the codebook.

Step 4 — Variance Reduction: CUPED + CUPAC (15 %)

Implement both variance-reduction methods covered in S05 §Ex 2 and compare them to the plain OLS baseline.

Required:

  1. CUPED. Compute \(\theta = \mathrm{Cov}(Y, X) / \mathrm{Var}(X)\) where \(X\) = prior_minutes, \(Y\) = minutes_28d. Build \(Y^{\text{CUPED}}_i = Y_i - \theta \cdot (X_i - \bar X)\). Refit the ATE regression on the adjusted outcome and report the new standard error for arm.

  2. CUPAC. Fit a small ML regressor (ranger in R or sklearn.ensemble.RandomForestRegressor in Python) on the control arm only, predicting minutes_28d from prior_minutes, prior_skip_rate, tenure_days, country, device_type. Predict \(\hat Y_i^{(0)}\) for every unit (treated and control). Build the CUPAC-adjusted outcome \(Y^{\text{CUPAC}}_i = Y_i - \hat Y_i^{(0)}\) and refit the ATE regression. Report the new SE.

  3. Comparison table. Stack the three rows — plain OLS, CUPED, CUPAC — with columns SE, variance ratio (squared SE relative to plain OLS), MDE at the realised \(n\), and the experiments-per-quarter multiplier \(1 / \text{VR}\).

  4. Interpretation (one paragraph). Under what conditions does CUPAC beat CUPED on this dataset? Cite the DGP intuition in your own words: which pre-period covariates are doing the work, and why does the ML regressor extract additional variance beyond what the linear CUPED θ captures?

TipWhat we expect

Industry-deployed CUPED in 2020+ typically reduces SE by 30–50 % (VR ≈ 0.3–0.5). CUPAC with rich pre-period features typically beats CUPED when those features contain predictive signal not linearly captured by the single CUPED covariate. On this dataset, CUPAC should beat CUPED meaningfully — but a one-paragraph explanation of why is required, not just the numerical comparison.

Step 5 — Anytime-Valid Inference & Peeking (15 %)

You and a colleague are watching the experiment from different vantage points:

  • Analyst 1 waits the full three weeks and produces one fixed- horizon 95 % confidence interval for the Variant A vs. control effect on minutes_28d.
  • Analyst 2 peeks at the data at the end of each week. At each peek she records:
    1. the naïve 95 % CI based on cumulative data through that week (using \(z = 1.96\) as if she were doing a one-shot analysis);
    2. an anytime-valid confidence sequence computed with the law-of-iterated-logarithm bound \[c(n_k) = \sqrt{2 \log(\log(n_k) + 2) + \log(2/\alpha)}\] from S05 §3c (no confseq package required — the formula is closed-form).

Required:

  1. Three-peek table + plot. Produce a table with the three weeks, point estimate, naïve CI, anytime-valid CI, and per-arm sample size used in the bound. Plot the point estimate plus both intervals at each peek.

  2. Discussion. Which interval would you ship the analysis on? What does the difference between the naïve CI and the CS look like at the earliest peek (small \(n_k\), smaller \(\log(\log n_k)\) term)?

  3. Bonus (+5 %, capped). Simulate the cumulative Type-I error under a true null (\(\tau = 0\)) across the 3-peek setup. Compare the naïve \(z = 1.96\) procedure to the LIL CS. (You should replicate the structure of S05 Ex 3 on the 3-peek cadence used here.)

Step 6 — Bayesian A/B with Expected Loss (20 %)

Construct Bayesian posteriors for both outcomes against control, and apply an expected-loss decision rule.

Required:

  1. Posteriors. For each (arm, outcome) pair:
    • minutes_28d — Normal model on each arm’s mean; use a non-informative prior (\(\mu \sim N(\hat\mu, 10^6)\), \(\sigma\) using sample SD). Draw \(M = 50{,}000\) samples per arm.
    • engaged_28d — Beta-Binomial with a uniform (non-informative) prior \(\theta_{\text{arm}} \sim \text{Beta}(1, 1)\). The conjugate posterior is then \(\theta_{\text{arm}} \mid s, n \sim \text{Beta}(1 + s,\; 1 + n - s)\), where \(s\) is the number of engaged users in the arm and \(n\) is the arm size. Draw \(M = 50{,}000\) samples per arm.
  2. For each contrast (A vs. C, B vs. C, on each outcome), report:
    • 95 % credible interval.
    • \(P(\tau_{\text{arm}} > 0 \mid \text{data})\).
    • Expected loss of shipping that arm: \[\mathbb{E}[\max(-\tau_{\text{arm}}, 0) \mid \text{data}].\]
    • Decision under the thresholds \(\varepsilon_{\text{minutes}} = 0.5\) min and \(\varepsilon_{\text{engaged}} = 5 \times 10^{-4}\).
  3. Product recommendation (one paragraph). Write the email you would send to the recommender team. Which arm(s) ship? Which hold? If your recommendation is not “ship the arm with the largest raw uplift on minutes,” explain why — connect to the gap between \(P(\tau > 0)\) and expected loss, and to the distinction between continuous-metric uplift and binary-metric risk.
TipWhat we expect

The slide-deck example from S05 §3d had \(P(B > A) \approx 0.87\) but expected loss of \(2.06 \times 10^{-4}\), breaching an \(\varepsilon = 10^{-4}\) threshold. A similar gap is expected here on one of the contrasts. Identifying that the largest-uplift arm is not always the one to ship is the headline learning of this step.

Step 7 — Reflection (5 %)

Write a one-page reflection covering:

  • Strengths and limitations of your analysis. Which assumptions carry the most weight? Where are they fragile?
  • What additional data (longer follow-up, network-level metrics, per-user cost data) would tighten your recommendation?
  • AI-use disclosure (per course policy). If you used an LLM, state which prompts, which output you verified independently, and what (if anything) you took on trust.

Rubric

Criterion Weight Excellent (1) Good (2) Needs Improvement (3) Missing/Inadequate (4–5)
SRM + Randomization (Step 1) 15 % Overall + stratified SRM + post-mitigation re-check + balance table + clear interpretation of the localised bug. Stratified diagnostic but weak interpretation, or balance check missing. Only overall SRM; stratified diagnostic missing or misread. No SRM analysis.
Power & Sample Size (Step 2) 10 % MDE for both outcomes converted to raw scale, required-n for both deltas, clear powered/under-powered verdict. MDE correct but verdict weak or sample-size-for-delta missing. One outcome only; conversion to raw scale missing. No power analysis.
ATE estimation + inference (Step 3) 20 % All four estimators presented with HC1 SE; bootstrap and randomization inference reproduce the SE; collider trap correctly identified and explained. Estimators correct but bootstrap or randomization missing, or collider explanation weak. Several estimators correct but bad-control variant ignored or treated as a “good” covariate. No ATE analysis.
CUPED + CUPAC (Step 4) 15 % Both implementations correct; comparison table with VR + experiments-per-quarter multiplier; clear interpretation of when CUPAC beats CUPED. Both implementations but interpretation generic; or one variance ratio computed inconsistently. Only CUPED, or CUPAC implementation broken. No variance reduction analysis.
Peeking + Anytime-Valid (Step 5) 15 % Both intervals at every peek; plot + table; clear discussion of when the CS-vs-naïve gap matters. Plot or table but discussion is mechanical, not connected to operational stopping. Only one interval, or the LIL bound mis-derived. No sequential analysis.
Bayesian + Expected Loss (Step 6) 20 % Posteriors for both outcomes and both contrasts; expected-loss table with decisions; product recommendation that uses the gap between \(P(\tau > 0)\) and expected loss. All numbers but the product recommendation does not exploit the loss-vs-probability gap. Bayesian posteriors only; no expected-loss step. No Bayesian analysis.
Reflection (Step 7) 5 % Honest, specific, actionable; AI use disclosed. Mostly clear; minor gaps. Generic or missing AI disclosure. Missing reflection.

In addition to step-specific scoring, the general evaluation criteria apply (correctness, code quality, communication clarity, reproducibility — see the syllabus for the full table).

Submission

Content

  • A written report of 8–10 pages (selectively displayed code, figures, references) clearly structured per the steps above.
  • Annotated, reproducible code (R or Python). The instructor must be able to open your source file, install the listed packages, and re-render your report end-to-end on a clean machine.

Format

  • Submit one source file plus the rendered PDF. The source file may be either a Quarto Markdown (.qmd) or a Jupyter notebook (.ipynb) — both are accepted, in either R or Python.
  • File names: [YourLastName]_Assignment_2.{qmd|ipynb} and [YourLastName]_Assignment_2.pdf.
  • Bundle the two files into a single zip: [YourLastName]_Assignment_2.zip.
  • Do not include streaming_ab_assignment.csv in the zip. When grading, the instructor will place the CSV in the same folder as your source file before re-rendering. Your code should therefore read the file with a plain relative path (read_csv("streaming_ab_assignment.csv")), not a path that traverses parent directories.
  • If your code reads from any additional external file you created (e.g., a saved bootstrap result, an intermediate adjustment), bundle it alongside the source file in the zip — but never the provided streaming_ab_assignment.csv.

Deadline

  • Sunday, 31 May 2026, 23:59 (Hamburg local time).
  • Upload to the StudIP folder (link in the course shell).

Quiz

  • A 15-minute closed-book paper quiz on this material at the start of the Monday, 1 June 2026 lecture. Quiz covers SRM diagnostics, the bias decomposition with HC-robust SEs, CUPED/CUPAC variance reduction, the peeking pathology and one cure (Pocock or LIL CS), and the Bayesian credible interval plus expected-loss decision.
WarningIf something breaks

If you cannot complete a step due to a technical problem, submit anyway. Document what you tried, what failed, and what you would have done. Partial credit is awarded generously when the reasoning is shown.