Assignment III — The Confounded Impact of Push Notifications

Overview

You are a data scientist at FitLife, a mobile fitness app with 5 million active users. FitLife sends personalised push notifications (“🏃 Time for your afternoon run!”, “🧘 Try a 5-minute mindfulness session”). Management wants rigorous evidence on what actually drives engagement so it can tune its notification strategy. Two distinct causal questions sit on top of one dataset:

  1. Should we keep sending? What is the effect of the algorithm sending a notification (notification_sent) on engagement? The send decision is made by an algorithm from observed user features — so this is a question about observed confounding, and the Part-III tools from S06 (matching, IPW, AIPW, sensitivity analysis) apply.
  2. What does opening do? What is the effect of a user opening a notification (notification_opened)? Whether a user opens is a choice driven by unobserved motivation — highly motivated users both open more and engage more, regardless of the app. No set of observed controls can close that backdoor, so this is a question about unobserved confounding that needs a design (S07).

To separate the mere act of sending from the user’s decision to open, FitLife runs a randomized encouragement design: half of all users are independently shown a random “Reminder Pop-Up” (pop_up_rem) nudging them to check that day’s notification. The pop-up makes opening more likely but — by design — should not change engagement except through getting the user to open. This randomized pop-up is your instrument.

Your report must estimate the same headline question multiple ways, reconcile the answers, and translate them into a recommendation. This assignment spans the whole of Part III: Steps 1–5 use the observed-confounding toolkit on the send decision; Steps 6–7 use instrumental variables on the open decision.

Dataset & Codebook

Download fitlife_a3.csv (≈ 640 KB) — save it into the same folder as your source file (.qmd or .ipynb), then read it with a plain relative path: read_csv("fitlife_a3.csv") (R) or pd.read_csv("fitlife_a3.csv") (Python).

Column Type Description
age integer User age (18–60)
gender categorical Male / Female
region categorical US / EU / Asia
sessions_per_day integer Average app sessions per day (pre-period)
past_engagement continuous Average daily engagement minutes before any notification
device_type categorical Android / iOS
app_version integer (1–3) Installed app version
notification_sent binary 1 if the algorithm sent a push notification (treatment for Steps 1–5)
pop_up_rem binary 1 if the user was shown the random reminder pop-up (the instrument \(Z\), ~50/50)
notification_opened binary 1 if the user opened the notification (treatment for Steps 6–7)
engagement continuous Minutes of app use after the notification window (the outcome \(Y\))
TipStarter code 
library(tidyverse)
fit <- read_csv("fitlife_a3.csv") |>
  mutate(app_version = factor(app_version))
glimpse(fit)
import pandas as pd
fit = pd.read_csv("fitlife_a3.csv")
fit["app_version"] = fit["app_version"].astype("category")
fit.info()

You may submit in R or Python. Report your results in a rendered Quarto document (.qmd → HTML/PDF) or Jupyter notebook. Show your code, your numbers, and your reasoning.

Step 1 — Conditional Outcome Regression (10 %)

Estimate the effect of notification_sent on engagement.

  • Regress engagement on notification_sent with no controls, then adjusting for all observed covariates \(X\) (age, gender, region, sessions_per_day, past_engagement, device_type, app_version). Use heteroskedasticity-robust (HC1) standard errors.
  • Interpret the adjusted conditional ATE of sending. State before running any “real” method why the naive estimate is suspect, and which kind of confounding adjustment can and cannot fix.

Step 2 — Matching: propensity score vs. covariate space (15 %)

The lecture’s verdict on matching was pointed: King & Nielsen (2019) show that matching on the propensity score can leave covariate imbalance untouched, and recommend covariate-space matching (Mahalanobis, CEM) instead — reserving the propensity score for the weighting you do in Steps 3–4. Put that verdict to the test on FitLife.

  • Estimate a logit propensity score for notification_sent and run 1:1 nearest-neighbour PSM (1:3 optional). Then run at least one covariate-space matcher on the same covariates — Mahalanobis nearest-neighbour matching, and ideally CEM (coarsened exact matching) as well.
  • For each method, report covariate balance before and after matching — a standardised-mean-difference table plus a love plot (R: MatchIt + cobalt; Python: a balance plot) — paying particular attention to the dominant confounder past_engagement.
  • Estimate the ATT under each method, with standard errors, \(t\)-statistics, and \(p\)-values.
  • Interpret the comparison. Does covariate-space matching actually balance past_engagement better than PSM here? What does CEM’s pruning do to the number of treated units retained — and therefore to the estimand and the ATT? Connect explicitly to the King–Nielsen critique, and to why the assignment turns to weighting in Step 3.

Step 3 — Inverse Probability Weighting (15 %)

  • Compute stabilised inverse-probability weights from your propensity model.
  • Assess covariate balance under the weights (weighted table + love plot).
  • Fit a weighted regression of engagement on notification_sent to estimate the ATE, and compare it to Steps 1–2. If the matching ATT and the weighting ATE differ, explain why (think about who the treated users are).

Step 4 — Doubly-Robust / AIPW Estimation (20 %)

  • Fit both an outcome model and a propensity model for notification_sent.
  • By hand, compute the AIPW doubly-robust pseudo-outcome \[\phi_i = \big(\hat m_1(X_i) - \hat m_0(X_i)\big) + \frac{D_i\,(Y_i - \hat m_1(X_i))}{\hat e(X_i)} - \frac{(1-D_i)\,(Y_i - \hat m_0(X_i))}{1 - \hat e(X_i)},\] and average it for the ATE. Wrap the estimator in a function and bootstrap (≥ 1000 resamples) for the standard error, \(t\)-statistic, and \(p\)-value.
  • Explain the double-robustness property and compare your AIPW estimate to Steps 1–3.

Step 5 — Sensitivity Analysis: the pivot to a design (10 %)

  • Run a sensitivity analysis (sensemakr and/or an E-value) on your adjusted regression of sending. Benchmark the strength of a hypothetical unobserved confounder against your strongest observed covariate (past_engagement); report the robustness value.
  • Then make the pivot explicit: argue why, for the opening decision, no amount of adjustment can identify the effect — i.e. name the plausible unobserved confounder and explain why it defeats Steps 1–4. This motivates the instrumental-variable design in Step 6.

Step 6 — Instrumental-Variable Estimation (20 %)

Now switch the treatment to notification_opened, which is confounded by unobserved motivation.

  • Show that the naive (even covariate-adjusted) OLS of engagement on notification_opened is biased.
  • Compute the LATE by hand as \(\text{ITT} / \text{first stage}\) (the Wald ratio), ignoring covariates. Then run 2SLS — without and with covariates — using pop_up_rem as the instrument \(Z\).
  • Assess instrument strength (report the first-stage / effective / Kleibergen–Paap \(F\)) and the complier share.
  • State the four IV assumptions (relevance, exclusion, independence, monotonicity), say which is testable, and argue the credibility of each in this setting. Explain precisely whose effect your 2SLS estimate identifies (a LATE on compliers — not the ATE).

Step 7 — Executive Summary & Estimand Reconciliation (10 %)

Write a concise, decision-relevant summary for FitLife management:

  • The two business questions, the methods you used, and the headline numbers.
  • An explicit reconciliation: contrast the effect of SENDING (your ATE/ATT from Steps 1–4) with the LATE of OPENING among compliers (Step 6). Explain why they answer different questions and must not be collapsed into a single number.
  • A clear recommendation for FitLife’s notification strategy.

Optional bonus (+5, total capped at 100): A fuzzy regression-discontinuity extension. FitLife also auto-shows the pop-up once a user’s weekly activity score crosses a threshold. Using fitlife_a3_rdd.csv (columns weekly_activity, notification_opened, engagement; cutoff at 50), treat \(\mathbb{1}\{\text{weekly\_activity} \ge 50\}\) as an instrument and estimate the fuzzy-RDD LATE with rdrobust. Compare it to your Step-6 encouragement LATE and discuss why both are local.

Rubric

Criterion Weight Excellent (1) Good (2) Needs Improvement (3) Missing/Inadequate (4–5)
Conditional outcome regression (Step 1) 10% Naive and covariate-adjusted OLS both reported with HC1 SEs; the confounding gap interpreted; correctly framed as the total effect of sending. Both estimates present; interpretation thin. One estimate missing, wrong adjustment set, or no interpretation. No regression.
Matching: PS vs covariate space (Step 2) 15% PSM and a covariate-space matcher (Mahalanobis and/or CEM); before/after balance (table + love plot) focused on past_engagement; ATT with SE for each; a correct read of the comparison — whether covariate-space matching balances better than PSM, CEM’s pruning/estimand tradeoff, tied to King–Nielsen and the pivot to weighting. PSM + one covariate-space matcher + balance + ATT, but the comparison interpretation is thin. Only PSM, or no balance comparison, or ATT and ATE conflated. No matching.
Inverse probability weighting (Step 3) 15% Stabilized IPW targeting the ATE; near-zero weighted SMDs shown; weighted regression for the estimate; lands on the ATE. IPW + estimate, but the balance-under-weights check is missing or weights are not stabilized. Unstabilized/extreme weights with no diagnostic, or wrong estimand. No weighting analysis.
Doubly-robust / AIPW (Step 4) 20% Correct AIPW augmentation term; both nuisance models refit inside a ≥ 1000-rep bootstrap; OLS/IPW/AIPW three-way agreement noted; double robustness explained. AIPW point estimate correct, but the bootstrap is under-specified (models not refit) or the DR property is not explained. AIPW formula wrong, or no valid inference. No AIPW.
Sensitivity analysis (Step 5) 10% Robustness value + benchmark against past_engagement + E-value, all on the adjusted SEND estimate; the pivot argument made (adjustment defensible for sending, broken for opening). RV/E-value computed, but the benchmark or the send-vs-open pivot is weak. One sensitivity statistic only, applied to the wrong estimate, or no pivot reasoning. No sensitivity analysis.
Instrumental-variable estimation (Step 6) 20% Naive OPEN bias shown; Wald-by-hand = 2SLS-without-covariates; 2SLS with covariates; first-stage \(F\) + complier share; all four IV assumptions stated, the testable one named, and the estimand made explicit as a LATE on compliers. 2SLS + \(F\) + most assumptions, but Wald-by-hand missing, one assumption mis-stated, or LATE-vs-ATE blurred. 2SLS produced but instrument strength ignored, assumptions absent, or the estimand mis-named as the ATE. No IV analysis.
Executive summary & estimand reconciliation (Step 7) 10% Decision-relevant summary with headline numbers; explicit send-vs-open reconciliation explaining why ATE/ATT/LATE answer different questions and must not collapse to one number; clear, actionable recommendation. Summary + recommendation, but the reconciliation is thin or one estimand is conflated. Numbers restated without estimand interpretation or recommendation. No summary.

In addition to step-specific scoring, the general evaluation criteria apply (correctness, code quality, communication clarity, reproducibility, and AI-use disclosure per course policy — see the syllabus for the full table).

Submission

Content

  • A written report of 8–10 pages (selectively displayed code, figures, references) clearly structured per the steps above.
  • Annotated, reproducible code (R or Python). The instructor must be able to open your source file, install the listed packages, and re-render your report end-to-end on a clean machine.

Format

  • Submit one source file plus the rendered PDF. The source file may be either a Quarto Markdown (.qmd) or a Jupyter notebook (.ipynb) — both are accepted, in either R or Python.
  • File names: [YourLastName]_Assignment_3.{qmd|ipynb} and [YourLastName]_Assignment_3.pdf.
  • Bundle the two files into a single zip: [YourLastName]_Assignment_3.zip.
  • Do not include fitlife_a3.csv in the zip (nor fitlife_a3_rdd.csv, if you attempt the bonus). When grading, the instructor will place the provided CSV(s) in the same folder as your source file before re-rendering. Your code should therefore read each file with a plain relative path (read_csv("fitlife_a3.csv")), not a path that traverses parent directories.
  • If your code reads from any additional external file you created (e.g., a saved matching output, an intermediate weight table), bundle it alongside the source file in the zip — but never the provided FitLife datasets.

Deadline

  • Sunday, 14 June 2026, 23:59 (Hamburg local time).
  • Upload to the StudIP folder (link in the course shell).

Quiz

  • A 15-minute closed-book paper quiz on this material at the start of the Monday, 15 June 2026 lecture (Session 08). The quiz covers the observed-confounding toolkit (matching, IPW, AIPW / double robustness, sensitivity analysis), the four IV assumptions with 2SLS / LATE and weak-instrument diagnostics, and the send-vs-open estimand reconciliation. See the syllabus for the quiz-gate policy.
WarningIf something breaks

If you cannot complete a step due to a technical problem, submit anyway. Document what you tried, what failed, and what you would have done. Partial credit is awarded generously when the reasoning is shown.