Assignment IV — Targeting Proactive Care

Overview

You are a data scientist at MediHere, a health insurer with several million members. The care-management team runs a proactive adherence-outreach program for members on a chronic medication — a refill reminder plus a copay waiver — meant to keep members on therapy and head off expensive acute events (ER visits, hospitalisations). The program is expensive to run (nurse time is the bottleneck), and management suspects it is not worth treating everyone. They want rigorous evidence on for whom outreach actually lowers 12-month downstream cost, and a deployable targeting rule the care team can apply.

There is a catch baked into the data. Outreach was not randomized: a care-management triage model already enrols members it judges high-risk — the sickest, most frequently hospitalised, polypharmacy members. Those members also have the highest baseline cost regardless of any outreach. So a naive comparison of treated vs. untreated members is badly confounded, and — as you will see in Step 1 — it points the wrong way. This is precisely the selection-on-observables problem the Part-IV tools are built for.

This assignment is the Part-IV capstone: it synthesises Session 08 (Double Machine Learning) — to de-confound the average effect — and Session 09 (Treatment Effect Heterogeneity) — to estimate who benefits and to learn an optimal targeting policy. You will take one observational dataset from a misleading naive number all the way to a validated, audited, deployable rule, and reconcile the three different estimands you compute along the way.

Sign discipline — the outcome is a COST (lower is better)

Unlike the S09 lecture/lab, where the outcome was profit (higher is better) and you “treat where \(\hat\tau > 0\)”, here the outcome downstream_cost_12m is a cost: a beneficial outreach makes the treatment effect negative (it saves money). Steps 1–3 estimate the effect on cost, so a helpful program produces a negative ATE. From Step 4 on, the cleanest move is to define savings \(s_i = -Y_i\) (positive = good) so the standard targeting tools read naturally. If you copy “treat where \(\hat\tau>0\)” from the S09 lab without flipping the sign, you will target exactly the wrong members. And carry the savings scale all the way through: the policy tree in Step 6 must be fed the flipped doubly-robust score \(\psi^{\text{sav}} = -\psi\), not the cost-scale \(\psi\) from Step 2. A held-out value that comes out near-zero or negative is the classic symptom of an un-flipped score — audit it, do not narrate it away. Getting this right is part of what Step 6 assesses.

Dataset & Codebook

Download medihere_adherence.csv (≈ 410 KB, 10,000 members) — save it into the same folder as your source file, then read it with a plain relative path: read_csv("medihere_adherence.csv") (R) or pd.read_csv("medihere_adherence.csv") (Python).

The data are simulated from a calibrated structural model so that a ground-truth answer exists; focus your effort on the methodology, not on hunting for the “real” numbers.

Column	Type	Description
`age`	integer	Member age (40–92)
`sex`	categorical	`Female` / `Male`
`region`	categorical	`North` / `Central` / `South`
`clinical_risk_score`	continuous	Composite morbidity/risk score (HCC-style; higher = sicker)
`num_comorbidities`	integer	Count of chronic conditions (0–8)
`prior_hospitalizations`	integer	Inpatient admissions in the prior 12 months (0–5)
`num_medications`	integer	Distinct active medications — polypharmacy (1–15)
`baseline_adherence`	continuous	Prior-year medication adherence, PDC in [0, 1]
`digital_engagement`	binary	1 if the member uses the insurer’s app/portal
`outreach`	binary	1 if the member was enrolled in proactive outreach (the treatment \(D\))
`downstream_cost_12m`	continuous	Total downstream medical cost over the next 12 months, EUR (the outcome \(Y\) — lower is better)

Starter code

library(tidyverse)
med <- read_csv("medihere_adherence.csv")
glimpse(med)
# treatment D = outreach ; outcome Y = downstream_cost_12m (LOWER is better)
# covariates X = everything except outreach and downstream_cost_12m

import pandas as pd
med = pd.read_csv("medihere_adherence.csv")
med.info()
# treatment D = outreach ; outcome Y = downstream_cost_12m (LOWER is better)

You may submit in R or Python. Report your results in a rendered Quarto document (.qmd → HTML/PDF) or Jupyter notebook. Show your code, your numbers, and your reasoning.

Step 1 — Setup and the confounding trap (10 %)

Frame the problem before estimating anything.

State the estimand — the average effect of outreach on 12-month downstream cost, \(\tau = E[Y(1) - Y(0)]\) — and the two assumptions it rests on: unconfoundedness given \(X\) ([A1]) and overlap ([A2]).
Compute the naive treated-minus-control difference in cost (and an OLS of cost on outreach with no controls). Report it.
Diagnose the trap. Show why the naive number is misleading — compare baseline characteristics (clinical_risk_score, prior_hospitalizations, cost) across the two arms. Explain in words why the triage targeting confounds the comparison, and which kind of adjustment can and cannot fix it.

Step 2 — Honest ATE via Double Machine Learning (20 %)

De-confound the average effect with cross-fitted DML.

Estimate the ATE on cost with at least one estimator built by hand: the partially-linear (PLR) partialling-out estimator (residualise \(Y\) and \(D\) on cross-fitted \(\hat\ell(X), \hat m(X)\), then regress the residuals) and/or the AIPW / interactive (IRM) doubly-robust score \[\psi_i = \big(\hat\mu_1(X_i) - \hat\mu_0(X_i)\big) + \frac{D_i\,(Y_i - \hat\mu_1(X_i))}{\hat e(X_i)} - \frac{(1-D_i)\,(Y_i - \hat\mu_0(X_i))}{1 - \hat e(X_i)}.\] Use flexible ML nuisances (random forests) with K-fold cross-fitting. Keep your per-member AIPW score \(\psi_i\) — you will reuse it in Steps 4 and 6. Report the influence-function standard error and a 95 % CI.
Contrast your DML estimate against (a) the naive number from Step 1 and (b) a linear-controls OLS. The naive, the linear-OLS, and the DML estimates will disagree sharply — explain why (think about the shape of the cost–risk relationship), and why orthogonalisation + sample-splitting are needed (Neyman orthogonality, regularisation bias).
Verify your by-hand estimate against the DoubleML package (R: DoubleML + mlr3/ranger; Python: doubleml + scikit-learn). They should agree up to cross-fitting noise.

Step 3 — Overlap and sensitivity to unobserved confounding (10 %)

DML is only as trustworthy as its assumptions.

Overlap [A2]: plot the estimated propensity \(\hat e(X)\) by treatment arm. Is there a region where one arm is essentially absent? Decide whether trimming is warranted and say what trimming does to the estimand.
Unconfoundedness [A1] is untestable — so quantify the exposure. Run a sensitivity analysis: how strong would an unobserved confounder (e.g., a frailty or functional-status index that triage cannot see) have to be to overturn your estimate? Python: doubleml’s native sensitivity_analysis() → report the robustness value. R has no native API in DoubleML 1.0.2, so pick either equally-accepted route: (i) report an E-value (EValue) computed from your DML point estimate and CI — an analogous “how strong a confounder” summary; or (ii) get the same robustness value by running this one step through Python’s doubleml (e.g. via reticulate).¹ Interpret: is the average effect robust, or fragile?

Step 4 — Estimate the conditional effects (CATEs) (20 %)

Now the per-member effect. Flip to savings \(s_i = -Y_i\) (positive = beneficial) so the targeting tools read naturally (see the sign-discipline warning above).

Estimate \(\hat\tau(x)\) with a causal forest (R: grf::causal_forest; Python: econml.dml.CausalForestDML). This dataset is larger and far noisier than the S09 lab’s profit data, so tune the forest for it — report your num.trees / min.node.size (or n_estimators / min_samples_leaf) and why; as a sanity check, confirm sd(\(\hat\tau\)) is commensurate with a plausible effect range (hundreds of EUR), not an order of magnitude wider.
Build at least one more learner yourself and compare: the DR-learner (regress your Step-2 AIPW score on the full covariate vector — grf::regression_forest / econml.dr.DRLearner) and/or a T-learner (one outcome model per arm). Plot the CATE distribution and at least one covariate-conditional profile (e.g. \(\hat\tau\) against baseline_adherence and against clinical_risk_score), and name the most- and least-responsive subgroups.
Which learner best captures the heterogeneity here, and why? Compare the learners not just on a fit metric but on the mechanism: a causal/DR-learner targets the effect via an orthogonal score, while a T-/S-learner subtracts two fitted outcome surfaces. On a small effect buried in a noisy, high-magnitude outcome, what does that do to the subtraction-based learner’s ranking and to its spread sd(\(\hat\tau\))? Explain in terms of correlated arm errors.

Step 5 — Validate the heterogeneity (15 %)

A wiggly \(\hat\tau(x)\) is not yet evidence — a flexible learner will draw heterogeneity even from noise. Pass three gates:

Real? A best-linear-predictor / calibration test (R: grf::test_calibration + best_linear_projection; Python: regress your AIPW score on the demeaned \(\hat\tau\) for the \(\beta_2\) coefficient). Does it reject “no heterogeneity”?
Worth it? A RATE / AUTOC targeting curve (R: rank_average_treatment_effect; Python: a manual TOC), with a random-priority placebo whose CI should cover zero. Does prioritising by \(\hat\tau\) beat random targeting?
Sensible? CLAN — compare the covariate means of the most- and least-affected \(\hat\tau\) deciles. Do the extremes resolve into business-sensible personas a medical director would recognise?

Step 6 — Build and evaluate a targeting policy (15 %)

Turn \(\hat\tau\) into a deployable rule the care team can apply under fixed nurse capacity.

Fit a depth-2 policy tree on the doubly-robust savings scores (flipped to positive = good) (R: policytree::policy_tree; Python: econml.policy.DRPolicyTree). Mind the sign — the objective is to reduce cost (maximise savings), so confirm your action values are in units of savings (EUR), not cost. Read off the two-question if/else rule.
Estimate the rule’s honest held-out value: fit the policy on one split and evaluate it on a separate held-out split (never in-sample), against treat-all and treat-none baselines. Report the per-member gain over treat-all with a 95 % CI — is the targeted policy distinguishably better than a blanket campaign?
Audit the rule before shipping. Run CLAN on the policy itself (who does it treat vs. withhold?) and a fairness check: is the treated share roughly balanced across sex and region, or does the rule silently ration care by a demographic group?

Step 7 — Reconcile the estimands and write the memo (10 %)

Write a concise, decision-relevant summary for MediHere’s medical director.

An explicit reconciliation of the three numbers you computed: the naive difference, the DML ATE, and the policy value (gain over treat-all). State what question each answers and why they must not be collapsed into one number.
Two honest caveats: the winner’s-curse / overfitting risk of targeting on estimated effects (and why you evaluated the policy out-of-sample), and the unconfoundedness assumption the whole analysis shares (tie this back to your Step-3 robustness value / E-value).
A clear, actionable recommendation: the headline targeting rule, its capacity implication, and what additional data would strengthen the case.

Optional bonus (+5, total capped at 100): Derive and implement the policy-learning objective by hand. The optimal policy maximises \(\frac{1}{n}\sum_i (2\pi(x_i) - 1)\,\psi^{\text{sav}}_i\) — a weighted classification that treats (\(\pi=1\)) where the doubly-robust savings score is positive. Compute this objective for your policy tree’s rule, for treat-all, and for treat-none, and confirm the ordering matches your held-out policy value. Discuss the gap to an infeasible oracle that treats every truly-beneficial member.

Rubric

Criterion	Weight	Excellent (1)	Good (2)	Needs Improvement (3)	Missing/Inadequate (4–5)
Setup & the confounding trap (Step 1)	10%	Estimand and [A1]/[A2] stated; naive difference reported; the confounding diagnosed with an across-arm baseline comparison and explained as triage-targets-the-sick.	Naive estimate + some diagnosis, but the mechanism is thinly argued.	Naive number only, or no diagnosis of why it is biased.	No naive estimate / framing.
Honest ATE via DML (Step 2)	20%	A correct by-hand PLR and/or AIPW with cross-fitting and influence-function SE; contrasted against naive and linear-OLS with a correct explanation of the disagreement; verified against the DoubleML package.	DML point estimate + package check, but the by-hand inference is under-specified or the linear-OLS contrast is unexplained.	One estimator with no cross-fitting/SE, or no contrast/verification.	No DML estimate.
Overlap & sensitivity (Step 3)	10%	Propensity-overlap diagnostic with a trimming decision; a robustness value (`doubleml`, in Python or called from R for this one step) and/or an E-value (`EValue` in R) computed on the ATE and correctly interpreted as robust/fragile.	Overlap + one sensitivity statistic, but interpretation or trimming reasoning is weak.	Only one of overlap / sensitivity, or no interpretation.	Neither overlap nor sensitivity.
Estimate the CATEs (Step 4)	20%	Causal forest and a self-built learner (DR- and/or T-learner); CATE distribution + a covariate-conditional profile; a correct read of which learner wins and why (signal-to-noise / direct-targeting).	Forest + one learner + a plot, but the comparison interpretation is thin.	A single learner, no comparison, or no profile.	No CATE estimation.
Validate the heterogeneity (Step 5)	15%	All three gates — BLP/calibration, RATE/AUTOC with placebo, and CLAN — run and correctly interpreted (real / worth it / sensible).	Two gates done well, or all three but one mis-interpreted (e.g. placebo omitted).	One gate only, or results stated without interpretation.	No validation.
Targeting policy (Step 6)	15%	Depth-2 policy tree with the sign handled correctly (fed \(\psi^{\text{sav}}\)); honest held-out value vs treat-all/treat-none, reported as a per-member gain whose 95 % CI excludes zero (or, if it does not, the sign/effect-size is explicitly audited rather than narrated away); CLAN-on-policy + fairness audit.	Policy tree + held-out value, but the CI, the sign handling, or the audit is missing/weak.	In-sample value only, wrong sign, or no policy rule.	No policy analysis.
Reconcile & memo (Step 7)	10%	Decision-relevant memo with the three estimands explicitly reconciled (different questions, do not collapse); winner’s-curse and unconfoundedness caveats; an actionable recommendation.	Memo + recommendation, but the reconciliation or a caveat is thin.	Numbers restated without estimand interpretation or recommendation.	No summary.

In addition to step-specific scoring, the general evaluation criteria apply (correctness, code quality, communication clarity, reproducibility, and AI-use disclosure per course policy — see the syllabus for the full table).

Submission

Content

A written report of 8–10 pages (selectively displayed code, figures, references) clearly structured per the steps above.
Annotated, reproducible code (R or Python). The instructor must be able to open your source file, install the listed packages, and re-render your report end-to-end on a clean machine.

Format

Submit one source file plus the rendered PDF. The source file may be either a Quarto Markdown (.qmd) or a Jupyter notebook (.ipynb) — both are accepted, in either R or Python.
File names: [YourLastName]_Assignment_4.{qmd|ipynb} and [YourLastName]_Assignment_4.pdf.
Bundle the two files into a single zip: [YourLastName]_Assignment_4.zip.
Do not include medihere_adherence.csv in the zip. When grading, the instructor will place the provided CSV in the same folder as your source file before re-rendering. Your code should therefore read it with a plain relative path (read_csv("medihere_adherence.csv")), not a path that traverses parent directories.
If your code reads from any additional external file you created (e.g., a saved score table), bundle it alongside the source file in the zip — but never the provided dataset.

Deadline

Sunday, 28 June 2026, 23:59 (Hamburg local time).
Upload to the StudIP folder (link in the course shell).

Quiz

A 15-minute closed-book paper quiz on this material at the start of the Monday, 29 June 2026 lecture (Session 10). The quiz covers DML (cross-fitting, Neyman orthogonality, the AIPW score, why linear adjustment can fail), sensitivity / robustness value, CATE estimation and the three validation gates (BLP, RATE/AUTOC, CLAN), and optimal policy learning (policy trees, honest held-out value, and the ATE-vs-CATE-vs-policy-value reconciliation). See the syllabus for the quiz-gate policy.

If something breaks

If you cannot complete a step due to a technical problem, submit anyway. Document what you tried, what failed, and what you would have done. Partial credit is awarded generously when the reasoning is shown.

A GitHub-only package, dml.sensemakr, implements the OVB-for-ML robustness value directly in R, but it refits the DML with its own engine, so its ATE and RV will not match the estimate you computed in Step 2. If you use it, present it as a separate cross-check, not as the sensitivity of your fitted model.↩︎