Assignment I — Booking Cancellations

Overview

A mid-range hotel chain wants to know whether upselling guests to a meal package at booking time affects the rate at which those bookings are later cancelled. Marketing argues the upsell increases commitment and should reduce cancellations; revenue management worries the higher all-in price causes sticker shock and raises cancellations. Both effects are plausible. The chain has handed you a snapshot of recent reservations and asked: what is the causal effect of the meal upsell on the cancellation rate?

This assignment walks you through the full graphical-causal-inference pipeline on this dataset:

  1. Develop and justify a Directed Acyclic Graph (DAG) for the booking process, including at least one unobserved variable.
  2. Run causal-discovery algorithms (PC and GES) and compare the discovered structure to your hand-drawn DAG.
  3. Apply identification criteria (backdoor, frontdoor, do-calculus) to decide whether and how the causal effect can be estimated, and identify any bad controls (post-treatment colliders, descendants of colliders) that must be excluded from any adjustment set.
  4. Compute four numeric estimates of the Average Treatment Effect (ATE) — naïve, backdoor with observed controls, frontdoor, and at least one deliberate-bad-control variant — and explain why they differ.
  5. Reflect on what your analysis can and cannot conclude.

The dataset is bilingual-ready: students may submit in R or Python. The case is motivated by Bach et al. (2024)’s sensitivity analysis of Booking.com cancellations and the DoubleML hotel-booking case study. The data are simulated from a Structural Causal Model that has been calibrated to look realistic; this is disclosed so that you focus on the methodology rather than on real-world data quirks.

Dataset & Codebook

The dataset contains 5,000 reservations with 11 columns.

Download booking_assignment.csv (≈ 240 KB) — save this file into the same folder as your source file (.qmd or .ipynb). Your code should then read it with a plain relative path: read_csv("booking_assignment.csv") (R) or pd.read_csv("booking_assignment.csv") (Python).

Column Type Description
lead_time integer (days) Days between booking and check-in.
prior_bookings integer (count) Number of prior bookings the guest has made with the chain.
market_segment factor Booking channel: Direct, TA-TO (travel agent / tour operator), Corporate, Online-TA.
country factor Top-5 source country (DEU, GBR, FRA, ESP, ITA) plus Other.
deposit_type factor NoDeposit, Refundable, NonRefund.
meal_upsell binary (0/1) Treatment. 1 = guest accepted half-board / full-board upsell at booking; 0 = bed & breakfast (default).
expected_total_spend continuous (EUR) Forecast total revenue per stay (room + ancillaries).
pre_checkin_engagement integer (count) Number of guest-initiated touches before check-in (app opens, email replies, etc.).
is_canceled binary (0/1) Outcome. 1 = booking was cancelled before check-in; 0 = guest arrived.
upgrade_offered binary (0/1) 1 = hotel sent the guest a free room-upgrade offer pre-arrival; 0 = no offer. The hotel’s offer-decision system uses both the booking’s meal-upsell signal and an internal cancellation-risk score.
upgrade_clicked binary (0/1) 1 = guest clicked the upgrade-offer email; 0 = no click.
WarningImportant — what the dataset does NOT contain

There is no column for guest price-sensitivity in the dataset. Internally, the booking system does not record this. You should suspect that price-sensitivity exists as a hidden trait that influences both whether a guest accepts the meal upsell and whether they later cancel — and you must reason about it explicitly when you build your DAG and apply the identification criteria.

WarningImportant — temporal ordering of the columns

Not all observed columns are safe to control for. Some are recorded before the treatment (lead_time, prior_bookings, etc.) and some are recorded after (mediators, downstream signals). Adjusting for a post-treatment variable can introduce bias rather than reduce it — particularly if the post-treatment variable is a collider (a common child of treatment and outcome) or a descendant of a collider. Step 3 of this assignment asks you to identify which observed columns are safe controls and which are not.

library(tidyverse)
dat <- readr::read_csv("booking_assignment.csv")
glimpse(dat)
import pandas as pd
dat = pd.read_csv("booking_assignment.csv")
print(dat.shape)
dat.head()

Step 1 — Develop a Causal DAG (20%)

Before touching the data, draw the DAG you believe describes the booking process. Use ggdag (R) or networkx (Python) to render it inline in your report.

Required:

  • At least one unobserved variable (latent node), justified from domain reasoning.
  • An explicit story for every edge you draw: cite a course concept, a paper, or a plausible booking-system mechanism. Edges drawn without justification will lose points.
  • A clear distinction between the treatment node, the outcome node, the mediators (if any), the observed confounders, the unobserved confounder(s), and any post-treatment colliders or their descendants.

Hints (not a checklist; think first, then read):

  • The treatment is meal_upsell; the outcome is is_canceled. What plausibly affects both?
  • Are expected_total_spend and pre_checkin_engagement causes of the upsell or consequences of it? What does the codebook tell you?
  • A latent variable is one that exists but is not in the CSV. Name it and motivate it.
  • For each observed column, ask: is it recorded before meal_upsell, between meal_upsell and is_canceled, or after is_canceled? Read the codebook description carefully — the timing matters for which arrows are admissible.
  • Watch for colliders (nodes that have two or more incoming arrows from variables on the \(D\)–\(Y\) paths). They are easy to spot in the DAG once drawn — and dangerous to condition on.

Step 2 — Causal Discovery and Comparison (20%)

Run two discovery algorithms on the observed columns of the dataset:

  1. PC algorithm (constraint-based): R pcalg::pc(), Python causallearn.search.ConstraintBased.PC.
  2. GES algorithm (score-based): R pcalg::ges(), Python causallearn.search.ScoreBased.GES.

Categorical variables (market_segment, country, deposit_type) must be one-hot encoded before either algorithm sees them. Use fastDummies::dummy_cols() (R) or pd.get_dummies() (Python).

Required output:

  • The CPDAG returned by each algorithm (visualised or printed as an edge list).
  • A side-by-side comparison table with three columns: edge in your hand-drawn DAG, edge in PC’s output, edge in GES’s output. Mark agreements and disagreements.
  • A diagnostic discussion (~1 page) on the disagreements:
    • Where do the algorithms agree with your hand-drawn DAG? Why is that reassuring?
    • Where do they disagree? What is the most plausible explanation for each disagreement (algorithm assumptions violated, latent confounder not modelled, sample size, finite-sample noise, etc.)?
    • Crucially: does either algorithm recover your latent variable? If not, what does this tell you about discovery’s limits?
TipWhat we expect

Discovery rarely recovers latent confounders without explicit latent-variable extensions (FCI, RFCI). When the algorithm cannot represent a latent, it typically encodes the residual confounding as a direct edge between the two variables that the latent confounds. Spotting and explaining this signature is part of the assignment.

Step 3 — Identification (20%)

Apply the three identification criteria to your DAG:

3a — Backdoor

  • Use dagitty::adjustmentSets() (R) or pgmpy.inference.CausalInference.get_all_backdoor_adjustment_sets() (Python) to find all minimal sufficient adjustment sets in your DAG.
  • Discuss whether each set is achievable in practice — i.e., are all variables in the set actually in the dataset?

3a-bis — Bad controls (mandatory)

For each observed column in the dataset, classify it as one of:

  1. Pre-treatment confounder — safe to include in the adjustment set.
  2. Mediator on the \(D \to Y\) path — must not be in the backdoor adjustment set; relevant for frontdoor identification.
  3. Post-treatment collider or descendant of a collider — must not be in any adjustment set; conditioning on it introduces spurious dependence.
  4. Other (irrelevant) — neither helps nor hurts.

State your classification in a small table, and verify your answer programmatically using dagitty::isAdjustmentSet() (R) or pgmpy.inference.CausalInference.is_valid_backdoor_adjustment_set() (Python): show that adding the bad-control candidate to your backdoor set returns FALSE.

3b — The frontdoor-forcing sub-question

ImportantMandatory sub-question

Identify a sub-DAG (or your full DAG with the latent variable hidden) where backdoor with observed-only variables FAILS — i.e., no observed-variable subset blocks all backdoor paths. Then derive the frontdoor formula through your chosen mediator(s) for that sub-DAG. Show all three frontdoor conditions and verify them programmatically. State the closed-form expression for \(P(Y \mid do(D))\).

3c — Do-calculus

Encode your DAG as an ADMG (with bidirected edges representing the latent confounder), pass it to dosearch in R, and report the closed-form identifying expression for \(P(\text{is\_canceled} \mid do(\text{meal\_upsell}))\). Python users may rely on R for this step (no Python implementation has feature parity with dosearch for ADMGs); cite y0 as a developing alternative.

Step 4 — Four-Way Numeric ATE (15%)

Estimate the ATE four ways and put the numbers in a table:

Estimator What it computes Required formula / code
Naïve diff-in-means \(\bar Y_{D=1} - \bar Y_{D=0}\) One-line mean(...) - mean(...) (R) or .groupby().mean() (Python).
Backdoor with observed X (good controls only) OLS of is_canceled on meal_upsell + your observed pre-treatment adjustment set lm(is_canceled ~ meal_upsell + ...) (R) / statsmodels.formula.api.ols(...) (Python).
Bad-control variant (deliberately wrong) Same OLS as the good-controls row, but add at least one of the bad-control candidates you identified in 3a-bis to the regression lm(is_canceled ~ meal_upsell + good_controls + <bad_control>). Report the coefficient on meal_upsell.
Frontdoor through your mediator(s) Plug-in estimator from Step 3b Empirical \(P(M \mid D)\), outcome regression \(E[Y \mid M, D]\), then sum over \(m\) and \(d'\) as the formula prescribes.

Required discussion:

  • Which estimator do you trust most and why?
  • If the four numbers differ markedly, what does that tell you about which assumptions are doing the work in each estimator?
  • Compare the bad-control number to the good-control backdoor number. Is the bad-control estimate closer to or further from what you eventually conclude is the right ATE? Interpret in one or two sentences why adding the wrong control changed the answer in the direction it did.
  • If the sign of the estimate flips between estimators, that is a major finding — interpret it carefully. Which causal direction would you report to the hotel chain’s product team?
TipBonus credit (+5%, capped)

Implement the do-calculus expression from Step 3c numerically (i.e., not just symbolically) and produce a fourth point estimate. Compare to the frontdoor estimate from the table.

Step 5 — Reflection (15%)

Write a one-page reflection covering:

  • Strengths and limitations of your analysis. What assumptions are doing the most work? Where are they fragile?
  • What additional data (variables, experiments, external sources) would close the identification gap?
  • Discovery vs. identification. What is the appropriate role of an algorithmic CPDAG in a causal pipeline like this one?
  • AI-use disclosure (per course policy). If you used an LLM to explain a concept, debug code, or draft prose, state where and what you verified independently.

Rubric

Criterion Weight Excellent (1) Good (2) Needs Improvement (3) Missing/Inadequate (4–5)
Dataset & question framing 10% Clear causal question, codebook respected, latent-variable suspicion stated upfront. Clear question, minor gaps. Question vague or codebook ignored. No causal question or wrong column used as treatment.
DAG (Step 1) 20% 8–12 nodes, every edge justified, latent variable stated and motivated. Most edges justified. Several edges unjustified; latent missing or unmotivated. DAG missing or shows misunderstanding of d-separation.
Discovery & comparison (Step 2) 20% Both algorithms run, encoded correctly, full comparison table, lucid diagnostic of disagreements. One algorithm + comparison; diagnostic acceptable. Only one algorithm or no comparison; weak diagnostic. Neither algorithm runs / output ignored.
Identification (Step 3) 20% All three criteria correctly applied; mandatory frontdoor sub-question solved; bad-control candidates correctly classified and verified programmatically (isAdjustmentSet / is_valid_backdoor_adjustment_set returns FALSE for the proposed bad sets). Minor errors in one criterion, or missing frontdoor sub-question, or bad-control classification incomplete. Multiple criteria misapplied; bad-control variants ignored or conflated with good controls. No identification analysis.
Four-way ATE (Step 4) 15% Four numeric estimates presented (naïve, good-controls backdoor, deliberate-bad-control variant, frontdoor) with sign- and magnitude interpretation, including a one- or two-sentence comparison of the bad-control vs. good-control estimate. One estimate missing or weakly interpreted. Estimates produced but not interpreted, or bad-control variant omitted. No numeric estimates.
Reflection (Step 5) 15% Honest, specific, actionable; AI use disclosed. Mostly clear; minor gaps. Generic or missing AI disclosure. Missing reflection.

In addition to step-specific scoring, the general evaluation criteria apply (correctness, code quality, communication clarity, reproducibility — see the syllabus for the full table).

Submission

Content

  • A written report of 8–10 pages (selectively displayed code, figures, references) clearly structured per the steps above.
  • Annotated, reproducible code (R or Python). The instructor must be able to open your source file, install the listed packages, and re-render your report end-to-end on a clean machine.

Format

  • Submit one source file plus the rendered PDF. The source file may be either a Quarto Markdown (.qmd) or a Jupyter notebook (.ipynb) — both are accepted, in either R or Python.
  • File names: [YourLastName]_Assignment_1.{qmd|ipynb} and [YourLastName]_Assignment_1.pdf.
  • Bundle the two files into a single zip: [YourLastName]_Assignment_1.zip.
  • Do not include booking_assignment.csv in the zip. When grading, the instructor will place the CSV in the same folder as your source file before re-rendering. Your code should therefore read the file with a plain relative path (read_csv("booking_assignment.csv")), not a path that traverses parent directories.
  • If your code reads from any additional external file you created (e.g., a hand-built lookup table, a saved discovery output), include those alongside the source file in the zip — but never the provided booking_assignment.csv.

Deadline

  • Sunday, 17 May 2026, 23:59 (Hamburg local time).
  • Upload to the StudIP folder (link in the course shell).

Quiz

  • A 15-minute closed-book quiz on this material at the start of the Monday 18 May 2026 PBL session. Quiz covers DAGs, identification criteria, and the interpretation of your three-way ATE comparison.
WarningIf something breaks

If you cannot complete a step due to a technical problem, submit anyway. Document what you tried, what failed, and what you would have done. Partial credit is awarded generously when the reasoning is shown.