Measuring Advertising Experience Quality in Streaming:
A Longitudinal Research Framework

Why Advertising Experience Research Matters

Ad-supported streaming is no longer a niche product tier. Netflix's ad-supported plan reached approximately 94 million monthly active users in 2025, with U.S. members averaging around 41 hours of monthly viewing. By 2026, Netflix reported that its ads plan reached more than 250 million global monthly active viewers — confirming that this growth has already arrived, not merely projected. Note: Netflix's public advertising metrics shifted from "monthly active users" to "monthly active viewers" between these reporting periods; the latter reflects household-level viewership and should be interpreted as an indicator of scale rather than a directly comparable subscriber count. The rapid scale of this shift makes advertising experience quality a timely strategic research problem, not just a technical modeling exercise. How platforms balance monetization with member experience will be a defining competitive variable in the next phase of streaming growth.

As streaming platforms shift toward ad-supported models, understanding how advertising experience shapes member satisfaction and retention becomes a strategic research priority. Yet the core mechanism — how ad fatigue accumulates over time and at what point it begins to damage the viewing experience — remains poorly understood in a longitudinal context.

This demonstration builds a transparent, evidence-grounded research framework to answer three questions: (1) How do ad load and personalization independently affect member satisfaction over a 12-month period? (2) Does ad fatigue mediate the effect of ad load on churn intent? (3) Is the relationship between fatigue and satisfaction bidirectional, and if so, which direction dominates?

I generated a synthetic longitudinal panel (3,150 members × 12 months) using parameters calibrated to published research and industry benchmarks, then applied a full measurement and modeling pipeline including Confirmatory Factor Analysis, Generalized Estimating Equations, Weibull Survival Analysis, and Cross-Lagged Panel Modeling.

End-to-End Research Lifecycle

The Netflix Ads Experiences role calls for "expert navigation of the full research lifecycle from foundational and generative exploration to tactical execution." This portfolio is organized along that lifecycle. Each phase answers a distinct research question and feeds the next.

Phase 0 is anchored by my prior published computational qualitative work (NLP/topic modeling on large-scale social media data) — see About section. Phases 1–3 are demonstrated below.

Research Framework & Design

Factorial Structure

The design crosses three levels of ad load (2.5, 4.5, 6.5 min/hr) with three levels of personalization (low, medium, high), yielding 9 experimental conditions with 350 simulated members each. A third factor, ad diversity (creative variety), was varied within conditions to assess its fatigue-mitigating effect.

Longitudinal Dynamics

Unlike cross-sectional designs, this framework models two time-dependent processes that make ad experience fundamentally different from a single-exposure study:

Structural Model

The theoretical backbone integrates three published frameworks:

How Parameters Were Calibrated

Every numerical value in the simulation has an explicit justification. Parameters were either (a) taken directly from published estimates, (b) derived by back-calculation from a defensible target outcome, or (c) stated explicitly as design assumptions. No key parameter was left undocumented; each was sourced, calibrated, or explicitly treated as a modeling assumption subject to sensitivity analysis.

Core Parameter Table

Theoretical Frameworks Integrated

Data Generation & Analysis Pipeline

All code is available below. The full pipeline runs in a single Python environment. Click any block to expand.

# Evidence-Informed Synthetic Longitudinal Data Generator
# Parameters calibrated from published literature — see Parameter Table above

from dataclasses import dataclass
import numpy as np

@dataclass
class EffectProfile:
    # All coefficients in 5-point scale absolute units (not standardized)
    # This ensures between-condition differences are visible in output
    name: str = "base"
    personalization_to_relevance: float = 1.20   # Google/Ipsos 2019: +40% tolerance
    diversity_fatigue_mitigation: float = 0.49   # Back-calc: 40% lifecycle extension
    adload_to_intrusiveness:      float = 0.28   # Li, Edwards & Lee (2002)
    adstock_to_fatigue_increment: float = 0.28   # Back-calc: SS fatigue ≈ 0.48 at 4.5min/hr
    relevance_to_satisfaction:    float = 0.40   # Ducoffe (1996); Kim & Han (2014)
    intrusiveness_to_satisfaction: float = -0.35
    fatigue_to_satisfaction:      float = -0.32
    trust_to_satisfaction:        float = 0.35
    continuance_to_churn_logodds: float = -0.80
    fatigue_to_churn_logodds:     float = 1.20   # Parks Associates 2024: ~2%/mo baseline

def simulate_one_scenario(cfg, ep):
    """
    Core longitudinal data generator.
    Time-varying dynamics:
      [1] Adstock: adstock_t = exposure_t + λ × adstock_{t-1}  [Koyck 1954]
      [2] Fatigue: AR(1) in bounded [0,1] state with variety mitigation
      [3] Latent constructs: generated in absolute 5-pt scale units
      [4] Behavioral outcomes: logistic models calibrated to benchmarks
    """
    fatigue_state = rng.uniform(0.02, 0.06, size=n)  # New subscriber: minimal prior fatigue
    adstock_state = np.zeros(n)

    for month in range(1, cfg.n_months + 1):
        # Adstock update (Koyck carryover)
        adstock_state = ad_exposure + cfg.adstock_lambda * adstock_state
        adstock_norm  = adstock_state / (adstock_state + 180.0)

        # Fatigue update: AR(1) with variety mitigation (Kronrod & Huber 2019)
        fat_increment = (ep.adstock_to_fatigue_increment * adstock_norm
                        * (1 - ep.diversity_fatigue_mitigation * diversity))
        fatigue_state = np.clip(cfg.fatigue_persistence * fatigue_state
                                + fat_increment + noise, 0, 1)

        # Satisfaction structural model (Ducoffe 1996)
        satisfaction_lat = (3.00
            + ep.relevance_to_satisfaction      * (relevance_lat    - 3.0)
            + ep.trust_to_satisfaction          * (trust_lat        - 3.0)
            + ep.intrusiveness_to_satisfaction  * (intrusiveness_lat - 2.5)
            + ep.fatigue_to_satisfaction        * (fatigue_lat      - 2.5)
            + re_satisfaction)

        # Churn intent: logistic, calibrated to ~2%/month (Parks Associates 2024)
        churn_logit = (-4.20
            + ep.continuance_to_churn_logodds * (continuance_lat - 3.0)
            + ep.fatigue_to_churn_logodds     * fatigue_state
            + 0.25 * price_sensitivity)

# ── CFA: Confirmatory Factor Analysis ──────────────────────────────
model_spec = """
Relevance     =~ relevance_1 + relevance_2 + relevance_3
Intrusiveness =~ intrusiveness_1 + intrusiveness_2 + intrusiveness_3
Trust         =~ trust_1 + trust_2 + trust_3
Fatigue       =~ fatigue_1 + fatigue_2 + fatigue_3
Satisfaction  =~ satisfaction_1 + satisfaction_2 + satisfaction_3
Continuance   =~ continuance_1 + continuance_2 + continuance_3
"""
model = semopy.Model(model_spec)
model.fit(df_month6[item_cols])
fit_stats = semopy.calc_stats(model)

# ── GEE: Generalized Estimating Equations ──────────────────────────
# GEE chosen over LMM because ad condition variables are constant
# within members (between-subject design), causing singularity in
# LMM's random effect estimation. GEE with exchangeable correlation
# structure provides valid inference for this design.
from statsmodels.genmod.generalized_estimating_equations import GEE
from statsmodels.genmod.families import Gaussian
from statsmodels.genmod.cov_struct import Exchangeable

gee_m2 = GEE.from_formula(
    "satisfaction_score ~ time_c + load_z + pers_z + fatigue_score",
    groups=df["member_id"], data=df,
    family=Gaussian(), cov_struct=Exchangeable()
).fit()

# ── Cross-Lagged Panel Model ─────────────────────────────────────── 
# Within-person demeaning removes between-subject confounds,
# isolating the lagged temporal relationship.
for col in ["fatigue_score", "satisfaction_score", "fat_lag1", "sat_lag1"]:
    m = df.groupby("member_id")[col].transform("mean")
    df[col + "_dv"] = df[col] - m

path_A = smf.ols("satisfaction_score_dv ~ fat_lag1_dv + sat_lag1_dv + time_c",
                 data=df).fit(cov_type="HC3")
path_B = smf.ols("fatigue_score_dv ~ sat_lag1_dv + fat_lag1_dv + time_c",
                 data=df).fit(cov_type="HC3")

# ── Survival Analysis ────────────────────────────────────────────── 
# Weibull AFT for covariate inference; Kaplan-Meier for visualization.
waf = WeibullAFTFitter(penalizer=0.01)
waf.fit(surv_df, duration_col="duration", event_col="event")

Full source code: ads_experience_synthetic_v2.py (simulation engine) · ads_analysis_pipeline.py (analysis pipeline) · Python 3.12 · statsmodels 0.14.6 · lifelines 0.30.3 · semopy 2.3.11

Confirmatory Factor Analysis

What Each Construct Measures

Model Fit Indices

χ²(120) = 110.67, p = .718. A non-significant χ² indicates good model-data fit. Note on perfect fit indices: CFI = 1.000 and RMSEA = .000 are mathematically inevitable when the data-generating process exactly mirrors the assumed factor structure — as it does here by construction. This is a known limitation of evaluating measurement models on synthetic data. In practice, real survey data from streaming members would introduce correlated residuals (adjacent items sharing method variance), measurement drift over time, and respondent heterogeneity. A realistic expectation for a well-designed real study would be CFI ≈ .95–.98 and RMSEA ≈ .03–.06. The Trust construct's weaker performance (AVE = .387) is the more meaningful finding here — it is not an artifact of the simulation, but reflects genuine multi-dimensionality in how members conceptualize ad credibility.

Factor Loadings & Convergent Validity

AVE = Average Variance Extracted (threshold ≥ .50); CR = Composite Reliability (threshold ≥ .70). Trust falls below both thresholds, indicating its three items do not converge sufficiently on a single factor. This likely reflects the multi-dimensional nature of ad trust (brand trust vs. data privacy trust vs. ad content credibility). A revised instrument would separate these facets.

Discriminant Validity (Fornell-Larcker)

Each diagonal cell = AVE. Off-diagonal cells = squared inter-factor correlation (r²). Discriminant validity holds when the diagonal exceeds all values in the same row and column.

The Satisfaction ↔ Continuance pair fails the criterion (r² = .830 exceeds both AVEs of .712 and .592). This is expected — in consumer research, overall satisfaction and continuance intention are conceptually very close constructs. Practically, they can be treated as a combined "loyalty" dimension. Trust also fails against Satisfaction (r² = .428 > AVE .387), consistent with its convergent validity weakness.

Longitudinal Analysis Results

6.1 — Satisfaction & Fatigue Trajectories

Across all conditions, satisfaction declines monotonically from month 1 to month 12 as ad fatigue accumulates. The effect is most pronounced under high ad load (6.5 min/hr), where average satisfaction drops from 2.84 to 2.20 — a 0.64-point decline on the 5-point scale.

6.2 — GEE Longitudinal Model

Generalized Estimating Equations were used to model repeated satisfaction measurements while accounting for within-member correlation (exchangeable working correlation structure). Three nested models were compared using QIC (Quasi-likelihood under the Independence model Criterion).

M2 Coefficient Table

*** p < .001. β values are unstandardized GEE estimates. Load and Personalization z-scored (SD=2 min/hr and SD=0.25 respectively) for comparability.

6.3 — Factorial Comparison at Month 12

Two-way ANOVA on month-12 satisfaction confirms large independent effects of both ad load (η² = .297) and personalization (η² = .083), with no significant interaction (η² = .000, p = .855). Ad load and personalization operate as additive, independent levers.

The range from worst condition (high load + low personalization: 1.91) to best condition (low load + high personalization: 3.56) spans 1.65 points on the 5-point scale — a substantial effect with meaningful real-world implications for member experience.

6.4 — Survival Analysis: Churn Intent

A Weibull AFT model regressed time-to-first-churn-intent on member-level average fatigue, satisfaction, ad load, personalization, and price sensitivity. Key finding: ad load does not directly predict when members develop churn intent. Fatigue and satisfaction do.

6.5 — Cross-Lagged Panel Model

To test the temporal directionality of the fatigue–satisfaction relationship, I applied a within-person demeaned cross-lagged model. This removes all stable between-person differences (including condition effects) and tests whether changes in one variable precede changes in the other.

Both paths are statistically significant, confirming a bidirectional relationship. However, Path A is approximately 3× stronger than Path B (|−0.098| vs |−0.034|). Within the simulated longitudinal model, fatigue emerges as the dominant temporal antecedent of satisfaction — members who feel more fatigued in month t−1 report meaningfully lower satisfaction in month t, even after controlling for their prior satisfaction level. This pattern would need to be replicated in real member data before treating it as a generalizable finding.

Product Experiment: Creative Rotation Frequency

The longitudinal phase identified a clear product opportunity: creative diversity reduces fatigue accumulation, and fatigue mediates the ad-load-to-churn pathway. But correlational evidence — even causal-ordered evidence — has limits for product decisions. The next question is sharper and more actionable:

This is the kind of question only a controlled experiment can answer. The longitudinal model says "yes, in theory." Product needs to know "yes, on real members, at this magnitude, for these segments, over this timeframe."

7.1 — Proposed Pre-analysis Plan

Note: This is an illustrative pre-analysis plan demonstrating how I would lock in design decisions before running the experiment. It is not registered with an external preregistration repository (e.g., OSF, AsPredicted); in production this plan would be formally registered.

7.2 — Power Analysis (Simulation-Based)

Rather than relying on closed-form approximations that assume independence (which doesn't hold in this longitudinal design), I conducted simulation-based power analysis: generate 100 synthetic trials at each candidate sample size, fit the planned DiD model, and count the proportion of trials that detect the effect at α = 0.05.

Conclusion: 300 members per arm achieves 96% power for the expected effect size. The actual study uses 1,500/arm for comfortable margin and to enable subgroup analyses.

7.3 — Validation Checks

Two checks before touching the outcomes:

Sample Ratio Mismatch (SRM)

The classic A/B test pitfall: silent assignment bugs that cause the arms to be unbalanced. A chi-square test on observed vs. expected assignment proportions: χ² = 0.00, p = 1.00 → PASS. No assignment irregularities.

Baseline Balance (with SMD)

A subtle methodological point worth highlighting: with 4,500 members, traditional F-tests on baseline covariates can show "statistical significance" even when the actual differences are practically negligible. This is a Type I error inflation at large N — the test is too sensitive. Best practice is to report Standardized Mean Differences (SMD) instead, with conventional thresholds: |SMD| < 0.10 indicates negligible imbalance.

All F-tests are "significant" by p-value, but all SMDs are well below the 0.10 threshold for meaningful imbalance. This is the large-sample randomization-check artifact — reporting only F-tests here would be misleading.

7.4 — Primary Analysis: Difference-in-Differences

The DiD design isolates the treatment effect from any pre-existing differences between arms. Coefficient estimates are the period × arm interaction, fitted via GEE with exchangeable working correlation to handle within-member dependence.

7.5 — CUPED Variance Reduction

CUPED (Controlled-experiment Using Pre-Experiment Data) is a variance reduction technique from Microsoft (Deng et al., 2013). The idea: members who are naturally higher on the outcome at baseline will also be higher post-treatment. By adjusting each member's post-treatment outcome for their baseline, we strip out predictable variance and tighten the treatment effect estimate.

7.6 — Heterogeneous Treatment Effects

Average treatment effects can mask important variation across member segments. By splitting members into pre-treatment fatigue tertiles, we can ask: which members benefit most from faster rotation?

All three segments show statistically significant benefit, but the Mid-fatigue segment shows the largest effect (β = −0.136). This is the actionable nuance: low-fatigue members have less room to improve, and high-fatigue members may have other drivers (e.g., long-term subscriber frustration we saw in the survival analysis). The Mid-fatigue segment — members who are starting to feel the load but haven't given up — is the optimal target for this intervention.

7.7 — Secondary Outcomes & FDR Correction

Running multiple outcome tests inflates the Type I error rate. The Benjamini-Hochberg procedure controls the false discovery rate (FDR) — the expected proportion of false positives among rejected nulls — at q < 0.05.

7.8 — Product Recommendations

Translating findings into actionable product strategy is where research either earns trust or doesn't. Based on this experiment, my recommendations to the product team would be:

Summary of Demonstration Findings

All findings below are derived from synthetic data generated under the modeling assumptions documented in §3. They illustrate what the analytic pipeline would reveal under those assumptions, not validated claims about real Netflix members. Each pattern is a testable hypothesis for a future study with real member data, not a generalizable conclusion.

Implications for Streaming Ad Experience Research

For Product & Ads Experience Teams

If the mediation pattern observed in this simulation replicates in real-member data, it would have a direct product implication: optimizing fatigue management may be a more tractable target than ad load reduction, because it can be addressed through product levers (break placement, creative rotation frequency, personalization quality) that don't directly reduce ad inventory or revenue. A 40% reduction in fatigue accumulation through creative diversity — estimated from OTT industry benchmarks — could meaningfully extend the time before members reach dissatisfaction thresholds. This is a hypothesis the framework is designed to test, not a validated effect.

For Measurement Design

Ad fatigue should be measured as a time-varying construct within longitudinal survey designs, not a single cross-sectional measure. The AR(1) dynamics observed here suggest that surveys administered at months 3–4 will underestimate eventual steady-state fatigue, while surveys administered after month 8 will better capture the plateau. Diary study designs or embedded in-app pulse surveys (1–2 items per session) would provide higher temporal resolution than traditional monthly surveys.

For Research Methodology

This demonstration illustrates the value of connecting attitudinal survey data (what members say) with behavioral signals (what they do: drop-off rates, episode completion, churn). The Weibull AFT finding — that attitudinal fatigue predicts churn timing more directly than ad load — would be impossible to detect from behavioral data alone, and invisible from survey data without longitudinal follow-up.

Limitations of This Demonstration

Researcher Profile & Purpose of This Demonstration