A four-arm randomised trial of savings reminders

Context

Mobile wallets have placed dedicated savings tools directly into the hands of populations previously excluded from the formal banking sector. In many low-literacy, mobile-first markets, however, this feature is activated far more frequently than it is sustained: an account is created at registration, receives an initial deposit, and then falls dormant. Service providers and public financial-inclusion initiatives routinely turn to the most cost-effective intervention available: automated text messages (SMS).

This rationale is well founded. Across a series of field experiments conducted with retail banks in Bolivia, Peru, and the Philippines, Karlan, McConnell, Mullainathan, and Zinman demonstrated that monthly reminders increased overall savings balances, with messages highlighting both an individual's personal goal and the associated financial incentive proving particularly effective (Management Science, 2016; NBER working paper). The underlying psychological mechanism is limited attention: individuals genuinely intend to save, and timely reminders reactivate that intention at moments when liquidity is on hand.

It does not follow, however, that any reminder design will succeed across any market context. Phrasing, delivery channel (text messages offer little utility to users who cannot read), timing relative to cash inflows, and the underlying financial product all vary substantially. Before deploying millions of communications, an implementing organization must determine whether its specific messages change user behavior, and which framing yields the strongest response.

Methodological Pitfalls Addressed by the Protocol

Field trials evaluating reminder interventions frequently suffer from four recurring methodological failures:

1. Outcome metrics that capture courtesy rather than capital accumulation. A respondent who has received twelve consecutive prompt messages will predictably tell an enumerator that they are saving more diligently. Self-reported survey data is systematically biased in favor of the treatment effect. Reliable evaluation demands objective, administrative transaction records.
2. Underpowered multi-arm designs. Every additional message variant introduces a distinct comparison against the control group. In the absence of multiple-testing adjustments, an uncorrected four-arm trial carries approximately a 14% probability (under independence: $1 - 0.95^3$) of identifying at least one null intervention as statistically significant. Applying appropriate multiplicity corrections, however, substantially increases the required sample size per arm.
3. Severely underpowered framing contrasts. The primary operational question raised by practitioners—"does the goal-framed message outperform the loss-framed message?"—is typically the exact comparison that the trial is least equipped to detect.
4. Absence of pre-committed decision rules. When empirical results arrive with borderline significance (e.g., one arm achieving $p = 0.04$ and another missing the threshold), debates over scaling strategies typically begin only after stakeholders have inspected the estimates, introducing substantial confirmation bias.

Approach

This protocol is structured as a formal pre-analysis plan (available for download above) intended for public registry prior to field randomization. Key design specifications include:

Treatment Arms. The experimental design evaluates four distinct arms:

• Control: No reminder messages delivered;
• Plain Reminder: A neutral prompt to deposit funds;
• Goal Frame: A reminder referencing the individual account holder's personal savings target;
• Loss Frame: A reminder emphasizing the cost of inaction ("each week without a deposit puts your goal further away").

The loss-framed variant leverages the behavioral principle of loss aversion, which posits that objectively equivalent outcomes impose greater psychological weight when framed as losses rather than forgone gains (Tversky & Kahneman, Science, 1981). All communications are held constant in length, schedule (bi-weekly over six months), and sender identity. Messages are delivered as short voice notes to subscribers who selected voice notifications during account setup.

Unit of Randomization and Stratification. Randomization occurs at the level of the individual account (one unique phone number), allocated in equal proportions across four strata: gender $\times$ whether the account logged any savings deposit during the preceding three months. Stratifying on an established baseline predictor of the outcome variable enhances statistical precision and guarantees balance across treatment arms (Bruhn & McKenzie, 2009).

Primary Endpoint. Regular Saver Status, defined as making at least one voluntary deposit into the savings wallet during at least three of the six post-randomization months, derived directly from administrative transaction logs. Secondary endpoints comprise cumulative deposit frequency, sixth-month balance (winsorized at the 99th percentile to mitigate leverage from extreme outliers), behavioral persistence across months 7–12, and net withdrawal volumes. Message opt-outs (STOP requests) and user grievances are monitored continuously as operational safeguards.

Econometric Specification. Estimation follows an intention-to-treat (ITT) framework estimated via ordinary least squares (OLS) with stratum fixed effects, a pre-specified baseline control variable, and heteroskedasticity-robust (HC2) standard errors:

$$Y_i = \alpha + \sum_{k=1}^{3} \tau_k T_{ki} + \sum_s \gamma_s S_{si} + \theta\, Y_i^{\text{pre}} + \varepsilon_i$$

where $T_{ki}$ represents an indicator for assignment to treatment arm $k$, $S_{si}$ denotes stratum fixed effects, and $Y_i^{\text{pre}}$ controls for baseline savings activity. The three primary confirmatory hypotheses ($\tau_k = 0$) are evaluated using Holm’s step-down procedure at a family-wise error rate of $\alpha = 0.05$ (Holm, 1979). Secondary outcomes are evaluated using Benjamini–Hochberg false discovery rate (FDR) q-values (1995).

Sample Size and Statistical Power

For a binary primary endpoint, the required sample size per treatment arm compared against the control condition at corrected planning level $\alpha^\ast$ is given by:

$$n_t = \left\lceil \frac{(z_{1-\alpha^\ast/2} + z_{1-\beta})^2\,\big[p_1(1-p_1) + p_0(1-p_0)/r\big]}{\delta^2}\cdot\frac{1+(m-1)\rho}{1-a} \right\rceil$$

where $p_1 = p_0 + \delta$, $r$ represents the control-to-treatment allocation ratio, $m$ and $\rho$ denote cluster size and intra-cluster correlation respectively (here $m = 1$), and $a$ represents the anticipated rate of unmatchable administrative records at endline. Because Holm’s initial step evaluates against an adjusted alpha of $\alpha/K$ and subsequent steps are strictly less conservative, calibrating sample size to $\alpha^\ast = \alpha/K$ guarantees at least the target statistical power across all pairwise comparisons.

Assuming a 30% baseline rate, a minimum detectable effect (MDE) of 5 percentage points, $K = 3$ pairwise comparisons, 80% statistical power, and 5% data linkage attrition, the design requires 1,929 accounts per arm, or 7,716 accounts in total. The table below illustrates the sensitivity of required sample sizes across alternative baseline rates and effect sizes.

Total Sample Size ($N$, 4 arms)	MDE: 3 pts	MDE: 4 pts	MDE: 5 pts	MDE: 7 pts
Baseline: 20%	16,512	9,436	6,128	3,216
Baseline: 30%	21,116	11,968	7,716	3,988
Baseline: 40%	23,760	13,404	8,596	4,400

Two analytical results from the companion notebook illustrate the core trade-offs:

1. Without multiple-testing adjustments, the trial would appear to require only 5,784 participants—highlighting both the superficial appeal of uncorrected designs and their tendency to inflate false-positive rates.
2. Detecting a subtle 2-percentage-point difference between the goal-framed and loss-framed messages would demand approximately 12,700 participants per arm (6.6 times the planned study size). The pre-analysis plan acknowledges this constraint explicitly, formally relegating framing contrasts to secondary, exploratory status.

Interactive Planning Tool and Simulation Checks

The interactive planning tool implements the analytical equations detailed above. Users can adjust baseline rates, target effect sizes, the number of experimental arms, multiple-testing corrections, clustering structures, attrition, and allocation ratios to evaluate required sample sizes, design effects, and statistical power curves. Toggling to "Minimum Detectable Effect" reveals what a fixed operating budget can detect: with 1,200 accounts per arm, the design achieves an MDE of 6.4 percentage points.

The companion computational notebook evaluates these analytical formulas against simulated trials. Drawing on 2,000 synthetic replications analyzed via stratified OLS with HC2 robust standard errors and Holm adjustment:

• Under the benchmark planning scenario (one framing is active, while remaining treatments exert zero effect), simulated power reaches 79.3% (against an 80.0% analytical target, well within standard Monte Carlo margin of error).
• Under the sharp null hypothesis of zero treatment effect across all arms, the family-wise false-positive rate remains controlled at 4.6%.
• When all three reminder variants exert true effects, Holm-adjusted power per comparison rises to approximately 86%.
• When strata covariates display strong predictive power, model precision increases effective power to 87.8%—treated in this design as an added safety margin rather than justification to reduce recruiting targets.

Pre-Specified Decision Rules

To eliminate post-hoc rationalization once experimental data is unblinded, policy triggers are established a priori:

• Scale Intervention: An arm is approved for full roll-out if it achieves statistical significance under the Holm procedure on the primary endpoint, yields an estimated gain of at least 3 percentage points, and generates an opt-out rate no more than 2 percentage points above the neutral reminder. If multiple variants meet these criteria without statistically distinguishing themselves from one another, the operational default is to scale the plain reminder.
• Discontinue Reminders: The organization ceases message-based savings prompts if no arm achieves statistical significance and all 95% confidence intervals exclude effects above 3 percentage points. Resources are then redirected toward structural product innovations, such as the commitment savings devices evaluated in the discrete choice experiment.
• Inconclusive Findings: If empirical results fall between these thresholds, the intervention is not scaled, and leadership convenes to assess the viability of an adapted replication trial.

Operational thresholds should be customized by programme leadership prior to formal registry, calibrated against unit messaging delivery costs and the net present value of an incremental regular saver.

Ethical Safeguards and Informed Consent

Participants explicitly opt in to informational savings communications upon activating the savings feature, via plain-language digital prompts or voice prompts. Opt-out requests (STOP commands) are processed immediately and tracked as a primary safety metric. The loss-framed message is pretested for potential psychological distress or confusion through a small-scale feasibility test before message copy is finalized.

To protect user confidentiality, mobile wallet operators pseudonymize all unique user identifiers prior to data transmission. The research team accesses only variables pre-specified in the protocol, and published analytical tables suppress cells representing fewer than ten individual accounts. Independent Institutional Review Board (IRB) ethical approval and prospective trial registration must precede message delivery.

Limitations

This design serves as a methodological reference case. Baseline rates, anticipated effect magnitudes, and attrition rates represent calibrated assumptions rather than empirical estimates from an active deployment; an operational trial must replace these placeholders with historical operator records.

Additionally, this trial does not evaluate communication channels cross-sectionally (channel delivery reflects participant preference), preventing direct causal comparisons between voice and SMS efficacy. Furthermore, because outcomes are tracked exclusively within a single mobile wallet product, recorded increases in balances may partly capture liquidity reallocation away from alternative channels (such as informal community savings groups or cash stored at home). Finally, behavioral persistence following the cessation of messaging is monitored over a six-month window only.

Operational Implementation Roadmap

In an active programme setting, the project execution roadmap proceeds through six sequential phases:

1. Validate baseline parameters and record linkage rates using historical provider data;
2. Conduct user acceptance and phrasing feasibility testing across a focus cohort of 12–20 participants;
3. Finalize and publicly register the pre-analysis plan;
4. Execute centralized randomization via an auditable, seeded script archived by the research team;
5. Track weekly message delivery rates, opt-out triggers, and customer grievances;
6. Conduct primary econometric analysis at month six against pre-committed decision rules, followed by longer-term persistence reporting at month twelve.