# Pre-analysis plan (template): randomised test of savings reminders for mobile-wallet users **Worked example by Ahmed Beddi. Not a registered plan.** It describes a hypothetical trial on a hypothetical mobile-money savings product in a low-literacy, mobile-first setting. No institution, provider or programme is involved. Numbers come from the companion notebook `power_savings_reminders.ipynb`; planning assumptions are labelled as such. Version 0.1, 3 November 2026. In a real trial this document would be registered (for example on the AEA RCT Registry, ) before randomisation, and any later change would be dated and justified in an amendment log. --- ## 1. Question and decision **Policy question.** Do short, scheduled reminders increase regular saving among holders of a mobile-wallet savings feature, and does the framing of the reminder matter? **Decision the trial informs.** Whether the provider (or the public programme supporting it) should send reminders to all eligible users, which version to send, or whether to stop investing in reminders and work on product design instead. The decision rule is in section 10 and is fixed now. ## 2. Arms | Arm | Content (illustrative wording, to be pretested) | Schedule | |---|---|---| | C — Control | No message (business as usual) | — | | T1 — Plain reminder | "Reminder: you can add to your savings this week. Dial [code]." | Twice a month, 6 months | | T2 — Reminder + goal | "You are saving for [goal chosen at enrolment]. Adding a little this week keeps you on track." | Same | | T3 — Reminder + loss | "Each week without a deposit puts your [goal] further away. Add a little this week." | Same | All messages are sent in the user's preferred language and, where the user chose it at enrolment, as a short voice message instead of text. Channel (SMS or WhatsApp) follows what the user already uses; it is recorded, not randomised. Messages are identical across arms in length, timing and sender identity, so the contrasts isolate content. Wording is fixed after a feasibility test (comprehension, tone, no distress from the loss frame) and frozen before randomisation. ## 3. Population, eligibility and unit of randomisation * **Eligible:** adult holders of an active wallet with the savings feature switched on, who consented to receive messages, one account per phone number. * **Excluded:** staff and agents; accounts flagged for fraud review; numbers shared by more than one registered account (to limit spillovers, see section 7). * **Unit of randomisation:** the individual account. * **Strata:** sex (from registration records) × baseline activity (any savings deposit in the 3 months before randomisation), 4 strata. Complete randomisation in equal shares to the four arms within each stratum, with a seeded, logged script run by the research team, not by the implementer. ## 4. Outcomes (all from administrative transaction records) Self-reported saving is not used for any confirmatory outcome: respondents in a reminder arm have been told, repeatedly, that saving is expected of them, so self-report would be biased in the direction of the treatment. **Primary outcome** * **Y1 — Regular saver:** 1 if the account made at least one deposit into the savings feature in at least 3 of the 6 calendar months after randomisation, 0 otherwise. Accounts closed during the period are coded from their record (a closed account that stopped depositing is a 0, not a missing value). **Secondary outcomes** (one family, see section 8) * S1 — Number of savings deposits, months 1–6. * S2 — Savings balance at the end of month 6, winsorised at the 99th percentile of the pooled distribution. * S3 — Regular saver in months 7–12 (persistence after reminders stop). * S4 — Net withdrawals from savings, months 1–6 (checks that deposits are not simply recycled). **Safeguard outcomes** (reported, not tested for benefit) * Opt-out rate (user replies STOP or unsubscribes), complaints logged by customer service. * Use of costly short-term credit, if the provider offers it (checks that people do not borrow to save). **Exploratory:** goal attainment among T2 users who set a goal; heterogeneity (section 9). ## 5. Sample size and minimum detectable effect Planning assumptions: control regular-saver rate $p_0 = 0.30$ (to be replaced by the provider's pre-period figure); minimum effect worth detecting $\delta = 5$ percentage points; two-sided family-wise $\alpha = 0.05$ across three contrasts; power 80 %; 5 % of records not linkable at endline; equal allocation. $$n_t^{\text{raw}} = \frac{(z_{1-\alpha/(2K)} + z_{0.8})^2\,[p_1(1-p_1) + p_0(1-p_0)]}{\delta^2},\quad K=3 \;\Rightarrow\; n_t^{\text{raw}} \approx 1{,}832.1,\qquad n_t = \lceil 1{,}832.1 / 0.95 \rceil = 1{,}929 .$$ **Enrol 1,929 per arm, 7,716 in total.** The notebook confirms this by simulating the section 6 model with stratum fixed effects but without the baseline covariate, the conservative case (simulated power 79.3 % for the planning scenario, against 80.0 % analytic; Monte Carlo standard error about 0.9 points). When strata are prognostic, simulated power rises to 87.8 %; the plan does not count on that gain. With a smaller budget of 1,200 per arm the MDE rises to 6.4 points. Framing contrasts (T2 vs T3) are not powered: detecting a 2-point difference between two active arms would need about 12,700 per arm. ## 6. Estimation Intention-to-treat. For outcome $Y_i$ of account $i$: $$Y_i = \alpha + \sum_{k=1}^{3} \tau_k\,T_{ki} + \sum_{s} \gamma_s\,S_{si} + \theta\,Y_i^{\text{pre}} + \varepsilon_i$$ * $T_{ki}$: assignment to arm $k$; $S_{si}$: stratum indicators; $Y_i^{\text{pre}}$: number of months with a savings deposit in the 6 months before randomisation (the only pre-specified covariate besides strata). * Linear probability model for binary outcomes; OLS throughout. Heteroskedasticity-robust HC2 standard errors (individual randomisation, no clustering). * $\tau_k$ is the effect of being *assigned* to arm $k$. Message delivery failures are not excluded. * Robustness (reported, not confirmatory): without $Y^{\text{pre}}$; logit marginal effects; randomisation inference p-values from 10,000 re-draws of the assignment within strata. * Treatment-on-the-treated (effect of receiving at least one message) is estimated only as a secondary instrumental-variables analysis, using assignment as the instrument for delivery. ## 7. Threats and how they are handled **Attrition.** Outcomes come from records, so attrition means records that cannot be linked at endline (ported numbers, merged accounts). We report linkage rates by arm and test the difference. If linkage differs by arm by more than 2 points, we add Lee (2009) bounds for the primary outcome. **Spillovers.** Messages can be shown to or forwarded to household members who are in another arm, which biases contrasts toward zero. One account per phone number limits this; the estimand remains the intention-to-treat effect including any spillover. If the provider can link accounts to households, we would randomise at household level instead and cluster standard errors by household. **Implementation.** Delivery receipts by arm are monitored weekly. If more than 10 % of messages fail in any arm for two consecutive weeks, the implementer pauses and fixes the queue; the pause is logged. **Seasonality.** All arms run over the same calendar period, so seasonality affects levels, not contrasts. ## 8. Multiple hypotheses * **Primary family:** $H_k: \tau_k = 0$ for $k = 1,2,3$ on Y1. Holm (1979) step-down procedure at family-wise $\alpha = 0.05$. Sample size is planned at $\alpha/3$, Holm's first step. * **Secondary family:** S1–S4 for each of the three contrasts (12 tests). Benjamini–Hochberg false-discovery-rate q-values reported alongside unadjusted p-values. * **Framing contrasts** $\tau_2 - \tau_1$, $\tau_3 - \tau_1$, $\tau_2 - \tau_3$ on Y1: reported with confidence intervals, labelled as secondary. ## 9. Heterogeneity (pre-specified, exploratory) Interactions of each arm with: sex; baseline activity (active vs dormant); preferred channel (voice vs text). No other subgroup is reported as a finding. ## 10. Decision rule Fixed before data are seen. "Significant" means Holm-adjusted p < 0.05 on the primary outcome. 1. **Scale a reminder** if at least one arm is significant, its point estimate is at least 3 points, and its opt-out rate is not more than 2 points above the plain reminder. If several arms qualify and the framing contrasts are not significant, scale the plain reminder (simplest wording, least risk). 2. **Stop reminders as a savings lever** if no arm is significant and the upper bound of every 95 % confidence interval is below 3 points. Effort moves to product features (for example commitment options) instead. 3. **Inconclusive** otherwise: report as such, do not scale; decide on a replication with the provider's actual pre-period base rate. 4. **Safeguard override:** any arm whose opt-out or complaint rate exceeds control by more than 5 points is not scaled, whatever its effect. The thresholds (3 points; 2 and 5 points for safeguards) are placeholders that the programme owner sets from message cost and the value of an additional regular saver before registration. ## 11. Ethics, consent and data * Ethics review by a competent research ethics committee before any message is sent. * **Consent:** users opt in to savings messages when they switch on the savings feature, in plain language and by voice for users who choose voice; the consent text says messages may vary and that anonymised transaction data are used to improve the service. STOP is honoured immediately and is itself an outcome. * The loss frame is pretested for distress; no message mentions debt, shame or other users. * **Data:** the provider pseudonymises account identifiers before extraction; the research team receives only the variables listed in sections 3–4, under a data-sharing agreement; no data leave the provider's environment without de-identification; small cells (fewer than 10 accounts) are suppressed in any published table. * Control users receive the best-performing message after the trial if the rule in section 10 says to scale. ## 12. Timeline (illustrative) | Weeks | Step | |---|---| | 1–4 | Feasibility test of message wording (12–20 users, comprehension and tone only); data-sharing agreement; ethics review | | 5 | Registration of this plan; baseline extraction; randomisation | | 6–31 | Reminders (6 months); weekly delivery monitoring | | 32–57 | Persistence window (months 7–12) | | 36 | Primary analysis (month-6 outcomes) and decision | | 60 | Persistence analysis (S3) and final report | ## References * Karlan, D., McConnell, M., Mullainathan, S. and Zinman, J. (2016). "Getting to the Top of Mind: How Reminders Increase Saving." *Management Science* 62(12): 3393–3411. Working paper: * Holm, S. (1979). "A Simple Sequentially Rejective Multiple Test Procedure." *Scandinavian Journal of Statistics* 6(2): 65–70. * Benjamini, Y. and Hochberg, Y. (1995). "Controlling the False Discovery Rate." *Journal of the Royal Statistical Society B* 57(1): 289–300. * Lee, D. S. (2009). "Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects." *Review of Economic Studies* 76(3): 1071–1102. Working paper: * Bruhn, M. and McKenzie, D. (2009). "In Pursuit of Balance: Randomization in Practice in Development Field Experiments." *American Economic Journal: Applied Economics* 1(4): 200–232.