A cluster-randomised field trial through savings groups

Context

Community savings groups—such as village savings and loan associations (VSLAs), record-keeping tontines, or women's collectives affiliated with agricultural cooperatives—represent some of the few financial delivery channels capable of reaching individuals with limited formal education, volatile incomes, and no physical bank branch nearby. This makes them a natural platform for financial-capability programming: groups already convene on a weekly schedule, members share high baseline trust, and administrative ledgers already track individual deposits.

Much of the rigorous evidence on these models comes from cluster-randomised controlled trials, such as the landmark VSLA evaluation in Malawi (Ksoll et al., Journal of Development Economics, 2016).

Consider a programme team seeking to integrate eight short educational modules into routine meetings. Delivered by a trained facilitator, each session centres on an illustrated rule-of-thumb card reinforcing three simple heuristics: separate business revenue from household cash, define a concrete savings goal, and deposit savings on the designated meeting day. This design builds on empirical evidence showing that simple heuristics can change financial practices more effectively than comprehensive accounting curricula among micro-entrepreneurs with lower baseline numeracy (Drexler, Fischer, & Schoar, American Economic Journal: Applied Economics, 2014).

The primary policy question is straightforward: does this heuristic-based intervention produce a statistically measurable increase in member savings?

Why the group is the unit of randomisation

Because sessions are delivered collectively to the entire group, individual-level randomisation is impossible: a control member attending a treated meeting would directly receive the treatment. The savings group must therefore serve as the cluster unit of randomisation. This experimental structure introduces three methodological constraints that must be addressed at the design stage:

1. Statistical power depends primarily on the number of groups, not the number of members. Individual observations within a group are not independent; they share a facilitator and local economic conditions, creating intra-cluster correlation ($\rho$).
2. Standard errors must be clustered at the group level. In the synthetic dataset analysed here, failing to adjust for clustering underestimates standard errors by a factor of 1.77. That distortion is large enough to convert an inconclusive p-value of 0.07 into a spuriously significant result well below 0.01 (Abadie, Athey, Imbens, & Wooldridge, Quarterly Journal of Economics, 2023; Cameron & Miller, Journal of Human Resources, 2015).
3. The risk of accidental sample imbalance is substantially higher. With 40 groups per arm, the standard error of a standardised difference between arms is roughly 0.22. Consequently, a single unconstrained random draw will frequently exhibit notable baseline imbalances across several observable covariates.

From a feasibility test to a powered trial

These two evaluation phases address fundamentally different questions. Skipping the feasibility phase introduces operational vulnerabilities that cause large field trials to fail; conversely, treating a feasibility pilot as an impact study leads programmes to scale up based on statistical noise.

Dimension	Feasibility test	Powered field trial
Core question	Can the intervention be delivered, understood, and reliably measured in this setting?	Does the intervention change savings behavior, and by what margin?
Sample size	12–20 participants across 1–2 groups	80 groups, roughly 1,500 members (in this worked example)
Counterfactual	None, or an observational before-and-after benchmark	Randomised, concurrent control groups
Key metrics	Rule comprehension (teach-back accuracy), module duration, facilitator fidelity, attendance, recruitment and consent rates, ledger legibility and completeness, image capture duration, item non-response	Ledger-recorded savings contributions, mobile-money records (where consented), loan repayment rates, financial capability index
Valid claims	"Members understood rules 1 and 3, but struggled with rule 2; modules took 25 minutes rather than 15; ledgers are fully legible across 90% of pages"	An unbiased average treatment effect estimate paired with a confidence interval
Invalid claims	Any causal claim regarding sustained behavioural change	Out-of-sample claims extending beyond the study population

A feasibility test produces operational decisions rather than effect estimates: clarifying ambiguous guidelines, adjusting session pacing, standardising ledger photography workflows, or removing redundant survey items. It also yields planning inputs (recruitment rates, attendance benchmarks, ledger completeness), but it cannot estimate the intra-cluster correlation or effect size—parameters that a sample of 12–20 people is underpowered to identify.

Transitioning to a full-scale trial is justified only when every pre-specified progression criterion (go / no-go benchmark) is satisfied (for example: at least 80% correct teach-back per rule, at least 70% average attendance, and legible ledger archives for at least 90% of members). The companion piece, A feasibility test is not a small experiment, elaborates on this framework.

Design of the powered trial

Sample design. 80 groups comprising 1,474 total members in the synthetic register (mean cluster size $\bar m = 18.4$ members; coefficient of cluster size variation $\text{CV} = 0.23$), distributed across three administrative regions.

Stratified randomisation. Groups are stratified by region and urban/rural classification (yielding six strata). Within each stratum, half the clusters are assigned to treatment using a logged, reproducible pseudo-random script with a fixed seed. Stratifying on baseline covariates that correlate with the primary outcome enhances precision and guarantees orthogonal balance across those dimensions (Bruhn & McKenzie, American Economic Journal: Applied Economics, 2009).

Rerandomisation under an explicit decision rule. The experimental protocol specifies a transparent decision rule in advance: draw sequential stratified assignments from a seeded pseudo-random stream and accept the first iteration where the maximum absolute standardised difference across seven baseline covariates is below 0.25. A single draw satisfies this condition roughly 19% of the time (an expected wait time of approximately five draws).

Using seed 1, the algorithm accepts the eleventh draw, which achieves a maximum baseline difference of 0.234 (compared to 0.390 in the unconstrained first draw). Rerandomisation under a documented, pre-registered rule remains formally valid provided the downstream statistical inference reflects the constrained assignment mechanism (Morgan & Rubin, Annals of Statistics, 2012). The accompanying notebook implements this by running randomisation-based inference that re-draws allocations under the exact same algorithm.

Primary outcome. The average monthly savings contribution per member across months 4 through 9, extracted directly from physical group ledgers photographed on a regular schedule in both trial arms. Maintaining identical data-collection schedules across treatment and control groups prevents Hawthorne-style observation biases, ensuring that treated groups do not maintain better records simply because a field officer visits them.

Statistical power and sample size. For $J$ groups with average size $\bar m$, treatment proportion $P$, intra-cluster correlation $\rho$, and baseline explanatory power $R^2$, the standardised minimum detectable effect ($\text{MDE}$) is given by:

$\text{MDE} = \left(t_{1-\alpha/2,\,J-2} + t_{1-\beta,\,J-2}\right)\sqrt{\frac{(1-R^2)\,\big[\rho + (1-\rho)/\bar m\big]}{P(1-P)\,J}}$

The design effect incorporates an adjustment for unequal cluster sizes equal to $1 + [(\text{CV}^2+1)\bar m - 1]\rho$ (Eldridge, Ashby, & Kerry, International Journal of Epidemiology, 2006).

Assuming $\rho = 0.10$ and baseline correlation $R^2 = 0.40$, an 80-group sample can detect an effect size of 0.19 standard deviations at 80% statistical power. Expanding cluster size yields rapidly diminishing returns: increasing membership to 1,000 individuals per group would only reduce the MDE to 0.16. Increasing the number of clusters ($J$) remains the primary operational lever for boosting statistical power.

Econometric estimation model (ANCOVA). Analysis is conducted on an intention-to-treat (ITT) basis using an individual-level ANCOVA model with cluster-robust standard errors:

$y_{ij} = \alpha + \tau T_j + \beta\, y^{0}_{ij} + \sum_s \gamma_s S_{sj} + \varepsilon_{ij}$

The model is estimated via Ordinary Least Squares (OLS) with standard errors clustered at the group level and Student-$t$ critical values evaluated at $J-2$ degrees of freedom. In contexts where savings flows exhibit low serial autocorrelation over time, controlling for baseline savings $y^{0}_{ij}$ is substantially more statistically efficient than a difference-in-differences estimator (McKenzie, Journal of Development Economics, 2012). The corresponding randomisation-inference p-value is reported alongside. If the sample size fell below roughly 40 clusters, a wild cluster bootstrap would replace standard cluster-robust asymptotic approximations.

Methodological takeaways from the simulation

The interactive simulation reproduces the synthetic sample of 80 groups. It allows practitioners to assign treatment, inspect balance across seven covariates, and evaluate the efficiency of the rerandomisation algorithm. For instance, attempting to stratify across four binary variables simultaneously generates 23 micro-strata across 80 groups, leaving many micro-strata to be resolved by isolated binary randomisation.

The notebook then simulates endline outcomes where the true effect at full implementation fidelity is 150 units in peri-urban groups and 250 units in rural groups, diluted by actual session delivery rates. Three key lessons emerge:

• A single experimental realisation can fail to detect true effects due to dilution: The point estimate in the simulated trial is 141 units (cluster-robust $\text{SE} = 76.5$; $p = 0.069$; randomisation-inference $p = 0.082$). Incomplete field delivery diluted the true average effect down to 178 units—falling below the pre-specified MDE threshold of 216 units. Under a disciplined decision rule, this outcome is classified as inconclusive rather than ineffective, with implementation fidelity identified as the primary operational constraint.
• Analytic power formulas align closely with empirical simulation: Across 300 simulated trial iterations configured at the analytical MDE ($0.189\text{ SD}$, $\rho = 0.122$, and $R^2 = 0.50$), the complete estimation pipeline rejects the null hypothesis in 81% of cases, validating the theoretical 80% power target.
• Subgroup heterogeneity estimates exhibit high statistical noise: Although the true underlying rural premium is 100 units, the empirical estimate is 120 with an error margin of $\text{SE} = 164$. Similarly, an interaction by participant gender—set to zero by definition in the data-generating process—emerges spuriously as $-186$ ($p = 0.09$). This variability demonstrates why subgroup analyses must be pre-specified in limited numbers and interpreted strictly as exploratory findings.

Implementation fidelity

Implementation fidelity must be audited through empirical monitoring rather than assumed: tracking sessions delivered (facilitator self-reports verified via random supervisory spot-checks), member attendance (group attendance logs), and a five-point adherence checklist per session (card shown, all rules reviewed, teach-back completed, individual goals documented).

In the simulation, treated groups completed a median of seven of the eight scheduled sessions, with an average individual attendance rate of 81%. Dose-response regressions must be interpreted descriptively rather than causally, as groups with higher voluntary participation rates systematically differ along unobserved characteristics.

Operational timeline (illustrative)

Timeline	Operational milestone
Months 1–2	Feasibility pilot across 1–2 groups; refinement of pedagogical materials and facilitation guides; institutional ethics review
Month 3	Comprehensive group listing, baseline ledger photography audit, formal registration of the pre-analysis plan (PAP)
Month 3	Stratified rerandomisation executed under the pre-registered decision rule
Months 4–6	Intervention delivery in treated groups; bi-weekly fidelity monitoring
Months 4–9	Standardised ledger photography across all study arms (primary outcome window)
Months 10–11	Data consolidation, single-blind econometric analysis, final policy reporting

Methodological limitations

All numerical estimates in this worked example reflect a synthetic data-generating process calibrated for training purposes. In a live programme evaluation, the ICC ($\rho$) and baseline predictive power ($R^2$) must be derived from administrative baselines or comparable historical trials rather than from a feasibility pilot.

Furthermore, ledger records capture internal group contributions rather than overall household savings; measured increases may reflect the substitution of funds from informal cash reserves rather than new capital generation. Finally, geographic spillovers between treated and control groups residing in the same community would bias treatment effect estimates toward zero. Field designs must randomise at the village level whenever groups share overlapping members or market connections.

Guidelines for programme implementation

To implement this experimental design effectively, funding agencies and programme leaders should follow a structured progression:

1. Conduct an initial feasibility test to validate field instruments and eliminate operational bottlenecks.
2. Define explicit progression criteria (go / no-go benchmarks) that must be met before committing resources to an impact trial.
3. Establish an administrative baseline and register the pre-analysis plan (PAP) prior to treatment assignment.
4. Execute randomisation using an auditable, reproducible script governed by a pre-specified balance rule.
5. Monitor implementation fidelity continuously, ensuring symmetrical data collection across treatment and control groups.
6. Conduct a single econometric analysis governed strictly by the pre-registered decision protocol.