A programme team designs a new savings reminder, an illustrated budgeting card, or a simplified account-opening form. Before committing a significant budget, an intuitive proposal emerges: run "a small pilot" with fifteen people to see how it works. Three weeks later, the internal report notes that eleven of the fifteen participants saved more after the intervention. The steering committee immediately begins planning a national roll-out.
That report has answered a question the study was structurally unequipped to address. Testing with fifteen people was genuinely valuable—but for an entirely different purpose.
Two studies, two distinct questions
A feasibility test asks: can this intervention be delivered, understood, and reliably measured in this specific operational context?
An adequately powered experiment asks: does it change behaviour, and by what measurable margin?
The first is an essential prerequisite for the second. It is in no way a miniature version of it.
This distinction is well established in clinical trial methodology. The CONSORT extension for randomised pilot and feasibility trials requires authors to formulate explicit feasibility objectives and interpret findings exclusively against those benchmarks—never as preliminary estimates of intervention effectiveness (Eldridge et al., BMJ, 2016). Leon, Davis, and Kraemer stated the principle directly: a pilot study is not designed for hypothesis testing, and its observed effect sizes should never be used to power a definitive trial (Journal of Psychiatric Research, 2011). A widely cited methodological framework categorises the legitimate aims of a pilot: testing operational processes, estimating resource requirements, assessing management feasibility, and confirming scientific prerequisites such as outcome measurability (Thabane et al., BMC Medical Research Methodology, 2010). In public policy and development programmes, evaluation teams rarely enforce this same discipline.
Why fifteen people cannot demonstrate an effect
Consider the most optimistic scenario: twenty participants, randomly split into two groups of ten, evaluating a binary outcome with a baseline rate near 50%. The standard error of the difference between the two groups is approximately:
The resulting 95% confidence interval spans roughly ±44 percentage points. An entirely ineffective intervention and one that doubles the savings rate are both fully consistent with virtually any observed outcome. Without a concurrent control group—the case in most informal pilots—there is no causal estimate at all: the before-and-after change conflates the intervention with seasonal cash-flow variations, the novelty of being observed, natural regression to the mean after an atypical month, and social desirability bias (participants telling field staff what they believe the team wants to hear).
The arithmetic of a credible impact evaluation operates on a completely different scale. In the worked example of a four-arm reminder trial, detecting a 5-percentage-point increase from a 30% baseline—evaluating three treatment arms against a control group with adjustments for multiple comparisons—requires 1,929 participants per arm. In a cluster randomized trial across community savings groups, 80 distinct groups are required to detect an effect size of approximately 0.19 standard deviations. Sample sizes of this scale cannot be approximated by simply expanding a pilot around the margins.
Small samples also generate the most misleading effect sizes. When a tiny pilot randomly registers an artificially large effect and is subsequently used to parameterize the main evaluation, the resulting trial will be severely underpowered for the true, more modest effect. Conversely, when a small pilot shows no statistically visible movement amidst the noise, a high-potential intervention risks being abandoned prematurely.
What a feasibility test should measure
A feasibility test provides value by auditing the operational foundations that a large-scale evaluation depends on and cannot correct post-launch. A robust pre-trial audit checklist should assess:
- Comprehension: Can participants explain, in their own words, what the message, educational card, or form directs them to do? Teach-back must be scored against a structured rubric rather than judged by subjective impression.
- Usability: Where do users pause, misread instructions, or abandon the process? Nielsen’s foundational usability research indicates that five users typically uncover the vast majority of interface problems within a given user segment (Nielsen, 2000). Testing with twelve to twenty participants adequately covers two or three distinct target profiles, such as digitally literate users and individuals with limited literacy.
- Delivery and fidelity: Was the session executed as designed and within the scheduled duration? Were automated SMS messages successfully delivered? What was the actual unit cost per participant?
- Recruitment and consent: What proportion of approached individuals agreed to enroll, and what specific reasons did non-participants give? Was the informed consent protocol genuinely understood?
- Measurement integrity: Can the primary outcome be captured objectively? Can administrative transaction logs be linked to participants without ambiguity? Are physical savings group ledgers legible and consistently maintained? How long does each survey section take, and do specific questions yield high non-response rates or zero variance?
- Acceptability and unintended friction: Did any part of the intervention cause psychological discomfort or customer friction? A loss-framed notification, for instance, should be evaluated for unintended distress on a small group before being broadcast to tens of thousands of clients.
Each dimension must be tied to an explicit, pre-specified progression criterion (go / no-go threshold)—for example: at least 80% correct teach-back on all core rules; at least 70% session attendance; and verified transaction matching for at least 90% of participants. The deliverable of a feasibility study is an operational decision (proceed, modify protocol, or stop), never a premature impact estimate.
Self-reported data: the bias a feasibility test must expose
In informal pilot tests, the default outcome measure is often a single post-intervention survey question: "Have you saved more since the session?" This is the most fragile metric available. Beneficiaries who have just received training on the value of savings from a visitor who clearly hopes the project succeeded have every incentive to answer affirmatively. Moreover, the simple act of surveying participants can itself alter subsequent behaviours and contaminate downstream measurements (Zwane et al., PNAS, 2011).
A rigorous feasibility test manages this challenge through three deliberate practices:
- It strictly refrains from treating self-reported changes as evidence of impact.
- It verifies the availability of objective measurement pipelines for the eventual trial: administrative core-banking records, time-stamped ledger photography protocols, or agent cash-in receipts.
- It evaluates the measurement instrument itself when self-reporting is unavoidable (e.g., measuring beliefs, intentions, or intra-household constraints): are survey questions interpreted consistently across local languages? Does an independent enumerator elicit different patterns than the programme facilitator? And do the responses exhibit meaningful variance across respondents?
When to transition to a powered trial
A programme should transition to a full-scale impact trial only when four cumulative conditions are satisfied:
- Every progression criterion is met, or any observed deficit has an operational remedy that does not require re-testing.
- The primary outcome can be tracked objectively, identically, and without differential bias across all trial arms.
- Empirical planning parameters are established: the operational inputs a feasibility test can provide (recruitment rates, attendance, record linkage success, unit costs) are documented. Meanwhile, parameters it cannot reliably estimate (baseline prevalence, population variance, intra-cluster correlation) are derived from administrative ledgers, historical datasets, or a dedicated baseline survey—never from the pilot sample.
- The institutional decision is formalised: the policy or product decision that the evaluation will inform is documented in writing, along with the specific effect thresholds that would trigger action.
At that stage, the research team can register a formal pre-analysis plan, complete a verifiable power calculation, and specify transparent decision rules. If multiple intervention variants remain under consideration, a discrete choice experiment can systematically narrow a wide range of potential product designs down to the two or three versions that warrant field testing.
The institutional imperative
Treating a feasibility pilot as proof of impact is rarely an error of competence; it is an error of project sequencing. A feasibility test often takes place when team members and budgets are already heavily invested, producing preliminary numbers that superficially resemble positive results.
The solution is institutional discipline: clearly define the purpose of the study at the outset, specify what claims it can and cannot make, and lock in the progression thresholds before anyone examines the field data. A feasibility test that concludes with: "the instructional card requires an additional visual prompt, ledger digitization requires a structured photography protocol, and 13 of 15 participants accurately understood the withdrawal rules" has fulfilled its role effectively.