Methodology & Sample

The methodology of the NSP evaluation is structured around a comparison, over a two- to three-year period, of changes in outcomes of interest between treatment and control villages. Villages are to be surveyed multiple times over a three-year period, beginning with the baseline survey conducted prior to NSP activities, followed by the first follow-up survey after partial completion of program activities, and concluded by a second follow-up survey after the completion of NSP-funded projects in treatment villages, but before the initiation of NSP activities in control villages. The evaluation is being led by Andrew Beath of Harvard University; Professor Fotini Christia of M.I.T.; Shahim Kabuli of the World Bank; and Professor Ruben Enikolopov of the New Economic School, and is being implemented in conjunction with the Vulnerability Analysis Unit (VAU).

The 500 villages included in the evaluation are located across ten districts in Balkh, Baghlan, Daykundi, Ghor, Herat, and Nangarhar provinces. The ten districts were selected based on size, security conditions, the consent of the assigned FP, and the constraint that no villages in the district had previously received NSP activities. The ten districts provide what is considered to be a representative sample of Afghanistan’s geographic, ethnic, and economic diversity, although security conditions have precluded the inclusion of southern provinces in the evaluation. Seven Facilitating Partners (FPs) are contracted to implement NSP the ten evaluation districts, including two local FPs, as well as major international NGOs such as IRC and Oxfam.

Within each of the 10 evaluation districts, 50 ‘evaluation villages’ were selected by the assigned FP, with the understanding that 25 of the 50 villages would be randomly selected for participation in NSP and that the remaining 25 villages would form the control group and not receive NSP until following the completion of the evaluation. The evaluation team then used existing data to form 25 ‘matched village pairs’, grouping villages with similar pre-treatment characteristics, and employed a computer algorithm to then randomly select one of each matched village pair to receive NSP. As such, the evaluation employs the ‘matched-pair cluster randomization’ design developed by Professor Gary King of Harvard University.

Estimates of the impact of NSP will be based upon a comparison of changes in outcomes of interest from the baseline and the follow-up surveys between the treatment and control groups. As the 500 villages in the evaluation sample were randomly assigned to either the treatment or control groups, the pre-NSP characteristics of villages selected to receive NSP are, on average, statistically indistinguishable from outcomes of interest in those villages not selected to not receive NSP. Accordingly, should any differences in the average level of outcomes of interest arise between the 250 treatment villages and the 250 control villages, it can be assured that those differences reflect the impact of NSP and not any differences in starting conditions between the treatment and control villages.