Chapter 3 Randomised Assignment
Having conducted two impact assessments using potentially biased estimators of the counterfactual in chapter 3 (with conflicting policy recommendations), you decide to go back to the drawing board to rethink how to obtain a more precise estimate of the counterfactual. After further deliberations with your evaluation team, you are convinced that constructing a valid estimate of the counterfactual will require identifying a group of villages that are as similar as possible to the 100 treatment villages in all respects, except that one group took part in HISP and the other did not. Because HISP was rolled out as a pilot, and the 100 treatment villages were selected randomly from among all of the rural villages in the country, you note that the treatment villages should, on average, have the same characteristics as the untreated rural villages in the country. The counterfactual can therefore be estimated in a valid way by measuring the health expenditures of eligible households in rural villages that did not take part in the program.
Luckily, at the time of the baseline and follow-up surveys, the survey firm collected data on an additional 100 rural villages that were not offered the program. Those 100 villages were also randomly selected from the population of rural villages in the country. Thus the way that the two groups of villages were chosen ensures that they have statistically identical characteristics, except that the 100 treatment villages received HISP and the 100 comparison villages did not. Randomized assignment of the treatment has occurred.
Given randomized assignment of treatment, you are quite confident that no external factors other than HISP would explain any differences in outcomes between the treatment and comparison villages.
To validate this assumption, test whether eligible households in the treatment and comparison villages have similar characteristics at baseline.
df_elig %>%
filter(round == 0) %>%
dplyr::select(treatment_locality, locality_identifier,
age_hh, age_sp, educ_hh, educ_sp, female_hh, indigenous,
hhsize, dirtfloor, bathroom, land, hospital_distance) %>%
tidyr::pivot_longer(-c("treatment_locality","locality_identifier")) %>%
group_by(name) %>%
do(tidy(lm_robust(value ~ treatment_locality, data = .))) %>%
filter(term == "treatment_locality") %>%
dplyr::select(name, estimate, std.error, p.value) %>%
kable()| name | estimate | std.error | p.value |
|---|---|---|---|
| age_hh | -0.6354625 | 0.3759583 | 0.0910361 |
| age_sp | -0.0386302 | 0.3120790 | 0.9014911 |
| bathroom | 0.0149907 | 0.0132340 | 0.2573724 |
| dirtfloor | -0.0129497 | 0.0118744 | 0.2755159 |
| educ_hh | 0.1607976 | 0.0697576 | 0.0211978 |
| educ_sp | 0.0289107 | 0.0670018 | 0.6661273 |
| female_hh | -0.0041155 | 0.0070493 | 0.5593691 |
| hhsize | 0.0596953 | 0.0530454 | 0.2604833 |
| hospital_distance | 2.9087631 | 1.1323148 | 0.0102288 |
| indigenous | 0.0091048 | 0.0131969 | 0.4902756 |
| land | -0.0402168 | 0.0704607 | 0.5681787 |
You observe that the average characteristics of households in the treatment and comparison villages are in fact very similar. The only statistically significant differences are for the number of years of education of the head of household and distance to hospital, and those differences are small (only 0.16 years, or less than 6 percent of the comparison group’s average years of education, and 2.91 kilometers, or less than 3 percent of the comparison group’s average distance to a hospital). Even with a randomized experiment on a large sample, a small number of differences can be expected because of chance andthe properties of the statistical test. In fact, using standard significance levels of 5 percent we could expect differences in about 5 percent of characteristics to be statistically significant, though we would not expect the magnitude of these differences to be large.
Estimate the average household health expenditures for eligible households in the treatment and comparison villages for each period. What is the impact of the program?
out_round0 <- lm_robust(health_expenditures ~ treatment_locality,
data = df_elig %>% filter(round == 0),
clusters = locality_identifier)
out_round1 <- lm_robust(health_expenditures ~ treatment_locality,
data = df_elig %>% filter(round == 1),
clusters = locality_identifier)
htmlreg(list(out_round0, out_round1), doctype = FALSE,
custom.coef.map = list(`treatment_locality` = "Treatment Village",
`(Intercept)` = "Intercept"),
custom.model.names = c("Baseline", "Follow Up"),
caption = "Health Expenditures by Treatment Locality")| Baseline | Follow Up | |
|---|---|---|
| Treatment Village | -0.08 | -10.14* |
| [-0.51; 0.34] | [-10.93; -9.35] | |
| Intercept | 14.57* | 17.98* |
| [14.26; 14.89] | [ 17.36; 18.60] | |
| R2 | 0.00 | 0.30 |
| Adj. R2 | -0.00 | 0.30 |
| Num. obs. | 5628 | 5629 |
| RMSE | 4.30 | 7.73 |
| N Clusters | 197 | 197 |
| * Null hypothesis value outside the confidence interval. | ||
With the validity of the comparison group established, you can now estimate the counterfactual as the average health expenditures of eligible households in the 100 comparison villages. Table X shows the average household health expenditures for eligible households in the treatment and comparison villages. You note that at baseline, the average household health expenditures in the treatment and comparison groups are not sta- tistically different, as should be expected under randomized assignment.
Given that you now have a valid comparison group, you can find the impact of the HISP simply by taking the difference between the average out-of-pocket health expenditures of households in the treatment villages and randomly assigned comparison villages in the follow-up period. The impact is a reduction of US$10.14 over two years.
Re-estimate using a multivariate regression analysis that controls for the other observable characteristics of the sample households. How does your impact estimate change?
out_round1_nocov <- lm_robust(health_expenditures ~ treatment_locality,
data = df_elig %>% filter(round == 1),
clusters = locality_identifier)
out_round1_wcov <- lm_robust(health_expenditures ~ treatment_locality +
age_hh + age_sp + educ_hh + educ_sp +
female_hh + indigenous + hhsize + dirtfloor +
bathroom + land + hospital_distance,
data = df_elig %>% filter(round == 1),
clusters = locality_identifier)
htmlreg(list(out_round1_nocov, out_round1_wcov), doctype = FALSE,
custom.coef.map = list(`treatment_locality` = "Treatment Village",
`(Intercept)` = "Intercept"),
custom.model.names = c("No Covariate Adjust.", "With Covariate Adjust."),
caption = "Evaluating HISP: Randomized Assignment with Regression Analysis")| No Covariate Adjust. | With Covariate Adjust. | |
|---|---|---|
| Treatment Village | -10.14* | -10.01* |
| [-10.93; -9.35] | [-10.70; -9.32] | |
| Intercept | 17.98* | 27.57* |
| [ 17.36; 18.60] | [ 25.83; 29.30] | |
| R2 | 0.30 | 0.43 |
| Adj. R2 | 0.30 | 0.43 |
| Num. obs. | 5629 | 5629 |
| RMSE | 7.73 | 6.98 |
| N Clusters | 197 | 197 |
| * Null hypothesis value outside the confidence interval. | ||
Replicating this result through a linear regression analysis yields the same result, as shown in Table X. Finally, you run a multivariate regression analysis that controls for some other observable characteristics of the sample house- holds, and you find that the program has reduced the expenditures of the enrolled households by US$10.01 over two years, which is nearly identi- cal to the linear regression result.
With randomized assignment, we can be confident that no factors are present that are systematically different between the treatment and comparison groups that might also explain the difference in health expenditures. Both sets of villages started off with very similar average characteristics and have been exposed to the same set of national policies and programs during the two years of treatment. Thus the only plausible reason that poor households in treatment communities have lower expenditures than households in compari- son villages is that the first group received the health insurance program and the other group did not.
Why is the impact estimate derived using a multivariate linear regression basically unchanged when controlling for other factors, compared to the simple linear regression and comparison of means?
Because the treatment was assigned randomly, the comparison and treatment groups should have identical characteristics and be exposed to the same external factors over time. The only difference between the two groups is that the treatment group received HISP. Because of the randomized assignment process, the characteristics (controls) of the treatment and comparison group are unrelated to treatment status, so controlling for additional characteristics in the multivariate linear regression is not expected to change the estimated impact by much.
Based on the impact estimated with the randomized assignment method, should the HISP be scaled up nationally?
Yes, based on this result, the HISP should be scaled up nationally because it decreased health expenditures by more than the $10 threshold level.