Causal Effects in Observational Studies

STAT 20: Introduction to Probability and Statistics

Concept Questions

To study the impact of receiving permanent resident status on mental health, we compare answers to a psychiatric survey from people who entered and won the US green card lottery to answers from others who entered but did not win.

What kind of study is this?

1. A randomized trial.
2. A natural experiment.
3. An observational study that requires matching.
4. None of the above.

01:00

This is the first of a series of questions that each ask students to classify a study. You may want to consider going through all of these questions before reviewing answers and then having a broader discussion about all the examples.

For the first question, the best answer is B, since the lottery is purportedly random but the assignment mechanism is not under the researcher’s control. Note that it is important that the comparison is to those who entered the lottery but did not win; a more general comparison to immigrants who did not receive green cards would be more of an observational study.

To study the impact of childhood trauma on later academic performance, we compare GRE scores for students who lost a close family member in an automobile accident before the age of 8 to GRE scores for students who did not lose a close family member before age 8.

What kind of study is this?

1. A randomized trial.
2. A natural experiment.
3. An observational study that requires matching.
4. None of the above.

01:00

The answer is C, since automobile accidents to one’s parents are not assigned randomly. For example, we’d expect children whose parents work in driving-intensive blue-collar occupations such as trucking, driving for rideshare companies, and commercial delivery to be overrepresented in the treated group, and these subjects are also less likely to pursue graduate school at baseline giving that their parents may not have attended college at all.

To study the effectiveness of a blood pressure medication, we enroll 500 patients. We take the blood pressure of all patients before anyone receives medication. We assign the 200 patients with the highest blood pressure readings to get the medication, assigning the others to be controls.

What kind of study is this?

1. A randomized trial.
2. A natural experiment.
3. An observational study that requires matching.
4. None of the above.

01:00

This is a terrible study design, which builds in confounding on an important covariate. Although you could attempt matching here, it would likely be ineffective since it would be very hard to create many close matches on pre-treatment blood pressure. So “none of the above” seems like the most reasonable answer here.

For this study it’s a good idea to talk about what the randomized trial would have looked like – clearly you’d want to pick the 200 treaated subjects by simple random sampling rather than this deterministic rule.

In the next slide, you will see the first few rows of a dataset containing demographic information on California counties. Scroll to see all of the rows.

We are interested in determining whether a difference in median_edu has a causal effect on homeownership using matching. Which county serves as the best counterfactual match to Fresno County?

1. Kern County
2. Alameda County
3. Contra Costa County
4. Shasta County
5. Del Norte County

The challenge in this question is to remember the distinction between covariates, treatment, and response. When forming matched pairs, you want to look for covariates that are the same and treatments that are different, while you should be ignoring the response completely.

The answer is (A), since both covariates (metro and smoking_ban) are identical to those of Fresno County, but the treatment (med_edu) is different.

You may want to mention that this was a final question in a previous semester.

01:00

name	homeownership	median_edu	metro	smoking_ban
Fresno County	55.0	some_college	yes	none
Colusa County	64.4	hs_diploma	no	none
Del Norte County	60.9	hs_diploma	no	none
Alameda County	55.1	some_college	yes	none
Contra Costa County	69.5	some_college	yes	partial
Glenn County	67.5	hs_diploma	no	none
Shasta County	66.0	some_college	yes	none
Kern County	61.4	hs_diploma	yes	none
San Luis Obispo County	61.4	some_college	yes	none

In this table there are nine counties, five with some_college values for median_edu and four with hs_diploma values.

How many counties of each type will remain after we conduct optimal matching on metro and smoking_ban?

1. some_college: 4, hs_diploma: 4.
2. some_college: 5, hs_diploma: 4.
3. some_college: 2, hs_diploma: 2.
4. some_college: 2, hs_diploma: 4.
5. Can’t tell without more information.

01:00

This question is designed to shift students away from thinking about individual matched pairs toward thinking about how matching reshapes an entire dataset. The correct answer is (A) since every county with hs_diploma is matched to a single county with some_college.

This question is also a good jumping off point for a mini-lecture about matching. The county example is not ideal because there are not a lot of close matches and because there are a lot of ties among the distances so the best match is not unique (although Contra Costa County is probably the one that will get dropped since it looks the least like any of the hs_diploma counties).

A better source for material in the mini lecture is the “matching_mini_lecture.docx” file about final exam scores and attending review sessions. Eventually this example might be a good thing to incorporate into the notes.

Which R command correctly performs matching on covariates to measure the impact of median_edu on homeownership?

1. matchit(homeownership ~ median_edu, data = county, method = ‘optimal’, distance = ‘euclidean’)
2. matchit(median_edu ~ homeownership, data = county, method = ‘optimal’, distance = ‘euclidean’)
3. matchit(median_edu ~ metro + smoking_ban, data = county, method = ‘optimal’, distance = ‘euclidean’)
4. matchit(homeownership ~ median_edu + metro + smoking_ban, data = county, method = ‘optimal’, distance = ‘euclidean’)

01:00

The correct answer is (C). This is a good chance to underline the idea that the response variable is ignored completely when creating matched pairs and also to drill students on the syntax “treatment ~ covariates,” which is shared by both matchit and bal.tab.

Assuming that metro and smoking_ban variables are the only ones we have measured, name an unmeasured variable that could introduce confounding between median_edu and homeownership.

02:00

There’s a free response question on PollEV if you want to use it; either I think this works best as a “pair-and-share” group discussion question followed by a brief class discussion (either with groups volunteering answers or you reviewing their submitted short-answers from Poll EV). Some interesting variables to discuss might include median age and whether or not a county contains a large university.