want to improve your qualitative research? try using representative sampling and working in teams
In the mid-1990s, the federal government launched a high-profile social experiment in five cities to determine if escaping concentrated neighborhood poverty could change the lives of the nation’s poorest families. The program—Moving to Opportunity (MTO)—would rely on housing vouchers as the policy tool to accomplish such change. In the Baltimore MTO site, over 600 families signed up, and they were randomly assigned to one of three groups. Those in the experimental group received modest housing counseling and a housing voucher that could only be used in a low-poverty neighborhood; another group received a voucher with no restrictions; and the control group received no voucher. Over the next ten years, researchers and policymakers would follow these families to determine whether the program worked to improve such outcomes as adult self-sufficiency and children’s educational prospects.
Our team has primarily used qualitative methods to study the Baltimore site for more than ten years, beginning in 2003. Our original goal was to understand the mechanisms behind the experimental results from the MTO survey, as well as the role of family, school, and neighborhood in the lives of poor households. The focus in our book, Coming of Age in the Other America, and in this essay, stems from in-depth interviews we conducted in 2010. We drew a stratified random sample of 200 15-24 year old youth from the final evaluation survey, which had occurred in the previous year. We interviewed 150 out of these 200 youth, for a response rate of 75 percent. Sampling from “shared origins”—the survey of all Baltimore MTO participants—meant that we knew the population our qualitative study applied to (poor African-American families living in Baltimore public housing in the 1990s). But because we did not sample on “shared outcomes,” it also ensured heterogeneity in pathways into adulthood.
To ensure that we gathered consistent information from each youth, interview conversations covered a predetermined set of topics around the transition to adulthood (e.g., education, employment, family formation). However, while questions were phrased in ways that were broad and open-ended, details were revealed through skillful probing during the conversations, not usually via direct queries, such as those used in surveys. Right from the start, we needed to signal to these youth that our interaction would be very different from the MTO survey, which was still fresh in many participants’ minds. To this end, we began each conversation with the question: “Tell me the story of your life.” As each youth’s narrative unfolded, we would follow their lead, probing as areas of interest emerged, rather than following a prescribed order.
Sampling and in-depth interviews allow for a way to get around the thorny problem for qualitative researchers that not all research participants are available and willing to be observed in the course of their daily routines. Given the frequency with which ethnographers have written about men hanging out at a particular street corner, diner, or takeaway (e.g., Streetcorner Society, A Place on the Corner, Tally’s Corner, Slim’s Table), it is probably the case that the method has under-represented their peers who may prefer to spend their leisure time at home with their families or hanging out at the library reading books. By employing standard sampling techniques that allow one to calculate a response rate, and by comparing the demographic characteristics of those sampled to the larger population one is generalizing to, we can at least know something about who is not represented in our study.
Researchers who use qualitative methods usually do not attempt to get representative samples because the depth of the observations they employ often restrict their focus to a few individuals, or a narrow range of activities, or a single venue. Ethnographers often go into the field to study a particular group of people, thus they “sample on the dependent variable” by choosing the outcome they intend to portray. This is why Sudhir Venkatesh focused on the power brokers of the Robert Taylor Homes—the gang leader and tenant council head in American Project. Alice Goffman’s meticulous recounting of the experiences of a few young men—whose lives she documented in her book, On the Run—allowed limited time to focus on the women in their lives, or on the law-abiding members of the community. This is not a criticism; indeed, we applaud the rich portraits of the inner city from these accounts. But if the goal is to illustrate a wide range of a group of people in a community, sampling on the dependent variable is not the way to achieve it. A systematic, in-depth interview study that follows standard sampling procedures to ensure representativeness, or, at the very least heterogeneity, can offer a wider lens on diverse individuals in the community. It should be said that some ethnographies at least partially accomplish this, such as Mary Pattillo’s classic Black Picket Fences, and her more recent book, Black on the Block.
Such an approach can be even more effective if the sample is prospective. We were fortunate to be able to identify our sample in early childhood from a population of children whose parents’ characteristics, collected in a baseline survey, were largely representative of those raising their children in public housing in the mid-1990s. Their shared origins, and not their outcomes, are what got them into our study. This allows us to observe diverse pathways and give equal voice to those youth who are seldom, if ever, to be found on the corner and who spend their time elsewhere—their homes, the library, or a job.
One hallmark of qualitative research—especially ethnography—is that it is often done by a lone researcher who conducts the fieldwork, analysis, and writeup. In contrast, our study was truly a group effort. Over the course of the ten years we were in the field—between 2003 and 2013, over forty individuals were involved—graduate students, undergraduates, postdoctoral researchers, professional staff, and faculty collaborators. Together, we: recruited participants, conducted interviews, and wrote field notes; logged, transcribed, de-identified, coded, and analyzed data; wrote, and edited papers. We also shared chapters of our book for feedback from others on the team, some who knew certain details better than we did.
Teamwork was woven into all aspects of the project. For example, we often interviewed in pairs. This was partly for safety, but also because it was easier for one person to drive while the other navigated, to call a youth to let him know we were on our way, or to begin writing the field notes on the return trip home. But the primary reason is that it improved the quality of the data. We have all watched each other conduct interviews and given feedback on how the other person could have done it better, or what was missed. We debriefed as a team on a weekly (sometimes daily) basis, talking about the day’s interviews, what we noticed, what surprised us, and what patterns we were observing. Was there something in the interview guide that needed updating because of recent events? Did the team understand the terminology someone used when describing her sexual or romantic relationships?
An additional advantage to team research is that it’s much harder to make a mistake when others who read and comment on your work have all been in the same neighborhoods, interviewed others from the same family, and have read the same transcripts. Often, during the course of analyzing and writing, we were challenged by colleagues who questioned whether a given conclusion is warranted. That kind of feedback sent us back to the drawing board more than a few times.
Certainly not every qualitative study warrants a representative sample, nor is random sampling feasible in each study. Exploring heterogeneity in a community does not always need to be the research goal. However, recognizing diverse pathways within a group of people broadens our understanding and translates into policy prescriptions that are a better fit for more people. Goffman’s richly detailed focus on a small group of men and their immediate networks gives us invaluable information about the impact of mass incarceration on everyday life. But it doesn’t offer insight for the majority of young adults in the same neighborhood who don’t get “caught up.” Employing a representative sampling frame makes it less likely that researchers mistake the exceptions for the norm. And while larger-scale team research is more expensive, it makes collecting data from a wider sample possible, as well as increasing the reliability of the findings.
Stefanie DeLuca and Kathryn Edin are in the sociology department at Johns Hopkins University. Susan Clampet-Lundquist is in the sociology department at St. Joseph’s University. They are the authors of Coming of Age in the Other America.
Comments 4
ankitsaini
May 22, 2018Your game knowledge depend on what type of the website actually you are using for it because if you are not getting the good or relevant knowledge then it is not possible to make the highest free spades score in your whole game. And after it we comes on the second and the most important point which is how you apply tricks after read about it.
Alan Perry
September 10, 2019An excellent educational platform which students from all over the world will help you improve your work and grades. Thanks to Essay Review Expert, you can find honest reviews on many student services, like papersowl, that help a lot to make your job easier.
suka chu
November 7, 2019Thank you for providing this interesting content, I love the posts from you. driving directions
Greatt
June 23, 2020Thank you for sharing. I often read educational portals becouse I want to learn more or improve some of my skills. This year I graduated from high school and now I need to prepare for admission. I want to try applying at Florida State University. I know that it’s not easy to enter this university, so now I’m looking for a way to increase my chances. I would like you to share your experience in writing an admission essay, maybe you could give me some tips on how I can write a more impressive essay. This article https://admission-writer.com/blog/writing-fsu-essay has a lot of good recommendations and cool ideas on how I can improve my essay, maybe it will be interesting for you too. It seems to me that choosing the right strategy for writing an essay at such a prestigious university is very difficult, because you are trying to write about what sets you apart from other candidates, but you can never be sure that this is exactly what the selection committee members are looking for in their students.