Poking, Prodding, Prying and the Failure to Saturate
I recently read an article about the experiences and expectations of women living with fistula in sub Saharan Africa. The study used a mixed-methods approach, marrying closed-answer questionnaires with in-depth qualitative interviews.
While over one hundred short questionnaires were collected, as for many studies like this one, the exigencies on the ground often mean that relatively few time-intensive, open-ended interviews can be conducted. The authors interviewed just 8 women, justifying that “Recruitment was stopped after the data achieved saturation, a which point answers from women seemed to repeat information gained earlier and little new information was forthcoming” (Mselle et al. 2012: 929). Another article by the same authors a year later claims to have reached saturation after 16 interviews: “The sample size of the women affected by obstetric fistula was not predetermined. However, saturation was achieved after 16 interviews, where answers from women seemed to repeat information gained earlier and little new information was attained.” (Mselle et al. 2013: 3).
Apparently, the vast diversity of experience and expectation was covered after only a handful of interviews. After listening to eight or sixteen women, the researchers felt that they had nothing left to learn. No more information was to be gained.
I read these articles after returning home from a particularly arduous interview. An interview with a quiet, 30 year-old Fulani woman who had returned to the fistula center for a prophylactic C-Section three years after a successful surgery had rendered her continent. She spoke softly and frequently smiled from the corner of her mouth, looking down at her interlaced fingers which rested languidly on her lap. Although she readily accepted to be interviewed, her pithy (often one-word) answers left me wondering if she actually wanted to tell me her story or if she had accepted only to be polite.
I always know an interview isn’t going well when the majority of the interview consists of me rambling away. And boy, was I rambling — rephrasing, reposing, rewording…
Among the many non-committal responses, she initially said that she had made no attempt to hide her fistula, that everyone knew, and that she experienced no mistreatment. “Fistula isn’t a sickness that you can hide”, she told me. “No one said a word to me.” “No one told me not to sit with them. No one told me not to share their food.” One might have gathered that during the six years that she lived with fistula, her family, friends and neighbors supported her and overlooked her injury. And for some women, this certainly is the case. But, I’ve now heard the stories of 77 women before her – hundreds of hours of experiences of women who fall on every shade of the spectrum, and as the interview went on, small contradictions nagged at me. I knew that something was missing. I just didn’t believe that I had the whole story.
So I poked.
And I pried.
And I prodded.
And eventually, well after my research assistant became uncomfortable posing the same question in several different ways and long after I might have ended the interview 6 months previously, my perception of her story began to shift. Certainties seemed less certain. Although she gave no indication of it, I asked her directly if people thought that she was healed. She told me that after initially being quite ill for the first two months (and incapable of hiding her condition), she was able to take care of herself; she learned how to keep herself clean and dry. Soon, no one smelled the urine. No one saw her wetness. So, yes, people in her village began to assume that she was healed. Rumors spread. She encouraged the assumptions. Like so many other women with fistula, she lived six years at home carefully guarding her secret from everyone but her husband and her mother. So, no, she didn’t experience mistreatment. She didn’t experience shame. But not because people accepted her. They were never given the chance to. They didn’t know that she wasn’t just like them.
“Not even your co-wife knew?” I asked. (She had told us that she was very close with her co-wife, so close that many people teased that they were not co-wives but sisters).
“No”, she replied. “She might have suspected, but I was very careful. Once she asked our husband how I was healed, what treatment I had taken. But my husband was very forceful with her; he told her it wasn’t her concern. He told her to stop her questions and to stop her curiosity.”
I often re-read, and sometimes re-listen to, my first dozen (and even two dozen) interviews and am taken aback by the short and shallow responses, by how many questions went unasked, how often I declined to probe when in retrospect I can tell a woman is withholding. I wonder how different these stories would read if I had pushed a little harder or asked a different question. Indeed, if I drew conclusions based on these first 8, or even first 16 interviews, I would have a very different (and less nuanced) picture of the complex ways in which fistula affects women’s lives.
The accuracy of qualitative data attained from open-ended interviews can be difficult to judge. How do we gauge the quality of data processed through opaque statistical computations, or run through software analysis, rendering what appears to be good, strong, veracious renditions of reality? And how can we audit this data? Unlike quantitative measurements (which certainly have their own inherent problems), with qualitative data, we are often left to trust that the interview was conducted well, that the person was at ease, was forthcoming, that the interviewer asked all of the right questions in the right order.
Could “saturation” after only a few interviews mean that the right questions weren’t being asked? Would it be more honest to explain that the reason for only eight interviews was that in-depth interviews are hard work, that processing 25 hours of spoken data takes time and is costly?
One thing that fieldwork seems to teach you is that you will never understand everything. You will always have more questions. You will question the strength of your own work and the veracity of you own conclusions. You will wonder what you overlooked. And then when it is time to write -up, you pretend that you don’t. You cover up the questions with pseudo answers. You hedge. You lean on explanations like “saturation” to cover-up shortcomings and to camouflage holes.
Indeed, over-simplification and the pretense of certainty aren’t just for journalists (as I’ve previous critiqued of Nick Kristof, CNN , and others). Even we academics are guilty in our own way.
Mselle, L. T., Moland, K. M., Evjen-Olsen, B., Mvungi, A., & Kohi, T. W.
2012 “Hoping for a Normal Life”: Reintegration After Fistula Repair in Rural Tanzania. J
Obstet Gynaecol Can 34(10): 927–938
Mselle, L. T., Moland, K. M., Evjen-Olsen, B., Mvungi, A., & Kohi, T. W.
2013 Why give birth in a Health Facility? Users’ and Providers’ Accounts of Poor Quality of
Birth Care in Tanzania. BMC Health Services Research 13:174