Ernestina Coast, Natalie Mondain and Clementine Rossier

Qualitative research in demography:

quality, presentation and assessment

Conference Item [eg. keynote lecture, etc.]

Original citation:
Coast, Ernestina and Mondain, Natalie and Rossier, Clementine (2009) Qualitative research in
demography: quality, presentation and assessment. In: XXVI IUSSP International Population
Conference, 27th September - 2nd October, 2009, Marrakech, Morocco. (Unpublished)

This version available at: http://eprints.lse.ac.uk/36788/

Available in LSE Research Online: July 2011

© 2009 The Authors

LSE has developed LSE Research Online so that users may access research output of the
School. Copyright © and Moral Rights for the papers on this site are retained by the individual
authors and/or other copyright owners. Users may download and/or print one copy of any
article(s) in LSE Research Online to facilitate their private study or for non-commercial research.
You may not engage in further distribution of the material or use it for any profit-making activities
or any commercial gain. You may freely distribute the URL (http://eprints.lse.ac.uk) of the LSE
Research Online website.
 Qualitative research in demography: quality, presentation and assessment1
E. Coast2
N. Mondain3
C. Rossier4
L. Bernardi5
S. Randall6

1. Context: qualitative approaches in demography

There has been an increase in the number published studies, and ongoing
research programmes, that use qualitative approaches (in full or in part) in
recent years in the broad field of demography (Coast, 2003; Randall &
Koppenhaver, 2004). Randall & Koppenhaver (2004) found 24 articles for 1991-
1995 and 43 articles for 1996-2000 meeting criteria of qualitative research in
demographic literature7. In an update, we found 54 articles meeting these
criteria for 2001-2005. Changes have occurred in recent years in demography,
mainly related to the need for more explanatory (rather than descriptive and
predictive) frameworks, leading both to the improvement of quantitative
methods as well as the emergence of qualitative approaches (Riley and
McCarthy, 2003).

This rise in qualitative contributions to research endeavours has been mirrored

in other, traditionally quantitative, disciplines (Daly, 2007). This paper
acknowledges that the simple division of research into qualitative and/or
quantitative is not particularly helpful, not least because they have
associations with research paradigms with which they are not necessarily
linked, but that discussion is beyond the scope of this paper. Since the 1980s,
population studies scientists have increasingly used qualitative methods. The
methods used have themselves shifted over time, from ethnographic
approaches (Caldwell micro-approaches book ref.), towards increasing reliance
on focus groups and in-depth interviews (Randall and Koppenhaver 2004).
Qualitative approaches are used in demography at multiple points in the

1
Paper for presentation at IUSSP 2009, Marrakech, 27th September – 2nd October
2
Senior Lecturer in Population Studies, London School of Economics [email protected]
3
Assistant Professor, University of Ottawa [email protected]
4
Institut National d'Études Démographiques (INED) [email protected]
5
Associate Professor, University of Lausanne [email protected]
6
Reader, University College London [email protected]
7
Demography, Perspectives in Sexual and Reproductive Health, International Family Planning
Perspectives, International Migration Review, Population –English edition, Population Studies,
Population and Development Review, Studies in Family Planning. All these journals are on
JSTOR, except for International Migration Review. Key words used to search abstracts:
anthropology / gical / gist, ethnography / phic / pher, qualitative, focus groups, in-depth /
semi-structured interviews, participant observation
production of knowledge, including: design and testing of quantitative
questionnaires; to understand unexpected survey results; and, to grasp
sensitive issues, perceptions, “cultural contexts”, and other elements of the
social world which are difficult to measure quantitatively (Bozon 2006).

This use of qualitative methods raised a number of criticisms on the part of

anthropologists (Greengage 1997, Kreutzer & Fricke 1997, Base & Abby 1998,
Coast 2003). While demographers borrowed the tools of sociology and
anthropology, they often left out these disciplines’ theories (i.e. they did not
necessarily theorize demographic behaviours as socially embedded individual
actions), nor did the necessarily embrace a non-positivist epistemology. This
disconnection is problematic since methods, theories and epistemologies are
linked (Riley & McCarthy 2003). Anthropological demography and historical
demography tried in the last decades to reconcile these diverging perspectives
by developing unified interpretative frameworks and methodology for
population studies (Bernardi and Hutter 2007)

Within demography, qualitative approaches have tended to focus on certain

topics, particularly those that are considered “sensitive”. Of course, the
choice of method is dependent on whether it is the best approach to answering
a question, and within demography qualitative approaches have tended to be
used in the study of family and partnership dynamic, fertility and family
planning, and lifestage transitions e.g.: the transition to adulthood.
Furthermore there is a plethora of qualitative research around proscribed
behaviour such as unsanctioned sexual activity, illegal, illicit and ‘undesirable’
behaviour (e.g. adolescent and extra marital sexual activity, abortions etc).
These issues tend to be poorly captured by surveys and where data on attitudes
and actual behaviour is needed especially for guiding subsequent interventions.
The problems posed by HIV/AIDS research and interventions have driven much
of the movement towards qualitative research methods.

Within demography, qualitative approaches incorporate a richness of

information that cannot be captured using only quantitative data (Randall &
Koppenhaver, 2004; De Loenzien & Yana, 2005). As such, qualitative methods
have increasingly drawn the attention of demographers and led to the
emergence (or recognition) of a new field: anthropological demography
(Kreutzer & Fricke, 1997; Bledsoe, 1998; Greengage, 1997) even if it must be
made clear that the qualitative methods used by demographers are not
anthropological (Randall & Koppenhaver, 2004). In demography, the most
frequently used qualitative methods are focus group discussions (Barker & Rich.
1992; Calves, 2000; Gueye, M. et al, 2001; Castle et al, 1999), in-depth
individual semi-structured interviews (LeGrand et al., 2003; Randall & LeGrand,
2003; Mondain & Delaunay, 2006; Mondain et al., 2007); and, life histories
(Randall and Mondain, 2009). Combinations of methods are also used,
attempting data triangulation (LeGrand et al. 2003; Randall & LeGrand, 2003),
together with other “systematic” methods such as pile sorts. The choice of
such methods is likely to be driven by the fact that demographers are highly
concerned by statistical representativeness and as such have been reluctant to
use qualitative methods involving non-random selection of small(er) numbers of
participants. Therefore, when acknowledging the advantages of using
qualitative methods, demographers have ‘naturally’ turned towards those
methods allowing them to gather a greater number of participants. Methods
privileged by ‘pure’ anthropologists, such as participant observation, are less
(well) used.

There is a wide body of work on how to collect, analyse and present qualitative
data in the social sciences in general (Marshall & Rossman, 2006; Huberman &
Miles, 2002; Patton, 2002; Denzin & Lincoln, 2000; Strauss & Corbin, 1990] and
demography in particular [Obermeyer, 1997; Axinn & Pearce, 2006). The body
of work that deals with assessing the quality of qualitative evidence is much
less well developed, and, to date, not found within the broad discipline of
demography (for an exception see Matthews).

Mixed methods, the combined use of quantitative and qualitative methods, are
increasingly widely used in social science in general (Tashakkori & Teddlie,
1998; Onwuegbuzie & Teddlie. 2002; Onwuegbuzie & Leech, 2005). In
demography, broadly, examples remain scarce and in most cases, when both
methods are used, authors have difficulties to articulate the statistical and
qualitative results with each other and thus describe them separately (Mondain
et al., 2004; Mondain et al, 2007)8. Approaches for assessing the quality of
mixed-methods research in general are very poorly developed (Bryman, Sale &
Brazil, 2004; Tashakkori & Teddlie, 2003), and are not explored explicitly in
this paper.

Researchers need to produce, identify, assess, aggregate, interpret and

disseminate the highest quality evidence (Curran et al, 2007), whilst
acknowledging that quality is, in and of itself, an abstract concept (Boaz &
Ashby, 2003). Let us consider the routine tasks of a demographer that require
assessing both these. In their own research, an evaluation of data quality is an
integral part of the whole process and will influence what analyses are
subsequently undertaken. However the demographer must also assess both the
data and analyses of others: reading published research; and, reviewing journal
articles, funding proposals, and student theses. Some of this evidence may
draw on qualitative data. Demographers have an arsenal of techniques for
assessing the quality of quantitative data, but they are unlikely to have been
trained in methods of qualitative data collection or analysis, and even less
likely to have been trained to assess their quality. At the heart of this issue is
that evidence produced by a qualitative study is likely to look quite different
8
For example, workshop on the articulation of quantitative and qualitative research in
population studies was convened at the Institut Superieur des Sciences de la Population,
University of Ouagadougou 18-20th October, Ouagadougou, Burkina Faso
from that produced by a quantitative study. For a novice qualitative
researcher (who might be a highly experienced quantitative researcher), the
methodological minutiae of qualitative research might be overwhelming (Daly
et al, 2007). Experienced qualitative researchers know, or think they know,
good qualitative research when they see it although they approach qualitative
evidence with variable levels of experience, skills and self-confidence. So how
can the demographer without such qualitative experience know “high quality”
when they see it, for the equating of peer review publication with high quality
does not always necessarily hold (Grayson, 2002)?

For any study, regardless of ontology or epistemology, there are standard

processes for doing research, such as matching method to research question(s)
and ensuring ethical conduct. Our focus in this paper is on the production and
reporting of qualitative research in demography, drawing on two perspectives:
views from primary data collection; and, experiences of reading and reviewing
published studies. We begin with a brief review of how quality is assessed,
drawing on frameworks from other disciplines.

2. What is quality? Should it be assessed? How might it be assessed?

Qualitative research encompasses many different traditions and methods, and
the entry points to appraise quality are many: research design; methodological
approach; data collection; analysis; and, insight and interpretation. There are
extensive concerns about how quality or rigour or thoroughness might be made
manifest. These concerns are being driven in part by an agenda for inclusion of
qualitative research in systematic reviews alongside those traditionally
established for quantitative research.

Criteria for assessing the quality of quantitative research are well-developed,

and there is much debate about whether criteria developed for quantitative
evidence (for example, reliability, validity (both internal and external) and
replicability) might also be used for qualitative evidence (Hope & Waterman,
2003; Bryman, 2001; Popay et al, 1998; Rolfe, 2006). Other authors argue that
the very language of “scientific” endeavour is inappropriate for qualitative
research. Sandelowski, for example, argues that the criterion of
“trustworthiness” should replace notions of “validity”. Criteria for judging the
quality of qualitative evidence are highly contested. Indeed, for some, the
very notion of developing such criteria undermine the endeavour of qualitative
research; the very reflexivity and flexibility that characterizes qualitative
research might be seen as at odds with the development of criteria for the
evaluation of its quality. Or, should concerns about the quality of qualitative
research be made on its own terms, with the case being made for a
reformulation of notions of quality being quite different to those used for
quantitative research? Dixon-Woods et al (2004) argue that the diversity or
“near anarchy” in the range of qualitative approaches makes it difficult to
achieve consensus as to what makes for a flawed piece of research.
In the same way that qualitative research is underpinned by a wide variety of
epistemological and methodological approaches, so too is the agenda related to
its quality appraisal. Many disciplines have developed criteria to assess the
quality of qualitative studies, notably in the area of health-related research.
Debates about what criteria are appropriate for assessing the quality of
qualitative research are longstanding (Holloway & Wheeler, 1996; Perakyla,
1997; Seale, 1999). Some learned societies and journals now produce
guidelines for the review of explicitly qualitative research9, and the Evidence-
based Medicine Working Group has gone so far as to identify an “ideal
qualitative study” (Guyatt & Rennie, 2001).

Much of the research about research and evidence quality has focused on
methodological quality, not least because this is relatively straightforward to
assess (Boaz & Ashby, 2003). Spencer et al (2003) reviewed 29 frameworks for
assessing the quality of qualitative research, the majority of which were
developed in the fields of medical or health-related research. They identified
heterogeneity in terms of framework purpose (e.g.: assessment of articles,
proposal review), applicability to different methods, and their length, format
and coverage. The purpose of their review was to develop a framework for
assessing quality of qualitative research, which resulted in the identification of
4 central principles10, articulated by 18 “Appraisal questions11” dealing with
research design, findings, sample, data collection, analysis, reporting,
reflexivity and neutrality. Each of these 18 questions has a series of linked
quality indicators.

Dixon-Woods et al (2004) identified in excess of 100 sets of proposals for

quality in qualitative research. They suggest that there is little emerging
9
Medical Sociology Group of the British Sociological Association; Canadian Journal of Public
Health
10
Contributory in advancing wider knowledge or understanding about policy, practice, theory
or a particular substantive field; Defensible in design by providing a research strategy that can
address the evaluative questions posed; Rigorous in conduct through the systematic and
transparent collection, analysis and interpretation of qualitative data; credible in claim
through offering well-founded and plausible arguments about the significance of the evidence
generated.
11
How credible are the findings? How has knowledge/understanding been extended by the
research? How well does the evaluation address its original claims and purpose? Scope for
drawing wider inference – how well is this explained? How clear is the basis for evaluative
appraisal? How defensible is the research design? How well defended is the sample design/
target selection of cases / documents? Sample composition/ case inclusion – how well is the
eventual coverage described? How well was the data collection carried out? How well has the
approach to, and formulation of, the analyses been conveyed? Contexts of data sources – how
well are they retained and portrayed? How well has the diversity of perspective and content
been explored? How well has detail, depth and complexity (ie¨richness") of the data been
conveyed? How clear are the links between data, interpretation and conclusions i.e.: how well
can the route to any conclusions be seen? How clear and coherent is the reporting? How clear
are the assumptions/ theoretical perspectives/ values that have shaped the form and output of
the evaluation? What evidence is there of attention to ethical issues? How adequately has the
research process been documented?
consensus either as to whether quality assessment of qualitative research is
achievable or even desirable. They suggest that what is critical in the
development of any quality criteria is the need to distinguish “fatal flaws”
from “minor errors”. More recently, Walsh & Downe (2006) reviewed 8
frameworks (including those that themselves incorporated reviews of other
frameworks e.g.: Spencer et al, 2003), synthesised and removed ‘redundant’
criteria, and produced a list of 12 essential criteria12, identified by 53 specific
prompts13. In a recent review of frameworks to assess the quality of qualitative
12
Clear statement of, and rationale for, research question/aims/purposes; Study thoroughly
contextualised by existing literature; Method/design apparent, and consistent with research
intent; Data collection strategy apparent and appropriate; Sample and sampling method
appropriate; Analytic approach appropriate; Context described and taken account of in
interpretation; Clear audit trail given; Data used to support interpretation; Researcher
reflexivity demonstrated; Demonstration of sensitivity to ethical concerns; Relevance and
transferability evident
13
Clarity of focus demonstrated; Explicit purpose given, such as descriptive/explanatory
intent, theory building, hypothesis testing; Link between research and existing knowledge
demonstrated; Evidence of systematic approach to literature review, location of literature to
contextualise the findings, or both; Rationale given for use of qualitative design; Discussion of
epistemological/ontological grounding; Rationale explored for specific qualitative method (e.g.
ethnography, grounded theory, phenomenology); Discussion of why particular method chosen is
most appropriate/sensitive/relevant for research question/aims; Setting appropriate; Were
data collection methods appropriate for type of data required and for specific qualitative
method? Were they likely to capture the complexity/diversity of experience and illuminate
context in sufficient detail? Was triangulation of data sources used if appropriate? Selection
criteria detailed, and description of how sampling was undertaken; Justification for sampling
strategy given; Thickness of description likely to be achieved from sampling; Any disparity
between planned and actual sample explained; Approach made explicit (e.g. Thematic
distillation, constant comparative method, grounded theory); Was it appropriate for the
qualitative method chosen? Was data managed by software package or by hand and why?
Discussion of how coding systems/conceptual frameworks evolved; How was context of data
retained during analysis; Evidence that the subjective meanings of participants were Portrayed;
Evidence of more than one researcher involved in stages if appropriate to
epistemological/theoretical stance; Did research participants have any involvement in analysis
(e.g. member checking); Evidence provided that data reached saturation or
discussion/rationale if it did not; Evidence that deviant data was sought, or discussion/
rationale if it was not; Description of social/physical and interpersonal contexts of data
collection; Evidence that researcher spent time ‘dwelling with the data’, interrogating it for
competing/alternative explanations of phenomena; Sufficient discussion of research processes
such that others can follow ‘decision trail’; Extensive use of field notes entries/verbatim
interview quotes in discussion of findings; Clear exposition of how interpretation led to
conclusions; Discussion of relationship between researcher and participants during fieldwork;
Demonstration of researcher’s influence on stages of research process; Evidence of self-
awareness/insight; Documentation of effects of the research on researcher; Evidence of how
problems/complications met were dealt with; Ethical committee approval granted; Clear
commitment to integrity, honesty, transparency, equality and mutual respect in relationships
with participants; Evidence of fair dealing with all research participants; Recording of
dilemmas met and how resolved in relation to ethical issues; Documentation of how autonomy,
consent, confidentiality, anonymity were managed; Sufficient evidence for typicality
specificity to be assessed; Analysis interwoven with existing theories and other relevant
explanatory literature drawn from similar settings and studies; Discussion of how explanatory
propositions/emergent theory may fit other contexts; Limitations/weaknesses of study clearly
outlined; Clearly resonates with other knowledge and experience; Results/conclusions
evidence for inclusion in systematic reviews, Dixon-Woods et al (2007)
concluded that while the use of frameworks tended to produce less agreement
between assessors, they did sensitise them to methodological issues.

What does it mean to try to put into practice, both in terms of primary data
collection, and also in terms of reading and reviewing the work of others, the
assessment of quality in qualitative research? The next section involves two
attempts at reviewing what it means to put into practice the assessment of
quality. The first draws on primary data collection experiences of
demographers, illuminating the sorts of researcher reflexivity around data
collection, analysis and presentation that many of the frameworks for quality
assessment refer to. The second part demonstrates the sorts of issues inherent
in trying to assess or read for quality when examining published literature in
demography.

3.1 Dealing with quality: perspectives from the demographic field

At every stage of qualitative research a key dimension is the inductive and
iterative nature of the research process. Words and speech are fundamental
data source and thus it is essential that respondents are willing to talk. Part of
the research process is therefore identifying the best ways to induce
respondents to talk openly or debate the issues under study. However, the
ways people talk, what they are willing to talk about and with whom and under
what circumstances are integral not only to data collection, but also to analysis
and interpretation. Good qualitative research needs to allow for the new and
unexpected to emerge, be identified – hopefully by both field workers and
researchers if they are not one and the same – and then be developed. In the
same way that qualitative data coding should be inductive and guided by the
data, so too should data collection. This need for responsiveness and flexibility
is present throughout the research process, not only, but particularly for
qualitative research in demography dealing with powerful but and personal
issues in people’s lives: death, sexuality, reproduction, marriage.

Under such conditions, how should qualitative demographic research be

designed? And what factors and risks should be considered during this planning
process? For each of the following dimensions – choosing the appropriate
qualitative instruments; number and sampling of respondents; selection and
training of interviewers; the fieldwork; data management and analysis – we
briefly describe the main issues and use examples of our own work. The
examples we use are mainly taken from studies conducted in sub-Saharan
Africa or with African populations (Appendix 1).

obviously supported by evidence; Interpretation plausible and ‘makes sense’; Provides new
insights and increases understanding; Significance for current policy and practice outlined;
Assessment of value/empowerment for participants; Outlines further directions for
investigation; Comment on whether aims/purposes of research were achieved
3.1.1: Which qualitative methods to choose?
Method choice is context sensitive and the inappropriateness of certain
methods may only emerge with hindsight after fieldwork. In the 1999 Senegal
research project our purpose was to combine different qualitative methods to
investigate fertility decision making strategies. As part of a comparative study
with Zimbabwe the original intention had been to use identical methods in both
countries. The Senegal research was to repeat the approach used in Zimbabwe
where the whole team spent two weeks in each research site using diverse
methods such as free-listing, in-depth interviews, photo-stories and focus
groups. Interview guides were directed towards the demographic research
hypotheses around links between child mortality and reproductive decisions.
But with collaborators trained in contrasting academic disciplines (demography
in Zimbabwe, anthropology in Senegal), different expertise of the students
allocated to the fieldwork and a strong belief amongst the Senegalese team
that a more indirect approach was best in the Senegalese context, the Senegal
approach allocated pairs of fieldworkers to spend two months in each of the
three research sites (Dakar, a small town and a village) with a phase of
observation and immersion before initiating in-depth semi-structured
interviews and focus groups.

The methods were not equally successful in each site: focus group discussions
worked well with urban women but were inappropriate in the village where
potential participants all knew each other well and would have been
uncomfortable addressing the study topics in front of kin and neighbours.
Informal group discussions were effective in the village when young people
gathered in the researcher’s room in the evenings to chat. Focus groups with
urban men were not a success: men disliked the informality of a method which
encouraged them to debate and discuss amongst themselves and wanted a
formal situation whereby the facilitator asked a question and each answered in
turn. Furthermore men would not challenge or question accepted social norms
in public, especially those around religious interpretations of appropriate
reproductive behaviour. In private in-depth interviews a few men were more
open to debating such issues. Whereas in Senegal respondents seemed more
inclined to be frank and honest in private in-depth interviews, in Zimbabwe the
interviewers believed respondents were far more likely to tell the truth in the
more public focus groups. An essential stage in qualitative research is a
conscious reflection on the quality of the data produced by different methods:
cultural traditions and norms and gender roles may have a substantial impact
on the success or failure of individual approaches.

3.1.2: Sample size and strategies to identify respondents

Qualitative research does not involve big numbers - close attention must be
paid to the selection or sampling of respondents - it is essential to document
these choices and to reflect on particular biases that may have been
generated. A snowballing approach to respondent recruitment is often used
which can be very effective if a homogenous sample is needed with people with
similar characteristics or attitudes. However it becomes inappropriate if a
wider understanding is required from a diverse socio-economic range of
respondents.

Such diversity was sought in recent work on the effects of out-migration on

family dynamics in a small town in Senegal (Randall and Mondain, 2009;
Mondain and Randall, 2009). In the absence of a sampling frame we used the
six neighbourhoods in the town as clusters. On the first day the interviewers
went out with pre-selected characteristics of respondent (by age and sex);
starting from the neighbourhood chief’s house they counted 10 houses and at
the 10th house asked if a respondent corresponding to the criteria lived there
and could be interviewed then or later; if not they continued to the next
house. Subsequent respondents were identified by counting a further 10
houses from the preceding interview This achieved a far more heterogeneous
sample than would have been achieved through snowballing although there was
probably a bias towards unemployed individuals who were at home in the day.
If several individuals of the required characteristics were resident there was a
tendency to select the person present rather than the absent one. However in
a town where there is little formal employment and few fixed working hours
this was probably not a major bias but would have been a major problem in a
large city.

If sampling frames are available they can help in selection of respondents;

however, the small target numbers of respondents in most qualitative studies
often require quotas to ensure adequate socio-demographic diversity of the
participants. Representativeness is not the key issue in most qualitative
research however: what is much more important is a willingness to participate
in the study. Given that often the sample size will be less than 50 respondents,
the one or two individuals representing a particular age/sex/education
category cannot and should not be seen as representing that group: overall,
qualitative data is aiming to understand perceptions, processes, constraints,
dilemmas and uncertainties and should not be used to infer to the wider
population of similar individuals. Since a successful social interaction between
researcher and respondent is an essential dimension to good qualitative data,
important issues in sample selection are (a) identifying respondents who are
willing to talk and generating an environment where they are able to do so
freely and (b) reflecting consciously on possible biases in the data, where
possible taking steps to address these.

The issue of selecting key informants is complex. One possibility is to decide on

appropriate key informants a priori, based on discussions with local colleagues.
In the African context, key informants are usually the local leaders, political or
religious, social and health workers, teachers, etc. Qualitative approaches
leave the door open for others to emerge as key informants during fieldwork
and the identification of key informants should be an iterative process
throughout, with interviews identifying new, influential characters in the
community who can be followed up. In the particular (but typically
demographic) case of Demographic Surveillance Systems (DSS) the permanent
interviewers working in these sites often become key informants as well
(Mondain & Bologo, 2007).

3.1.3: Selection and training of interviewers

Given that qualitative research is largely about exploring understanding,
perceptions and dilemmas, communication in an appropriate language is
incredibly important. Good mastery of the local language is essential not only
for basic communication but also for the success of the whole qualitative
research endeavour. This requires good local interviewers. Herein lies a
dilemma, especially if working in rural areas or with poorly educated urban
dwellers. Either one selects research assistants with primary or unfinished
secondary education who are best able to develop excellent rapport with
respondents with whom they are likely to be socio-economically and
educationally close, or one selects university students or graduates who are
better able to understand the intellectual dimensions of the study and may
even want to integrate the work into their own dissertations, but who may not
always be as good at developing rapport. There are arguments for and against
both strategies. Furthermore, whichever strategy is chosen and however good
the training, there are some individuals who are just excellent at inducing
respondents to talk and others, who because of their innate personality, find it
much harder. Good training and team building (see below) can go some way to
improving the quality of data from less gifted interviewers.
A key issue in qualitative research is translation and ensuring that issues in
translation are at the forefront of interviewers’ minds. Key concepts around
demographic research dealing with birth, death and life transitions are often
difficult to translate, and yet most qualitative research in Africa ultimately
depends on translation both for analysis and publication. In the 2007 Senegal
research project a critical dimension of the data collection process was
identifying and discussing local core expressions which had very specific
meanings. For example the Wolof term ‘mougn’, mostly used by women and
referring to their need for being patient and obedient in the context of their
daily lives and relations with in-laws and husbands could not be translated by
one French/English word (namely ‘patience’) as this would have led to the loss
of the complex meaning of ‘mougn’. Regular interactions between researchers
and interviewers, who simultaneously become key informants, are vital; good
transcripts of qualitative interviews will often retain words in the local
language annotated with long detailed explanations of meaning.
Furthermore, producing qualitative data implies specific skills in the art of
discussion and building rapport. Matthews (2005) discussing the issue of
studying marriage emphasizes the fact that researchers should consider “the
interaction not as an interview but as an episode of participant observation.”
By this she refers to the need to consider the specificities of local
communication and modes of interaction14; those may indeed vary significantly
between different socio-cultural contexts, and depend not just on the language
spoken but also to belonging to certain ethnic, social or caste groups.

For example, in the research undertaken in the Sereer community in 1999,

interviews were conducted jointly by the principal researcher and two
interpreters (male and female to match with the sex of the respondent); it
became obvious that as soon as a praise-singer or blacksmith (caste) was
interviewed, interpreters showed a lack of interest and would not make the
same efforts to generate good rapport. Because they were local but not
themselves casted, their relationship with the respondent was largely
determined by the acceptable social relations in their daily lives between
themselves and a caste person from their community. Had interpreters been
outsiders, with more education and less embedded in the community, these
problems would not have occurred; on the other hand, it was made clear that
most participants from this area would have not talked openly to an outsider
(Gokah, 2006). Pursuing this dilemma of outsider/insider in the community, the
quality of the interviews can also be jeopardized when respondents are aware
that the interviewer is asking questions on topics he should know about. This
excerpt from an interview conducted with a 43 year-old man in French by the
researcher and translated into the local language by the interpreter
demonstrates some of the complexities around selection of interviewers:
should they belong to the community or be external while at the same time
mastering the language?
Q [in Sereer] : She asks why did your family prefer your first wife
for your marriage ? What were their criteria?
A [in Sereer] : Yes, … and you know that. In the past, people
would say: go and see the daughter of this man because she is well
brought up, her father has such and such qualities, her mother is as
such. (…). It is for this reason that our elders made a rigorous selection
of the girl to marry. I guess you remember this old selective method of
the girl to marry. (Senegal, Sereer community 1999)
In order to maximize the quality of qualitative data it is essential to focus on
understanding the goals of the research during the training of interviewers.
Here again there are trade offs between well-educated students from outside
and local interviewers with less education. Interviewing requires skill at
probing and identifying ‘pockets’ of potential information left open by
informants and windows of opportunity to pursue potentially interesting
avenues which is only possible if the interviewers can see this potential
because they understand the issues. In selecting interviewers, different
criteria must be considered: academic training, former fieldwork experience
(not always an advantage if different research methods are being used), and
study-specific criteria such as age, sex, marital status, ethnicity. Training
14
Studying this was the aim of a project conducted in five different DSS to explore the
problems related to standardized inquiries in various socio-cultural contexts (Mondain &
Bologo, 2007).
interviewers in qualitative methods is particularly challenging as it is related to
the art of communicating, itself culturally specific. We need to consider the
extent to which external research teams (often from the North) are able to
provide adequate training in interviewing in a context that is foreign to them.
Certain general guidelines can be discussed in common (such as the art of
probing, allowing time for silences, etc.) but in the end, interviewers are those
who know how to explain, ask and create a climate of mutual trust – and some
are inevitably better than others.

3.1.4: Fieldwork
The typical demographic qualitative inquiry is based on an interview guide
generating fluid situations in terms of adapting the questions and conducting
the interviews. It is essential for the principal researchers to maintain close
contact with interviewers both in order to identify key contextual dimensions
while analyzing the data later on and to ensure that everyone is aware of new
issues which emerge as fieldwork progresses. Regular field meetings are an
essential component of the fieldwork and generate team spirit and good
collaboration. .
Although qualitative research training usually emphasizes the need to avoid
leading questions and those with yes/no answers it is also important to find
ways of getting interviewers to encourage informants to focus on specific
incidents rather than their more general ‘feelings’ (Matthews, 2005) which can
end up being very vague and indeterminate – with little idea of how they
impinge on actual life. This was a challenge for the 2007 fieldwork in Senegal
where we wanted to establish the differences between marriages and family
life within migrant households versus households with no migrants. The
interviewers, knowing our interest in migration, found it difficult to avoid
direct questions which just generated opinions about migrant households
totally ungrounded in any specific events or experience. Good qualitative data
on demographic issues usually needs to be well grounded in personal
experience.

Systematic transcription after each interview is best undertaken by the

interviewers themselves as rapidly as possible after the interview. This allows
them to include good contextual information as well as non-verbal
communication which can be critical in interpreting what people say and how
they say it.
Q Could the fact of losing a child push a husband to impose another
pregnancy on his wife?
A (in a very serious tone of voice) What are you saying? But it’s
not linked. The death of a child and wishing for a pregnancy, it’s
not linked. One makes a child with God’s blessing. If He takes it
back, you can only wait until He gives it back to you...I can see no
possible calculation. Dakar
1999
 I Uhuh. An if you have children yourself are you going to do the
same thing [foster out your child]?
R (very quickly) Ah no, no. That never, never, never. Never foster
out my children. I would not even foster my children to my own
sister. I am going to bring up my children myself.
Small town Senegal 2007

In both these cases the non-verbal communication adds to the intensity of the
statements. Such additions to transcripts are only likely whilst the transcriber
still remembers the details of the interview. Interviewers should also be
encouraged to report the quality of the interview and whether they felt the
respondent was telling the truth, was happy to talk or was disinterested,
without being made to feel that a poor interview necessarily reflects badly
upon themselves. Other contextual information is very important and
contributes to the quality of interpretation.

The quality of the data management during the fieldwork is a key preliminary
step to good data quality and therefore analysis. We give three examples, one
where the data management was closely followed up during the fieldwork, one
where mistakes were made, and one where power relations within the team
made it difficult to obtain good quality transcriptions.

During the 2007 fieldwork in Senegal, the two principal researchers were
present in the field, supervising the interviewers who had been hired and
trained by one of them. Every day, each interviewer undertook one interview
in order to translate and transcribe the same day. Transcription had to be
completed before conducting further interviews. The transcript was then read
by the researchers generating comments on the way the interview was
conducted, where probes had been useful and also for questions on issues they
were not familiar with (especially cultural dimensions) and specific words used.
Every evening a team debriefing involved discussion of the day’s work and
comments on previous days’ transcripts; there were frequent suggestions about
good questions to ask or issues that should be pursued. This continuous process
of discussion, reading and immediate follow up of the interviews made the
further data management and coding easier and more efficient and was also an
effective way of providing training to the weaker and less confident team
members.

In 2005-6 research was conducted in Montreal (Canada) on the transition to

adulthood of first generation young African immigrants using a team of four
trained interviewers supervised by a PhD student. For various reasons a
different person transcribed the interviews, leading to long delays between
interview and researchers reading the transcripts, rendering it impossible to
pursue gaps or lack of clarity. Without regular and timely follow up there was
no opportunity for comments on interview quality and no improvements over
the course of the project.
In the 1999 Sereer marriage behaviour project interviews were conducted by
the principal researcher with two interpreters who were local residents who
spoke French but did not write it well enough to translate and transcribe
interviews which was done by another. Here there was an issue of
transcription quality which reflected relations between different team
members: the transcriber was older and better educated than the two
interpreters working with the principal researcher, who at that time, was a
young female PhD student. Power relationships based on age, gender and
education level rendered it difficult to persuade the transcriber to remain
faithful to the recording because he argued the level of Sereer language was
poor and therefore a direct translation into French could not be understood by
the researcher. As a result he ‘re-wrote’ the interviews into an academic
French style totally dissonant with local peasant forms of expression.

3.1.5: Presentation of results

Presenting demographic qualitative research in a way which allows the reader
to assess the quality of the research must include detailed description and
reflection on methods. Who are the respondents: how were they selected?
What are the potential biases in selection? The inherent qualities and
characteristics of respondents allow the authors to discuss the potential for
generalizations from their data and its limits.

In most cases, qualitative demographic research analysis is backed up using

quotes to illustrate key analytic points, with contextual socio-demographic
information about the respondent. The core question here is: how much data
should be included and how should it be selected? It is important to note that
the data appears through the form of excerpts and those constitute evidence,
not examples; usually they are chosen from among a set of other similar
excerpts by the authors although rationale for selection of quotes must be
justified in the text with indications about whether the point being made was
heard frequently or is a rare exception.
Several questions in presenting the data emerge:
‐ Should qualitative data be quantified? Something that is easy with
software such as N6. In other words should statistics be generated about
the numbers of respondents who mentioned certain aspects or amount of
text on a certain topic? Such statistics give a false picture for various
reasons: the fluid nature of an in-depth interview along with following
up issues of interest to the respondent mean that not all questions are
systematically asked of everyone. Furthermore silence is not always a
sign of absence of importance (Randall & Koppenhaver 2004). On the
other hand indications of scale are helpful with typical ‘fluid’
descriptions such as ‘many respondents said …’; ‘most women mentioned
…’, etc. ?
 ‐ How long should the quotes be? If too long the paper may not be
publishable and the analytical dimension will be undeveloped; This is
critical for articles but can be got round in books where very long quotes
may be presented (see Cicourel 1974, Brand 2001)
‐ Should there be quotes for each point raised or can discourses be
summarized instead of quoted?
‐ Should the question which has led to the answer be systematically
included in the quote? This is an important issue because including
context allows the reader to assess data quality and identify leading
questions, but too much context increases the word count which may be
problematic for journal articles.
‐ Should the quotes be included in the written paragraphs or isolated in
the text? Although this is more an issue of style; including quotes in the
written paragraphs may appear as a fluid way to incorporate the data in
the text but requires that the excerpts are small soundbites taken out of
context.
‐ Should each quote be commented on and analysed or should they just
serve as illustrations allowing the reader to perform their own analysis.?
‐ Qualitative research often includes different types of supporting
information: field notes, photos, informal discussions, etc. To what
extent should these be reported on and used in the paper, thus breaking
with the traditions in demographic publications based on data gathered
systematically? This issue is important as our ability to analyse what
respondents say depends on this ‘parallel’ work of presence, informal
discussions, observation, etc. How do we include these dimensions into
the results and conclusions?

Most of these issues remain up for debate but may be important in trying to
publish qualitative research, especially in traditional demographic journals.

In order to assess the quality of qualitative research there are various

dimensions: the appropriateness of the methods used; identification and
selection of respondents; the organization of fieldwork; and the way the
analysis was conducted. The presentation of quotes is the visible part of the
iceberg of the research process and the choices made in their selection needs
to be documented. It is clear that all this detail cannot always be included in a
paper’s methods section because of space constraints. On the other hand these
dimensions are key in the assessment of the quality of the data and should at
least be addressed while reviewing papers; in other words, reviewers should be
made aware of these aspects and not hesitate to require authors to justify (not
in the paper necessarily) their research design in more details. Website
references can be a useful tool here where details of research methods can be
posted. Furthermore there are debates around the free access to the whole
qualitative database and researchers with confidence in the quality of their
data should find ways of making this available once suitably anonymised.

3.2 Assessing quality: published demographic literature

This section reviews a sample of articles using qualitative methods (both singly
and alongside quantitative methods) published between 2001 and 2005 in major
demographic journals15, assessing them for the quality of methodological
reporting.

3.2.1 Method
We selected a subset of articles to perform this analysis. One author did an
exploratory analysis for 13 articles chosen randomly (without paying attention
to the journal title). She read each article entirely and handled each article as
if she was doing an article review for a journal. From this analysis, she
developed a “rating tool”. She stopped reading new articles when she reached
saturation and her rating tool remained stable. She completed the rating tool
for each article, and analyzed the results of the rating item by item (see
below). The results are summarised separately for operation-research articles

15
To select our sample of articles, we replicated the search of Randall and Koppenhaver (2004)
for the 2001-2005 period. As these authors did, we searched the following journals:
Demography, Perspectives in Sexual and Reproductive Health, International Family Planning
Perspectives, International Migration Review, Population –English edition, Population Studies,
Population and Development Review, Studies in Family Planning. All these journals are on
JSTOR, except for International Migration Review. As they did we used the following key words
to search the abstracts: anthropology / gical / gist, ethnography / phic / pher, qualitative,
focus groups, in-depth / semi-structured interviews, participant observation. We then
experimented with different keywords in the same set of journals and time period. We
introduced the following keywords: sociology / gical / gist, mixed methods, and did not find
any additional papers using qualitative methods (we did not consider theoretical or review
articles whose abstracts mentioned one of these words). To be consistent, we eliminated 5
articles which were reviews of qualitative studies or referring only to anthropological theory,
without use of qualitative data. Also, noticing variations in the spelling of some of the
keywords of interest, and the use of “interviews” as a short-hand for qualitative interviews, we
introduced the following key words in the search: focus-group, focus-groups, semistructured
interviews, interviews. With these new key words, we found 21 additional articles using
qualitative data in the set of journals for the 2001-2005 period. Altogether, we thus identified
70 articles for the present analysis. Focusing only on articles available in JSTOR (that is,
eliminating International Migration Review), we worked with 61 articles.
Our sampling criteria maximize ease of access (of concern for the Ouagadougou-based author).
The drawback to the JSTOR database is that the articles selected are not among the most
recent ones, but the phenomenon of interest is not likely to change rapidly. Also, the JSTOR
database does not contain every “demographic” peer-reviewed journal. Finally, JSTOR over-
represents journals dealing with family planning. Out of the 61 articles, 16 were published in
either Demography, Population, Population Studies or Population and Development Review,
and the remaining 45 in International Family Planning Perspectives and Perspectives on Sexual
and Reproductive Health. To minimize this problem, we aimed at selecting for the analysis the
same number of articles from each journal.
(tests or design of reproductive health interventions) and for fundamental-
research articles (all other studies), because these two types of research have
very different methodological needs. Another author then applied this grid to
13 additional articles (chosen to even out the number of articles by journal
title); this second analysis modified only marginally the results of the first
analysis.

Each article is first graded for its overall methodological strength (evaluating
only the qualitative part). To assess this parameter, we ask whether the
research question has been answered with the methodology used. The answer
is either “poor”, “medium” or “good” (poor is taken here as the equivalent of
a “reject” diagnosis when reviewing an article submitted to a journal, medium
is taken as a “revise and resubmit” answer, and good is the equivalent of
“accepted”). Each article is then graded for the completeness of its
methodological description, along six items, scoring 0 or 0.5 or 1 according to
the degree of completeness. The six items are:
1) Rationale for the qualitative data collection tools chosen mentioned
2) Presence of elements which enable the reader to verify the quality of
the material collected: information on interviewers, their training,
information on data format, language and translation, on interview
conditions; assessment of the quality of the displayed qualitative data
3) Description of the choice of site, sample recruitment, sample size, and
rationale for site/ sampling criteria and size
4) Description of then analytical method used
5) Display of results, and possibility for the reader to verify the
demonstration
6) Cross-validation of the results.
Development of this grid was informed by the authors’ years of reviewing
qualitative studies submitted to peer-reviewed journals.

3.2.2 Results
Out of the 26 articles selected (from a total of 61), two made only minor use of
qualitative data, and we discarded them from the analysis. Of the remaining 24
articles, five were operation-research articles and 19 were fundamental
research studies (Appendix 2).

3.2.2.1 Qualitative data and analysis in operation-research

The reviewed action research studies (5) have overall a fairly appropriate use
of qualitative methods: 3 are rated as methodologically “good” and 2 as
“medium”.

I. Data collection method

Focus groups (FG) are used in all these studies, often combined with in-
depth interviews (IDI), and in one case ethnographic notes, and in another,
free listing techniques. However, the reason for the choice of data
collection tool is never discussed
II. Quality of the collected data
Most of these articles provide information on the FG composition and on the
characteristics of the moderator, on the interviewer who conducted the IDI,
on the format of the data (taped and transcribed), language and
translation. Short quotes are used in the display of data (except in one case
see below), which do not allow the reader to assess personally the quality
of the data, for example, to what extent responses are induced by the
interviewer.

III. Choice of site and respondents

The sampling criteria are usually explained (the full range of diversity
among clients and providers is usually aimed at), and possible impacts of
recruitment specificities on results are discussed. The size of the sample
however is never discussed, but the clinical design (a “before” “after”
design) usually renders the question of number irrelevant.

IV. Analytical method used

All these studies use thematic (content) analysis, and usually described it as
such. This analytical approach is adapted to the goal of these studies, which
is descriptive. The exception is the “medium” rated article, which does not
describe its analytical method. In that article, the quantitative results are
placed in the “results” section and the qualitative results are introduced
only in the discussion (no quotes), giving a different status to the two
bodies of data and rendering it impossible to verify the quality of the
qualitative data and demonstration. We return below to the issue of
researchers treating qualitative data, that is the content of individuals’
discourses, as “the final truth” or as “the scientific answer” to their
question.

V. Display of results
Qualitative results are usually illustrated with numerous short quotes (often
not contextualized beyond the source: provider or client). Usually one
quote is provided for every point made (every result), so that readers can
verify that the analysis of the interventions’ failure, success or planning is
sound. Often results are translated into concrete policy recommendation.

VI. Cross-validation of analysis and results

Four studies contrast the points of view of providers and clients on the
intervention, and 2 out of 5 also use quantitative data to rate programmatic
success or failure and contrast these results to the qualitative findings. In
one study the contrasted groups are differentiated by ethnic background. In
one study, cross-validation of thematic coding is performed, and in another
one, cross validation is done through a sequential type of data collection.
There is at least a minimal check on the validity of the results in every
operation-research study reviewed, and often more than one check.
3.2.2.2 Qualitative data and analysis in research
If the use of qualitative data and analysis is overall satisfactory in operation-
research published in demographic journals, more methodological problems
appear when researchers use a qualitative approach to study social phenomena
in the same discipline. Out of the 19 research articles reviewed, only 11 are
rated as “good”, 5 as “medium”, and 3 as “poor”. Studies rated as “good” or
“medium” vary substantially in the quality of the methodological description
(“good” articles have sometimes very short methodological description,
because of the use of short-hand such as “grounded theory”, “standard
ethnography”, “life histories”; the methodological description may be spread
throughout the text). However, it is very significant that all studies rated as
“poor” are lacking methodological information.

I. Data collection method

None of the 19 studies explains their choice of method (FG, IDI, participant
observation). However, these are exactly the kind of papers where you
would expect a thorough discussion of the tools. Since different types of
tools are adapted to different types of research question, these choices
need to be made explicit.

II. The quality of the collected data

Most article report on IDI data format, which are always taped and
transcribed and sometimes translated. Few articles (but two out of the
three “good” articles) report on who the interviewers are and their training
and interview conditions; this information is crucial to identify possible
interviewer bias (for example, in one case, the interviewer is a White male
researcher and the respondents are non-White young males). One test of
the quality of IDI data is first-hand access to the data themselves. This
entails displaying long quotes for IDI (entire interviews or interviews
extracts would be ideal, but not possible in an article format). Only 2 out of
the 19 articles display long quotes. In one case, the article is rated as
“good” and the data shown is of excellent quality; in the other case (a
study rated as “medium”), the data shown is of very poor quality, although
this information could not be deduced from the description of data quality.
Poor quality IDI data is to be inferred when answers are short and
interviewer’s questions or probes are numerous (the likelihood of
interviewer’s bias is then very high).

III. Choice of site and respondents

Site(s) and respondents choice are linked to the stage of the research
(exploratory, theory building, theory testing). In exploratory research, one
would choose a typical place and respondents (both engaging in and
avoiding the practice of interest). In theory-building studies, one would look
for maximum diversity given the phenomenon of interest. In theory testing,
 one needs to have the comparison groups necessary according to the
hypothesized factors at work. All studies describe the choice of sites and
sampling criteria. However, they do not usually relate their options to the
type of research conducted.

Sample size is related to the previous point: in exploratory research, the in-
depth interrogation of a few respondents or in-depth observation of a small
site is relevant. In theory-building or testing studies, maximal diversity has
to be reached, which implies a relatively large number of sites and
observations or interviews. Ideally, the size of the sample is not set in
advance, but is defined during fieldwork, when the researchers “reach
saturation”, that is, does not meet new situations when conducting new
interviews or observations. For planning and budgeting reasons, it is often
difficult to follow the principle of saturation, but a relatively large number
of cases can be assumed to enable saturation. Saturation is reached earlier
when the sample is limited to one group, and more interviews are necessary
to reach saturation in several targeted groups (and thus typically in theory-
testing studies). None of the 19 articles discusses sample size or saturation,
although all “good” and two of the “medium” studies feature a large
enough sample size to infer saturation purposes. One “poor” and one
“medium” study suffer from an ill-defined study purpose, mixing an
exploratory design (a typical place, typical respondents, few respondents,
many observations on the respondents) with a theory-building design (foster
as much diversity as possible by diversifying sites and recruitment criteria,
and a single interview).

IV. Analytical method used

The analytical method used is always described very succinctly: most articles
mention that they use “coding”, one mentions coding and “grounded theory”,
one mentions “typology building”, and one mentions coding and a comparative
study design, one mention the method of “emphasis of response”. The 4 latter
studies are rated as “good”. “Coding” is a vague notion, and insufficient to
describe qualitative data analysis in fundamental research. The analysis can
involve thematic coding in a first, descriptive analytical stage, but needs to go
beyond it, since the idea is to understand the structuring dimensions of the
phenomenon of interest. As mentioned already, the method used to get at the
underlying social factors of a phenomenon is comparative analysis. Theory
testing studies imply a comparative study design from the start, and narratives
coded into themes are thus compared across different groups. Thematic coding
is not sufficient to explain this procedure: a more exact wording would be
“thematic coding” and then “comparison of themes across pre-defined
groups”. Theory-building studies also involve comparisons. The groups of
comparison are identified through data analysis (typology building or grounded
theory): the cases or situations are compared one by one and grouped, and
contrasts between groups help the researcher understand what dimensions
differentiate the groups, that is, what social factors structure the practice or
behaviour. “Coding” is not enough to describe this analytical procedure,
although it is part of it; narratives are coded, and then cases are compared
across themes, and then grouped, and the groupings further analyzed. As
already mentioned, even exploratory type research involves comparisons:
researchers compare minimally between situations of norm compliance and
deviant situations to uncover the main social mechanisms underlying shared
representations and practices (see the importance of the study of deviancy in
anthropology); here, given the smaller number of cases involved, a formal
coding is not always necessary.

The majority of the studies relying on thematic coding use a qualitative data
coding software. However, using a qualitative software is not related to the
final grade received by the paper. Articles rated as “poor” use a software, and
articles rated as “good” do not use a software, and the reverse is true too. (As
a side note: one of the “medium” rated study using a software states that the
software processes the data and does the coding, which is wrong: these
software enable the researcher to code a text into the codes of his or her
choice, and to retrieve easily the texts coded under a given category).

V. Display of results
Most studies display multiple quotes (usually short quotes, which makes it
difficult to assess data quality); and the quotes are usually situated (at least
some characteristics, but some go into details into each case). Both the
content of the narrative and the characteristics and life history of the
respondent allow for the reader to check on the argumentation, and the more
details we have on both, the easiest it is for the reader to judge of the validity
of the results. Because of the constraints of the article format however, only
two articles provide large amounts of suited narratives (the two articles are
graded as “good”). A few studies prefer to summarize the results without
quoting “data”. They can as well and as satisfactorily answer the research
question, but the reader has less opportunity to judge the validity of the
results.

VI. Cross-validation of results

Most “good” or “medium” studies cross-validated coding or contrast qualitative
and quantitative results. Studies with no cross-validation strategy are rated as
“poor” or “medium”.

3.2.3 Discussion
Qualitative studies of poor quality usually have incomplete methodological
descriptions, suggesting that studies of lesser quality are conducted by
researchers who are less aware of the methodological standards of qualitative
research, and thus less able to describe and defend them. Some excellent
qualitative studies display complete but very short methodological information,
which is fine among knowledgeable readers, but does not help promote good
standards for qualitative research in a quantitative field. Editors, professors,
tutors, funders and other research standards setters could increase the quality
of qualitative research in demography by requesting complete and detailed
methodological description in the works they review.

Qualitative studies in the field of demography could also gain in quality by

paying more attention to issues that are important to quantitative studies in
the field: data quality and cross-validation. Key to qualitative data quality is
the interviewer or observer her/himself, so detailed description of the
interviewer – interviewee interactions and discussion of possible biases are to
be promoted. Similarly, systematic display of longer quotes (along with greater
awareness of the signs of poor qualitative data quality) will improve quality
checks. Editors need to be aware of the implication of short article length on
qualitative studies, and should be encouraged to allow for longer texts16. Cross-
validation of the analysis, and not only cross-validation of coding, encourages
reliability; multiple modes of cross-validation should be encouraged (existing
results, other data sources, confronting the narratives of respondents with
diverging interests, etc.)

But the main problem with qualitative research published in demography (aside
from operation-research) is that the methods of qualitative analysis (grounded
theory, typology building, comparative analysis) are poorly reported, and
appear to be poorly understood. The development of qualitative curricula and
teaching material may help here, especially since researchers who perform
these analyses well do not explain how they proceed, perhaps because their
readership is usually made of other specialists. Many of these authors have
developed these techniques after long immersions in another discipline:
sociology, gender studies, psychology, anthropology.

4. Discussion

When we read or review research using qualitative approaches, we will

inevitably apply criteria as to whether the research is successful or not. In the
same way that an experienced quantitative researcher can distinguish between
a good and a poor study, so can an experienced qualitative researcher. For a
quantitative researcher, or a novice qualitative researcher the evidence
presented by a piece of qualitative research is likely to look very different from
that produced by, say, secondary analysis of a large-scale quantitative dataset.

Extended “checklists” will be unlikely to gain (universal) endorsement from

those trained as ethnographers, but might be attractive to demographers who

16
Some journals, especially in the medical sciences, permit longer maximum word lengths for
qualitative as opposed to quantitative articles.
want to better understand how credible the qualitative data are. This category
not only includes reviewers, but also includes those who have not collected
their own data but have subcontracted data collection to others, which is often
done in multi-method research projects. Our aim here is not to achieve
consensus –unlikely and ultimately undesirable given the breadth of qualitative
epistemology – but to identify “points of entry” for the non-specialist. It is of
value to have a framework for appraising the quality of qualitative research, so
long as it is used with care. By drawing together the state of the art in terms
of frameworks for quality assessment with experiences of primary data
collection and research reviewing, we demonstrate how common-sense
approaches can improve the quality of qualitative research and its reviewing.
For many qualitative researchers, concerns about the “criteriology” of
assessing quality are well-founded if criteria become highly procedural,
mechanistic and prescriptive. What is at issue here is thinking about informed
judgement of quality, both when research is being done and when it is being
read by others. Many of the frameworks use open-ended questions to prompt
judgement, and take into account that quality standards will inevitably be
shaped by research question(s) and whether they have been answered.

There are, however, important practical issues (and solutions) raised by these
assessment of quality. For example, transparency in reporting is linked to
issues of under- and non-specification, especially when research reporting is
constrained by journal word counts. Transparent research can be presented in
a way that can be both appraised and (re-)used by others. Authors should be
able to make innovative use of internet-based repositories of their research
instruments and/ or data. At the very least, authors can make available copies
of their research instruments.

Assessing quality needs some expertise in the conduct and use of qualitative
research, necessitating improved support, education and guidance for non-
experts. Insight and interpretation arte key aspects of qualitative research,
and are the most difficult to appraise. But any research needs to have, and
justify sound methods and defensible conclusions. By focusing on both the
production and consumption of qualitative research in demography we hope to
shed light on what is involved in being a good researcher in general. When we
read, or review, research using qualitative approaches, it is inevitable that we
should apply criteria for deciding what a good study is and what is not. By
highlighting what is involved in producing and reviewing qualitative research in
demography we contribute to an articulation of what is often implicit – the
decision about what is good and what is not.
Bibliography
Allison, Paul D., 1984, Event History Analysis. Regression for Longitudinal
Event Data, Sage University Paper No 46.
Barker, Gary K et Susan Rich, 1992, « Influences on Adolescent Sexuality in
Nigeria and Kenya: Findings from Recent Focus Groups Discussions », Studies in
Family Planning, 23,3: 199-210.
Basu A. M. And P. Aaby (1998). The methods and uses of anthropological
demography. Oxford : Clarendon Press.
Bernardi L. and I. Hutter (2007). “Anthropological Demography”, Demographic
Research 17(18): 541-566.
Bledsoe C, F.Banja, A.G.Hill, 1998. Reproductive Mishaps and Western
Contraception: An African Challenge to fertility theory. PDR 24(1)
Boaz, A. and D. Ashby (2003). Fit for purpose? Assessing research quality for
evidence-based policy and practice. ESRC UK Centre for evidence based
policy and practice Working Paper: 18.
Bonvalet C. and Z. Andreyev (2003). « The Local Family Circle”, Population
(English Edition), 58(1), 9-42.
Bozon M. (2006). “L'apport des méthodes qualitatives en démographie ».
Chapter 136 in : / Démographie : analyse et synthèse. 8, Observation,
méthodes auxiliaires, enseignement et recherche . - Paris : INED, p.
439-463.
Brand Saskia (2001) Mediating means and fate, Leiden, Brill
Bryman, A., S. Becker, et al. (2008). Quality Criteria for Quantitative,
Qualitative and Mixed Methods Research: A View from Social Policy. 11:
261-276.
Calves, Anne-E. 2000. “Premarital childbearing in urban Cameroon: Paternal
recognition, child care and financial support”. Journal of Comparative Family
Studies 30(1): 443-452.
Castle S., M.Konate, P.Ulin, S.Martin, 1999. A qualitative study of clandestine
contraceptive use in urban Mali Studies in Family Planning 30 (3)
Cicourel A (1974) Theory and Method in a study of Argentine fertility New
York, Wiley
Cleves, Mario A., William W. Gould et Roberto G. Gutierrez, 2002, An
Introduction to Survival Analysis using STATA, Stata Press.
Coast, E. (2003). “An evaluation of demographer’s use of ethnographies”.
Population Studies 57(3), 337-346
Coast, E., Hampshire K., Randall S. (2007) “Disciplining anthropological
demography”, Demographic Research 16(16): 493-518.
De Loenzien Myriam, Simon Yana David, Les approches qualitatives en
démographie. Théories et applications, 2005
Dixon-Woods, M., R. L. Shaw, et al. (2004). "The problem of appraising
qualitative research." Qual Saf Health Care 13: 223-225.
Dixon-Woods M, Sutton A, Shaw R, Miller T, Smith J, Young B, Bonas S, Booth A
and Jones D (2007) ‘Appraising qualitative research for inclusion in systematic
reviews: a quantitative and qualitative comparison of three methods’ in J
Health Serv Res Policy vol 12 no 1 Jan 2007
Gagnon, A. & N. Mondain, 2008. “Démographie et santé mondiale : un
carrefour indispensable ». In: V. Ridde & K. Mohindra (eds) Développer et
pratiquer la recherche et l’enseignement en santé mondiale au Québec, Actes
du colloque Développer et pratiquer la recherche et l’enseignement en santé
mondiale au Québec Présenté dans le cadre du 75e Congrès de l’Acfas, à
l’Université du Québec à Trois-Rivières, le 11 mai 2007
Gokah, T. 2006. The Naïve Researcher: Doing Social Research in Africa, Int. J.
Social Research Methodology, Vol. 9, No.1, pp. 61–73
Greenhalgh, S. (1997) “Methods and meanings: Reflections on disciplinary
difference”. Population and Development Review, 23(4), 819-824.
Gueye, Mouhamadou, Sarah Castle et Mamadou Kani-Konate, 2001, « Timing of
First Intercourse Among Malian Adolescents: Implications for Contraceptive
Use », International Family Planning Perspectives, 27,2: 56-62.
Holloway & Wheeler (1996) Qualitative research for nurses Oxford: Blackwell
Science
Kertzer, David I et Tom Fricke, 1997, Anthropological Demography. Toward a
New Synthesis, The University of Chicago Press
LeGrand L., T. Koppenhaver, N. Mondain and S. Randall (2003). “Reassessing
the Insurance Effect: A Qualitative Analysis of Fertility Behavior in
Senegal and Zimbabwe”, Population and Development Review, 29(3),
375-403
Kertzer, D., & Fricke, T. (1997) Anthropological Demography. University of
Chicago Press: and other problems”, Demographic Research 11(3): 57-
94.Chicago.
Kirshenbaum S. B., A. E. Hirky, J. Correale, R. B. Goldstein, M. O. Johnson, M.
J. Rotheram-Borus and A. A. Ehrhardt (2004). “"Throwing the Dice":
Pregnancy Decision-Making among HIV-Positive Women in Four U.S.
Cities”, Perspectives on Sexual and Reproductive Health, 36(3), 106-113
Matthews, S. 2005. “Crafting Qualitative Research Articles on Marriages and
Families”, Journal of Marriage and Family 67: 799–808
Mondain, N., LeGrand, T. & Delaunay, V., 2004. L’évolution de la polygamie en
milieu rural sénégalais : institution en crise ou en mutation ? Cahiers Québécois
de Démographie, 33(2) : 271-308
Mondain, N., T. LeGrand et P. Sabourin. 2007. “Changing Patterns in Men’s
First Marriage among the Sereer in Rural Senegal”, Journal of Comparative
Family Studies, 38(4): 627-644
Mondain N. et Delaunay V. 2006. « La vie avant le mariage : les grossesses
prénuptiales chez les Sereer Siin au Sénégal. » In : Enfants d'aujourd'hui :
diversité des contextes, pluralité des parcours : tome 2. Paris (FRA) ; Paris :
AIDELF ; PUF, p. 799-814
Mondain, N., S. Lardoux, S. Bignami and A. Gagnon, 2008. “Transition to
adulthood among African immigrant adolescents in Western countries: evidence
from a qualitative study in Montreal, Canada”, paper presented at the EPC
Conference, Barcelona, July 2008
Mondain, N., S. Randall and A. Diagne, 2009. “Assessing the effects of out-
migration on those left behind in Senegal: local family dynamics between
change and continuity”, paper accepted for oral presentation at the IUSSP
Conference in Marrakech, Sept-Oct 2009
Mondain, N. & E. Bologo, 2007. “Ensuring the sustainability of DSS by making
scientific results available to lay persons: case studies in Senegal and Burkina
Faso”, paper presented at the UAPS conference, Arusha, December 2007.
Onwuegbuzie, Anthony J. & Nancy L. Leech, 2005. On Becoming a Pragmatic
Researcher: The Importance of Combining Quantitative and Qualitative
Research Methodologies, Int. J. Social Research Methodology, Vol. 8, No. 5, pp.
375–387
Popay, J. (2006). The systematic review of qualitative findings. The 17th
International Nursing Research Conference Focusing on Evidence-Based
Practice.
Randall, S. and KoppenhaverT. (2004) “Qualitative data in demography: The
sound of silence and other problems”, Demographic Research, 11(3): 57-
94.
Randall S. et T. LeGrand, 2003. “Reproductive Strategies and Decisions in
Senegal: the role of child mortality” Population-E, 58(6): 687-716
Randall, S., N. Mondain and A. Diagne, 2009. “Absent parents in modern
Senegal: consequences for child wellbeing”, paper accepted for oral
presentation at the IUSSP Conference in Marrakech, Sept-Oct 2009.
Riley N et J. McCarthy. 2003. “The essentials of demography”, in Demography
in the Age of the Postmodern, Chap.3, Cambridge University Press,
Cambridge, p.35-60.
Rolfe, G. (2006). "Validity, trustworthiness and rigour: quality and the idea of
qualitative research." Methodological Issues in Nursing Research: 304-
310.
Sandelowski, M., S. Docherty, et al. (1997). "Qualitative metasynthesis: issues
and techniques." Research in Nursing 20: 365-371.
Seale, C. (1999) The quality of qualitative research Sage
Spencer, L., R. J., et al. (2003). Quality in qualitative evidence: a framework
for assessing research evidence: a quality framework. London, Cabinet Office
Government Chief Social Researcher's Office.
Tashakkori, A., Teddlie, C., Mixed Methodology : Combining Qualitative and
Quantitative Approaches, London, Sage, 1998.
Walsh D and Downe S (2006) ‘Appraising the quality of qualitative research’ in
Midwifery vol 22
Appendix 1

Year Title of project Methods and team Publications

1999 Rock-Project: relation Individual and couple in- LeGrand T.,
between the decline in depth semi-structured Koppenhaver T.,
child mortality and interviews Mondain N., et Randall,
fertility decision making Focus groups. S., 2003.
in Senegal and Team: man/woman pairs Randall & LeGrand,
Zimbabwe in the 3 sites (Dakar, 2003
small town and village); Randall & Koppenhaver,
international team of 2004
principal researchers
1999 PhD project: changes in Individual in-depth semi- Mondain et al, 2004
marriage behaviour in structured interviews Mondain & Delaunay,
rural Senegal: the case of Focus group discussions 2006
the Sereer (only exploratory) Mondain, et al, 2007.
Individual structured
interviews
Team: the principal
researcher and 2
interpreters, one male,
one female
2005- Projet “Jeunes en Focus group discussions Gagnon & Mondain,
2006 transition” in Montreal, Individual in-depth semi- 2008
Canada: first generation structured interviews Mondain et al, 2008
African immigrant youth Team: 2 men and 2
transition to adulthood women, one of them
supervising the project
for her PhD thesis; one
principal researcher
2007 Projet: “Les effets de la Individual in-depth semi- Randall et al, 2009
migration sur les structured interviews Mondain et al, 2009
dynamiques familiales Life histories
locales : le cas d’une Team: 2 men and 2
petite ville au Sénégal » women; 2 principal
researchers
2008+ Observatoire de Preliminary qualitative None
Population phase characterizing
d’Ouagadougou residential districts prior
to establishment of DSS.
Rapid appraisal,
observations, key
informant interviews,
informal discussions,
both individual and
group
Appendix 2

1 Tools, and Description of Selection criteria Method of analysis Evidence, Cross- Overall methodological
rationale for data collection demonstration validation quality
choosing conditions and Number of IDI /
these tools interviewers, FG
language, data
format
Zulu 2001 FG: to elicit No Three areas of Use of a software Quotes are used Comparison MEDIUM
Demography norms information on the country, (Nudist). After a first (= “clearest with Argues that macro quali
who collected ethnically reading, development articulations of quantitative studies are not enough,
qualitative + IDI with the data and diverse, with of first broad codes statements and survey data micro quali studies are
quantitative individuals: how different post- of norms about views necessary to make causal
understand partum length abstinence and their expressed”) No cross- inferences (earlier work
Malawi relevance of Local rationale. Then, validation of showed longer abstinence
these norms to language, Random list of development of finer Short quotes, codes in patrilinear societies).
Post partum behaviour transcribed individuals in codes when coding many quotes Problem: he explain why
abstinence and translated the survey line by line in Nudist. abstinence is longer in the
IDI of into English at village (each= Goal: diversity of Northern region (the end of
“special the same time common norms within each abstinence is linked to
informants”: sampling frame), area, and contrasts return of menstruation
what norms (data recorded: first selection for between the three there), but no discussion of
they teach not stated, but 22 FG (10 areas. why that would be in
implied) participants ), patrilineal societies and not
we do not know in others.
how, the rest of
the list for IDI The explanation remains at
the individual level
22 FG, and 61
in-depth
interviews

No rationale for
why that
number, but
number high
Completeness 1 0,5 1 1 0,5 1 5
of
methodological
description
2 Tools, and Description of data Selection criteria Method of Evidence, Cross- Overall
rationale for collection analysis demonstration validation methodological
choosing conditions and Number of IDI / FG quality
these tools interviewers,
language, data
format
Otoide et al. 2001 20 FG with Team of Selected to represent the No method Some short quotes (in No POOR
IFPP women researchers diversity of youth in one city of analysis pidgin with
aged 15-24 conducted the FG. (geographic location within mentioned translation or
Nigeria (149 Nothing on their city, occupation of youth, ..). english) as
women in characteristics The sexual and contraceptive illustrations, within
Why do young total) (especially sex) practices of youth in the city the text, not
people use are said to be the same as in contextualized
abortion and not no In English and the rest of the country.
contraception? pidgin English
Nothing on why 20 FG, but
FG taped, and then presumable, same criteria
transcribed. (have enough diversity)

Notes taken during

the FG.

No need for
translation (local
investigators)
Completeness of 0 0,5 1 0 0,5 0 2
methodological
description
3 Tools, and rationale Description of data Selection criteria Method of analysis Evidence, Cross- Overall
for choosing these collection conditions demonstration validation methodological
tools and interviewers, Number of IDI / FG quality
language, data format
Legrand et al. Individuals and The urban and rural sites First coding from the two Numerous cross-validation GOOD
2003 PDR couples IDI (each Same sex were chosen to represent country teams put in quotes, on coding
spouse separately) interviewers; diverse socio eco common into one, codes relatively items, analysis
Test of the description of contexts, within the two on pre-defined questions long, situated performed in
insurance effect Focus groups interviewers training countries’ largest ethnic or themes and emerging each country
on fertility and supervision group area theme codes. and the
behaviors FG shown to be of NUD IST software. discussion on
little use among the Because field staff IDI Random choice the results
Zimbabwe and Wolof (do not like to was different, within set categories to Explanatory analysis:
Senegal discuss these issues variations across maximise diversity. comparison across rural
in public), but useful countries in the and urban and Z and S to
with the Shona (not topics most covered FG: not random choice, understand social-eco
the same problem) but respect pre-set change and reproductive
Discussions with categories of participants attitudes. Comparative
Less couple field staff to design not explained in the
interviews and focus understand what 72 IDI, 24 FG, 37 couple methodological appendix
groups in Senegal topics respondents interviews in Zimbabwe
too maximise did not like, silences
diversity because lots are part of the data to 122 IDI, 14 FG, 9 couple
of heterogeneity understand norms “ interviews in Senegal
the sound of silence”
Participant Numbers not justified,
observation in but large enough to
Senegalese villages insure saturation,
sampling strategy to
Gender (couple) maximise diversity
perspective not used
here
Completeness of 1 1 1 1 1 1 6
methodological
description
4 Tools, and Description of Selection criteria Method of Evidence, Cross-validation Overall
rationale for data collection analysis demonstration methodological
choosing these conditions and Number of IDI / quality
tools interviewers, FG
language, data
format
Schuler et al. IDI Nothing on Three rural and Thematic Lots of cases Providers and GOOD
2001 IFFP Some focus interviewers two urban sites; in coding using described to clients views are
groups each site sub- ethnographic illustrate points, contrasted,
Qualitative Some clinic Transcripts. areas close and software with always a quote discourses and
observations No mention of further from the SPdata within the observations are
Bangladesh During several language or clinique description of the contrasted
years to observe translation No further case
Evaluates the change, Selected through explanation
impact of a shift retrospective key informants
in national FP data prior and hospital
program change records, with set
Some criteria to
respondents maximize
interviewed diversity
several times
Unstructured Several hundred
ethnographic transcripts (IDI
notes and observations
in cliniques),
No rationale, but about 125 used
all tools used! for this article
Completeness of 1 0,5 1 0,5 1 1 5
methodological
description
5 Tools, and Description of Selection criteria Method of analysis Evidence, Cross- Overall
rationale for data collection demonstration validation methodological
choosing conditions and Number of IDI / quality
these tools interviewers, FG
language, data
format
Kirshenbaum et 56 IDI open-ended 4 cities Method based on grounded Many cases Of coding GOOD
al. 2004 PSRH questions, probes, Diverse theory. Two steps: a first described, with a (and of
No rationale different themes, recruitment primary coding was done. short quotes to analysis?
Abortion for the exact wording of points: clinics, Then a team of 8 researchers illustrate the Not clear)
decision making choice, but questions on associations, worked on 16 interviews to case
among HIV appropriate to reproduction, 2-3 adds,. refine coding, cross-
positive women the purpose hours long validation of coding
Discussion of the procedures. These
USA Information on implications of researchers coded 38
interviewers the age of the additional interviews (in
sample (relatively pairs), until saturation was
Interviews audio old compared to reached on analysis.
taped and similar studies) Then, researchers worked in
transcribed on the result pairs more in depth on
themes touching
No rationale for reproduction; new finer
the number, but coding, cross-validation of
large coding.
Explanatory analysis: results
compared across groups of
women, according to the
time of diagnosis and life
cycle stage
Completeness of 1 1 1 1 1 1 6
methodological
description
6 Tools, and Description of Selection Method of Evidence, Cross-validation Overall methodological
rationale data collection criteria analysis demonstration quality
for conditions and
choosing interviewers, Number of
these tools language, data IDI / FG
format
Gueye et al. 30 FG No description Male and Just mentions: Describes with With quantitative results POOR
2001 IFPP of interview female FG, “Coding and short unsituated
No guide urban and analysis using quotes the discuss the biases of Consider respondents’
rationale rural The reasons to have focus group discourse as being the
Quantitative and for this Nothing on data Ethnograph sex early (hypothetical cases are truth, not the reflection of
qualitative choice collection Purposive software” discussed, not personal an underlying social
procedures, on sample, cases), to explain order. The link between
Adolescent moderators screening for discrepancies with adolescents reason to
sexuality educational quanti data (where have sex early and macro
FG in local and marital women declare having social change is just
Mali languages taped, status early sex out of love, stated, not demonstrated.
translated into this reason does not
French and Nothing on appear in the FG)
transcribed choice of
number
Completeness of 0 0,5 0,5 0 0,5 1 2,5
methodological
description
7 Tools, and Description of Selection criteria Method of Evidence, Cross- Overall methodological
rationale for data collection analysis demonstration validation quality
choosing conditions and Number of IDI /
these tools interviewers, FG
language, data
format
Vanlandingham IDI with 10 Guide elaborated Selected from Coded and Numerous typical Cross- MEDIUM
et al. 2002, IPPF young men during 5 pre-test diverse entry analyzed using exchanges validation of
with lots of interviews. Open points (by one The Ethnograph (between coding (the Shows that young men
Taïland, male friends ended. Thai researcher software. Two interviewer and two authors have premarital sex less
premarital By lead author and Thai types of coding: respondent); long coded until often with prostitutes and
sexuality Complex with Thai research pre-formed quotes unables to they agreed, more with girlfriends
and intimate research assistant), codes rate data quality the first
topic assistant, in through personal (guidelines) and (which is poor) author coded The social context is one
exploratory secluded private contacts with emerging codes. the rest). of AIDS, but also
analysis area one level of Use of Thai urbanization and lowered
The impact of a unknown Use of results concept (in local parental control (women
Western between them matrix language in the having pre marital sex):
interviewer is (respondent x text) link with the context only
not discussed. Number not themes) stated, and the macro
No identifying discussed The goals is on features also just stated,
elements, different topics but well done (lots of
interviews to give the nuanced context and
recorded. dominant view references).
Transcribed into as well as the
Thai and then diversity of view Good information, and
translated into points. some good
English. methodological features
(double coding, etc.) but
big problems: very small
sample, and very directed
interviews
Completeness of 1 1 0,5 0,5 1 1 5
methodological
description
8 Tools, and Description of Selection criteria Method Evidence, demonstration Cross- Overall
rationale data collection of validation methodological
for conditions and Number of IDI / FG analysis quality
choosing interviewers,
these tools language, data
format
Adeokun et al. 24 focus No description of 24 focus groups (4 with nothing Results: only quantitative Yes, with MEDIUM
2002 IPPF 18 IDI with guide, nothing on clients, 4 with evidence. elaborated
providers format, interview providers, the rest in quantitative
Dual protection conditions, the community, Qualitative results brought up data
intervention, No interviewers different sex, age and in the discussion, no quotes,
Nigeria rationale socio-eco status to be explain the reasons for the Provider and
representative) quantitative results (some clients’ views
improvement but not that are contrasted
Nothing on IDI many)

The qualitative data explains

the relative lack success of the
program (time constraints for
providers, changing staff and
poor training on the job, and
most of all, men’s reluctance
to use condoms with their
wives, even female condoms).
Completeness of 1 0 0,5 0 0 1 2,5
methodological
description
9 Tools, and Description of Selection criteria Method of Evidence, Cross-validation Overall
rationale for data collection analysis demonstration methodological
choosing conditions and Number of IDI / quality
these tools interviewers, FG
language, data
format
Eastman et al. FG with Description of In different work Three researchers Lots of short cross-validation of GOOD
2005 PSRH parents (3) guide, by environments, defined major quotes in the text categorization by
and trained diversity of topics by reading (source not discussion between
Design of a adolescents facilitators, employees seeked, the transcripts; detailed, only researcher. Creation of
parental sexual (6) matched by groups stratified for each source of adolescents, a code book. Test of the
education gender to the by gender, child’s data, three parent or coding by having a new
program at work IDI with group age and gender. researchers then employer) researcher recode the
employers found the text transcripts with the code
US Interview and On a voluntary relevant to each book: Coehn’s kappas
No rationale, FG conditions base; this bias’ topic (written on to grade interrated
but choice described implication on the an index card), reliability.
appropriate results is discussed and classified
Transcripts them into themes
Adolescents within topics;
chosen in one
school with mixed
socio economic
and ethnic
students, groups
grade and gender
specific

Varied employers

No rationale for
numbers (but
exploratory stage
research)
Completeness of 1 1 1 1 1 1 6
methodological
description
10 Tools, and Description of Selection criteria Method of Evidence, demonstration Cross- Overall
rationale data collection analysis validation methodological
for conditions and Number of IDI / FG quality
choosing interviewers,
these tools language, data
format
Katz et al. 2002 IDI with Description of From the list of Reading of first Short quotes, stand alone, Quantitative GOOD
IFPP providers guide public health transcripts, many, no details on data (mystery
FG with facility with FP, coding scheme respondent (just provider or clients)
IUD use clients FG animated by random selection of designed, then client) compared to
a team of two 15 urban and 15 coding of the text qualitative
El Salavador people, a rural clinics; one of each transcript Explains well that the data
moderator and a provider selected with a softaware problem are a negative view
record keeper randomly in each (DTSearch) of this method by women, Clients and
clinic and the fact that provider do providers
IDI and FG in not talk about this method perspectives
Spanish, taped FG: randomly and encourage use, because compared
and transcribed selected 6 urban they do not feel confident in
into Spanish clinics and 4 rural inserting / removing it (lack
clinics; groups by of experience), and also
method used prescription writing implies
(sterilization, re- less effort.
supply method,
IUD)

No rationale for
numbers, but high
Completeness of 1 1 1 1 1 1 6
methodological
description
11 Tools, Description of Selection criteria Method of Evidence, Cross- Overall methodological
and data collection analysis demonstration validation quality
rationale conditions and Number of IDI / FG
for interviewers,
choosing language, data
these format
tools
Hardee et al. FG Guide described Two rural counties each Reading the Short quotes, Quantitative MEDIUM
2004 IPPF (in results in three provinces, transcripts and many; data for
No section!) different levels of eco identifying theme respondents contraception Concludes from the data
Link between rationale Moderated by development and FP and described to entrenched gender
one-child policiy staff of official success, within each representative inequalities (sex ratio,
and improved FP research county, seven township quotes. worse mortality for
women’s status center in China, randomly selected for women, many
met in various FG, local leaders helped In the results, we inequalities), but to
China, rural places, lasted 1 selected participants, see that results some improvements:
to 2 hours which included a bias are contrasted by decrease of son
(favourable towards state generation, to get preference, more day in
Not always FP policy), but not at social change, family decision,
possible to possible other ways; but not explained sometimes equal
match diverse opinions and in the analytical opportunities at work,
moderator with some negative ones show method possibility to earn
the group (sex that bias not total. money.
and age), which
would have Diverse groups of men Do not work on the link
been preferable and women, 5 to 12 between improved status
participants per group, and FP, which is just
FG secretly altogether 32 women and stated. (do not use the
observed by 232 men (?! Mistake), no links made by women
officials! number of FG, but themselves). Improved
Focus group apparently high. status may be a
discussion taped consequence of
(supposedly), economic growth alone.
transcribed into
Chinese, then
translated to
 English
Completeness of 0 1 1 0,5 0,5 0,5 3,5
methodological
description

12 Tools, and Description of data Selection criteria Method of Evidence, Cross- Overall
rationale for collection analysis demonstration validation methodological
choosing these conditions and Number of IDI / quality
tools interviewers, FG
language, data
format
Bonvalet 2003 IDI Very short 99 in-depth re- Construction of a Use of case to With GOOD
Population description: family interviews among typology of illustrate each quantitative
Bias of the and residential national survey families living type, with lots of data and other The information is
Why do people narrative (real histories, life respondents, of near extended quotes, very research on very short, but
live near their life is made of histories, but from which 37 lived family; not long. the same sufficient to
extended family? compromise) is the quotes, we see near extended explained beyond topic educated readers
discussed that the interviews family used for that. Enough quotes (life histories,
France were very open this analysis to verify the typologies);
Different types are analysis
Nothing on No explanation of explained in terms
interviewer and selection of of social
interview respondents within background;
conditions survey respondents ownership f family
(presumably: business, making
No indication on random) up for difficult
format (but there childhood.
must have been Nothing on
taped and number, but
transcribed; the reasonable given
quotes show very purpose
careful
transcription)
Completeness of 1 0,5 1 1 1 1 5,5
methodological
description
L1 Tools, and Description of Selection Method of Evidence, Cross-validation Overall
rationale for data collection criteria analysis demonstration methodological
choosing these conditions and quality
tools interviewers, Number of IDI /
language, data FG
format
Agadjanian FG: expert No information on Affiliation to Nothing about A few quotes Interpretation of GOOD
2001 interviews, who collected the the main methods (not always qualitative and
individual data and how religious explicitely specified whether quantitative part No argument for
Population interviews, and besides “research congregations The facto the from interviews of the study is not quail study at all: It
Studies participant team” (one guess of the Maputo author explores of FG) are used merged explicitly, is just presented as
observation in the authors did for suburb belt an hypothesis to illustrate but the conclusion an obvious
qualitative + specific events the use of the contrasting data findings which interpret survey component to
quantitative personal pronouns No information from differnt are mainly data using the understand the
“I” when about the religious summarized findings of the different interaction
Mozambique General describing the data number of communities qualitative study and communication
purpose: collection “I interviews or and in differnt implicitly going on within
Contracetptive understanding asked…”) FG settings different
Use the mechanisms congregations.
through which No information No reference to
church about the language previous
attendance or the publications by The research shows
affect transcriptions the author in that religiosity is
contraceptive (one guesses which the study relevant not per se’
use Portuguese from would be better but for it offers to
the use of words described women in particular
(specific aims in the quotes) opportunities for
are not specified social interaction
and results from (data recorded: and therefore
each method are not stated, but exposure to modern
presented implied by the behaviour like
together ) long quotes ) contraceptive use
Completeness of 0.5 0,5 0 0 0,5 0.5 2.0
methodological
description

L2 Tools, and Description of Selection Method of Evidence, Cross- Overall methodological

rationale for data collection criteria analysis demonstration validation quality
choosing these conditions and
tools interviewers, Number of
language, data IDI / FG
format
Perelli – Harris FG and No information Different Nothing about findings are the conclusions GOOD
2005 individual on who collected regions of methods of summarized , interpret survey
interviews: the data, but Ukraine and analyses no use of data using the The qualitative study is
PDR understanding the presumably various age quotes findings of the used to interpret
reasons for early research and education The author search qualitative puzzling demographic
qualitative + childbearing assistants 22 FG (15 for themes study in a trends like early
quantitative among women working in local with women explaning early convincing way childbearing and low
No differentiation language and 7 with childbearing with fertility.
Ukraine of the reasons to men with 8-10 little
use one or the No information people) differnetiation by
Fertility other method on language, social group (she Conclusions show that
transcription No works early childbearing and
information low fertility are
on the number consistent with cultural
of individual beliefs about health,
interviews gender, as well as
economic uncertainties
Recruitment : and practices of
Flyers intergenerational support
distributed at
metro stations
Completeness of 1 0 0.5 0 0.5 1 3
methodological !
description
L3 Tools, and Description of data Selection criteria Method of Evidence, Cross-validation Overall
rationale for collection conditions analysis demonstration methodological
choosing these and interviewers, Number of IDI / FG quality
tools language, data format
Schroder- 12 month Village was one of the It is not Individual Observation complete GOOD
Butterfill and participant Detailed information three extensive rural explicitly case studies and correct interview
Kreager observation and justification for communities in which mentioned but it used as data (where stronger A Great example of
interviewing men and field sites were conducted is clear from the illustration is the effect of social village study in which
PDR 2005 Semi-structured women separately and and in which levels of way in which desirability and methods are
interviews with for having at least two childlessness vary results are sensitive topics ) combined at all stages
Childlessness in all elderly over rounds, multiple (choice on east Java presented that of the research design
old age and 60 in one village perspectives and because highest summary tables A random sample of and data collection is
vulnerability of (203) triangulation with prevalence of elderly guided the choice households has been illustrated and
the elderly observation and daily childless) of case studies to given a questionnaire justified in a detailed
contacts (sensitive be studied in on intra household fashion
East Java Aim: collect life topics like adoption, depth economy and
Indonesia histories of all bad relationships etc..) 203 IDI in Kidul, Second exchanges and on
elderly over 60 round IDI with 45% of The authors elderly care and
in the village Childlessness is a the same sample and distinguish levels health so that a
and to compare stigma and a reason to interviews with one of support across typology of economic
ego and alter divorce. related person of the social parity and differences locally
information in elderly for 50% of the number of relevant could be
relation to Children with no sample marriages. developed
pattern of contact to parents were
support not mentioned in the Census of the elderly of
first interview the village (9 missing
justified)

Completeness of 1 1 1 0.5 1 1 5.5

methodological
description
L4 Tools, and Description of Selection Method of analysis Evidence, Cross-validation Overall
rationale for data collection criteria demonstration methodological
choosing these conditions and quality
tools interviewers, Number of IDI
language, data / FG
format
Johnson hanks Ethnographic Data collected 2 communities “standard A few quotes (not Interpretation of GOOD
2003 material by author where anthropological always specified qualitative and
through three maximum practice” = whether from quantitative part of The account of the
Population General stage fieldwork contrast of “establish the interviews of FG) the study is not socially meaningful
(english purpose: (described in school generative are used to merged explicitly, processes related to
edition) understanding another paper participation, principle of action illustrate findings but the conclusion education and
the relation referenced) marriage delay from disparate which are mainly interpret survey fertility is
qualitative + between and fertility cases and summarized data using the convincing and
quantitative education and examples” findings of the pertinent to the
childbearing main trend qualitative study initial question
Cameroon preferences exploratory implicitly
The examination of
Education and comparing the concepts of
fertility norms on selfhood,
premarital sex individuality
concept of self independence and
and marriage are
independence, examined in context
individuality and related to
reproductive
practices
Completeness 0.5 0,5 0.5 1 0,5 0.5 3.5
of
methodological
description
L5 Tools, and Description of Selection criteria Method of Evidence, Cross- Overall methodological
rationale for data collection analysis demonstration validation quality
choosing these conditions and Number of IDI / FG
tools interviewers,
language, data
format
Ville et al 2005 Exploratory semi- This information “Varied and relatively Not Extensive No GOOD
structured is in a footnote complex life mentioned quotes from the
qualitative + interviews to and does not say trajectories” main trends interviews The specific aim here
quantitative capture forms of anything about exploratory was to construct a
identities and data collection 22 interviews (11 men questionnaire which
Population themes composing and 11 women aged 23 would incorporate a
(english edition) them to 92) valid appraisal of
identity. This is well
France explained

Identity and
migration
Completeness of 1 0,5 0.5 0 1 0 3
methodological (because I think that a
description word on how these
variation is achieved
woud not have harmed
comprehension)
L7 Tools, and Description of Selection criteria Method of analysis Evidence, Cross- Overall
rationale data collection demonstration validation methodological
for conditions and Number of IDI / quality
choosing interviewers, FG
these tools language, data
format
Finer et al 2005 In depth 38 in-depth Hospital based Transcription, correction Summary by Constant MEDIUM
interviews interviews with sample selected and top down coding – authors of what comparison
Perspectives on (does not women in four for variation in validity check by cross respondents of coded The qualitative is used
Sexual and mention sites access to and authors reliability of the said and short interviews for illustration
Reproductive why Content parallel reimbursement for coding (software aid N6) illustrative and results purposes only – no
Health interviews to the one of the abortion – all quotes from survey major theory or
and not structured abortion patients Problem: the authors data analysis exploration is carried
qualitative + FG) survey - could participate mistakenly think they can out or cross validated
quantitative interviews get at the factors of with survey data
during medical Reflection on the abortions by asking
US visit – English descriptive of the women why they abort, Indeed, no real
language and sample and its age instead of analyzing the analysis since they
abortion cash bias and reasons situations which lead look for the answer in
compensation – for it women to abort, and the wrong place!
in depth concluding on the POOR? (I think it can
followed reasons to abort from the stay with medium
structured latter analysis (either by given that even it is
interviews focusing on main factors just illustrative it is
present in all situations, relatively decently
or by getting at certain carried out. >But if
factors by contrasting you feel strong about
different groups of POOR than change it)
abortion situations)

In fact, such an analysis

would have needed to
compare women who had
an unplanned pregnancy
and aborted and those
who did not…
Completeness of 0 1 1 0.5 0,5 1 4
methodological
description
L8 Tools, and Description of Selection Method of Evidence, Cross- Overall methodological quality
rationale for data collection criteria analysis demonstration validation
choosing these conditions and
tools interviewers, Number
language, data of IDI /
format FG
Donaldson 2002 Published and No No No No a part some Interviews GOOD
unpublished very short and reports
Population reports on India Exploratory main quotes, material are
Studies pop policy trend with some read in Contrast between the official
(target free care in parallel view of the Indian government
India system) distinguishing (policy is successful) and the
perspective of variety of implementaions across
Interviews with differnet actors teh country states. Health
Contraception senior Indian personnelperceive resistence to
and population and foreign certain type contraception
Policy officials and partially due ot poor quality of
pop specialists services.

Completeness of 1 0 0 0 0,5 0.5 2.0

methodological
description
L9 Tools, and Description of data Selection criteria Method of Evidence, Cross-validation Overall
rationale for collection analysis demonstration methodological
choosing these conditions and Number of IDI / FG quality
tools interviewers,
language, data
format
Luke and Secondary data Project interviews 5 case studies (4 out Read and code Extensive No (the paper is GOOD
Watkins 2002 Interviews with of the of original 8 of PP) transcriptions quotes and entirely based on
national elites Implementation of selected for regional summary the reported
PDR collected reproductive health variation in health Systematic reactions of
(Policy Project ) policies and services and in the comparison of interviewees
Developing programs in LDC Cairo program categories though critically
countries (5 cases Additional Critical implementation developed commented by
: Ghana, interviews appreciation of the through authors)
Bangladesh, collected normative Respondents thematic coding
Malawi, Senegal, independently in consensus effect selected by the and analytical
Jordan) Malawi (USAID, funds and Policy Projects staff grids
Cairo) (NGO, government
Population Range of and donors,
Policy reactions to 110 interviews academics, Great discussion
Cairo consultants) of the biases (i.e.
USAID funding
)
Completeness of 1 1 1 1 1 0 5.0
methodological
description
L9 Tools, and rationale for Description of Selection Method of Evidence, Cross- Overall
choosing these tools data collection criteria analysis demonstration validation methodological
conditions and quality
interviewers, Number of
language, data IDI / FG
format
Marcell et al FG to gain better Flyers/ lunch 32 boys in 3 * Standard Quotes Three methods GOOD
2003 understanding of breaks 2 FG in two content in a sequence Complex framework
reproductive health of Parental consent schools in S analysis for should have to appreciate
USA young males to develop 15 dollars Francisco FG improved reproductive health
interventions to promote compensation validity and topics in respondents
Reproductive responsible sexual Variation in Grid and reliability of lives
health behaviour and care ethnic Emphasis the FG
services + Free listing background of response Concrete results in
activities method terms of policy for
Perspectives on Self selection developing health
Sexual and Group interviews in of respondents care services for
Reproductive preparation of the FG (good male adolescents
Health guideline were interviews discussion of
with male students of limits of self
health classes selection)
Completeness of 1 1 1 1 0.5 0.5 5
methodological
description
L10 Tools, and Description of data Selection criteria Method of Evidence, Cross- Overall methodological
rationale for collection analysis demonstration validation quality
choosing conditions and Number of IDI /
these tools interviewers, FG
language, data
format
Nie 2005 (Semi- 47 interviews (2/3 No mention Quotes No POOR
structured ?) Interviews per mail women) 20-79 It is difficult to judge to
PDR Interviews and phone (more years old main trends in which extent what has been
Shanghai than once Mixed income the exploration presented as results are the
sometimes) and employment of the reception results of social desirability,
Population policy situation but of the policy acceptance or
(responses to one mostly college internalization.
child policy) educated
The problem is here the
Snowballing from limit of IDI as an
4 relatives and observation tool, participant
family friends observation, or systematic
known for > 5 repeat interviews may be
years more useful

Discussion is poor
Completeness of 0.5 0.5 0.5 0 0.5 0 2
methodological
description
L11 Tools, and Description of data Selection Method of Evidence, Cross- Overall methodological quality
rationale for collection conditions criteria analysis demonstration validation
choosing these and interviewers,
tools language, data format Number of
IDI / FG
Buckley et al FG discussion to In Russian and Ubzek Mothers Not said quotes DHS MEDIUM
2004 understand with but it tables Interpret DHS results
reasons for Cash honorarium children <3 , looks like
Studies in worsening sexual unmarried main
Family health despite Transcribed/double women and stream FG are not suited to go beyond a
Planning health and translation single men content description of dominant norms; the
reproductive analysis research question needed individual
Ubzekistan programs 18-24 level behavioural observations to
secondary understand access or non access to
Reproductive educated sexual health programmes and its
and Sexual link with health conditions( I so not
health Selected share this entirely: i believe that in
from this case they had a lower level
housing ambition that is to explore attitudes
blocks towards health and reproductive
Does not say programs so FG could be enough )
number of
FG

Completeness of 0.5 1 1 0 0.5 0.5 3.5

methodological
description
L12 Tools, and Description of Selection Method of Evidence, Cross- Overall methodological
rationale for data collection criteria analysis demonstration validation quality
choosing conditions and
these tools interviewers, Number of IDI
language, data / FG
format
Mills and Bertrand FG because Very good Transcription Quotes Reaction to the GOOD
2005 of description of the 10 chiefdoms and dissemination
“abundance research setting were the basis translation Discussion of seminar The multiple
Studies in Family of for 18 biases (social perspectives from all
Planning information Community purposively Initial coding desirability) type of actors involved in
about contact for groups with Atlas ti the health obstetric care
Ghana beliefs recruitment is useful to identify
opinions 18 FG reasons for behavioral
Health / obstretic and Details about FG patterns
care perceptions” sessions Groups
described in
Local language table
Completeness of 0.5 1 1 1 0.5 0.5 4.5
methodological
description