Quantitative Methods

Quantitative/Survey
Research Design
Dereje Teferi
[email protected]
Forms of quantitative data ‘collection’

Experiments / quasi-experiments (not

‘Controlled’ comparisons
common in some disciplines)

‘Talking’ with people Surveys

Observing Structured observation

Using existing quantitative Secondary analysis of existing ‘survey’

information: I (micro-)data

Using existing quantitative Analysis of existing published

information: II quantifiable data

Looking at ‘texts’ (books, (Some forms of) content analysis

films, web pages, adverts…)
Sample and population
A range of statistical analyses of a sample can be
carried out, including descriptive analyses.
However, the topic of interest/research question is
typically focused on some feature of the
population
statistical analyses usually involve the use of
techniques for making inferences from a sample
to the corresponding population.
Sampling error or bias?
A parameter is a quantity relating to a given variable for a
population (e.g. the average (mean) adult income in the
AA).
When researchers generalize from a sample they use
sample observations to estimate population parameters.
The sampling error for a given sample design is the
degree of error that is to be expected in making these
estimations.
So the parameter estimates generated by quantitative
research are equal to the population parameters, plus a
certain amount of sampling error, plus any bias arising
from the data ‘collection’ process.
Time as a key dimension in quantitative
research
Cross-sectional studies
Observations of a sample or ‘cross-section’ of a population
(or of other phenomena) are made at one point in time –
most surveys are cross-sectional.
 This leads to a common criticism of survey research:
that it is a historical/unsuited to the examination of social
processes.
Longitudinal studies
These permit observations of the same population or
phenomena over an extended period of time.
 These enable analysis of change. They may also facilitate
more credible assertions relating to causality.
 Types of longitudinal study
1. Trend studies – these examine change within
a population over time (e.g. the Census).
2. Cohort studies – these examine over time
specific subpopulations or cohorts (often,
although not necessarily, the same individuals)
e.g. a study might interview people aged 30 in
1970, 40 in 1980, 50 in 1990 and 60 in 2000.
3. Panel study – These examine the same set of
people each time (e.g. interview the same
sample of (potential) voters every month
during an election campaign.
Survey design issues
Form of implementation
Thinking about who or what you’re researching: case,
population, sample
Non-probability samples
Probability samples (‘Random samples’)
Sample size
Some general points…
Typically, individuals are the ‘units of analysis’.
(This is not always the case though: for example in
a survey of hospitals)
Individuals, referred to as respondents, provide
data by responding to questions.
The ‘research instrument’ used to gather data is
often referred to as a questionnaire.
Questionnaires/‘Interview schedules’:
collect standardised information.
are used to elicit information to be used in analyses.
Three types of surveys:
1. Self-administered questionnaires
Including:
1. Mail(ed) surveys (or e-mail surveys)
2. Web-based surveys
3. Group surveys (e.g. in a classroom)
2. Interview surveys (‘face-to-face’: including
CAPI(Computer Assisted Personal interview)
3. Telephone surveys (including CAT interviewing)
Method Advantages Disadvantages Tips to Remember
Self- •Cheap •Low response rate (and •Simplify questions
completion •Cover wide area possible bias from this) •Include covering letter
•Anonymity protected •Questions need to be simple •Include stamped addressed response
•Interviewer bias •No control over interpretation envelope
doesn’t interfere •No control over who fills it in •Send a reminder
•People can take their •Slow
time

Telephone •Can do it all from •People may not have home •Because you rely totally on verbal
survey one place phones/be ex-directory communication – questions must be
•Can clarify answers •You may get wrong person or short and words easy to pronounce
•People may be call at wrong time •Minimize number of response
relatively happy to •May be a bias from whose categories (so people can remember
talk on the phone name is listed/who’s at home them)
•Relatively cheap •Easy for people to break off
•Quick •No context to interview

Face-to- •High response rate •Slow •Important that interviewer is non-

face •High control of the •Expensive threatening
interview interview situation •Interviewer presence may •Interviewer can clarify questions, but
•Ability to clarify influence way questions are should be wary of elaborations that
responses answered affect the content
•If there is more than one •Aim to ask questions in a clear,
interviewer, they may have standardized way
different effects •If the list of possible responses is long,
show them to the respondent for them to
read while the question is read out
 Response rate
You must keep track of the response rate,
Response rate is calculated as the proportion of
people who are selected to take part in the
survey (i.e. who are part of the ‘desired’ sample)
who actually participate.
A more detailed example:
You are studying women over 50. You stop women in the
street, ask their ages, and, if they qualify, you ask to
interview them.
If you stop 30 women, but 20 are under 50 and only 10 over
50, your starting point (those qualified to take part) is thus
10.
If 5 of these are willing to talk to you, you have achieved a
50% response rate (5/10)
Strengths of survey research
Useful for describing the characteristics of a large
population.
Makes large samples feasible.
Flexible - many questions can be asked on a given
topic.
Has a high degree of reliability (and replicability).
Is a relatively transparent process.
 Weaknesses of survey research
Seldom deals with the context of social life.
Inflexible – cannot be altered once it has begun
(therefore poor for exploratory research).
Subject to artificiality – the findings are a product of
the respondents’ consciousness that they are being
studied.
Sometimes weak in terms of validity.
Can be poor at answering questions where the units of
analysis are not individual people,
Usually inappropriate for historical research.
Can be particularly weak at gathering at certain sorts
of information, e.g. about:
Thinking about what you’re researching: Case,
Population, Sample
Population – all the theoretically-relevant cases (e.g.
‘St George club supporters’). This is also often referred
to as the target population.
This may differ from the study population, which is
all of the theoretically-relevant cases which are actually
available to be studied (e.g. ‘all St. George club
members or season ticket holders’).
Sometimes you can study all
possible cases
(the total population that you are interested in)

For example:
Post Haileselassie Prime Ministers
Homeless people using a particular shelter on
Christmas Day
National football teams in the 2022 World Cup
Secondary schools in AA
How you sample depends (among other
things) on some linked issues:
What you are especially interested in (what you want
to find out)
The frequency with which what you are interested in
occurs in the population
The size/complexity of the population
What research methods you are going to use
How many cases you want (or have the resources
and/or time) to study
Probability and non-probability
sampling
Probability samples (‘Random samples’)
A probability sample has a mathematical relationship to
the (study) population: we can work out mathematically
what the likelihood (probability) is of the results found for
the sample being within a given ‘distance’ of what would be
found for the whole population (if we were able to examine
the whole population!)
 Such a sample allows us to make inferences about the
population as a whole, based on the sample results.
Non-probability samples
Formally, these do not allow us to make inferences about
the population as a whole.
However, there are often pragmatic reasons for their use,
and, despite this lack of statistical legitimacy, inferential
statistics are often generated
Types of Non-probability sampling:
1. Reliance on available subjects:
• Literally choosing people because they are available (e.g.
approaching the first five people you see outside the
library)
• Only justified if less problematic sampling methods are
not possible.
• Researchers must exercise considerable caution in
generalizing from their data when this method is used.
Types of Non-probability sampling:
2. Purposive or judgmental sampling
• Selecting a sample based on knowledge of a
population, its elements, and the purpose of the
study. Selecting people who would be ‘good’
informants (individually/collectively).
• Used when field researchers are interested in
studying cases that do not fit into regular
patterns of attitudes and behaviours (i.e. when
researching ‘deviance’).
• Relies totally on the researcher’s prior ability to
determine ‘suitable’ subjects.
Types of Non-probability sampling:
3. Snowball sampling
• Researcher collects data on members of the target
population s/he can access, and uses them to help
locate other members of the population.
• May be appropriate when members of a population
are difficult to locate (and/or access).
• By definition, respondents who are located by
snowball sampling will be connected to other
respondents, thus respondents are more likely to
share similarities with each other than with other
members of the population.
Types of Non-probability sampling:
4. Quota sampling
• Begin with a matrix of the population (e.g. assuming it is 50%
female and 9% minority ethnic, with a given age structure).
• Data is collected from people matching the defining
characteristics of each cell within the matrix.
• Each cell is assigned a weight matching its proportion of the
population (e.g. if you were going to sample 1,000 people, you
would want 500 of them to be female, and hence 45 to be
minority ethnic women).
• The data thus provide a representation of the population.
• However, the data may not represent the population well in
terms of criteria that were not used to define the initial matrix.
• You cannot measure response rates.
• And, crucially, the selection process may be biased.
The logic of probability sampling
Representativeness:
A sample is representative of the population
from which it is selected to the extent that it has
the same aggregate characteristics (e.g. same
percentage of women, of immigrants, of poor and
rich people…)
EPSEM (Equal Probability of Selection
Method):
Every member of the population has the same
chance of being selected for the sample.
Random sampling:
Each element in the population has a known, non-
zero chance of selection. ‘Tables’ or ‘lists’ of
random numbers are often used (in print form or
generated by a computer, e.g. in SPSS).
Sampling frame:
A list of every element/case in the population from
which a probability sample can be selected.
In practice, sampling frames may not include every
element. It is the researcher’s job to assess the
extent (and nature) of any omissions and, if
possible, to correct them.
Types of probability sampling:
1. Simple Random Sample
Feasible only with the simplest sort of sampling frame
(a comprehensive one).
The researcher enumerates the sampling frame, and
randomly selects people.
Despite being the ‘purist’ type of random sample, in
practice it is rarely used.
Types of probability sampling:
2. Systematic Random Sample
Uses a random starting point, with every kth
element selected (e.g. if you wanted to select
1,000 people out of 10,000 you’d select every 10 th
person: such as the 3rd, 13th, 23rd…).
The arrangement of cases in the list can affect
representativeness (e.g. if k is even, when
sampling pages from a book with chapters
starting on odd-numbered pages).
Types of probability sampling:
3. Stratified sampling
Rather than selecting a sample from the overall
population, the researcher selects cases from
homogeneous subsets of the population (e.g.
random sampling from a set of undergraduates,
and from a set of postgraduates).
This ensures that key sub-populations are
represented adequately within the sample.
A greater degree of representativeness in the
results thus tends to be achieved, since the
(typical) quantity of sampling error is reduced.
Types of probability sampling:
4. Multi-stage Sampling
This is often used when it is not possible or practical
to create a list containing all the elements within the
target population.
It involves the repetition of two basic steps: creating
lists of sampling units and sampling from them.
It can be highly efficient but less accurate.
Types of probability sampling:
5. Probability Proportional to Size (PPS)
sampling
A sophisticated form of multi-stage sampling.
It is used in many large-scale surveys.
Sampling units are selected with a probability
proportional to their size (e.g. in a survey where
the primary sampling units (PSUs) were cities, a
city 10 times larger than another would be 10 times
more likely to be selected in the first stage of
sampling).
Sample size
The sample size that is needed depends upon:
The heterogeneity of the population: the more
heterogeneous, the bigger the sample needed
The number of relevant sub-groups: the more sub-
groups, the bigger the sample needed
The frequency of a phenomenon that you are trying to
detect: the closer to 50% (of the time) that it occurs,
the bigger the sample needed
How accurately you want your sample statistics to
reflect the population: the greater accuracy that is
required, the bigger the sample needed.
How confident you want to be about your results!
Other considerations when you are
thinking about sample size
The response rate – if you think that a lot of
people will not respond, you need to start off by
sampling a larger number of people.
Form of analysis – some forms of statistical
analysis require a larger number of cases than
others. If you plan on using one of these you will
need to ensure that you’ve got enough cases.
Generally (given a choice): Bigger is better!
(hence the sample size often reflects costs/resources.)
Observation(al) studies
Observation is not just the preserve of qualitative
methods. Quantitative methods can be applied where
structured or systematic observation is carried out.
Like qualitative observation studies (and surveys), this
involves cross-sectional data (we can only observe the
present).
Unlike qualitative observation, structured or
systematic observation is not inductive but requires
the prior determination of what to observe (although
this may be suggested by initial unstructured
observations)
The observation schedule
 To produce quantitative data an observation schedule or coding
scheme is required.
 This describes what is to be observed and how what is observed
should be coded.
 For example, if I were observing in the Library Café and was
interested in interactions between students and the staff working at
the cash-registers I could code each student’s behaviour in the
following way:

1. No conversation, no eye contact, no smile

2. Eye contact and/or smile, no conversation
3. Conversation, only as required by the transaction
4. Conversation as required by the transaction and
polite thanks.
5. Conversation that goes beyond transaction and
polite thanks.
The observation schedule
The observations must be focused – and relevant to
the research question
The schedule (like closed questions in a
questionnaire) should have categories that are
mutually exclusive and exhaustive
Recording should involve as little observer
interpretation as possible – this is where reliability is
diminished.
Sampling in Structured Observations

It is important to be clear about the unit of analysis –

are you sampling events/situations, interactions, or
individuals?
Sampling must consider the dimension of time in
determining who, where, and when to make
observations. It may sometimes be appropriate to
sample at multiple time periods and in multiple sites.
Benefits and Drawbacks of
Structured Observation I
Like other ‘unobtrusive measures’ structured observation may

avoid researcher contamination – enabling the study of people

in their natural environment.
Unlike surveys it does not depend on the negotiation of

meaning between interviewer and interviewee (or the

interviewee’s accurate representation of her behaviour).
Unlike qualitative observation studies it can produce relatively

reliable data and since observation (with a schedule) can be

undertaken by more than one researcher, it enables large-scale
Benefits and Drawbacks of
Structured Observation II
 However the researcher will only ‘see’ the predetermined categories of

action that the schedule specifies. These may not be the categories of
action that are relevant to participants.
 Since structured observation precludes questioning participants about

their motives or opinions, it is wholly dependent on observing behaviour

and on the ability of the researcher to appropriately assess this.
 It is ahistorical, in that it can only assess behaviour in the moment (unlike

surveys which can ask, albeit imperfectly, about people’s pasts, or other
methods such as content analysis, historical or secondary data analysis).
Thank you