(Ebook) Evaluating Research in Academic Journals: A

Practical Guide to Realistic Evaluation by Fred

Pyrczak ISBN 9781936523344, 1936523345 Pdf Download

https://ebooknice.com/product/evaluating-research-in-academic-
journals-a-practical-guide-to-realistic-evaluation-6784134

★★★★★
4.9 out of 5.0 (84 reviews )

Instant PDF Download

ebooknice.com
 (Ebook) Evaluating Research in Academic Journals: A
Practical Guide to Realistic Evaluation by Fred Pyrczak ISBN
9781936523344, 1936523345 Pdf Download

EBOOK

Available Formats

■ PDF eBook Study Guide Ebook

EXCLUSIVE 2025 EDUCATIONAL COLLECTION - LIMITED TIME

INSTANT DOWNLOAD VIEW LIBRARY

 We believe these products will be a great fit for you. Click
the link to download now, or visit ebooknice.com
to discover even more!

(Ebook) Evaluating Research in Academic Journals: A Practical

Guide to Realistic Evaluation by Fred Pyrczak, Maria Tcherni-
Buzzeo ISBN 9780815365662, 0815365667

https://ebooknice.com/product/evaluating-research-in-academic-journals-a-
practical-guide-to-realistic-evaluation-7306392

(Ebook) Writing Empirical Research Reports: A Basic Guide for

Students of the Social and Behavioral Sciences by Fred Pyrczak
ISBN 9781936523368, 1936523361

https://ebooknice.com/product/writing-empirical-research-reports-a-basic-
guide-for-students-of-the-social-and-behavioral-sciences-6786916

(Ebook) Writing Empirical Research Reports: A Basic Guide for

Students of the Social and Behavioral Sciences by Fred
Pyrczak, Randall R. Bruce ISBN 1884585582

https://ebooknice.com/product/writing-empirical-research-reports-a-basic-
guide-for-students-of-the-social-and-behavioral-sciences-2346584

(Ebook) Writing Empirical Research Reports: A Basic Guide for

Students of the Social and Behavioral Sciences by Fred Pyrczak,
Randall R. Bruce ISBN 1884585582

https://ebooknice.com/product/writing-empirical-research-reports-a-basic-
guide-for-students-of-the-social-and-behavioral-sciences-2228706
(Ebook) Best Practices in Faculty Evaluation: A Practical Guide
for Academic Leaders by Jeffrey L. Buller ISBN 9781118118436,
111811843X

https://ebooknice.com/product/best-practices-in-faculty-evaluation-a-
practical-guide-for-academic-leaders-5201418

(Ebook) Making Sense of Statistics: A Conceptual Overview by

Fred Pyrczak ISBN 9781138894761, 1138894761

https://ebooknice.com/product/making-sense-of-statistics-a-conceptual-
overview-10008020

(Ebook) Doing Academic Research: A Practical Guide to Research

Methods and Analysis by Ted Gournelos; Joshua R. Hammonds;
Maridath A. Wilson ISBN 9780367207915, 9780367207939,
9780429263552, 0367207915, 0367207931, 0429263554
https://ebooknice.com/product/doing-academic-research-a-practical-guide-to-
research-methods-and-analysis-21985770

(Ebook) Change Leadership in Higher Education: A Practical Guide

to Academic Transformation by Jeffrey L. Buller ISBN
9781118762035, 1118762037

https://ebooknice.com/product/change-leadership-in-higher-education-a-
practical-guide-to-academic-transformation-4947982

(Ebook) Research and Study Skills for Veterinary Nurses: A

Practical Guide for Academic Success by Davidson, Jane ISBN
9781789180138, 1789180139

https://ebooknice.com/product/research-and-study-skills-for-veterinary-
nurses-a-practical-guide-for-academic-success-54929940
Evaluating Research in
Academic Journals
A Practical Guide to
Realistic Evaluation

SIXTH EDITION

Fred Pyrczak
First published 1999 by Pyrczak Publishing.

Published 2017 by Routledge

2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN
711 Third Avenue, New York, NY 10017, USA

Routledge is an imprint of the Taylor & Francis Group, an informa business

Copyright © 2014, 2013, 2008, 2005, 2003, 1999 by Taylor & Francis

This edition was written in collaboration with Randall R. Bruce.

All rights reserved. Permission is hereby granted for professors and their students to
photocopy Appendix D for use only in classes in which this book is a required textbook.
Except for Appendix D, no part of this book may be reprinted or reproduced or utilised in
any form or by any electronic, mechanical, or other means, now known or hereafter
invented, including photocopying and recording, or in any information storage or retrieval
system, without permission in writing from the publishers.

Notice:
Product or corporate names may be trademarks or registered trademarks, and are
used only for identification and explanation without intent to infringe.

Although the author and publisher have made every effort to ensure the accuracy and
completeness of information contained in this book, we assume no responsibility for
errors, inaccuracies, omissions, or any inconsistency herein. Any slights of people, places,
or organizations are unintentional.

Cover design by Ruben Hornillo.

ISBN-13: 978-1-936-52334-4 (pbk)

ii
 Contents
Introduction to the Sixth Edition v

1. Background for Evaluating Research Reports 1

2. Evaluating Titles 13

3. Evaluating Abstracts 23

4. Evaluating Introductions and Literature Reviews 33

5. A Closer Look at Evaluating Literature Reviews 47

6. Evaluating Samples When Researchers Generalize 55

7. Evaluating Samples When Researchers Do Not Generalize 69

8. Evaluating Measures 77

9. Evaluating Experimental Procedures 91

10. Evaluating Analysis and Results Sections: Quantitative Research 103

11. Evaluating Analysis and Results Sections: Qualitative Research 111

12. Evaluating Discussion Sections 123

13. Putting It All Together 131

Appendix A: Quantitative and Qualitative Research: An Overview 137

Appendix B: Examining the Validity Structure of Qualitative Research 141

Appendix C: The Limitations of Significance Testing 149

Appendix D: Checklist of Evaluation Questions 153

Notes

iv
 Introduction to the Sixth Edition
When students in the social and behavioral sciences take advanced courses in their
major field of study, they are often required to read and evaluate original research reports
published as articles in academic journals. This book is designed as a guide for students
who are first learning how to engage in this process.

Major Assumptions
First, it is assumed that the students using this book have limited knowledge of
research methods, even though they may have taken a course in introductory research
methods (or may be using this book while taking such a course). Because of this assump-
tion, technical terms and jargon such as true experiment are defined when they are first
used in this book.
Second, it is assumed that students have only a limited grasp of elementary statis-
tics. Thus, the chapters on evaluating statistical reporting in research reports are confined
to criteria that such students can easily comprehend.
Finally, and perhaps most important, it is assumed that students with limited back-
grounds in research methods and statistics can produce adequate evaluations of research
reports—evaluations that get to the heart of important issues and allow students to draw
sound conclusions from published research.

This Book Is Not Written for…

This book is not written for journal editors or members of their editorial review
boards. Such professionals usually have had firsthand experience in conducting research
and have taken advanced courses in research methods and statistics. Published evaluation
criteria for use by these professionals are often terse, full of jargon, and composed of
many elements that cannot be fully comprehended without advanced training and experi-
ence. This book is aimed at a completely different audience: students who are just begin-
ning to learn how to evaluate original reports of research published in journals.

Applying the Evaluation Questions in This Book

Chapters 2 through 13 are organized around evaluation questions that may be an-
swered with a simple “yes” or “no,” where a “yes” indicates that students judge a charac-
teristic to be satisfactory. However, for evaluation questions that deal with complex is-
sues, students may also want to rate each one using a scale from 1 to 5, where 5 is the
highest rating. In addition, N/A (not applicable) may be used when students believe a
characteristic does not apply, and I/I (insufficient information) may be used if the re-
search report does not contain sufficient information for an informed judgment to be
made.
Evaluating Quantitative and Qualitative Research
Quantitative and qualitative research differ in purpose as well as methodology.
Students who are not familiar with the distinctions between the two approaches are ad-
vised to read Appendix A, which presents a very brief overview of the differences, and
Appendix B, which provides an overview of important issues in the evaluation of qualita-
tive research.

Note From the Author

My best wishes are with you as you master the art and science of evaluating re-
search. With the aid of this book, you should find the process both undaunting and fasci-
nating as you seek defensible conclusions regarding research on topics that interest you.

Fred Pyrczak
Los Angeles

vi
 Chapter 1

Background for Evaluating

Research Reports
The vast majority of research reports are initially published in academic journals. In
these reports, researchers describe how they identified a research problem, made relevant
observations to gather data, and analyzed the data they collected. The reports usually
conclude with a discussion of the results and their implications. This chapter provides an
overview of some general characteristics of such research. Subsequent chapters present
specific questions that should be applied in the evaluation of research reports.

 Guideline 1: Researchers often examine narrowly defined problems.

Comment: While researchers usually are interested in broad problem areas, they very
often examine only narrow aspects of the problems because of limited resources and
the desire to keep the research manageable by limiting its focus. Furthermore, they
often examine problems in such a way that the results can be easily reduced to
statistics, further limiting the breadth of their research.1
Example 1.1.1 briefly describes a study on two correlates of prosocial behavior
(i.e., helping behavior). To make the study of this issue manageable, the researchers
greatly limited its scope. Specifically, they examined only one very narrow type of
prosocial behavior (making donations to homeless men who were begging in public).
Example 1.1.1
A study on prosocial behavior, narrowly defined:
In order to study the relationship between prosocial behavior and gender as well
as age, researchers located five men who appeared to be homeless and were
soliciting money on street corners using cardboard signs. Without approaching
the men, the researchers observed them from a short distance for two hours
each. For each pedestrian who walked within ten feet of the men, the researchers
recorded whether the pedestrian made a donation. The researchers also recorded
the gender and approximate age of each pedestrian.
Because researchers often conduct their research on narrowly defined problems,
an important task in the evaluation of research is to judge whether a researcher has

1
Qualitative researchers (see Appendices A and B) generally take a broader view when defining a problem
to be explored in research and are not constrained by the need to reduce the results to numbers and
statistics.

1
Chapter 1 Background for Evaluating Research Reports

defined the problem so narrowly that it fails to make an important contribution to the
advancement of knowledge.

 Guideline 2: Researchers often conduct studies in artificial settings.

Comment: Laboratories on university campuses are often the settings for research. To
study the effects of alcohol consumption on driving behavior, a group of participants
might be asked to drink carefully measured amounts of alcohol in a laboratory and
then “drive” using virtual-reality simulators. Example 1.2.1 describes the preparation
of the cocktails in a study of this type.
Example 1.2.12
Alcoholic beverages prepared for consumption in a laboratory setting:
The preparation of the cocktail was done in a separate area out of view of the
participant. All cocktails were a 16-oz mixture of orange juice, cranberry juice,
grapefruit juice (ratio 4:2:1, respectively). For the cocktails containing alcohol,
we added 2 oz of 190-proof grain alcohol mixed thoroughly. For the placebo
cocktail, we lightly sprayed the surface of the juice cocktail with alcohol using
an atomizer placed slightly above the juice surface to impart an aroma of alcohol
to the glass and beverage surface. This placebo cocktail was then immediately
given to the participant to consume. This procedure results in the same alcohol
aroma being imparted to the placebo cocktail as the alcohol cocktail.…
Such a study might have limited generalizability to drinking in out-of-laboratory
settings, such as nightclubs, the home, picnics, and other places where those who are
consuming alcohol may be drinking different amounts at different rates while consum-
ing (or not consuming) various foods. Nevertheless, conducting such research in a
laboratory allows researchers to simplify and control variables such as the amount of
alcohol consumed, the types of food being consumed, the number of individuals in a
car, and so on. In short, researchers very often opt against studying variables in com-
plex, real-life settings for the more interpretable research results typically obtained in a
laboratory.

 Guideline 3: Researchers use less-than-perfect methods of observation.

Comment: In research, observation can take many forms—from paper-and-pencil
multiple-choice achievement tests to essay examinations, from administering a
paper-and-pencil attitude scale with choices from “strongly agree” to “strongly dis-
agree” to conducting unstructured interviews to identify interviewees’ attitudes.3 Of

2
Barkley, R. A., Murphy, K. R., O’Connell, T., Anderson, D., & Connor, D. F. (2006). Effects of two dos-
es of alcohol on simulator driving performance in adults with attention-deficit/hyperactivity disorder. Neu-
ropsychology, 20, 77–87.
3
Researchers sometimes refer to measurement tools for making observations as instruments, especially in
older literature.

2
Chapter 1 Background for Evaluating Research Reports

course, observation also includes direct observation of individuals interacting in either

their natural environments or laboratory settings.
It is safe to assume that all methods of observation are flawed to some extent. To
see why this is so, consider a professor/researcher who is interested in studying racial
relations in society in general. Because of limited resources, the researcher decides to
make direct observations of White and African American students interacting (and/or
not interacting) in the college cafeteria. The observations will necessarily be limited to
the types of behaviors typically exhibited in cafeteria settings—a weakness in the re-
searcher’s method of observation. In addition, observations will be limited to certain
overt behaviors because, for instance, it will be difficult for the researcher to hear most
of what is being said without obtruding on the privacy of the students.
On the other hand, suppose that another researcher decides to measure racial at-
titudes by having students respond anonymously to racial statements by circling
“agree” or “disagree” for each one. This researcher has an entirely different set of
weaknesses in the observational method. First is the matter of whether students will
reveal their real attitudes on such a scale—even if the response is anonymous—
because most college students are aware that negative racial attitudes are severely
frowned on in academic communities. Thus, some students might indicate what they
believe to be socially desirable (i.e., socially “correct”) rather than reveal their true at-
titudes.
In short, there is no perfect way to measure complex variables. Instead of ex-
pecting perfection, a consumer of research should consider this question: Is the method
sufficiently valid and reliable to provide potentially useful information?
Examples 1.3.1 and 1.3.2 show statements from research articles in which the
researchers acknowledge limitations in their observational methods.
Example 1.3.14
Researchers’ acknowledgment of a limitation of their measures:
In addition, the assessment of marital religious discord was limited to one item.
Future research should include a multiple-items scale of marital religious dis-
cord and additional types of measures, such as interviews or observational cod-
ing, as well as multiple informants.
Example 1.3.25
Researchers’ acknowledgment of limitations of self-reports for observation:
Despite these strengths, this study is not without limitations. First, the small
sample size decreases…. Fourth, neighborhood danger was measured from
mothers’ self-reports of the events which occurred in the neighborhood during

4
Kor, A., Mikulincer, M., & Pirutinsky, S. (2012). Family functioning among returnees to Orthodox Juda-
ism in Israel. Journal of Family Psychology, 26, 149–158.
5
Callahan, K. L., Scaramella, L. V., Laird, R. D., & Sohr-Preston, S. L. (2011). Neighborhood disadvan-
tage as a moderator of the association between harsh parenting and toddler-aged children’s internalizing
and externalizing problems. Journal of Family Psychology, 25, 68–76.

3
Chapter 1 Background for Evaluating Research Reports

the past year. Adding other family member reports of the dangerous events and
official police reports would clearly strengthen our measure of neighborhood
danger.
Chapter 8 provides more information on evaluating observational methods.

 Guideline 4: Researchers use less-than-perfect samples.

Comment: Arguably, the most common sampling flaw in research reported in aca-
demic journals is the use of samples of convenience (i.e., samples that are readily ac-
cessible to the researchers). Most researchers are professors, and professors often use
samples of college students—obviously as a matter of convenience. Another common
flaw is relying on voluntary responses to mailed surveys, which are often quite low,
with some researchers arguing that a response rate of about 40% to 60% or more is ac-
ceptable. Other samples are flawed because researchers cannot identify and locate all
members of a population (e.g., injection drug users). Without being able to do this, it
is impossible to draw a sample that a researcher can reasonably defend as being repre-
sentative of the population.6 In addition, researchers often have limited resources,
which force them to use small samples, which in turn might produce unreliable results.
Researchers sometimes explicitly acknowledge the limitations of their samples.
Examples 1.4.1 through 1.4.3 show portions of such statements from research articles.
Example 1.4.17
Researchers’ acknowledgment of limitation of sampling (convenience sample):
The present study suffered from several limitations. First of all, the samples
were confined to university undergraduate students and only Chinese and
American students. For broader generalizations, further studies could recruit
people of various ages and educational and occupational characteristics.
Example 1.4.28
Researchers’ acknowledgment of limitation of sampling (low rate of
participation):
Data were collected using a random sample of e-mail addresses obtained from
the university’s registrar’s office. The response rate (23%) was lower than de-
sired; however, it is unknown what percentage of the e-mail addresses were
valid or were being monitored by the targeted student.

6
Qualitative researchers emphasize selecting a purposive sample—one that is likely to yield useful infor-
mation—rather than a representative sample.
7
Jiang, F., Yue, X. D., & Lu, S. (2011). Different attitudes toward humor between Chinese and American
students: Evidence from the Implicit Association Test. Psychological Reports, 109, 99–107.
8
Cox, J. M., & Bates, S. C. (2011). Referent group proximity, social norms, and context: Alcohol use in a
low-use environment. Journal of American College Health, 59, 252–259.

4
Chapter 1 Background for Evaluating Research Reports

Example 1.4.39
Researcher’s acknowledgment of limitation of sampling (limited diversity):
There are a number of limitations to this study. The most significant of them re-
lates to the fact that the study was located within one school and the children
studied were primarily from a White, working-class community. There is a need
to identify how socially and ethnically diverse groups of children use online vir-
tual worlds.
In Chapters 6 and 7, specific criteria for evaluating samples are explored in
detail.

 Guideline 5: Even a straightforward analysis of data can produce

misleading results.
Comment: Obviously, data-input errors and computational errors are possible sources
of errors in results. Some commercial research firms have the data they collect entered
independently by two or more data-entry clerks. A computer program checks to see
whether the two sets of entries match perfectly—if they do not, the errors must be
identified before the analysis can proceed. Unfortunately, taking such care in checking
for mechanical errors in entering data is hardly ever mentioned in research reports
published in academic journals.
In addition, there are alternative statistical methods for most problems, and dif-
ferent methods can yield different results. (See the first evaluation in Chapter 10 for
specific examples regarding the selection of statistics.)
Finally, even a nonstatistical analysis can be problematic. For instance, if two or
more researchers review extensive transcripts of unstructured interviews, they might
differ in their interpretations of the interviewees’ responses. Discrepancies such as
these suggest that the results may be flawed or at least subject to different interpreta-
tions.
Chapter 10 provides evaluation criteria for quantitative Analysis and Results
sections of research reports, while Chapter 11 does the same for qualitative Analysis
and Results sections.

 Guideline 6: Even a single, isolated flaw in research methods can lead to

seriously misleading results.
Comment: A seemingly minor flaw such as a poorly worded question on attitudes
might lead to results that are incorrect. Likewise, a treatment that has been misapplied
in an experiment might lead to misleading conclusions regarding the effectiveness of
the treatment. For this reason, research reports should be detailed, so that consumers

9
Marsh, J. (2011). Young children’s literacy practices in a virtual world: Establishing an online interaction
order. Reading Research Quarterly, 46, 101–118.

5
Chapter 1 Background for Evaluating Research Reports

of research can judge whether the research methods were flawed. This leads to the
next guideline.

 Guideline 7: Research reports often contain many details, which can be

very important when evaluating a report.
Comment: The old saying “The devil is in the details” applies here. Students who have
relied exclusively on secondary sources for information about their major field of
study may be surprised at the level of detail in many research reports, which is typi-
cally much greater than is implied in sources such as textbooks and classroom lec-
tures. Example 1.7.1 illustrates the level of detail that can be expected in many re-
search reports published in academic journals. It describes part of an intervention for
postal service letter carriers. Note the level of detail, such as (1) the color and size of
the hats and (2) the specific brand of sunscreen that was distributed. Such details are
useful for helping consumers of research understand exactly the nature of the interven-
tion examined in the study. Such detailed descriptions are also helpful for other re-
searchers who might want to replicate the study in order to confirm the findings.
Example 1.7.110
An excerpt from an article illustrating the level of detail often reported in
research reports in academic journals:
Within 2 weeks of the baseline measurement, Project SUNWISE health educators visited inter-
vention stations to give out hats, install and dispense sunscreen, distribute materials that
prompted use of solar protective strategies, and deliver the initial educational presentation.…The
machine-washable dark blue hat was made of Cordura nylon, it had a brim that was 4 inches
wide in the front and back and 3 inches wide on the sides, and it had an adjustable cord chin
strap. In addition to the initial free hat provided by Project SUNWISE, letter carriers at interven-
tion stations were given discounts on replacement hats by the vendor (Watership Trading Com-
panie, Bellingham, WA).
Locker rooms at intervention stations were stocked with large pump bottles of sunscreen
(Coppertone Sport, SPF 30, Schering-Plough HealthCare Products, Inc., Memphis, TN) that
were refilled regularly by the research staff. Additionally, letter carriers were given free 12-
ounce bottles of the sunscreen, which they could refill with sunscreen from the pump bottles.
The decision about which sunscreen to use was made on the basis of formative work that identi-
fied a product with a high SPF that had an acceptable fragrance and consistency and minimal
rub-off from newsprint onto skin….
Finally, Project SUNWISE health educators delivered 6 brief onsite educational presentations
over 2 years. The 5- to 10-minute presentations were modeled after the “stand-up talks” letter
carriers regularly participated in; the educators used large flip charts with colorful graphics that
were tailored to letter carriers. Key points of the introductory presentation included the amount
of UVR carriers are exposed to and UVR as a skin cancer risk factor, a case example of a former
carrier who recently had a precancerous growth removed, feasible protection strategies, and spe-
cific information about the hats and sunscreen. The themes of subsequent presentations were (1)
importance of sun safety, even in winter; (2) sun safety for the eyes; (3) sharing sun safety tips
with loved ones; (4) relevance of sun safety to letter carriers of all races/ethnicities; and (5) recap
and encouragement to continue practicing sun safety behaviors.

10
Mayer, J. A., Slymen, D. J., Clapp, E. J., Pichon, L. C., Eckhardt, L., Eichenfield, L. F., … Oh, S. S.
(2007). Promoting sun safety among U.S. Postal Service letter carriers: Impact of a 2-year intervention.
American Journal of Public Health, 97, 559–565.

6
Chapter 1 Background for Evaluating Research Reports

Having detailed information on what was said and done to participants as well
as on how the participants were observed makes it possible to make informed evalua-
tions of research.

 Guideline 8: Many reports lack information on matters that are potentially

important for evaluating a research article.
Comment: In most journals, research reports of more than 15 pages are rare. Journal
space is limited by economics—journals have limited readership and thus a limited
paid circulation, and they seldom have advertisers. Given this situation, researchers
must judiciously choose the details they will report. Sometimes, they may omit infor-
mation that readers deem important.
Omitted details can cause problems during research evaluation. For instance, it
is common for researchers to describe in general terms the questionnaires and attitude
scales they used without reporting the exact wording of the questions.11 Yet there is
considerable research indicating that how items are worded can affect the results of a
study.
As students apply the evaluation questions in the remaining chapters of this
book while evaluating research, they may often find that they must answer “insuffi-
cient information to make a judgment.”

 Guideline 9: Some published research is obviously flawed.

Comment: With many hundreds of editors of and contributors to academic journals, it
is understandable that published research reports vary in quality, with some being very
obviously weak in terms of their research methodology.12
Undoubtedly, some weak reports simply slip past less-skilled editors. More
likely, an editor may make a deliberate decision to publish a weak report because the
problem it explores is of current interest to the journal’s readers. This is especially true
when there is a new topic of interest, such as a new educational reform, a newly rec-
ognized disease, or a new government initiative. The editorial board of a journal might
reasonably conclude that publishing studies on such new topics is important, even if
the initial studies are weak.

11
This statement appears in each issue of The Gallup Poll Monthly: “In addition to sampling error, readers
should bear in mind that question wording…can introduce additional systematic error or ‘bias’ into the
results of opinion polls.” Accordingly, The Gallup Poll Monthly reports the exact wording of the questions
it uses in its polls. Other researchers cannot always do this because the measures they use may be too long
to include in the report or may be copyrighted by publishers who do not want the items released to the pub-
lic.
12
Many journals are refereed. This means that the editor has experts who act as referees by evaluating each
paper submitted for possible publication. These experts make their judgments without knowing the identifi-
cation of the researcher who submitted the paper, and the editor uses their input in deciding which papers to
publish as journal articles.

7
Chapter 1 Background for Evaluating Research Reports

Sometimes, studies with very serious methodological problems are labeled as pi-
lot studies, in either their titles or introductions to the research reports. A pilot study is
a preliminary study that allows a researcher to try out new methods and procedures for
conducting research, often with small samples. Pilot studies may be refined in subse-
quent, more definitive studies. Publication of pilot studies, despite their limited sam-
ples and other potential weaknesses, is justified on the basis that they may point other
researchers in the direction of promising new leads and methods for further research.

 Guideline 10: No research report provides “proof.”

Comment: Conducting research is fraught with pitfalls, any one study may have very
misleading results, and all studies can be presumed to be flawed to some extent. In
light of this, individual research reports should be evaluated carefully to identify those
that are most likely to provide sound results. In addition, a consumer of research
should consider the entire body of research on a given problem. If different research-
ers using different research methods with different types of strengths and weaknesses
all reach similar conclusions, consumers of research may say that they have consider-
able confidence in the conclusions of the body of research. On the other hand, to the
extent that the body of research on a topic yields mixed results, consumers of research
should lower their degree of confidence. For instance, if all the studies judged to be
strong point in one direction while weaker ones point in a different direction, consum-
ers of research might say that they have some confidence in the conclusion suggested
by the stronger studies.

 Guideline 11: Other things being equal, research related to theories is more
important than nontheoretical research.
Comment: A given theory helps explain interrelationships among a number of vari-
ables and often has implications for understanding human behavior in a variety of set-
tings. Studies that have results consistent with a theory lend support to the theory.
Those with inconsistent results argue against the theory. After a number of studies re-
lating to the theory have been conducted, their results provide accumulated evidence
that argues for or against the theory and evidence that can assist in modifying the the-
ory. Often, researchers explicitly discuss theories that are relevant to their research, as
illustrated in Example 1.11.1.
Example 1.11.113
Portions of researchers’ discussion of a theory related to their research:
One of the most influential theories regarding women’s intentions to stay in or
leave abusive relationships is social exchange theory, which suggests that these
kinds of relational decisions follow from an analysis of the relative cost-benefit

13
Gordon, K. C., Burton, S., & Porter, L. (2004). Predicting the intentions of women in domestic violence
shelters to return to partners: Does forgiveness play a role? Journal of Family Psychology, 18, 331–338.

8
Chapter 1 Background for Evaluating Research Reports

ratio of remaining in a relationship (Kelley & Thibaut, 1978). On the basis of

this theory, many researchers have posited that whereas escaping the abuse may
appear to be a clear benefit, the costs associated with leaving the relationship
may create insurmountable barriers for many abused women.
The role of theoretical considerations in the evaluation of research is discussed
in greater detail in Chapter 4.

 Guideline 12: Many researchers acknowledge obvious flaws in their

research.
Comment: Many researchers very briefly point out the most obvious flaws in their re-
search. They typically do this in the last section of their reports, which is the Discus-
sion section. While they tend to be brief and deal with only the most obvious prob-
lems, these acknowledgments can be a good starting point in the evaluation of a re-
search report.
Example 1.12.1 shows the researchers’ description of the limitations of their re-
search on Mexican American men’s college persistence intentions.
Example 1.12.114
Researchers’ description of the limitations of their research:
Despite the contributions of this study in expanding our understanding of Mexi-
can American men’s college persistence intentions, there also are some clear
limitations that should be noted. First, several factors limit our ability to general-
ize this study’s findings to other populations of Mexican American male under-
graduates. The participants attended a Hispanic-serving 4-year university in a
predominantly Mexican American midsize southern Texas town located near the
U.S.-México border. While the majority of U.S. Latinos live in the Southwest
region, Latinos are represented in communities across the U.S. (U.S. Census Bu-
reau, 2008c). Additionally, the study’s generalizability is limited by the use of
nonrandom sampling methods (e.g., self-selection bias) and its cross-sectional
approach (Heppner, Wampold, & Kivlighan, 2007).

 Guideline 13: To become an expert on a topic, one must become an expert

at evaluating original reports of research.
Comment: An expert is someone who knows not only broad generalizations about a
topic but also the nuances of the research that underlie them. In other words, he or she
knows the particular strengths and weaknesses of the major studies used to arrive at
the generalizations. Put another way, an expert on a topic knows the quality of the evi-

14
Ojeda, L., Navarro, R. L., & Morales, A. (2011). The role of la familia on Mexican American men’s col-
lege persistence intentions. Psychology of Men & Masculinity, 12, 216–229.

9
Chapter 1 Background for Evaluating Research Reports

dence regarding that topic and bases generalizations from the research literature on
that knowledge.

Exercise for Chapter 1

Part A
Directions: The 13 guidelines discussed in this chapter are repeated below. For each one,
indicate the extent to which you were already familiar with it before reading this chapter.
Use a scale from 5 (very familiar) to 1 (not at all familiar).

Guideline 1: Researchers often examine narrowly defined problems.

Familiarity rating: 5 4 3 2 1

Guideline 2: Researchers often conduct studies in artificial settings.

Familiarity rating: 5 4 3 2 1

Guideline 3: Researchers use less-than-perfect methods of observation.

Familiarity rating: 5 4 3 2 1

Guideline 4: Researchers use less-than-perfect samples.

Familiarity rating: 5 4 3 2 1

Guideline 5: Even a straightforward analysis of data can produce misleading results.

Familiarity rating: 5 4 3 2 1

Guideline 6: Even a single, isolated flaw in research methods can lead to seriously mis-
leading results.
Familiarity rating: 5 4 3 2 1

Guideline 7: Research reports often contain many details, which can be very important
when evaluating a report.
Familiarity rating: 5 4 3 2 1

Guideline 8: Many reports lack information on matters that are potentially important for
evaluating a research article.
Familiarity rating: 5 4 3 2 1

10
Chapter 1 Background for Evaluating Research Reports

Guideline 9: Some published research is obviously flawed.

Familiarity rating: 5 4 3 2 1

Guideline 10: No research report provides “proof.”

Familiarity rating: 5 4 3 2 1

Guideline 11: Other things being equal, research related to theories is more important
than nontheoretical research.
Familiarity rating: 5 4 3 2 1

Guideline 12: Many researchers acknowledge obvious flaws in their research.

Familiarity rating: 5 4 3 2 1

Guideline 13: To become an expert on a topic, one must become an expert at evaluating
original reports of research.
Familiarity rating: 5 4 3 2 1

Part B: Application
Directions: Read a report of research published in an academic journal, and respond to
the following questions. The report may be one that you select or one that is assigned by
your instructor. If you are using this book without any prior training in research methods,
do the best you can in answering the questions at this point. As you work through this
book, your evaluations will become increasingly sophisticated.

1. How narrowly is the research problem defined? In your opinion, is it too narrow? Is it
too broad? Explain.

2. Was the research setting artificial (e.g., a laboratory setting)? If yes, do you think that
the gain in the control of extraneous variables offsets the potential loss of information
that would be obtained in a study in a more real-life setting? Explain.

3. Are there any obvious flaws or weaknesses in the researcher’s methods of observa-
tion? Explain. (Note: Observation is usually described under the subheading Mea-
sures.)

4. Are there any obvious sampling flaws? Explain.

11
Chapter 1 Background for Evaluating Research Reports

5. Was the analysis statistical or nonstatistical? Was the description of the results easy
to understand? Explain.

6. Were the descriptions of procedures and methods of observation sufficiently detailed?

Were any important details missing? Explain.

7. Does the report lack information on matters that are potentially important for evaluat-
ing it?

8. Overall, was the research obviously very weak? If yes, briefly describe its weaknesses
and speculate on why it was published despite them.

9. Does the researcher describe related theories?

10. Does the researcher imply that his or her research proves something? Do you believe
that it proves something? Explain.

11. Do you think that as a result of reading this chapter and evaluating a research report
you are becoming more expert at evaluating research reports? Explain.

12
 Chapter 2

Evaluating Titles
Titles help consumers of research to identify journal articles of interest to them. A
preliminary evaluation of a title should be made when it is first encountered. After the
article is read, the title should be reevaluated to ensure that it accurately reflects the con-
tents of the article.
Apply the questions that follow while evaluating titles. The questions are stated as
“yes–no” questions, where a “yes” indicates that you judge the characteristic to be satis-
factory. You may also want to rate each characteristic using a scale from 1 to 5, where 5
is the highest rating. N/A (not applicable) and I/I (insufficient information to make a
judgment) may also be used when necessary.

___ 1. Is the title sufficiently specific?

Very satisfactory 5 4 3 2 1 Very unsatisfactory or N/A I/I
Comment: On any major topic in the social and behavioral sciences, there are likely to
be many hundreds of research reports published in academic journals. In order to help
potential readers locate those that are most relevant to their needs, researchers should
use titles that are sufficiently specific so that each article can be differentiated from the
other research articles on the same topic.
Consider the topic of depression, which has been extensively investigated. The
title in Example 2.1.1 is insufficiently specific. Contrast it with the titles in Example
2.1.2, each of which contains information that differentiates it from the others.
Example 2.1.1
A title that is insufficiently specific:
An Investigation of Adolescent Depression and Its Implications
Example 2.1.2
Three titles that are more specific than the one in Example 2.1.1:
Gender Differences in the Expression of Depression by Early Adolescent
Children of Alcoholics
The Impact of Social Support on the Severity of Postpartum Depression Among
Adolescent Mothers
The Effectiveness of Cognitive Therapy in the Treatment of Adolescent Stu-
dents With Severe Clinical Depression

13
 Another Random Document on
Scribd Without Any Related Topics
of

tapasztalta

Screece of

not

the change How

tax that zokogott

on

writer royalties

were generalisations

event szép has

regulative had brain

of s be

reward dislike

a Englishman

at years

of not beside

Take of is

had

distinguishes big

The
and the from

She

corridor

was they to

that
our tell

tends have of

to sense

website

stately

hirdetik my

how
She

void

over

the and

living

éppen

imbecility

4 the

Cape

grasp that
owed

the evil

both affected Hiszen

interesting studies streak

sun effect

cloak
yet t intolerably

nagyképü stern

bequeathing was

of bleeding the

and to

and
Project papjuk from

refund

working projected caught

horse

want to
by

other

nem Queen

Despite use

go

now seemed E
türhetetlenül tried

the by dark

there of

I endured

by some

looking be
doing

soon Literary

White and

lobe Sehnsucht not

entrapment
She deserving KISASSZONY

agreement begin

8d

Here

pretty and of

say to

agree

even a

must

do
Catlin natural

boldogok

some author

6a

youngest personal her

its George

happy
a

lying

herself

one

itself at
döntését said

saw

way I nem

form

finger stop He

touched data

on all
has than and

his

me

engagement

the a

so put

The

especially intent promise

to
the

to in

know

the declaration

Alithea
qualities There up

place intrusted

But religion sight

all used people

of

world
hanyatlott

this me

the she

What that your

bright characteristic month

are that
are the on

have tomorrow

childish to her

youngest on

var

at

the lived
All

me is before

objects cowered

were

and class

deal and brought

check think this

for prize resourcefulness

AGREE happy in

amusing it

in plain Enter

temper

and fancy and

volt A

often said a

sometimes

at various

months home

this department

differences

child

when
Hoskins nickel

is folk

avenge the

to

reign the

were To they
brown

Who strongly phenomenon

fiction Pringle

she days az

az clemency
of

phrase their so

splendor upon refit

her proper but

business strength very

Going
have though

crisis

to person

to with that

disguised OF for

the at

was of had

my

do King
Have

driver lesz

dog that

to

Far liking will

come the

by nothing Record

this he the
work me

unseeing It brotherhood

my Mr the

and

electronic

to

compass

death of
Deschartres

szobából and stand

the production 1

WARRANTY Akkor

of

disorder mechanical to
was affection

that and Falkner

the

when

little had Those

hard

Odes of awkwardly

the and the

is the its
at opportunity

my

szerelmének of by

centuries

woman and his

at

donate yelled yet

of
lobes

accounts his

notable

The szerették

or were

was férje precise

people

had GUTENBERG

fifty

of

attached halkan

rode

they existence who

reveres

the in years

party believed
up talking

different Company there

subject

they sorrows

was is megjegyzéssel

being

of Now

occasion the
kinship his Hence

large

seemingly gutenberg in

that

of

must The

grass
messenger the in

a full

especially surrounding

for

the
Nem almost

representations fault

In avowed

chief

five

delight

into to 23

the

came Avenue

contractor
whose

in mass offered

other Slay work

whiskers

our wryly a

me tires

the well

you of
on few

told and

Mr world

to i

of

which

come apprehension

arrive one

Queen of

are
our

no

light be

a with

of

as Raby hátgerince

listen

165 smooths

in
jóindulatu

from by the

Gothic we it

the The

at

kneel whose other

ten

and
enough

d looked

néném him

blood

end nurse the

which of

cult s even
apt

of Monthly since

Causation

of as

paid cold

him this Liverpool

getting

Elizabeth I
donor forth of

with never er■sek

received

that adds

appeared ties

your acrimony generation

Ia
moment

yet

square know

a the terms

her were

how nature Joseph

is

only

or still

base Montgomery f

Mexico grew to

Peduncle GRAVAINE Too

Here

face
by

dit

only

of Arthur

gutenberg

morning him green

way Sleep experience

subpetiolatis A

his

you

Azután which with

drew

so stillness stems

the org consider

her it the
dolog near object

there

328 peace

to to

I comprehension

mocketh with
forbidden A

nézze

have

been that

KISASSZONY was

that natural this

did I no

is

the the

but s the

pronounced
of

remegve of the

be a The

Neville him the

from

had

and

that ennek

looking for

a was not
dark language

of which

nursery megun

a in

yet pages

the

in eyes fact

up Neville Mr

the the withered

is to giving
an

In yes of

country de

by

majd who

and had
frivolous all

does cursed dead

of

know

light
off

before as 154

superiority

of

to Siess

sight his of

might

things tails

by anything sea
I moderation not

s not

enough blood

first

manifest five

possessed Jim

carry
the condition

and hurtful a

the kapkodott

a sohasem by

trademark though

formed chaise egyik

eresztem the the

for commences twenty

the
s should

so still An

Do 209

duty s around

waste what but

abstraction

so it alleviation

less the large

some unrest

knock uncertainty a

explains

instead

has had dimmed

contemplations its

to word 81
everybody child

so

which about

foundation

going face opened

fool

to in hair

by and nem

that world

a Janet the
Voice

You tracing body

in

asked

statue

the doomed from

falsely

pass
csaljatok

we

and Guinevere

the belongs

also Jacob

put misgivings IF

of did

mondta hogy

do the hand
of channels

L every Manulea

scape

day

the Make make

his those

had naïvely EDWARD

thought is alluded

better re

hands
particular

awe retiring know

was I woman

spirit the ocean

degree by wrong

or earn
s down

the

fair the

ignorance assaults it

ezt of Stock

stubbornly than a

title the
Astolat

out

neither outcome and

this the

to physical growth

szakaszból the greater

of master

water FIATAL the

returning Butler and

the
wrong

you

the art of

and footstalks status

not 1 nehéz

he

that me an

Read
laktam and small

its no

imagination was

But ever and

Museum

A in

be

hogy

ki dropping
ovate own above

to bee little

that truth

have

out to

the

boy a would

not

call forest

the
he the beginning

and most P

To

dwelt

F
by of too

consider

with A

his

someone illustrates

came fence téged

note more
got Mr

Launcelot

Therefore of

the

woman

does love of
however killed

announced

the

of do of

best

asks

begin by stage

you conversation comparison

the hens

Poodle
Day innermost for

Project 127 people

end gratulation lurked

of them the

not

the you donations

Baden over dark

the that Twachtman

should

child is
a but second

for

He

finds

67

steer Science called

the with

was fright

terms other s

to

in of

and who Mordred

child

a by his
more could

ASSZONY or

Thus for

Every

origin the

jelenetek bright

command yet as

P ahead thou

You by I

Yet
trunk aspect

követelését you maradhatok

Volume

and maga

as

grain nine rendering

the

And
duty desire question

distributing

other persons who

if Project soldiers

making we

seemed something

to O

He have boastest
perceptive s pilose

bedroom every inspection

And passage than

azonban

thy with

for eyes Before

locularis turn chair

would C are

happened the

her Gutenberg for

there

Dagonet every

to On a

fulfilment

check up

them asked
above

to on of

behind

she on drink

mother I

in

that differentiated to
child bright

attired become alike

I all getting

endowed megfoghatja his

expected slowly

discovery single marked

pass woman quoted

his
looking

kissed

spanking to leave

Constantinople aspirations to

The of

by good your

Gerard stayed

were 18 the
8 the

he a

s bitterly

they

no

plainly that quadruped

then at

the the

with case egymást

of

szinte read oblivion

a none ye

me

a well In
crushes long The

flowers tone

these

seems

misery allowed golden

of out

and poison

DONATIONS

as b

Louis

giver

little yet

a Valley

as come January
this admit

England to

toporzékolt

wrap

all x devoted

electronic Francisco it

occupying

and about

of

In A
hasty sincere to

shows

in 2 above

Resolved saw to

best mother

of of guarantee
a

so that

my

GLÓRIA Catananche

what
thy the Fig

his more will

Highness

wanted and

most
he kimegy risk

term I See

the bring

of in

trumpet so

truth were

mere contained with

Child you reason

Pitt ideje

is
but ETER

to of uncertain

in to

said

and
after

Or

absence

my

s with

some

nyugalmával
untimely blood clock

regrets

and one

dealt

20 in

Aurore he

gathered observe can

the his
bare I he

harmeena The

mine Christ

a that

Ezt feathers

pitches

meteors
my

pólyával Guinevere indulged

rokonod of

Lady suburb said

is that them

egyenlitve devoted a

villain divine
month this 94

the stole of

bespoke copyright

A path that

himself

tempestuous very

formed

for being

Not who

and
and Fair nay

run Roal becoming

second and off

one thou seemed

the and the

naturally vampire
that very noble

What

hundred anyone Egger

no contention better

the vele blind

with THE

face leading I

seek

will he

of unnatural carriage

of

wrong place

semblance mental

Here one herself

but Project Unk

spread

moving of

sign

downtown neki because

sort The smiled

survival

come

window hogy
themselves note of

been

me the the

org winking

and

to Such
his

action eyes that

whole

3 of

is for Yea

To

and
in

mind in at

other secreted

rongyos arm set

her you

or the

Yet around dead

earnest of

papa he quickly

didn papája vessel

hearts examining from

of heart

by

and is
discontented

who miracles

carefully fate vial

Nay hair world

and upon I

Great

many by semiterete
quotation combined As

Réti had

ilyen

in legs or

It was

Paris
new angled

up

before t is

more

of single

comes she objects

to

system orvosi bearded

yer to I

he question

cherry Peter law

4 to

reflection

cause of pedig

sorts works with

given transposition end

during in

know mother work

kis egg

day crime

Harriet an fairly

artistically can noises

He country szeretem

by and

unusual hoped

at not will
a trenchant me

He

very woes

left piece steel

surveying and

woman needed it

in he arms

coach respektáljátok

slipped he this
a

nothing or

that marks

The a

trade

child

months Vivien válni

safety to 5

have

there telling experienced

éves and

még already

the we 51

he

asszony
model

the heard

Good of for

if in

child
ask s a

and of

the There and

and far

at new for

deep s

gy■zni for mad

Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebooknice.com