Understanding by Design by Grant Wiggins and Jay McTighe

Backward Design
Why “backward” is best

Deliberate and focused instructional design requires us as teachers and curriculum writers to make an
important shift in our thinking about the nature of our job. The shift involves thinking a great deal,
first, about the specific learnings sought, and the evidence of such learnings, before thinking about
what we, as the teacher, will do or provide in teaching and learning activities. Though considerations
about what to teach and how to teach it may dominate our thinking as a matter of habit, the challenge
is to focus first on the desired learnings from which appropriate teaching will logically follow.

Our lessons, units, and courses should be logically inferred from the results sought, not derived from
the methods, books, and activities with which we are most comfortable. Curriculum should lay out
the most effective ways of achieving specific results. It is analogous to travel planning. Our
frameworks should provide a set of itineraries deliberately designed to meet cultural goals rather than
a purposeless tour of all the major sites in a foreign country. In short, the best designs derive
backward from the learnings sought.

The appropriateness of this approach becomes clearer when we consider the educational purpose that
is the focus of this book: understanding. We cannot say how to teach for understanding or which
material and activities to use until we are quite clear about which specific understandings we are after
and what such understandings look like in practice. We can best decide, as guides, what “sites” to
have our student “tourists” visit and what specific “culture” they should experience in their brief time
there only if we are clear about the particular understandings about the culture we want them to take
home. Only by having specified the desired results can we focus on the content, methods, and
activities most likely to achieve those results.

But many teachers begin with and remain focused on textbooks, favored lessons, and time-honored
activities—the inputs—rather than deriving those means from what is implied in the desired results—
the output. To put it in an odd way, too many teachers focus on the teaching and not the learning.
They spend most of their time thinking, first, about what they will do, what materials they will use,
and what they will ask students to do rather than first considering what the learner will need in order
to accomplish the learning goals.

Consider a typical episode of what might be called content-focused design instead of results-focused
design. The teacher might base a lesson on a particular topic (e.g., racial prejudice), select a resource
(e.g., To Kill a Mockingbird), choose specific instructional methods based on the resource and topic
(e.g., Socratic seminar to discuss the book and cooperative groups to analyze stereotypical images in
films and on television), and hope thereby to cause learning (and meet a few English/language arts
standards). Finally, the teacher might think up a few essay questions and quizzes for assessing student
understanding of the book.

This approach is so common that we may well be tempted to reply, What could be wrong with such
an approach? The short answer lies in the basic questions of purpose: Why are we asking students to
read this particular novel—in other words, what learnings will we seek from their having read it? Do

Team-Based Learning Institute Page 1 of 156

the students grasp why and how the purpose should influence their studying? What should students be
expected to understand and do upon reading the book, related to our goals beyond the book? Unless
we begin our design work with a clear insight into larger purposes—whereby the book is properly
thought of as a means to an educational end, not an end unto itself—it is unlikely that all students will
understand the book (and their performance obligations). Without being self-conscious of the specific
understandings about prejudice we seek, and how reading and discussing the book will help develop
such insights, the goal is far too vague: The approach is more “by hope” than “by design.” Such an
approach ends up unwittingly being one that could be described like this: Throw some content and
activities against the wall and hope some of it sticks.

Answering the “why?” and “so what?” questions that older students always ask (or want to), and
doing so in concrete terms as the focus of curriculum planning, is thus the essence of understanding
by design. What is difficult for many teachers to see (but easier for students to feel!) is that, without
such explicit and transparent priorities, many students find day-to-day work confusing and frustrating.

The twin sins of traditional design

More generally, weak educational design involves two kinds of purposelessness, visible throughout
the educational world from kindergarten through graduate school. We call these the “twin sins” of
traditional design. The error of activity-oriented design might be called “hands-on without being
minds-on”—engaging experiences that lead only accidentally, if at all, to insight or achievement. The
activities, though fun and interesting, do not lead anywhere intellectually. Such activity-oriented
curricula lack an explicit focus on important ideas and appropriate evidence of learning, especially in
the minds of the learners.

A second form of aimlessness goes by the name of “coverage,” an approach in which students march
through a textbook, page by page (or teachers through lecture notes) in a valiant attempt to traverse
all the factual material within a prescribed time. Coverage is thus like a whirlwind tour of Europe,
perfectly summarized by the old movie title If It's Tuesday, This Must Be Belgium, which properly
suggests that no overarching goals inform the tour.

As a broad generalization, the activity focus is more typical at the elementary and lower middle
school levels, whereas coverage is a prevalent secondary school and college problem. No guiding
intellectual purpose or clear priorities frame the learning experience. In neither case can students see
and answer such questions as these: What's the point? What's the big idea here? What does this help
us understand or be able to do? To what does this relate? Why should we learn this? Hence, the
students try to engage and follow as best they can, hoping that meaning will emerge.

The three stages of backward design

Stage 1: Identify desired results

What should students know, understand, and be able to do? What content is worthy of understanding?
What enduring understandings are desired? In Stage 1 we consider our goals, examine established
content standards (national, state, district), and review curriculum expectations. Because typically we
have more content than we can reasonably address within the available time, we must make choices.
This first stage in the design process calls for clarity about priorities.

Team-Based Learning Institute Page 2 of 156

Stage 2: Determine acceptable evidence

How will we know if students have achieved the desired results? What will we accept as evidence of
student understanding and proficiency? The backward design orientation suggests that we think about
a unit or course in terms of the collected assessment evidence needed to document and validate that
the desired learning has been achieved, not simply as content to be covered or as a series of learning
activities. This approach encourages teachers and curriculum planners to first “think like an assessor”
before designing specific units and lessons, and thus to consider up front how they will determine if
students have attained the desired understandings.

Stage 3: Plan learning experiences and instruction

With clearly identified results and appropriate evidence of understanding in mind, it is now the time
to fully think through the most appropriate instructional activities. Several key questions must be
considered at this stage of backward design: What enabling knowledge (facts, concepts, principles)
and skills (processes, procedures, strategies) will students need in order to perform effectively and
achieve desired results? What activities will equip students with the needed knowledge and skills?
What will need to be taught and coached, and how should it best be taught, in light of performance
goals? What materials and resources are best suited to accomplish these goals?

Note that the specifics of instructional planning—choices about teaching methods, sequence of
lessons, and resource materials—can be successfully completed only after we identify desired results
and assessments and consider what they imply. Teaching is a means to an end. Having a clear goal
helps to focus our planning and guide purposeful action toward the intended results.

Conclusion

Backward design may be thought of, in other words, as purposeful task analysis: Given a worthy task
to be accomplished, how do we best get everyone equipped? Or we might think of it as building a
wise itinerary, using a map: Given a destination, what's the most effective and efficient route? Or we
might think of it as planning for coaching: What must learners master if they are to effectively
perform? What will count as evidence on the field, not merely in drills, that they really get it and are
ready to perform with understanding, knowledge, and skill on their own? How will the learning be
designed so that learners' capacities are developed through use and feedback?

This is all quite logical when you come to understand it, but “backward” from the perspective of
much habit and tradition in our field. A major change from common practice occurs as designers must
begin to think about assessment before deciding what and how they will teach. Rather than creating
assessments near the conclusion of a unit of study (or relying on the tests provided by textbook
publishers, which may not completely or appropriately assess our standards and goals), backward
design calls for us to make our goals or standards specific and concrete, in terms of assessment
evidence, as we begin to plan a unit or course.

The rubber meets the road with assessment. Three different teachers may all be working toward the
same content standards, but if their assessments vary considerably, how are we to know which
students have achieved what? Agreement on needed evidence of learning leads to greater curricular
coherence and more reliable evaluation by teachers. Equally important is the long-term gain in
teacher, student, and parent insight about what does and does not count as evidence of meeting
complex standards.

Team-Based Learning Institute Page 3 of 156

Team-Based Learning Institute Page 4 of 156
Introduction to Team-Based Learning
It’s like a courtroom jury ...
TBL is a uniquely powerful form of small
group learning. It provides a complete Think of a courtroom jury that sifts through large amounts
coherent framework for building a of evidence, statements, and transcripts to come up with a
flipped course experience. simple decision: guilty or not guilty. Imagine your work on a
jury; you rise to state the jury’s verdict, but another person
TBL lets you achieve two rises from a different jury team in the same courtroom and
states a different verdict. You naturally want to talk to them;
important things: you naturally want to ask “why?” This simple comparability
between decisions, and the natural tendency to ask the
1. Students come to class prepared question “why” is at the heart of TBL. This “why” motivation
by using TBL’s ingenious Readiness provides the instructional fuel to power insightful debates
Assurance Process. between student teams.
2. Students learn how to apply the
course concepts to solve interesting,
authentic, real-world problems
using TBL’s 4 S framework.

The rhythm of TBL students for the activities that follow, and then moving
to Application Activities that often grow in complexity
TBL courses have a recurring pattern of instruction that and length as the module progresses. As the module is
is typical of many flipped classrooms. Students prepare ending, you provide some closure and reinforcement.
before class and then students spend the bulk of Module length varies in different contexts. In some
class time solving problems together. TBL gives you a courses an entire cycle is completed in one long
straightforward whole course framework to design and session and in other courses the cycle may be spread
implement your flipped classroom. across multiple class meetings.

A typical TBL course is divided into five to seven As the next module begins, the familiar TBL rhythm
modules. Each module has a similar rhythm, opening starts to build: out-of-class preparation, the Readiness
with the Readiness Assurance Process that prepares the Assurance Process, followed by Application Activities.

Typical TBL Cycle

Readiness Assurance Multiple

Process Application Activities

Readings
iRAT
tRAT
Appeals
Mini-lecture

Team-Based Learning Institute Page 5 of 156

How TBL Works

Readiness Assurance Preparatory Materials

Pre-Readings
Initial Preparation

Getting Your Students Ready

Individual Readiness
During this 5 stage process at the beginning of each Assurance Test
module, students progress from initial preparation to true
readiness to begin problem-solving.
Team Readiness
Following the Readiness Assurance Process, the bulk of Assurance Test
class time is spent with students applying course concepts
and solving problems.
Appeals Process

1 Pre-Class Preparation
Students are assigned preparatory materials to review Mini-lecture
True Readiness
before start of each module. The preparatory materials can Clarification
be textbook chapters, articles, videos, or PowerPoint slides.
The preparatory materials should highlight foundational
vocabulary and the most important concepts the students
need to begin problem solving, but not everything they
need to know by module end.

2 Individual Readiness Assurance Test 4 Appeals

To begin the classroom portion of the RAP process, During the closing of the team test, the instructor
students complete a 15-20 multiple-choice question circulates around the room and encourages teams to
test. They first complete the test individually (iRAT), and consider creating a written appeal for questions they
then repeat the same exact test with their team (tRAT). got incorrect. This forces students back into the reading
The iRAT holds students accountable for acquiring material exactly where they are still having difficulty. The
important foundational knowledge from the preparatory team then researches the “right” answer and may choose
materials that will prepare them to begin problem-solving. to complete the appeals form with their rationale and
The questions are typically written at Bloom’s levels: defense for their alternate answer. The appeal must consist
remembering, understanding and simple applying. of (a) a clear statement of argument, and (b) evidence cited
from the preparation materials. The instructor collects
3 Team Readiness Assurance Test these forms and considers them after class.
The Team Readiness Assurance Process Test (tRAT) is the
exact same test as the iRAT. A special type of scoring card 5 Mini-lecture
known as an IF-AT should be To conclude the Readiness Assurance Process, the
used (scratch and win style instructor focuses a short mini-lecture only on the
testing). With IF-AT’s, the Team #3 2
concepts that are still problematic for the students.
teams must negotiate which
answer to choose, they then 4
2
scratch off an opaque coating 4 In the words of Bob Philpot at South University, “TBL helps
over their answer choice, 1 me understand the 10-15% of the course material I really
hoping to find a star that need to talk to the students about.”
indicates a correct answer. If
the team does not discover a
star, they continue to discuss
the question and sequentially
select other choices. The tRATs [Link]
are high energy learning
events.

Team-Based Learning Institute Page 6 of 156

In Class Activities
4S Problem-Solving Framework
4S Significant
Problem

Framework
• Significant Problems
• Same Problem Same
• Specific Choice Problem
• Simultaneous Report
Specific
Choice
In the TBL classroom, the bulk of class time is spent having
student teams solve, report, and discuss solutions to
relevant, significant problems. Structuring the problems Simultaneous
using TBL’s 4S Framework lets you leverage the power Report
of team processing without many of the problems that
are inherent in other forms of small-group learning. The
structure of the TBL activities gives individuals, and teams,
many opportunities to make decisions and get timely
feedback on the quality of their thinking and their process
teams examine and critique other teams decisions and
for arriving at their answer.
defend their own.
Examples of Specific Choice
1 Significant Problem • Which of these is the best example of X?
Examples of Significant Problem
• Most important piece of evidence in support of Y?
• A historian reconciles conflicting sources.
• Which statement would the author most agree with?
• A doctor decides the best course of action.
• A businessperson picks the best location for a
business.
4 Simultaneously Report
• A writer identifies the most powerful passage or best Simultaneous reporting is most simply accomplished
example. with holding up of a coloured card indicating a particular
choice. When a team sees that another team has made a
You must use a significant, relevant problem that captures different decision, they naturally want to challenge the
the interest of students. The quality of the problem other teams’ decision. In the ensuing conversation, the
ultimately controls the effectiveness of an application teams challenge each other and defend their own thinking.
activity. Problems must require students to use course The reporting requires teams to articulate their thinking to
concepts to solve them. other teams – putting their thoughts into words. This helps
cognitively with the process of creating enduring, deep
2 Same Problem understanding. The feedback from their peers is immediate
and focused on “how did you arrive at your decision” and
Teams work on the same problem. This ensures the not “which is the right answer.”
comparability of team solutions and this naturally acts as
a potent discussion starter. Having students work on the
same problem lets you create reporting opportunities for A
teams to defend, challenge, discuss, and examine each
other’s thinking and problem-solving process. Working on
the same problem, ensures that students are interested in
what other teams decided.
B
3 Specific Choice A
Teams select the best choice from a limited list of
options. This ensures that teams can easily compare their
final decisions to the decisions of other teams. It is this
comparability that drives the rich reporting discussion as

Team-Based Learning Institute Page 7 of 156

4 Essential Elements of TBL
Teams must be properly Getting Students Ready
formed and managed
The magic of the Readiness
For more information
TBL works best with large, diverse Assurance Process is that it builds on on how to get started.
teams. TBL teams should have 5-7 the initial preparation, changing it
students. Teams should be created by into true readiness to begin problem- Visit
solving. At the simplest level, the RAP [Link]
instructor and uniformly distribute
the student assets you feel are is a series of multiple-choice tests.
Classroom materials, books,
important for team success. Teams First the test is taken individually, and videos, workshop schedules
need to be permanent so team then the same test is immediately and more!
cohesion has time to build. retaken in teams.

Applying course concepts Making students accountable

Teachers Say It Works! Use the 4 S problem-solving Making students truly accountable is
framework to have students make key. There is individual accountability
The enthusiasm and energy of complex decisions and then get rich, from the iRAT, but what is most
students. It’s just so much fun! immediate, and specific feedback on motivating is the accountability to
the quality of their decisions. The give- teammates during the tRAT’s and
Larry Michaelsen and-take discussions that follows after Application Activities. Peer evaluation
University of Central Missouri
teams publically report their decisions is key to giving the grading scheme
is a powerful opportunity deepen enough teeth to motivate students.
students understanding.
Students excited about learning and
faculty falling in love with teaching.
The way learning should be.

Holly Bender
Iowa State University
The Literature Says It Works!
Students are more engaged
Students are so engaged in
conversation with each other and Students reported higher level of engagement in TBL courses (Chung et al., 2009; Clark et
the task that, literally, they don’t al., 2008; Kelly et al., 2005; Levine et al., 2004).
know I am there. My favorite days
are when I have to tell them to Increased excitement in the TBL classroom
leave.
Teachers report increased excitement and engagement in their classrooms (Andersen et
Laura Madson al., 2011; Dana, 2007; Jacobson, 2011; Letassy et al.; 2008; Nicoll-Senft, 2009).
New Mexico State University
Teams outperform best members
I think the genius of TBL is that it The worst team typically outperforms the best student. In 20 years of results Michaelsen
maximizes the advantages of group (1989) found that 99.95% of teams outperformed their best member by an average of
learning while minimizing the 14%.
disadvantages.
Students perform better on final and standardized exams
Brent Maclaine
University of Prince Edward Island TBL students outperform non-TBL students on examinations (Grady, 2011; Letassy et al.,
2008; Persky, 2012, Zingone et al.; 2011, Koles et al., 2005; Koles et al., 2010; Thomas &
Bowen, 2011).

A large class can be an asset

Michaelsen, Knight, Fink (2002) found that students actually perceived a larger class size
as beneficial to their learning with TBL.

Team-Based Learning Institute Page 8 of 156

1 TBL is a collection of practices that support one another
for powerful instructional effect. This chapter describes
the building blocks of team-based learning and the steps
necessary to put them into place.

The Essential Elements of Team-Based

Learning
Larry K. Michaelsen, Michael Sweet

Team-based learning (TBL) possibly relies on small group interaction more

heavily than any other commonly used instructional strategy in postsecondary
education (for comparative discussion of different approaches, see Fink, 2004;
Johnson, Johnson, and Smith, 2007; Millis and Cottell, 1998). This conclu-
sion is based on three facts. First, with TBL, group work is central to expos-
ing students to and improving their ability to apply course content. Second,
with TBL, the vast majority of class time is used for group work. Third,
courses taught with TBL typically involve multiple group assignments that
are designed to improve learning and promote the development of self-
managed learning teams.
This chapter begins with a brief overview of TBL. Next, we discuss the
four essential elements of TBL and then walk through the steps required to
implement them. Finally, we examine some of the benefits that students,
administrators, and faculty can expect from a successful implementation
of TBL.

A Broad Overview of TBL

The primary learning objective in TBL is to go beyond simply covering con-
tent and focus on ensuring that students have the opportunity to practice
using course concepts to solve problems. Thus, TBL is designed to provide
students with both conceptual and procedural knowledge. Although some
time in the TBL classroom is spent ensuring that students master the course

NEW DIRECTIONS FOR TEACHING AND LEARNING, no. 116, Winter 2008 © Wiley Periodicals, Inc.
7
Published online in Wiley InterScience ([Link]) • DOI: 10.1002/tl.330

Team-Based Learning Institute Page 9 of 156

8 TEAM-BASED LEARNING: SMALL-GROUP LEARNING’S NEXT BIG STEP

content, the vast majority of class time is used for team assignments that
focus on using course content to solve the kinds of problems that students
are likely to face in the future. Figure 1.1 outlines generally how time in one
unit of a TBL course is organized.
In a TBL course, students are strategically organized into permanent
groups for the term, and the course content is organized into major units—
typically five to seven. Before any in-class content work, students must
study assigned materials because each unit begins with the readiness assur-
ance process (RAP). The RAP consists of a short test on the key ideas from
the readings that students complete as individuals; then they take the same
test again as a team, coming to consensus on team answers. Students receive
immediate feedback on the team test and then have the opportunity to write
evidence-based appeals if they feel they can make valid arguments for their
answer to questions that they got wrong. The final step in the RAP is a lec-
ture (usually very short and always very specific) to enable the instructor to
clarify any misperceptions that become apparent during the team test and
the appeals.
Once the RAP is completed, the remainder (and the majority) of the
learning unit is spent on in-class activities and assignments that require stu-
dents to practice using the course content.

The Four Essential Elements of Team-Based Learning

Shifting from simply familiarizing students with course concepts to requir-
ing that students use those concepts to solve problems is no small task.
Making this shift requires changes in the roles of both instructor and stu-
dents. The instructor’s primary role shifts from dispensing information to
designing and managing the overall instructional process, and the students’
role shifts from being passive recipients of information to one of accepting
responsibility for the initial exposure to the course content so that they will
be prepared for the in-class teamwork.
Changes of this magnitude do not happen automatically and may even
seem to be a dream rather than an achievable reality. They are, however, achiev-
able when the four essential elements of TBL are successfully implemented:

• Groups. Groups must be properly formed and managed.

• Accountability. Students must be accountable for the quality of their indi-
vidual and group work.
• Feedback. Students must receive frequent and timely feedback.
• Assignment design. Group assignments must promote both learning and
team development.

When these four elements are implemented in a course, the stage is set for
student groups to evolve into cohesive learning teams.

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 10 of 156

 Figure 1.1. Team-Based Instructional Activity Sequence

Note: This sequence is repeated for each major instructional unit—typically five to seven per course.

Team-Based Learning Institute

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Page 11 of 156
10 TEAM-BASED LEARNING: SMALL-GROUP LEARNING’S NEXT BIG STEP

Element 1: Properly Formed and Managed Groups. TBL requires

that the instructor oversee the formation of the groups so that he or she can
manage three important variables: ensuring that the groups have adequate
resources to draw from in completing their assignments and approximately
the same level of those resources across groups, avoiding membership coali-
tions that are likely to interfere with the development of group cohesiveness,
and ensuring that groups have the opportunity to develop into learning
teams.
Distributing Member Resources. In order for groups to function as effec-
tively as possible, they should be as diverse as possible. Each group should
contain a mix of student characteristics that might make the course easier
or more difficult for a student to do well in the course (for example, previ-
ous course work or course-related practical experience) as well as demo-
graphic characteristics like gender and ethnicity. The goal here is to equip
groups to succeed by populating them with members who will bring differ-
ent perspectives to the task.
Findings in both group dynamics research (Brobeck and others, 2002)
and educational research (Chan, Burtis, and Bereiter, 1997) illuminate the
positive impact of diverse input in problem-solving discussions on both learn-
ing and performance. When group members bring many different perspec-
tives to a task, their process of collaborative knowledge building in pursuit of
consensus is powerful to watch. In addition, although member diversity ini-
tially inhibits both group processes and performance, it is likely to become an
asset when members have worked together over time and under conditions
that promote group cohesiveness (Watson, Kumar, and Michaelsen, 1993).
Minimizing Barriers to Group Cohesiveness: Avoiding Coalitions. Coalitions
within a group are likely to threaten its overall development. In newly formed
groups, either a previously established relationship between a subset of mem-
bers in the group (such as a boyfriend and girlfriend or fraternity brothers) or
the potential for a cohesive subgroup based on background factors such as
nationality, culture, or native language is likely to burden a group with insider-
outsider tension that can plague the group throughout the term. Because it is
human nature to seek out similar others, allowing students free rein in form-
ing their own groups practically ensures the existence of potentially disrup-
tive subgroups (Fiechtner and Davis, 1985; Michaelsen and Black, 1994).
Time. Any group dynamics textbook will tell you that groups need time
to develop into high-performing teams, regardless of whether you favor
sequential or life cycle models (Tuckman, 1965; Tuckman and Jensen,
1977), cyclical models (Worchel, Wood, and Simpson, 1992), or adaptive
or nonsequential models (McGrath, 1991). For this reason, students should
stay in the same group for the entire course. Although even a single well-
designed group assignment usually produces a variety of positive outcomes,
only when students work together over time can their groups become cohe-
sive enough to evolve into self-managed and truly effective learning teams.

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 12 of 156

 THE ESSENTIAL ELEMENTS OF TEAM-BASED LEARNING 11

Element 2: Student Accountability for Individual and Group

Work. In lecture classes, there is no need for students to be accountable to
anyone other than the instructor. By contrast, TBL requires students to be
accountable to both the instructor and their teammates for the quality and
quantity of their individual work. Furthermore, teams must accountable for
the quality and quantity of their work as a unit. (For a review of the effects
of accountability on an array of social judgments and choices, see Lerner
and Tetlock, 1999.)
Accountability for Individual Preclass Preparation. Lack of preparation
places clear limits on both individual learning and team development. If sev-
eral members of a team come unprepared to contribute to a complex group
task, then the team as a whole is far less likely to succeed at that task, cheat-
ing its members of the learning that the task was designed to stimulate. No
amount of discussion can overcome absolute ignorance. Furthermore, lack
of preparation also hinders the development of cohesiveness because those
who do make the effort to be prepared will resent having to carry their
peers. As a result, the effective use of learning groups clearly requires that
individual students be made accountable for class preparation.
Accountability for Contributing to The Team. The next step is ensur-
ing that members contribute time and effort to group work. In order to
accurately assess members’ contributions to the success of their teams, it is
imperative that instructors involve the students themselves in a peer assess-
ment process. That is, members should be given the opportunity to evalu-
ate one another’s contributions to the activities of the team. Contributions
to the team include activities such as individual preparation for teamwork,
reliable class attendance, attendance at team meetings that may have
occurred outside class, positive contributions to team discussions, and valu-
ing and encouraging contributions from fellow team members. Peer assess-
ment is essential because team members are typically the only ones who
have enough information to evaluate one another’s contributions accurately.
Accountability for High-Quality Team Performance. The third significant
factor in ensuring accountability is developing an effective means to assess
team performance. There are two keys to effectively assessing teams. One is
using assignments that require teams to create a product that can be read-
ily compared across teams and with “expert” opinions, and the other is
using procedures to ensure that such comparisons occur frequently and in
a timely manner.
Element 3: Frequent Immediate Student Feedback. Immediate feed-
back is the primary instructional lever in TBL for two very different reasons.
First, feedback is essential to content learning and retention—a notion that not
only makes intuitive sense but is also well documented in educational research
literature (Bruning, Schraw, and Ronning, 1994; Kulik and Kulik, 1988; Hat-
tie and Timperley, 2007). Second, immediate feedback has tremendous impact
on group development (for a review, see Birmingham and McCord, 2004).

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 13 of 156

12 TEAM-BASED LEARNING: SMALL-GROUP LEARNING’S NEXT BIG STEP

Element 4: Assignments That Promote Both Learning and Team

Development. The most fundamental aspect of designing team assignments
that promote both learning and team development is ensuring that they
truly require group interaction. In most cases, team assignments generate a
high level of interaction if they require teams to use course concepts to
make decisions that involve a complex set of issues and enable teams to re-
port their decisions in a simple form. When assignments emphasize mak-
ing decisions, most students choose to complete the task by engaging each
other in a give-and-take content-related discussion. By contrast, assignments
that involve producing complex output such as a lengthy document often
limit both learning and team development because they typically inhibit
intrateam discussions in two ways. First, discussions are likely to be much
shorter because students are likely to feel an urgency to create the product
that is to be graded. Second, instead of focusing on content-related issues,
they are likely to center on how to divide up the work. Thus, complex prod-
uct outputs such as a lengthy document seldom contribute to team devel-
opment because they are likely to have been created by individual members
working alone on their part of the overall project.
Summary. By adhering to the four essential elements of TBL—careful
design of groups, accountability, feedback, and assignments—teachers create
a context that promotes the quantity and quality of interaction required to
transform groups into highly effective learning teams. Appropriately form-
ing the teams puts them on equal footing and greatly reduces the possibility
of mistrust from preexisting relationships between a subset of team members.
Holding students accountable for preparation and attendance motivates team
members to behave in prosocial ways that build cohesiveness and foster trust.
Using RAPs and other assignments to provide ongoing and timely feedback
on both individual and team performance enables teams to develop confi-
dence in their ability to capture the intellectual resources of all their mem-
bers. Assignments that promote both learning and team development
motivate members to challenge others’ ideas for the good of the team. Also,
over time, students’ confidence in their teams grows to the point that they are
willing and able to tackle difficult assignments with little or no external help.

Implementing Team-Based Learning

Effectively using TBL typically requires redesigning a course from beginning
to end, and the redesign process should begin well before the start of the
school term. The process involves making decisions about and designing
activities at four different times: before class begins, the first day of class,
each major unit of instruction, and near the end of the course. In this sec-
tion, we discuss the practical steps a TBL instructor takes at each of these
points, but for a treatment that is even detailed and practical, we direct read-
ers to Michaelsen, Knight, and Fink (2004).

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 14 of 156

 THE ESSENTIAL ELEMENTS OF TEAM-BASED LEARNING 13

Before Class Begins. Traditional education, particularly in undergrad-

uate programs, has tended to separate knowledge acquisition from knowl-
edge application both between and within courses. In a typical biology
course, for example, students listen to lectures through which they are
expected to absorb a great deal of knowledge that they will then later be
asked to put to use in a biology lab. In fact, even within higher-level courses,
students often spend much of the term absorbing knowledge that they do
not put to use until a project that is due just prior to the final exam.
TBL uses a fundamentally different knowledge acquisition and knowl-
edge application model. With TBL, students repeat the knowledge acquisi-
tion and knowledge application cycle several times within each individual
course. They individually study the course content, discuss it with their
peers and the instructor, and immediately apply it in making choices that
require them to use their knowledge. Thus, students in TBL courses develop
a much better sense of the relevance of the material because they seldom
have to make unreasonably large inferences about when and how the con-
tent might become useful in the real world. Rather than being filled with
libraries of “inert knowledge” (Whitehead, 1929), from which they then
later must extract needed information with great effort, students walk away
from TBL courses having already begun the practical problem-solving
process of learning to use their knowledge in context.
This benefit, however, does not occur by accident. Designing a successful
TBL course involves making decisions related to first identifying and cluster-
ing instructional objectives and then designing a grading system around them.
Identifying Instructional Objectives. Designing a TBL course requires
instructors to “think backward.” What is meant by “think backward”? In
most forms of higher education, teachers design their courses by asking
themselves what they feel students need to know, then telling the students
that information, and finally testing the students on how well they absorbed
what they were told. In contrast, designing a TBL course requires instruc-
tors to “think backward”—backward because they are planned around what
they want students to be able to do when they have finished the course; only
then do instructors think about what students need to know. Wiggins and
McTighe (1998) used the term backward design to describe this method of
course design, which enables the instructor to build a course that provides
students both declarative and procedural knowledge (in other words, con-
ceptual knowledge and the ability to use that knowledge in decision mak-
ing). This is a useful distinction, but if you have taught only with
conceptual familiarization as your goal, it can be surprisingly difficult to
identify what exactly you want students to be able to do on completion of
a course. The following question is a good a good place to start.
What are the students who really understand the material doing that
shows you they get it? Imagine you are working shoulder-to-shoulder with
a former student who is now a junior colleague. In a wonderful moment,

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 15 of 156

14 TEAM-BASED LEARNING: SMALL-GROUP LEARNING’S NEXT BIG STEP

you see that colleague do something that makes you think, “Hey! She really
got from my class what I wanted her to get. There’s the evidence right
there!” When you are designing a course backward, the question you ask
yourself is: “What specifically is that evidence? What could a former
student be doing in a moment like that to make it obvious she really in-
ternalized what you were trying to teach her and is putting it to use in a
meaningful way?”
For every course, there are several answers to this question, and these
different answers correspond to the units of the redesigned version of the
course. A given real-world moment will likely demand knowledge from one
part of a course but not another, so for any given course, you should brain-
storm about a half-dozen of these proud moments in which a former student
is making it obvious that she really learned what you wanted her to. For
now, do not think about the classroom; just imagine she is doing something
in an actual organizational context. Also, do not be afraid to get too detailed
as you visualize these moments. In fact, come up with as many details as you
can about how this former student is doing what she is doing, what deci-
sions she is making, in what sequence, under what conditions, and so on.
These detailed scenarios become useful in three ways. First, the actions
taking place in the scenarios will help you organize your course into units.
Second, the scenarios will enable you to use class time to build students’
applied knowledge instead of inert knowledge. Third, the details of the sce-
nario will help you design the criteria for the assessments on which you can
base students’ grades.
Once you have brainstormed the scenarios and the details that accom-
pany them, you have identified your instructional objectives, which often
involve making decisions that are based on insightful applications of the
concepts from your course. Now you are ready to ask three more questions:

• What will students need to know in order to be able to do those things?

Answers to this question will guide your selection of a textbook, the con-
tents of your course packet, experiential exercises, and are likely to
prompt you to provide supplementary materials of your own creation or
simple reading guides to help students focus on what you consider most
important in the readings or lab findings. In addition, the answers will be
key in developing questions for the readiness assurance process.
• While solving problems, what knowledge will students need to make deci-
sions? Answers to this question will help you import the use of course
knowledge from your brainstormed real-world scenarios into the class-
room. You may not be able to bring the actual organizational settings in
which your scenarios occurred into the classroom, although computer
simulations, video (including full-length feature films), and requiring stu-
dents to learn by doing (see Miller, 1991, and Michaelsen and McCord,
2006) are coming much closer to approaching the real world. But you
can provide enough relevant information about those settings to design
NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 16 of 156

 THE ESSENTIAL ELEMENTS OF TEAM-BASED LEARNING 15

activities that require students to face the same kinds of problems and
make the same kinds of decisions they will make in clinical and labora-
tory settings.
• What criteria separate a well-made decision from a poorly made decision using
this knowledge? Answers to this question will help you begin building the
measures you will use to determine how well the students have learned
the material and how well they can put it to use under specific conditions.

In summary, TBL leverages the power of action-based instructional

objectives to not only expose students to course content but also give them
practice using it. When you are determining an instructional objective, it is
crucial to know how to assess the extent to which students have mastered
that objective. Some teachers feel that designing assessments first removes
something from the value of instruction—that it simply becomes teaching
to the test. With TBL the view is that you should teach to the test as long as
the test represents (as closely as possible) the real use to which students will
ultimately apply the course material: what they are going to do with it, not
just what they should know about it.
Designing a Grading System. The other step in redesigning the course
is to ensure that the grading system is designed to reward the right things.
An effective grading system for TBL must provide incentives for individual
contributions and effective work by the teams, as well as address the equity
concerns that naturally arise when group work is part of an individual’s
grade. The primary concern here is typically borne from past group work
situations in which students were saddled with free-riding team members
and have resented it ever since. Students worry that they will be forced to
choose between getting a low grade or carrying their less able or less moti-
vated peers. Instructors worry that they will have to choose between grad-
ing rigorously and grading fairly.
Fortunately, many of these concerns are alleviated by a grading system
in which a significant proportion of the grade is based on individual perfor-
mance, team performance, and each member’s contributions to the success
of the teams. As long as that standard is met, the primary remaining con-
cern is that the relative weight of the factors is acceptable to both the
instructor and the students.
The First Day of Class. Activities that occur during the first few
hours of class are critical to the success of TBL. During that time, the
teacher must accomplish four objectives: ensure that students understand
why you (the instructor) have decided to use TBL and what that means
about the way the class will be conducted, form the groups, alleviate stu-
dents’ concerns about the grading system, and set up mechanisms to
encourage the development of positive group norms.
Introducing Students to TBL. Because the roles of instructor and stu-
dents are so fundamentally different from traditional instructional practice,
it is critical that students understand both the rationale for using TBL and
NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 17 of 156

16 TEAM-BASED LEARNING: SMALL-GROUP LEARNING’S NEXT BIG STEP

what that means about the way the class will be conducted. Educating stu-
dents about TBL requires at a minimum providing them with an overview
of the basic features of TBL, how TBL affects the role of the instructor and
their role as students, and why they are likely to benefit from their experi-
ence in the course. This information should be printed in the course syl-
labus, presented orally, and demonstrated by one or more activities.
In order to foster students’ understanding of TBL, we recommend two
activities. The first is to explain the basic features of TBL using overhead
transparencies (or a PowerPoint presentation) and clearly spelling out how
the learning objectives for the course will be accomplished through the use
of TBL, compared to how the same objectives would be achieved using a
lecture-discussion course format. The second activity is a demonstration of
a readiness assurance process using the course syllabus, a short reading on
TBL, or some potentially useful ideas, such as what helps and hinders team
development or strategies for giving helpful feedback (see Michaelsen and
Schultheiss, 1988) as the content material to be covered. (In a class period
of less than an hour, this activity might occur on day 2.)
Forming the Groups. When forming groups, you must consider the
course-relevant characteristics of the students and the potential for the emer-
gence of subgroups. As a result, the starting point in the group formation
process is to gather information about specific student characteristics that will
make it easier or more difficult for a student to succeed in the class. For a par-
ticular course, characteristics that could make it easier for a student to succeed
might include previous relevant course work or practical experience or access
to perspectives from other cultures. Most commonly, characteristics making
it more difficult for students to succeed are the absence of those that would
make it easier, but might include such things as a lack of language fluency.
We recommend forming the groups in class in the presence of the stu-
dents to eliminate student concerns about ulterior motives the instructor may
have had in forming groups. (For a depiction of how to form groups quickly
and effectively, see Michaelsen and Sweet, 2008, and for a more detailed
explanation and video demonstration, go to [Link].)
Alleviating Student Concerns About Grades. The next step in getting
started on the right foot with TBL is to address student concerns about the
grading system. Fortunately, student anxiety based on previous experience
with divided-up group assignments largely evaporates as students come to
understand two of the essential features of TBL. One is that two elements of
the grading system create a high level of individual accountability for pre-
class preparation, class attendance, and devoting time and energy to group
assignments: counting individual scores on the readiness assurance tests and
basing part of the grade on a peer evaluation. The other reassuring feature
is that team assignments will be done in class and will be based on think-
ing, discussing, and deciding, so it is highly unlikely that one or two less-
motivated teammates members can put the entire group at risk.

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 18 of 156

 THE ESSENTIAL ELEMENTS OF TEAM-BASED LEARNING 17

Many instructors choose to alleviate student concerns about grades by

directly involving students in customizing the grading system to the class.
Students become involved by participating in setting grade weights
(Michaelsen, Cragin, and Watson, 1981; Michaelsen, Knight, and Fink,
2004). Within limits set by the instructor, representatives of the newly
formed teams negotiate with one another to reach a consensus (all of the
representatives must agree) on a mutually acceptable set of weights for each
of the grade components: individual performance, team performance, and
each member’s contributions to the success of the team. After an agreement
has been reached regarding the grade weight for each component, the stan-
dard applies for all groups for the remainder of the course.
Each Major Unit of Instruction. Each unit of a TBL course begins with
a readiness assurance process (RAP), which occurs at least five to seven times
each term. The RAP provides the foundation for individual and team account-
ability and has five major components: (1) assigned readings, (2) individual
tests, (3) team tests, (4) an appeals process, and (5) instructor feedback.
Assigned Readings. Prior to the beginning of each major instructional unit,
students are given reading and other assignments that should contain infor-
mation on the concepts and ideas that must be understood to be able to solve
the problem set out for this unit. Students complete the assignments and come
to the next class period prepared to take a test on the assigned materials.
Individual Test. The first in-class activity in each instructional unit is
an individual readiness assurance test (iRAT) over the material contained in
the preclass assignments. The tests typically consist of multiple-choice ques-
tions that enable the instructor to assess whether students have a sound
understanding of the key concepts from the readings. As a result, the ques-
tions should focus on foundational concepts, not picky details, and be dif-
ficult enough to stimulate team discussion.
Team Test. When students have finished the iRAT, they turn in their
answers (which are often scored during the team test) and immediately pro-
ceed to the third phase of the readiness assurance process, the tRAT. During
this third phase, students retake the same test, but this time as a team, and the
teams must reach agreement on the answers to each test question. They then
immediately check the correctness of their decision using the intermediate
feedback assessment technique (IF-AT), a self-scoring answer sheet (see Fig-
ure 1.2) that provides feedback on each team decision. With the IF-AT answer
sheets, students scratch off the covering of one of four (or five) boxes in search
of a mark indicating they have found the correct answer. If they find the mark
on the first try, they receive full credit. If not, they continue scratching until
they find the mark, but their score is reduced with each unsuccessful scratch.
This allows teams to receive partial credit for proximate knowledge.
The answer sheets are an effective way to provide timely feedback on
the team RATs (not the iRATs—otherwise members would know the answers
before the team test and discussion would be pointless). Furthermore, using

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 19 of 156

18 TEAM-BASED LEARNING: SMALL-GROUP LEARNING’S NEXT BIG STEP

Figure 1.2. Immediate Feedback Assessment Technique

the answer sheets makes it possible to provide real-time content feedback to

multiple teams without requiring them to maintain the same work pace.
Getting real-time feedback from the IF-AT provides two key benefits to
the teams. First, it enables members to correct their misconceptions of the
subject matter. Finding a star immediately after scratching the choice con-
firms the validity of it, and finding a blank box lets them know they have
more work to do. Second, it promotes both the ability and the motivation for
teams, with no input from the instructor, to learn how to work together effec-
tively. In fact, those who have used the IF-ATs for their tRATs have learned
that doing so virtually eliminates any possibility that one or two members
might dominate team discussions. “Pushy” members are only one scratch
away from embarrassing themselves, and quiet members are one scratch away
from being validated as a valuable source of information and two scratches
away from being told that they need to speak up.
The impact of the IF-AT on team development is immediate, powerful,
and extremely positive. In our judgment, using the IF-ATs with the tRATs is
the most effective tool available for promoting both concept understanding
and cohesiveness in learning teams. Anyone who does not use them will
miss a sure-fire way to implement TBL successfully.
Appeals Process. At this point in the readiness assurance process, students
proceed to the fourth phase, which gives them the opportunity to refer to their
assigned reading material and appeal any questions missed on the group test.
That is, students are allowed to do a focused restudy of the assigned readings
(this phase is “open book”) to challenge the teacher about their responses on
specific items on the team test or about confusion created by either the qual-
ity of the questions or inadequacies of the preclass readings.
Discussion among group members is usually very animated while the
students work together to build a case to support their appeals. The students
must produce compelling evidence to convince the teacher to award credit for
the answers they missed. Teachers listening to students argue the fine details
NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 20 of 156

 THE ESSENTIAL ELEMENTS OF TEAM-BASED LEARNING 19

of course material while writing team appeals report being convinced their
students learn more from appealing answers they got wrong than from con-
firming the answers they got right. As an integral part of the readiness assur-
ance process, this appeals exercise provides yet another review of the readings.
Instructor Feedback. The fifth and final part of the readiness assurance
process is oral feedback from the instructor. This feedback comes immediately
after the appeals process and allows the instructor to clear up any confusion
students may have about any of the concepts presented in the readings. As a
result, input from the instructor is typically limited to a brief, focused review
of only the most challenging aspects of the preclass reading assignment.
The Readiness Assurance Process in Summary. This process allows
instructors to minimize class time that often is used instead to cover mate-
rial that students can learn on their own. Time is saved because the instruc-
tor’s input occurs after students have individually studied the material, taken
an individual test focused on key concepts from the reading assignment,
retaken the same test as a member of a learning team, and completed a
focused restudy of the most difficult concepts. A cursory review of team test
results illuminates for instructors which concepts need additional attention
so that they can correct students’ misunderstandings. In contrast to the con-
cerns many instructors express about “losing time to group work” and not
being able to cover as much content, many others report being able to cover
more with the readiness assurance process than they can through lectures
(Knight, 2004). Leveraging the motivational power and instructional effi-
ciency of the readiness assurance process leaves the class a great deal of class
time to develop students’ higher-level learning skills as they tackle multiple
and challenging application-oriented assignments.
Beyond its instructional power, the readiness assurance process is the
backbone of TBL because it promotes team development in four specific
ways. First, starting early in the course (usually the first few class hours),
students are exposed to immediate and unambiguous feedback on both in-
dividual and team performance. As a result, each member is explicitly
accountable for his or her preclass preparation. Second, because team mem-
bers work face-to-face, the impact of the interaction is immediate and per-
sonal. Third, students have a strong vested interest in the outcome of the
group and are motivated to engage in a high level of interaction. Finally,
cohesiveness continues to build during the final stage of the process when
the instructor is presenting information. This is because unlike lectures, the
content of the instructor’s comments is determined by students’ choices and
actions during the readiness tests. Thus, the instructor’s comments provide
either positive reinforcement (they celebrate together) or corrective instruc-
tion (which, particularly in the presence of other groups, can be experienced
as embarrassing and, in this way, provide an “external threat” that builds
cohesiveness within a group). Although the impact of the readiness assur-
ance process on student learning is limited primarily to ensuring that they
have a solid exposure to the content, it also increases students’ ability to
NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 21 of 156

20 TEAM-BASED LEARNING: SMALL-GROUP LEARNING’S NEXT BIG STEP

solve difficult problems for two reasons. First, by encouraging preclass

preparation and a lively discussion, the process builds the intellectual com-
petence of team members. Second, because they have immediate perfor-
mance feedback, the experience of working together during the group and
in preparing appeals heightens their ability and willingness to provide high-
quality content feedback to one another. As a result, the readiness assurance
process provides a practical way of ensuring that even in large classes, stu-
dents are exposed to a high volume of immediate feedback that in some
ways can actually be better than having a one-on-one relationship between
student and instructor.
Promoting Higher-Level Learning. The final stage in the TBL instruc-
tional activity sequence for each unit of instruction is using one or more
assignments that provide students with the opportunity to deepen their
understanding by having groups use the concepts to solve a problem. These
application assignments must foster both accountability and give-and-take
discussion first within and then between groups. Designing these assign-
ments is probably the most challenging aspect of implementing TBL.
The key to creating and implementing effective group assignments is
following what TBL users fondly refer to as the 4 S’s: (1) assignments should
always be designed around a problem that is significant to students, (2) all of
the students in the class should be working on the same problem, (3) stu-
dents should be required to make a specific choice, and (4) groups should
simultaneously report their choices (Figure 1.3). Furthermore, these pro-
cedures apply to all three stages in which students interface with course
concepts—individual work prior to group discussions, discussions within
groups, and whole-class discussion between groups. The 4 S’s are explained
in the following paragraphs.

Figure 1.3. Keys to Creating Effective Group Assignments

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 22 of 156

 THE ESSENTIAL ELEMENTS OF TEAM-BASED LEARNING 21

• Significant problem. Effective assignments must capture students’ inter-

est. Unless assignments are built around what they see as a relevant issue,
most students will view what they are being asked to do as busywork and
will put forth the minimum effort required to get a satisfactory grade. The
key to identifying what will be significant to students is using backward
design. If you identify something you want students to be able to do and
give them the chance to try, it is likely that your enthusiasm will carry
over to your students in a way that rarely happens when you organize
your teaching around what you think students should know.
• Same problem. Group assignments are effective only to the extent that they
promote discussion both within and between groups. Assigning students
to work on different problems practically eliminates meaningful discus-
sions because students have little energy to engage in a comparison of
apples and oranges, and students will not be exposed to feedback on the
quality of their thinking as either individuals or teams. In order to facili-
tate a conceptually rich and energetic exchange, students must have a
common frame of reference that is possible only when they are working
on the same problem, that is, the same assignment or learning activity.
• Specific choice. Cognitive research shows that learning is greatly enhanced
when students are required to engage in higher-level thinking (Mayer,
2002; Pintrich, 2002; Scandura, 1983). In order to challenge students to
process information at higher levels of cognitive complexity, an educa-
tional adage (sometimes attributed to William Sparke) is that teaching
consists of causing people to go into situations from which they cannot
escape except by thinking.

In general, the best activity to accomplish this goal is to require students

to make a specific choice. Think of the task of a courtroom jury: members are
given complex information and asked to produce a simple decision: guilty or
not guilty. As a result, nearly one hundred percent of their time and effort is
spent digging into the details of their content. In the classroom, the best way
to promote content-related discussion is to use assignments that require
groups to use course concepts to make decisions on questions such as these:

• Which line on this tax form would pose the greatest financial risk due to
an IRS audit? Why?
• Given a set of real data, which of the following advertising claims is least
(or most) supportable? Why?
• What is the most dangerous aspect of this bridge design? Why?
• Given four short paragraphs, which is the best (or worst) example of an
enthymeme? Why?

For a much more thorough discussion of assignments and a rationale

as to why they work so well in promoting both student learning and team
development, see Michaelsen, Knight, and Fink, 2004).
NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 23 of 156

22 TEAM-BASED LEARNING: SMALL-GROUP LEARNING’S NEXT BIG STEP

• Simultaneous reports. Once groups have made their choices, they can share
the result of their thinking with the rest of the class sequentially or simul-
taneously. The problem with sequential reporting is that the initial response
often has a powerful impact on the subsequent discussion because later-
reporting teams tend to change their answer in response to what seems to
be an emerging majority view—even if that majority is wrong.

This phenomenon, which we call answer drift, limits both learning and
team development for a variety of reasons. One is that it is most likely to
occur when the problems being discussed have the greatest potential for pro-
ducing a meaningful discussion. That is because the more difficult or ambigu-
ous the problem is, the greater the likelihood is that the initial response
would be incomplete or even incorrect, and subsequent groups would be
unsure about the correctness of their answer. Another is that answer drift dis-
courages give-and-take discussions because later responders deliberately
downplay differences between their initial answer and the one that is being
discussed. Finally, sequential reporting limits accountability because the only
group that is truly accountable is the one that opens the discussion.
Requiring groups to simultaneously reveal their answers virtually elimi-
nates the main problems that result from sequential reporting. Consider the
question in a tax accounting course on an assignment requiring teams to
choose a specific line on a tax form that would pose the greatest financial risk
due to an IRS audit. One option would be for the instructor to signal the teams
to simultaneously hold up a card with the line number corresponding to their
choice (others simultaneous report options are discussed in Sweet, Wright, and
Michaelsen, 2008). Requiring a simultaneous public commitment to a specific
choice increases both learning and team development because each team is
accountable for its choice and motivated to defend its position. Moreover, the
more difficult the problem, the greater the potential is for disagreements that
are likely to prompt give-and-take discussion, and the teams become more
cohesive as they pull together in an attempt to defend their position.
Near the End of the Course. Although TBL provides students with
multiple opportunities for learning along the way, instructors can solidify
and extend student understanding of both course content and group process
issues by reminding students to reflect on what the TBL experience has
taught them about course concepts, the value of teams, the kinds of inter-
action that promote effective teamwork, themselves, and how certain
aspects of the course have encouraged positive group norms.
Reinforcing Content Learning. One of the greatest benefits of using TBL
is also a potential danger. Since so little class time is aimed at providing stu-
dents with their initial exposure to course concepts, many fail to realize how
much they have learned. In part, this seems to result from the fact that with
TBL, the volume of their lecture notes is far less than in typical courses. As
a result, some students are a bit uneasy—even if they are aware that the

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 24 of 156

 THE ESSENTIAL ELEMENTS OF TEAM-BASED LEARNING 23

scores from TBL sections on common midterm exams were significantly

higher than scores from non-TBL sections. As a result, on an ongoing
basis—and especially near the end of the course—instructors should make
explicit connections between end-of-course exams and the RAT questions
and application assignments. In addition, an effective way to reassure stu-
dents is devoting a class period to a concept review. In its simplest form, this
involves (1) giving students an extensive list of the key concepts from the
course, (2) asking them to individually identify any concepts that they do
not recognize, (3) compare their conclusions in the teams, and (4) review
any concepts that teams identify as needing additional attention.
Learning About the Value of Teams. Concerns about better students being
burdened by less motivated or less able peers are commonplace with other
group-based instructional approaches. TBL, however, enables instructors to
provide students with compelling empirical evidence of the value of teams for
tackling difficult intellectual challenges. For example, in taking both individ-
ual and team tests, students generally have the impression that the teams are
outperforming their own best member, but are seldom aware of either the mag-
nitude or the pervasiveness of the effect. Near the end of each term, we create
a transparency that shows cumulative scores from the tests for each team—
the low, average, and high member score; the team score; and the difference
between the highest member score and the team score (see Michaelsen,
Knight, and Fink, 2004). Most students are stunned when they see the pattern
of scores for the entire class. In the past twenty years, over 99.9 percent of the
nearly sixteen hundred teams in our classes have outperformed their own best
member by an average of nearly 11 percent. In fact, in the majority of classes,
the lowest team score in the class is higher than the single best individual score
in the entire class (Michaelsen, Watson, and Black, 1989).
Recognizing Effective Team Interaction. Over time, teams get increas-
ingly better at ferreting out and using members’ intellectual resources in
making decisions (Watson, Michaelsen, and Sharp, 1991). However, unless
instructors use an activity that prompts members to explicitly think about
group process issues, they are likely to miss an important teaching oppor-
tunity. This is because most students, although pleased about the results,
generally fail to recognize the changes in members’ behavior that have made
the improvements possible.
We have used two approaches for increasing students’ awareness of the
relationship between group processes and group effectiveness. The aim of
both approaches is to have students reflect on how and why members’ inter-
action patterns have changed as their team became more cohesive. One
approach is an assignment that requires students to individually reflect on
how the interactions among team members have changed over time and for-
mulate a list of members’ actions that made a difference, share their lists
with team members, and create a written analysis that summarizes the bar-
riers to their team’s effectiveness and what was done to overcome them. The

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 25 of 156

24 TEAM-BASED LEARNING: SMALL-GROUP LEARNING’S NEXT BIG STEP

other, and more effective, approach is the same assignment, but students
prepare along the way by keeping an ongoing log of observations about how
their team has functioned (see Hernandez, 2002).
Learning About Themselves: The Critical Role of Peer Evaluations. One
of the most important contributions of TBL is that it creates conditions that
can enable students to learn a great deal about the way they interact with
others. In large measure, this occurs because of the extensive and intensive
interaction within the teams. Over time, members get to know each other’s
strengths and weaknesses. This makes them better at teaching each other
because they can make increasingly accurate assumptions about what a
given teammate finds difficult and how best to explain it to that person. In
addition, in the vast majority of teams, members develop such strong inter-
personal relationships that they feel morally obligated to provide honest
feedback to each other to an extent that rarely occurs in other group-based
instructional approaches (see Chapter Two, this volume, for examples).
Encouraging the Development of Positive Team Norms. Learning
teams will be successful only to the extent that individual members prepare
for and attend class. We have learned, however, that when we provide stu-
dents with ongoing feedback on attendance and individual test scores, the
link between preclass preparation and class attendance team performance is
so obvious that we can count on norms promoting preclass preparation and
attendance pretty much developing on their own. One simple yet effective
way to provide such feedback to students is the use of team folders. The fold-
ers should contain an ongoing record of each member’s attendance, along
with the individual and team scores on tests and other assignments
(Michaelsen, Knight, and Fink, 2004). The act of recording the scores and
attendance data in the team folders is particularly helpful because it ensures
that every team member knows how every other team member is doing. Fur-
thermore, promoting public awareness of the team scores fosters norms
favoring individual preparation and regular attendance because doing so
invariably focuses attention on the fact that there is always a positive relation-
ship between individual preparation and attendance and team performance.
Benefits of Team-Based Learning. In part because of its versatility in
dealing with the problems associated with the multiple teaching venues
in higher education, TBL produces a wide variety of benefits for students,
educational administrators, and individual faculty members who are en-
gaged in the instruction process.
Benefits for Students. In addition to ensuring that students master the
basic course content, TBL enables a number of outcomes that are virtu-
ally impossible in a lecture-based course format and rarely achieved with
any other small group–based instructional approach. When TBL is well
implemented, students can progress considerably beyond simply acquir-
ing factual knowledge and achieve a depth of understanding that can
come only through solving a series of problems that are too complex for

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 26 of 156

 THE ESSENTIAL ELEMENTS OF TEAM-BASED LEARNING 25

even the best students to complete through their individual effort. In addi-
tion, virtually every student develops a deep and abiding appreciation of
the value of teams for solving difficult and complex problems. They can
gain profound insights into their strengths and weaknesses as learners and
as team members.
Compared to a traditional curriculum, faculty members in a wide vari-
ety of contexts have observed that introducing TBL enables at-risk students
to successfully complete and stay on track in their course work, probably
because of the increased social support or peer tutoring.
Benefits from an Administrative Perspective. Many of the benefits for
administrators are related to the social impact of the fact that the vast major-
ity of groups develop into effective learning teams. When team-based learn-
ing is well implemented:

• Almost without exception, groups develop into effective self-managed

learning teams. As a result, faculty and other professional staff time used
for training facilitators and involved in team facilitation is minimal.
• TBL is cost-effective since it can be successfully employed in large classes
and across academic programs.
• The kinds of assignments characteristic of TBL reduce the potential for
interpersonal hostilities within teams to develop to a point where admin-
istrators must deal with the personal, political, and possibly even legal
aftermath.

Benefits for Faculty. There is tremendous benefit to faculty who use

TBL. Because of the student apathy that seems to be an increasingly com-
mon response to traditional lecture-based instruction, even the most dedi-
cated faculty tend to burn out. By contrast, TBL prompts most students to
engage in the learning process with a level of energy and enthusiasm that
transforms classrooms into places of excitement that are rewarding for both
them and the instructor. When team-based learning is well implemented:

• Instructors seldom have to worry about students not being in class or fail-
ing to prepare for the work that he or she has planned.
• When students are truly prepared for class, interacting with them is much
more like working with colleagues than with the empty vessels who tend
to show up in lecture–based courses.
• Because instructors spend much more time listening and observing than
making formal presentations, they develop many more personally reward-
ing relationships with their students.

When the instructor adopts the view that the education process is
about learning, not about teaching, instructors and students tend to become
true partners in the education process.

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 27 of 156

26 TEAM-BASED LEARNING: SMALL-GROUP LEARNING’S NEXT BIG STEP

References
Birmingham, C., and McCord, M. “Group Process Research: Implications for Using
Learning Groups.” In L. K. Michaelsen, A. B. Knight, and L. D. Fink (eds.), Team-Based
Learning: A Transformative Use of Small Groups in College Teaching. Sterling, Va.: Sty-
lus, 2004.
Brobeck, F. C., and others. “The Dissemination of Critical, Unshared Information in
Decision-Making Groups: The Effects of Pre-Discussion Dissent.” European Journal of
Social Psychology, 2002, 32, 35–56.
Bruning, R. H., Schraw, G. J., and Ronning, R. R. Cognitive Psychology and Instruction.
(2nd ed.) Upper Saddle River, N.J.: Prentice Hall, 1994.
Chan, C., Burtis, J., and Bereiter, C. “Knowledge Building as a Mediator of Conflict in
Conceptual Change.” Cognition and Instruction, 1997, 15(1), 1–40.
Fiechtner, S. B., and Davis, E. A. “Why Some Groups Fail: A Survey of Students’ Expe-
riences with Learning Groups.” Organizational Behavior Teaching Review, 1985,
9(4), 58–71.
Hattie, J., and Timperley, H. “The Power of Feedback.” Review of Educational Research,
2007, 77(1), 81–112.
Hernandez, S. A. “Team-Based Learning in a Marketing Principles Course: Cooperative
Structures That Facilitate Active Learning and Higher Level Thinking.” Journal of Mar-
keting Education, 2002, 24(1), 45–75.
Johnson, D. W., Johnson, R. T., and Smith, K. “The State of Cooperative Learning in
Postsecondary and Professional Settings.” Educational Psychology Review, 2007,
19(1), 15–29.
Knight, A. B. “Team-Based Learning: A Strategy for Transforming the Quality of Teach-
ing and Learning.” In Michaelsen, L. K., Knight, A. B., and Fink, L. D. (eds.), Team-
Based Learning: A Transformative Use of Small Groups in College Teaching. Sterling, Va.:
Stylus, 2004.
Kulik, J. A., and Kulik, C. C. “Timing of Feedback and Verbal Learning.” Review of Edu-
cational Research, 1988, 58(1), 79–97.
Lerner, J. S., and Tetlock, P. E. “Accounting for the Effects of Accountability.” Psycho-
logical Bulletin, 1999, 125(2), 255–275.
Mayer, R. E. “Rote Versus Meaningful Learning.” Theory into Practice, 2002, 41(4),
226–232.
McGrath, J. E. “Time, Interaction, and Performance (TIP): A Theory of Groups.” Small
Group Research, 1991, 22(2), 147–174.
Michaelsen, L. K., and Black, R. H. “Building Learning Teams: The Key to Harnessing
the Power of Small Groups in Higher Education.” In S. Kadel and J. Keehner (eds.),
Collaborative Learning: A Sourcebook for Higher Education. State College, Pa.: National
Center for Teaching, Learning and Assessment, 1994.
Michaelsen, L. K., Cragin, J. P., and Watson, W. E. “Grading and Anxiety: A Strategy for
Coping.” Exchange: The Organizational Behavior Teaching Journal, 1981, 6(1), 8–14.
Michaelsen, L. K., Knight, A. B., and Fink, L. D. Team-Based Learning: A Transformative
Use of Small Groups in College Teaching. Sterling, Va.: Stylus, 2004.
Michaelsen, L. K., and McCord, M. “Teaching Business by Doing Business: An Interdis-
ciplinary Faculty-Friendly Approach.” In D. Robertson and L. Nilson (eds.), To
Improve the Academy: Resources for Faculty, Instructional and Organizational Develop-
ment. Stillwater, Okla.: New Forums Press, 2006.
Michaelsen, L. K., and Schultheiss, E. E. “Making Feedback Helpful.” Organizational
Behavior Teaching Review, 1988, 13(1), 109–113.
Michaelsen, L. K., Watson, W. E., and Black, R. H. “A Realistic Test of Individual Ver-
sus Group Consensus Decision Making.” Journal of Applied Psychology, 1989, 74(5),
834–839.

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 28 of 156

 THE ESSENTIAL ELEMENTS OF TEAM-BASED LEARNING 27

Miller, J. A. “Experiencing Management: A Comprehensive ‘Hands-On’ Model for the

Introductory Management Course.” Journal of Management Education, 1991, 15(2),
151–173.
Millis, B. J., and Cottell, P. G. Cooperative Learning for Higher Education Faculty. Phoenix,
Ariz.: Oryx Press, 1998.
Pintrich, P. R. “The Role of Metacognitive Knowledge in Learning, Teaching, and Assess-
ing.” Theory into Practice, 2002, 41(4), 219–225.
Scandura, J. M. “Instructional Strategies Based on the Structural Learning Theory.” In
C. M. Reigeluth (ed.), Instructional Design Theories and Models. Hillsdale, NJ: Lawrence
Erlbaum Associates, 1983.
Sweet, M. “Forming Fair Teams Quickly.” In Michaelsen, L., McMahen, K., Levin, R.,
and Parmalee, D. (eds.) Team-Based Learning in Health Professions Education. Sterling,
VA: Stylus, 2008.
Sweet, M., Wright, C., and Michaelsen, L. K. “Simultaneous Report: A Reliable Method
to Stimulate Class Discussion.” Decision Sciences Journal of Innovative Education, 2008,
6(2), 469–473.
Tuckman, B. W. “Developmental Sequences in Small Groups.” Psychological Bulletin,
1965, 63, 384–399.
Tuckman, B. W., and Jensen, M.A.C. “Stages in Small Group Development Revisited.”
Group and Organizational Studies, 1977, 2, 419–427.
Watson, W. E., Kumar, K., and Michaelsen, L. K. “Cultural Diversity’s Impact on Group
Process and Performance: Comparing Culturally Homogeneous and Culturally
Diverse Task Groups.” Academy of Management Journal, 1993, 36(3), 590–602.
Watson, W. E., Michaelsen, L. K., and Sharp, W. “Member Competence, Group Inter-
action and Group Decision-Making: A Longitudinal Study.” Journal of Applied Psychol-
ogy, 1991, 76, 801–809.
Whitehead, A. The Aims of Education. Cambridge: Cambridge University Press, 1929.
Wiggins, G., and McTighe, J. H. Understanding by Design. Columbus, Ohio: Merrill Pren-
tice Hall, 1998.
Worchel, S., Wood, W., and Simpson, J. A. (eds.). Group Process and Productivity. Thou-
sand Oaks, Calif.: Sage, 1992.

LARRY K. MICHAELSEN is professor of management at Central Missouri, David

Ross Boyd Professor Emeritus at the University of Oklahoma, a Carnegie
Scholar, and former editor of the Journal of Management Education.

MICHAEL SWEET is a faculty development specialist for the Division of Instruc-

tional Innovation and Assessment at the University of Texas at Austin.

NEW DIRECTIONS FOR TEACHING AND LEARNING • DOI: 10.1002/tl

Team-Based Learning Institute Page 29 of 156

Team-Based Learning Institute Page 30 of 156
 A Compact Introduction to

Team-Based Learning
William D. Roberson and Larry K. Michaelsen (2016)

Welcome to Team-Based Learning

Various paths lead university instructors to Team-Based Learning, but we all have
in common two key motivators: 1) the desire to create deep, engaging learning
experiences that promote student independence as thinkers; and 2) the desire to
make our classrooms places of energy and enjoyment in learning, for our
students as well as for ourselves.

Larry Michaelsen originally developed a prototype of what was called “Team

Learning” as early as the 1970’s in an effort to ensure the benefits of small-group
learning in the face of rapidly expanding class size. Bill Roberson, after years of
struggle with inconsistent results from student “group work,” discovered Team-
Based Learning through Michaelsen’s early workshops and publications, and
finally found in them the means to ensure a positive group dynamic. He has been
an avid TBL practitioner ever since.

We have both worked with faculty colleagues who struggled because their
teaching failed to engage students, but then suddenly found the tools they
needed in the structures and processes of TBL. We also have worked with
colleagues who had come to realize, after years of teaching in traditional ways,
that their classrooms were having little impact on student learning. When they
turned to TBL out of frustration, they discovered in the process that their students
were, contrary to outward appearances, highly motivated and intelligent
individuals who were hungry for deep learning.

Team-Based Learning is a teaching strategy for creating classrooms that

foster student independence and enthusiasm for learning. This short
introduction to TBL is a descriptive summary of the essential perspectives, tools
and practices you’ll need for a first-time successful implementation of the TBL
model in your courses. This is not the definitive text on TBL, as it does not
provide many examples of specific TBL techniques. We encourage you to use this
text as a short-cut reference as you develop your course and lesson plans. For
more in-depth information we recommend four publications that provide more
fully developed explanations and examples:
 Michaelsen, Knight, and Fink (2004, Stylus), Team-Based Learning: A
transformative use of small groups in college teaching

Team-Based Learning Institute Page 31 of 156

  Michaelsen, Parmelee, McMahon and Levine (2008, Stylus), Team-Based
Learning for Health Professions Education
 Sweet and Michaelsen, eds. (2012, Stylus), Team-Based Learning in the
Social Sciences and Humanities
 Sibley, Ostafichuk, Roberson and Franchini (2014, Stylus), Getting Started
with Team-Based Learning.

Additional resources can be found at VIU’s web pages dedicated to TBL, on the
website of the Team-Based Learning Collaborative (TBLC),
[Link] and on Jim Sibley’s UBC website at
[Link]

What does a TBL classroom look like?

Here’s a class meeting in Physical Therapy that we recently watched in action.

Sylvia Mitchell enters her classroom on Thursday, just before 9AM. She is pleased to see that
all 30 of her students have already gathered and are chatting amongst themselves in their
permanent teams (there are 5 of them). She cheerfully makes small talk with students for a
few seconds, then starts the class meeting.

“This past Tuesday you took the individual and team RAT (Readiness Assessment Test) on the
basic ideas behind creating and choosing treatment plans for clients. Today we’re going to
look more closely at how those ideas work in reality. Please read the one-page patient data
sheet in your folders. Your job is to analyze the situation and condition of this client, and
make a determination about how you would respond with treatment, and why.”

Students read quietly for about 2 minutes. After students show signs of finishing, she places a
slide on the screen and says,

“Here are 4 possible treatment strategies. In your teams I’d like for you to compare and rank
them, from most justifiable to least justifiable in this case, given the client data. Be ready to
explain why, based on your reading from this past week. Feel free to refer back to your
textbooks if you need further clarification of any of the concepts we discussed last time.
Write down your team’s ranking of these strategies on a piece of paper. You have 10
minutes. Go.”

Suddenly the room is loud with a number of voices talking at once. Sylvia wanders about the
room listening to the team conversations, but not saying anything to the students. As the 10
minutes come to a close, she glances at the sheets where students are beginning to indicate
their ranking of the 4 plans. Most of the 5 teams are close to finished. To the whole room she
now says,

“OK, stop. Please take the blue card from your folder and write down the letter of the
treatment plan your team selected as most justifiable.” The teams finalize their decision

Team-Based Learning Institute Page 32 of 156

and write down the number corresponding to their choice. “On the count of 3, please hold
up your team’s response. One, Two, Three…Show!” 5 cards go up all at once. 3 of them show
the same answer, “B.” One team answers “C:” and another team answers “D.” Sylvia begins
the debriefing process by saying, “Interesting spread. Nobody chose A. There seems to be
some consensus on that. Mary, what’s the reason your team did not put A at the top of your
list?

Mary gives an explanation for how treatment plan A ignored a dimension of the case that,
according to their readings, is important. Sylvia invites another team to comment (“Did you
think the same thing?”), then moves to discussion of the other options: “Let’s hear from the
team that said “C.” Why did you guys conclude that this is a condition that should be treated
chemically…?” And so it goes for 20 more minutes, as Silvia asks the teams to respond to
each other’s reasoning. Through the exchange, Silvia questions (Why? What’s your evidence
for that? What’s your reasoning? How would you respond to that team’s argument?) and lets
students argue for their analysis and evaluation of the various plans. She facilitates so that
students challenge each other’s interpretations of key concepts or contradictions in
reasoning. She is careful not to divulge her own preference during this exchange.

More than 30 minutes pass in lively analysis and debate, and Silvia now moves toward
closure. “I’d like to draw your attention to several important things you said. First of all,
kudos to the groups who picked up on the implications of the blood test data. That led you to
Plan B, and you made a good case for it. Several of you focused on the patient’s present
physical condition. That’s important to consider, but I’d also suggest you look at the age and
history of the patient. That can tell us more than just a snapshot of the present. That longer
view might have led us to discount Plan C, and to a lesser extent Plan D. However, several of
you saying C and D brought up points worth considering, such as…”

After the summary, Sylvia introduces a new situation: “Let’s look at another case, this one a
bit more complicated…

What is Team-Based Learning?

In the TBL classroom described above, Sylvia has created conditions where her
students can apply and test their preliminary understanding of course content to
practice their thinking and gain deeper understanding of the subject matter. She
has built a 3-day learning sequence—1) reading, 2) assessment of reading, 3)
structured group analysis and decision, 4) whole-class discussion, 5) instructor
feedback—around getting students to ACT decisively and concretely on their
own in the kind of situational complexity they might face as experts in their field
or professional decision-makers.

Our example was taken from a course in Physical Therapy, but the same
principles apply to any discipline. For example, if we had shown a classroom from
a History course we would follow the same pattern:

Team-Based Learning Institute Page 33 of 156

 1. Students read to acquire preliminary factual knowledge of events, people,
and historical circumstances, as well as knowledge of historical principles,
theories and ways to conduct historical inquiry.
2. Students take a short test to verify their basic understanding of what they
have read.
3. Students working in teams are presented with a situation that requires
them to compare facts, claims, arguments, or artifacts and make a specific,
choice/decision about them using knowledge and informed judgements.
In a history course students might typically compare interpretations of a
document; or competing, contradictory historical accounts; or theoretical
arguments in support of various interpretations. In each case the groups
work to analyze the given circumstances of the challenge, then make a
claim in the form of a decision among options or within parameters
provided.
4. The groups’ claims are compared through whole class discussion.
5. The instructor facilitates the discussion and provides feedback at the end
of the discussion.

Through the use of carefully designed application activities (#3 in our example
from History) students are provided context for their learning, and are asked to
put concretely into use what they have learned abstractly from the readings.
Connecting abstract concepts from the readings to specific decisions and choices
during the team application activities is critical for consolidating student learning
and deepening their understanding. Our job as instructor is to find or create
these situations, cases and scenarios where what students “know” abstractly (via
their readings) is put to the test when they try to “use” it to address a specific
challenge.

Learning to use knowledge to inform and make significant, specific

choices/decisions is the central learning outcome of a Team-Based
Learning course.

The social framework of “teams” facilitates the outcome of improved decision-

making. The special TBL format of team application activities (see “4-S” below)
ensures that students are fully motivated and receive “immediate feedback” to
improve their thinking when confronted with specific choices.

The underlying principle of Team-Based Learning may be counter-intuitive at first

to some instructors. In more traditional classroom settings it’s a common
assumption that if we can get students to adopt productive behaviours (read,
prepare, take notes, attend class, pay attention, be quiet, etc.), then we can
improve their learning. We may even go so far as to impose rules, policies and

Team-Based Learning Institute Page 34 of 156

penalties whereby we try to coax or coerce these behaviours. When we focus on
“behaviours” rather than “outcomes” however, we can suddenly find ourselves on
a slippery slope, and land in the role of a policeman or parent trying to control
students.

Team-Based Learning classrooms focus, instead, on goals (outcomes) and

performance. The reasoning is simple: if we make the expected outcome clear,
indicate what a successful performance looks like, design effective, relevant
learning activities for practice, and provide tools for feedback and self-
assessment, students will figure out for themselves which behaviours are effective
or not. Once students take responsibility for their own performance, they will
usually abandon behaviours and attitudes that interfere with or undermine
success. It is common to see first-time TBL instructors look on with amazement
when students adopt, without being coached or coerced, the habits of careful
preparation, regular attendance, and open, respectful collaboration with peers.

What makes TBL work?

Few, if any, of the individual elements of the TBL method are unique to TBL:
homework reading, reading comprehension tests, small group work, class
discussion, peer feedback, etc. Most university instructors, however they teach,
use at least some of these techniques at any given moment. The special
learning dynamic that TBL produces, however, comes from the systematic,
way these practices function together, following a specific sequence and
leveraging a few key design principles.

Consider an analogy with aviation. An airplane pilot follows a specialized

protocol in order to land her airplane safely every time. This protocol includes
multiple steps, often in a specific, pre-determined sequence, each one
contributing to the stable, safe descent of the plane, in consideration of weight,
elevation, wind direction, speed, other approaching planes, messages from
ground control, etc. Following this explicit protocol—with each step supporting
specific dimensions of safety and stability—allows a pilot to accomplish the goal
of landing a plane securely in a wide range of dynamic or even hazardous
conditions.

Similarly, Team-Based Learning provides instructors with reliable protocols for

engaging students and promoting their learning. When understood and
respected, the protocols help any instructor—even a brand new one—
consistently create rich, engaging student experiences, in a wide range of
classroom situations, in any discipline.

Team-Based Learning Institute Page 35 of 156

CAUTION to NEW ADOPTERS: Improvising major changes to the protocols,
without anticipating the consequences downstream, can lead to disappointing
outcomes.

Here is an example: in a TBL course students read and take a test on what they
read before the instructor fully explains it. This practice is called “Readiness
Assurance Process” (RAP). Instructors new to TBL may feel uncomfortable with
this practice, and may feel obligated (in response to student complaints or pleas
for mercy) to give a “helping” lecture or provide another type of crutch before
the test.

This seemingly student-friendly change of protocol creates problems and will

have negative repercussions later on, in surprising ways. Here’s why: the
Readiness Assurance Process is not designed solely to check whether students
understood the assigned reading. The Readiness Assurance Process is also the
first step in socializing the teams—which occurs only through a certain amount of
student discomfort and struggle.

The Individual Readiness Assessment Test (iRAT) comes at the beginning of each
TBL learning sequence and occurs without handholding. It challenges students to
begin taking personal responsibility for their own learning behaviour. Also,
because it is designed to be a slightly difficult, it creates a small amount of
“productive frustration” so as to activate the perception that having peers to help
might be desirable.

The Team Readiness Assessment Test (tRAT—which is the same test taken again,
as a team) has a different function: it replaces the instructor’s handholding.
Students struggle together, learn to coach one another, give mutual feedback,
and provide the emotional support needed to prevail through the necessary
struggle.

The teams begin to bond when faced with real challenges that are beyond the
ability of any single individual. When the team performance beats any individual
performance (which is almost always the case), the team effort is validated and
team cohesion grows. Over the course of several RAP cycles, students begin to
experience their personal responsibility toward their teammates, and increasingly
realize that they need to come to class prepared in support of the team.

To non-TBL instructors the RATs might look like any other arbitrary use of a stick
to get students to read. In reality, the RAT allows and helps students to enter into
their teams as equals with confidence and mutual respect, a fundamental
condition for high-performing teams.

Team-Based Learning Institute Page 36 of 156

When teams fail to gel by mid-term, the reason can often be tracked back to a
less-than-rigorous implementation of the Readiness Assurance Process. When
students feel that other team members are not prepared and are not pulling their
weight—what we commonly call “social loafing” or “hitchhiking”—resentment
builds and undermines the team’s cohesiveness. In non-TBL uses of student
collaboration, this is the number one fatal error: failing to put in place
mechanisms to hold individuals accountable for their individual preparation, so
that they are able to grow into credible, responsible team members.

In short, a TBL instructor needs to stay fully aware of how the various elements of
the method are connected and interdependent. Even though it may feel
constricting, we recommend that instructors new to TBL adhere to the
protocols as closely as possible throughout the first implementation, before
improvising changes. Doing so will help you “land the plane” reliably, and
navigate toward a classroom that fully promotes student buy-in, self-sufficiency
and high-impact learning.

Later, after a first implementation, it should become clearer where creative

variations can be introduced without diminishing the targeted outcomes.

Team-Based Learning Institute Page 37 of 156

The 5 Pillars of Team-Based Learning
TBL is a whole course design and management strategy, not a set of techniques
to be sprinkled here and there. As a method, it is not complicated, but it requires
disciplined commitment to 5 “pillars” of practice, each being essential to
creating and reinforcing the conditions needed for learning at a high level:

A. Team formation by the instructor, to ensure the perception of fairness

among teams and diversity within teams
B. Readiness assurance to motivate individual preparation and promote
student competence for working with peers
C. Team assignments designed as student-owned decision-making
“applications” of content, both to promote deep learning and to ensure
high levels of student interest
D. Immediate feedback designed to be the natural consequence of the
activities, to stimulate engagement, provoke reflection and accelerate
learning
E. Student agency, self-determination and accountability, to promote a
fully adult culture of learning

This list is intended to summarize for new adopters the rationale behind specific
TBL practices. Each of the pillars is elaborated below, following the description of
the primary TBL Learning Protocol.

Team-Based Learning Institute Page 38 of 156

The TBL Learning Protocol
Immediately after Day One of a TBL course, once the permanent teams are
formed and students have been introduced to the TBL Model, it’s time to launch
the first learning sequence, (also called a “module” or “cycle,”) which is
determined by a specific TBL Protocol. This protocol will be repeated four to
seven times over the course of one typical, 12-14 week university term. On
average, modules extend over 2 weeks each. In most cases, the 4-7 modules
constitute the totality of a TBL course. Compressed courses might require fewer
modules, or a shorter timeline for each module.

Step One: Students Read (outside of class, before the in-class part of the module
begins)

Step Two: Readiness Assurance Process (RAP) (45-90 minutes)

a) Students take the individual Readiness Assessment Test (iRAT) (closed
book, in class)
b) Student teams take the team Readiness Assessment Test (tRAT) (closed
book, in class)
c) Students make appeals on unfair, erroneous or ambiguous questions
(open book, in class)
d) Instructor addresses student questions, lingering confusion and
uncertainties (in class)

Step Three: Application Activities and Assignments (2-4 class meetings)

 Student teams apply content to make analytical decisions (open book, in
class)
 Students continue to read, problem-solve and practice individually using
content (homework outside class)

Step Four: Assessment of Learning (not necessary for all modules)

 Assessment can include individual or team assignments, or a combination
 Traditional instruments (tests, papers, projects, etc) can be used for
individual assessment
 Specially formulated “capstone” case or scenario analysis can be used if
some assessment of learning by team is desired

Team-Based Learning Institute Page 39 of 156

Best Practices for the 5 Pillars of TBL

Pillar A: Team formation

Teams are the focal point for most classroom activities in a TBL course. They are
the social motor for learning. Ensuring that teams perform well is not a trivial
concern, but the key is in the set-up, not the continual coaching. Teams do not
need special training in collaboration or special team-building exercises when the
conditions are properly set by the instructor. Here are the key “socializing”
practices for every TBL instructor.

1. Teams are permanent over the whole term. Learning to function as a

team takes time. For some teams, it may take weeks for members to learn
to work together.
2. Teams need to be formed immediately at the outset of the course
(On Day One, if at all possible) to communicate clearly the new
expectations. You’ll need a plan for determining how late-registering
students will join existing teams.
3. Team membership needs to be assigned, not self-selected, to
establish a sense of fairness and to weaken existing alliances. It’s best if
the team formation process can happen in public. Transparency
contributes to the trust and confidence needed for rapid team cohesion.
4. Teams need to be large (optimally 5-7 members) in order to have
diversity of perspective and depth of resources.
5. Team formation should ensure even distribution of student assets
and liabilities relevant to the course outcomes. For example, in a
course in environmental science, you might want to distribute students
with a strong foundational background in chemistry or biology. See the
protocol for team formation, below, for ideas on how to accomplish this.
6. Optional Practice: Teams benefit from giving their team a name, to
reinforce their identity. Using this name regularly in classroom
conversations can further help with the process of bonding.
7. Teams need to be assigned an engaging, decision-based team
activity as soon as they have met for the first time. There are several
options. Setting grade weights for the course, determining criteria for
“helping behaviour,” doing a mini-RAP based on the course syllabus, or a
content-related team activity (See 4-S activity design, below) are all
potential candidates for a first-day activity that gives students their initial
taste of the TBL classroom.

Pillar B: Readiness Assurance

The protocol for the Readiness Assurance Process (RAP) includes 4 steps, which
are most powerful when they occur face-to-face and in immediate succession.

Team-Based Learning Institute Page 40 of 156

Each element of the RAP has at least one critical objective, and therefore should
not be omitted:
i. The iRAT (Individual Readiness Assessment Test) is a short, closed-book,
multiple-choice test that holds students individually accountable for their
own efforts to understand course content. The iRAT is the basis for effective
team member behaviour.

ii. The tRAT (Team Readiness Assessment Test) is a repeat of the iRAT, also
closed-book, written immediately after the iRAT, but taken as a team. It
fosters team skills by requiring negotiation, and furthers team development
by showing students the value of their teams, who almost always outscore
any individual.

“Immediate feedback” on team answers is critical for team development.

Immediate feedback can be provided by projecting the answer key on a
screen at the end of the tRAT. Alternatively, if available, the use of scratch-off
(IF-AT™) answer sheets (purchased through Epstein Education) is especially
effective. Visit [Link] to see a wide variety of “lottery-
style” answer sheets for different formats of multiple choice questions.

iii. The Appeals process follows immediately the tRAT, and invites students to
challenge test items that may be flawed, in order to show students that
they own their learning, and need to defend it. Teams must submit their
appeals in writing and provide evidence and sound reasoning for each
appeal.

iv. The instructor’s Clarification, after the submission of appeals, usually takes
the form of a class discussion or short mini-lecture, if needed. This step
allows students to get expert feedback directed to their specific questions
and concerns, but should not turn into an extended lecture. Do NOT
review the whole RAT—discuss only the questions that everyone
missed.

Key practices for the Readiness Assurance Process (RAP)

1. The RAP occurs, without exception, at the very beginning of every

cycle or module of the course. (A TBL course typically has 4-7 cycles or
modules, in contrast to many traditional courses that are organized by 12-
14 separate, weekly units of content).

2. Do not administer a RAP more often than 6-7 times in a typical 14-
week semester. Overuse of RATs will visibly erode student enthusiasm and
motivation for your course.

Team-Based Learning Institute Page 41 of 156

3. The reading amount for one RAP needs to provide enough substantial
material to feed team application activities over one whole module
(approximately 2-4 class meetings or 3-8 contact hours). The actual
amount will vary discipline by discipline and by density of material, so a
reading assignment could be 20-30 (dense, highly technical) pages on the
low end and 100-300+ pages (such as a whole collection of articles, several
textbook chapters, or a whole novel) on the high end. It may be necessary
to show students early in the course how to read strategically, for
broad, contextualized understanding when faced with larger reading
assignments.

4. Readiness Assessment Test (RAT) questions should target

understanding of important concepts, not picky details. RATs should
not be comprehensive exams, but rather a sampling of student
understanding of key ideas and critical differentiations. 10-20 items are
sufficient in most cases. A smaller number of high quality, carefully written,
challenging questions is the best approach.

5. Both the iRAT and tRAT are closed-book, in-class tests. The richness of
team discussion during the tRAT comes in part from the members’
struggle to reconstruct from memory their understanding of what they
read. If students need access to specific technical information from the
readings (formulas, equations, obscure values that should not be
memorized, for example) to support their thinking, these can be provided
with the questions, but be careful that the RAT does not focus on narrow
analyses with calculations or the like, as these types of questions will not
be highly effective in the tRAT discussion. Keep the questions at the level
of conceptual understanding of and differentiation among key concepts.

6. RAT questions are always in a multiple choice or true-false answer

format in order to create the conditions needed for dynamic team
decision-making and immediate feedback during the tRAT. Open-ended
questions do not force the kinds of negotiation and convergent thinking
that teams need in order to develop as teams. Remember that multiple
choice questions can be made highly complex and challenging, by basing
them on situations or mini-cases, rather than on factual recall.

7. Within one RAT, questions should be distributed over levels of

difficulty. See Bloom’s Taxonomy for ideas on writing questions at
different levels. A few items should be easy to build confidence; a few
items need to be hard (complex, ambiguous, nuanced) enough to elicit rich
team discussion.

Team-Based Learning Institute Page 42 of 156

8. RATs should be challenging. It’s better to administer RATs that are
slightly too difficult rather than too easy. Too easy means boredom and
low incentive because there’s nothing at stake—and one really gifted
member can carry the group. Excessively hard means loss of motivation as
students can begin to feel that success is out of reach. Target individual
scores averaging 50-70%, with team score averages targeting 80-95%.

9. RATs should be administered with time limits. A 10-question iRAT can

usually be written in 6-10 minutes; the tRAT discussion for the same test
will generally need 15-20 minutes. These will vary depending on difficulty
of the test and the specific population of students. It’s important to keep
some pressure on students, so aim for the low end, then adjust if needed.
A good rule for keeping students on track: once half the teams have
finished, the remaining teams have just two minutes more before grades
will be calculated and posted.

10. tRAT scores should be published on the board or screen for the whole
class to see, as part of the feedback process. This allows teams to monitor
their own learning and creates a fun, softly competitive atmosphere. In
many cases students will alter their expectations of themselves and
increase their commitment to preparation when they see how other teams
are performing.

11. The Appeals process should be conducted during the same class
meeting in which the RAP occurs. Do not skip the appeals step and do
not make it appear optional or unimportant. Students need to develop the
expectation that they, alone, are responsible for evaluating the quality and
accuracy of the RAT as a measure of their preliminary understanding of the
reading.

Do not ask, “Are there any appeals?” Rather, create the expectation for
appeals. At the end of the tRAT tell students they have 5 minutes to
determine which items they wish to appeal. Then give them another 10
minutes or more to write down and submit the reasons for the appeal. If
some teams elect not to appeal any questions, have a new assignment or
activity ready to keep them productive while the other teams finish.

Team-Based Learning Institute Page 43 of 156

Pillar C: Team Application Activities

Team application activities in the shape of comparative analysis leading to

concrete “decisions” or “specific choices” are the single most important
element of Team-Based Learning.

Decisions not only translate knowledge into actions, they are the mechanisms for
generating student interest, curiosity and engagement.

Because students have fulfilled their part of the bargain and are known to be
prepared (via the Readiness Assurance Process), they need to be challenged in
ways that allow them to see for themselves the usefulness of what they have
studied. Team application activities need to be hard enough, and contain enough
uncertainty or complexity, that the most diligent student cannot simply answer
because he/she knows a lot. Design team application activities around decisions
that require students to use not only their new knowledge, but also their
reasoning and their judgment.

The protocol for developing Team Application Activities is called “4-S”

(originally called “3-S” in Michaelsen’s original writings) .

I. Conceive the task so that it looks Significant and therefore interesting

from the student’s perspective. This means that the task will ask students
to USE their new knowledge (from the readings) actively in responding to
specific, concrete situations. Seeing the immediate utility and relevance of
what they have just read is highly empowering and motivating.

II. Require students to think comparatively and make a Specific choice

among several possible options. This forces students to weigh competing
priorities, values, arguments, interpretations, theories and the relevance of
specific facts, in making their decision. The answer parameters allow the
instructor to anticipate and target the specific terms and concerns of the
discussion.

III. Require all teams to work on the Same task, so that, when they report
their answers to the whole class, they will be able to compare their own
response to those of the other teams—for immediate feedback. In this
comparative framework, students will naturally and genuinely care about
how the other teams responded.

IV. Use report-out techniques allowing Simultaneous responses for all

teams. When all teams report simultaneously, the comparison is dramatic

Team-Based Learning Institute Page 44 of 156

 and the natural outcome is that groups are fully engaged: they want
feedback from and are willing to give feedback to each other. See the
practices described below for examples on how to conduct simultaneous
reporting.

When the teams have made a specific choice in relation to the

same significant problem, and then discover via a simultaneous
report that other teams made a different choice, they are both
motivated and intellectually prepared to challenge other teams’
answers and defend their own. The instructor then facilitates a
comparative, analytical discussion of all answers, where the teams
are asked to defend their thinking and respond to one another.

Proven best practices for Design of Team Application Activities

1. Start with a verb. If you can find the verb that represents a significant
action requiring knowledge of course content (evaluate, assess, diagnose,
predict, contrast, compare, rank, categorize, critique, etc.), you’re on your
way to a good application task. Do not design activities around verbs of
state, such as “understand,” and “know” or low-level tasks such as “identify”,
“find” or “match.” Team application activities need to be framed as concrete
actions in unfamiliar circumstances and new situations, so students can see
for themselves the applicability, portability and impact of their knowledge.

2. Find ideas for team application tasks by looking at what people who
work in your discipline do with their knowledge. Ask, “What kinds of
problems do we try to solve? What kinds of questions do we try to answer?
How do we use our discipline’s information and ideas?” Ask your students
to make the kinds of judgments, interpretations, evaluations, predictions
and other types of decisions that you, yourself, and other professionals or
academics do as the regular part of your work. For example, from our
opening story from Physical Therapy, “Look at the data summary provided
for this client. Assess and rank the various treatment plans according to
their relevance in this case.” From History: “Which of the various theoretical
explanations of this event is the most convincing?”

3. Develop Team Application tasks that…

…are based on responses to cases, scenarios, concrete problems, actual
questions and inquiries. The goal of a team task is to lead students to “test”
and stretch their knowledge by trying to use it in complex, realistic
situations.

Team-Based Learning Institute Page 45 of 156

 …ask for the comparative analysis and assessment of objects, statements,
claims, theories, arguments, representations, images and other products or
tools typical of your field.

4. Team responses, including those applied to complex scenarios and

questions, should be converted to a single, easily visible, focused
representation—letter; number; single word or phrase; image; chart, graph,
bullet list—so they can be easily compared across all teams for immediate
feedback during the Simultaneous Response phase. The most common
technique for simultaneous reporting is to ask teams to reveal their specific
choice responses using colored, numbered, or lettered cards. This can also
be done using blank sheets of paper, where students write down the letter
or number or word of their selection and hold them up at the moment of
reveal. Small, hand-held whiteboards also work for this purpose. Clickers
can be used, but they are less effective than cards, since they do not
communicate immediately and publicly teams’ visible ownership of their
answers.

For responses where students are asked to represent their decisions

graphically (draw an image or chart) it works well to have teams record their
work on poster paper, then simultaneously publish it by attaching it to the
wall at a given signal. In these cases, students can then conduct a “gallery
walk,” in which they roam the room assessing the other team’s answers,
before engaging in a whole-class, comparative discussion of all responses.
For more details on these and other reporting techniques, please see
Chapter Seven of Getting Started with Team-Based Learning.

5. Make sure some team application activities count for points or marks.
Mix application tasks that are “formative” or “developmental” (no points)
with those that are designed to be capstone-like “challenges” or “show us
what you can do” tasks that are scored for points. This ensures
accountability for the team’s work. A good practice is to do a series of non-
scored tasks leading up to the task that is graded.

Team-Based Learning Institute Page 46 of 156

Proven best practices for Management of Team Application Activities

1. Limit the time you allow teams for making their collective decision
during an activity, and if possible, use a visible timer (Power Point can be
adapted to this purpose). Tell the teams that they will need to produce an
answer at the end of the given, announced time limit, whether they have
finished discussing or not.

2. Do not assign teams a sequence of several tasks at the same time (as in
a worksheet or list of questions, for example) as this will kill all the energy of
both the team discussion and the whole-class discussion. Separate team
tasks into clear, single decisions, present them individually, one by one, and
discuss fully before moving to the next. For long sequences of activities that
include some non-4-S activities, consider alternating individual work (e.g.,
worksheets or problem sets) with focused team 4-S decision tasks that
require conceptual, convergent thinking that is built upon the individual
work.

3. Manage the task by projecting instructions, questions or other

prompts on a screen, or by using paper handouts. This keeps you from
having to shout over a loud classroom once discussion is underway.

4. Leave the teams alone while they are working on a task. Move around
so as to be seen, but so as not to be drawn into a conversation. Do not
invite yourself into a team conversation, and deflect questions asking for
special help.

5. If students from one team ask a question during a team activity, push
it back to the whole team to consider, if possible.

6. If you need to clarify an element of an activity, clarify for the whole

class, not for just one team.

7. Make the simultaneous report crisp. (On the count of 3: 1-2-3…SHOW).

This will help students see and benefit from the immediate feedback
provided by other teams’ responses, and will reduce fudging by teams who
are uncommitted.

Proven best practices for Facilitation of Application Activity

Discussions

While 4-S Activity Design ensures student engagement in high levels of thinking,
the actual learning itself is dependent upon effective facilitation by the

Team-Based Learning Institute Page 47 of 156

instructor. It is the facilitation process that frames and fosters the in-depth
analysis, feedback and reflection. The instructor’s role should be that of inquirer,
not director. Teams need to be invited to explain their reasoning and defend it
vis-à-vis the claims and evidence provided by other teams. An effective facilitator
will use questions and “naïve” re-statements of students’ claims, to entice
students to discover for themselves the consequences of their team decisions.
For that purpose, here are a few guidelines for facilitation.

1. Keep a poker face during facilitation. Maintain the appearance that all
responses could be valid or correct until all have been explained by the
teams who represent them. Many an excellent discussion has been
undermined by the instructor tipping her hand as to what she considers a
“correct” or best answer to be, even before all the teams have finished
reporting. It’s good practice to let students go down a wrong path, to fully
expose their reasoning. When it’s finally time for your feedback, point to the
strengths of the various team responses, even while pointing students
towards a “best” response.

2. It does not matter if all teams agree and report the same, “best” or
“correct” answer. The learning occurs during the discussion. Teams may have
different reasons for arriving at the same answer. Your first response to a
simultaneous report, no matter the spread of team answers, is some version
of “Why?” directed at one of the teams.

3. Cluster team answers when you debrief. If three teams answer the same
way, collect reasons from one of these, then ask the others if they have
something to add (Don’t proceed one by one). This avoids tedious, repeated
explanations.

4. Vary your order of collecting team answers. If you always start with the
worst one, students will catch on. Sometimes start with the best one. Starting
with the minority opinion is often a good strategy, as it ensures that
unpopular arguments will be heard.

5. Close the discussion by pointing to what has been learned. Make sure to
indicate any merit in students’ arguments, even if their overall reasoning was
flawed.

6. If a final, correct or best answer needs to be presented, offer it as “this

is what the experts would say,” so you, personally, will not always be
identified as the only source of knowledge and authority.

Team-Based Learning Institute Page 48 of 156

Pillar D: Immediate feedback

Seeing for ourselves the consequences and impact of our own actions is the most
powerful teacher that exists. This is the psychology that informs game design.
Games, like TBL classrooms, are learning systems, where each action by a player
(or team of players) generates consequences that provide the feedback that
teaches. A hockey player shoots at the goal and watches to see if the goalie can
stop the puck. A poker player makes a bet and watches to see how the other
players respond. A video gamer watches how opponents on the screen respond
to moves, then alters his strategy or tries a different tool. In any game, a player
watches and responds to the effects of his actions—immediate feedback—then
takes what he has learned into consideration when planning future moves.

TBL protocols and practices are specifically designed to create a

classroom experience rich with “immediate feedback.”

When the immediate feedback to a team is positive (“We got it right!” “We got
more points than the other teams!”) it validates team decisions that are sound,
and therefore helps the team bond through greater confidence and a stronger
sense of identity. When the feedback is negative (for example, when the team
misses a question on the tRAT) it can have a useful corrective effect, and help
team processes by affecting both the members who might be too assertive or
too quiet.

When teams receive feedback that their choice is incorrect, members who may
have had good ideas but were reluctant to speak up while the choice was being
made realize that they let their team down. Further, even if none of the other team
members have any idea about the fact that there was missing input, the quiet
members recognize the negative consequences of their inaction, and are motivated
to speak up in the future.

Also, during 4-S activity and discussion, when team members are struggling with
getting an overly assertive member to listen and have therefore ended up with a
problematic “team” answer, the immediate feedback provided by the entire class
in the simultaneous report and subsequent discussion helps them make their
point.

Here are the 3 primary practices of TBL that are designed to generate immediate
feedback:

1. Immediate feedback will occur for individuals, when transitioning

from iRAT to tRAT. When students finish their iRAT and turn to the tRAT,
they are bombarded with immediate feedback, as they begin comparing

Team-Based Learning Institute Page 49 of 156

 their own answers with those of their teammates. Disagreements among
team members lead immediately to analytic inquiry (Why did you say A?)
and self-assessment (Am I sure of what I read, understood or remember?).

2. Immediate feedback will occur for teams through the tRAT scratch-
off (IF-AT) process. The tRAT scratch-off form makes the consequences
for team decisions immediately visible. This ensures that a team will assess
its effectiveness at the end of each negotiation leading to an answer. The
immediacy of the feedback allows team members to evaluate the
effectiveness of their own decision-making, and to change any
behaviours—either collectively or of individuals—required to improve the
performance on subsequent items.

3. Immediate feedback will occur for everyone during team application

activities. The formatting of 4-S team application tasks for “same-
problem + simultaneous response” is specifically conceived for generating
immediate feedback.

“Same Problem” ensures that, however the teams respond, their

choice will become relevant feedback for the other teams.

“Simultaneous Response” ensures that each team will see

immediately where they stand vis-a-vis the other teams. No one can
hide from his own thinking.

The simultaneous response reveal is a critical moment of deep self-

assessment. When a team is alone in its report of an answer, it
immediately feels challenged, and will respond in a variety of ways, all
productive. It might argue forcefully and find value in defending itself
against the other teams. If the team felt unsure to begin with, the soul
searching begins when team members see the responses of all the other
teams. Because teams have had to commit to an answer and report it in
public, however, they have no choice but to make their case. In some
situations they will be vindicated, as the minority position may turn out to
be a good one.

Team-Based Learning Institute Page 50 of 156

Pillar E: Student agency, self-determination and
accountability

Students need to be treated like adults, who are free to act on their own
judgment, based on knowing what is expected of them to achieve a goal. This
means a major shift away from teaching practices characterized by “controlling”
students, where the instructor’s personal needs, preferences and even worries
and fears can inform how the course is experienced by students. A well-managed
TBL course ensures that students own the course. Students are assumed to be
self-motivated (even if they don’t appear to be on the surface!), intelligent,
capable, responsible individuals, and are, accordingly, objectively accountable at
multiple levels. Here are key practices that promote a learning culture for adults.

1. Course policies are written so as to place students in the role of “agent”

acting in his/her own interest. To achieve this culture means
communicating to students the choices they are free to make, and the
consequences that come with those choices, whatever they may be.
a) Eliminate “attendance requirements,” but replace them with “productivity
accountability.” Students who choose not to attend class are free to do so,
knowing that they accept as a consequence loss of the opportunity to
receive credit for work done during class. We recommend making sure
that something significant gets marked and recorded frequently in class.
b) Eliminate “make up” assignments. Instead, give students license (and
choice!) to drop a small, fixed number of scores in each category of their
grade, so that they can be responsible for managing their own options to
do or not do an assignment. (You’ll need policy language to deal with dire
cases of catastrophic illnesses or accidents).
c) Provide students with assignment deadlines expressed as “choices” tied to
“levels of “eligibility” to receive points. For example, assignments
submitted by a given date would be eligible for specific point values;
assignments submitted on a later date would be welcome, but eligible for
fewer points. Avoid the language of “penalties” for late assignments or
other infractions. Penalties are perceived by students as the instructor’s
arbitrary exercise of authority and control. Build your policy structure
around the choices students are free to make, knowing that they—and
they alone—are accountable for the consequences.
d) Explain in the course syllabus (and make sure students read it—such as
through a first-day “practice RAT” on the syllabus) how your course gives
students the tools and responsibility to manage their time as they find
necessary.

Team-Based Learning Institute Page 51 of 156

2. Students evaluate their teammates’ “helping behaviour” as a part of the
course grade. If students are going to become fully responsible and
accountable for their team’s learning, they will need leverage to motivate
their peers to be effective partners.
a) The Peer Evaluation component of the final course mark should be
somewhere around 5-10 % of course total, although there might be
circumstances in which more is appropriate. The amount needs to be
high enough to potentially influence a student’s final grade for the
course, but not so high as to directly determine the final grade, by itself.
b) Spend time on Day One or Day Two of the term working with students to
determine the criteria to be used in assessing peer helping behaviour.
This works well as a first team activity on day one. See Michaelsen,
Knight, and Fink (2004), for a description of the fishbowl exercise often
used to establish behaviour criteria for the whole class.
c) Schedule a first peer feedback and evaluation exercise approximately 1/3
way in the course. Use this activity to allow students to provide feedback
to one another, with no points at stake.
d) For the final peer evaluation, choose a process that requires students to
differentiate among individuals in assigning scores to their peers.

3. The overall grading scheme includes weights or percentages for

individual work, team work, and peer evaluation. The specific weights will
vary from course to course and from instructor to instructor, depending upon
learning outcomes and the students’ frame of reference. The culture of the
institution and age or maturity of the students, for example, might affect the
grading scheme and weighting strategy you use.
a) Give student teams the opportunity to determine at least some of the
grade weights (within parameters you give them) during the first week of
the course. A common, minimal practice is to let students decide the
weight of the iRAT vs the tRAT. Let them choose, for example, a 50-50, 60-
40 or 75-25 weight split, in either direction. All teams have to agree to the
same weighting scheme. See Sibley et. al. (2014) for elaboration on this
procedure.
b) As you become comfortable with this process, you can let teams help you
determine the value of grade weights for the whole course, within certain
parameters. A description of how to do this can be found in Michaelsen,
Knight, and Fink (2004).
c) For the course as a whole, it is recommended for new adopters to start
with an overall target weighting scheme of approximately 60-70% for
individual work and 30-40% for team-based assignments. As a rule, keep
the individual weight aggregate total well above 50%, to ensure individual
accountability. As you become more comfortable with the TBL model, you
might find reasons to shift the balance of weights in one direction or
another.

Team-Based Learning Institute Page 52 of 156

 d) Make sure that some of the daily team application activities result in a
score that counts toward the course grade. One way to do this is to have
students record in writing and turn in their team response and reasoning,
before the simultaneous response and subsequent class-wide
discussion. The instructor can then evaluate the team’s response and
enter a value in the grade book.

Here is a sample grading scheme for a TBL course. Note that there are several
traditional components. Individual tests and papers, for example, can still be part
of a TBL course.

20% RATs; (Individual vs. Team fraction to be determined in class)

10% Weekly in-class team “Challenge” activities
20% Individual in-class Essays/Midterms
25% Individual Capstone Essay/Memo (or “Final Exam”)
20% Capstone Team Case/Situation Analysis
5% Team Member Performance (Helping behaviour--peer graded)
100% Total

This will work!

For many instructors, the first implementation of a TBL course will be an
invigorating and satisfying intellectual challenge. Don’t worry if you don’t hit all
the marks perfectly the first time you use TBL. And don’t be deterred by the few
students who may struggle in response to the new expectations you have
communicated. It’s perfectly natural for a few students to push back at first—this
is a positive sign that you have gotten their attention and are challenging them in
a meaningful way. Be prepared to explain to students that your course is
designed to teach them how to use their knowledge, which will prepare them for
real-world challenges.

This is a learning process for both you and your students. Students in general will
be forgiving when they notice (and they will!) that you are trying to create for
them an engaging experience, and that they are learning more than they would
in a lecture-based course. We hear stories from faculty members who tell of how
their students became partners in the process, offering feedback during the
course on how to improve RATs as well as the design of 4-S activities that did not
work as planned.

Your goal for a first time effort is to put the basic TBL protocol in place,
respecting the 5 pillars of practice, and fine tune as you go. It’s common, for
example, to struggle at first with designing consistently effective team application
activities around meaningful choices and decisions that generate lively, relevant

Team-Based Learning Institute Page 53 of 156

discussion. For some instructors, calibration of RAT difficulty (too easy vs. too
hard) is also something to be learned through practice. Above all, it’s common to
struggle in adapting to an outcomes-driven course, where the design of student
work in their teams forces you to “think backward” to make sure all the pieces are
in real alignment, from the clarity of learning outcomes, to the design of 4-S
application activities, to the creation of the RATs, to the selection of content.

There will be genuine joy in the effort, however. We are reminded of a colleague
who recently commented to us, during her first semester of TBL, “The discussions
in class have been inspiring: this is the first time in my teaching career that I’ve
actually been able to see and hear my students learning!”

These students could be yours. Trust the method.

Team-Based Learning Institute Page 54 of 156

 2010; 32: 118–122

Twelve tips for doing effective Team-Based

Learning (TBL)
DEAN X. PARMELEE1 & LARRY K. MICHAELSEN2
1
Wright State University, USA, 2University of Central Missouri, USA

Abstract
Team-based learning (TBL) in medical education has emerged over the past few years as an instructional strategy to enhance
active learning and critical thinking – even in large, basic science courses. Although TBL consistently improves academic outcomes
by shifting the instructional focus from knowledge transmission to knowledge application, it also addresses several professional
competencies that cannot be achieved or evaluated through lecture-based instruction. These 12 tips provide the reader with a set
of specific recommendations which, if followed, will ensure the successful design and implementation of TBL for a unit of study.
Med Teach Downloaded from [Link] by University of Saskatchewan

Introduction each module begins with the readiness assurance process

(RAP). The RAP consists of a short test (over the key content and
Medical educators have long recognized two important realities. concepts from the readings or other activities, e.g. dissection)
One is that being able to recite all the subtle differences between which students first complete as individuals, then they take the
one form of a disease and another is a very different kind of exact same test again as a team, coming to consensus on each
knowledge than being able to quickly diagnose the correct form question. Students receive immediate feedback on the team test
of that disease suffered by a real, living patient. The other is that and they then have the opportunity to write evidence-based
For personal use only.

medical students must master both kinds of knowledge. appeals if they feel they can make valid arguments for their
In traditional medical education, students were exposed to answers to questions which they got wrong. The final step in the
the two different kinds of knowledge at different times and in RAP could be a ‘lecture’ (usually very short and always very
different settings. The content was typically taught in lecture- specific) to enable the instructor to clarify any misperceptions
based courses and, later (some years later) students learned to that become apparent during the team test and the appeals, but
use the content during their time in clinical rotations. also could be a between-team discussion about why the
Delaying students’ opportunity to learn to use the content, selected correct answers are best – fielded by the instructor.
however, does not fit well with what we now know about how Once the RAP is completed, the remainder (and the majority) of
adults learn best – the kind of learning that both ‘sticks’ and the learning module is spent on in-class activities and assign-
can be transferred to novel situations. As a result, medical ments that require students to practice using the course content
educators have experimented with a number of approaches by solving challenging problems.
for enabling students to more closely connect the content and
concept acquisition with its application – e.g. problem-based
TWELVE TIPS
learning (PBL), case presentation.
The purpose of this article is to describe an approach that, Tip 1: Start with good course design
like PBL, immediately and intensively engages students with TBL is an instructional strategy that works best when it is
the kinds of problems they will encounter in medical practice. integrated tightly with a course’s design. It can be the primary
With this approach, team-based learning (TBL), some class- mode of instruction or work alongside other learning activities,
room time is spent on ensuring that students master the course i.e. focused lecture, service learning, self-directed online
content. However, the major emphasis is on concept applica- tutorials. We recommend using Dee Fink’s Creating
tion, and the processes through which students learn both the Significant Learning Experiences: An Integrated Approach to
content and the applications are specifically designed so that Designing College Courses (2003) for guidance in defining a
student groups develop into self-managed learning teams. As course’s (or curriculum’s) contextual issues, goals, assessments,
a result, a single instructor can both provide content expertise learning activities, and feedback mechanisms. Often, instructors
and oversee the learning endeavors of an entire class. will ‘try out’ a TBL module or two in an existing course, either
For a course with TBL as part of its learning activities, replacing a set of lectures or small group sessions that had
students are strategically organized into permanent groups (for required recruiting and herding many faculties. This is a valid
the entire term of the course) and the course content is way to gain experience with how to implement it, but, usually, it
organized into major units (typically five to seven). Before each is hard to incorporate the peer evaluation component since the
in-class event, students must study assigned materials because number of meetings will be few.

Correspondence: Dean X. Parmelee, Academic Affairs, Boonshoft School of Medicine, Wright State University, PO Box 927, Dayton, OH 45401-
0927, USA. Tel: 1 937 775 3803; fax: 1 937 775 2842. email: [Link]@[Link]

118 ISSN 0142–159X print/ISSN 1466–187X online/10/020118–5 ß 2010 Informa Healthcare Ltd.
DOI: 10.3109/01421590903548562

Team-Based Learning Institute Page 55 of 156

 Twelve tips

Team-Based Learning
(Repeated 5 –7 times per course)

Advanced Readiness Assurance Application of Course Concepts

Peer Teaching &
Preparation Development of Students’ Critical Thinking Skills
Feedback
(pre-class) 1-hour + or – A few minutes to several hours (integrative)

Step1 Step 2 Step 5 Step 6

Application/Critical
Individual Individual Instructor
Thinking
Study Test Input
Activities & Problems

Step 3 Step 4
Team
Written Appeals
Test

Figure 1. Instructional activity sequence for TBL content unit.

Med Teach Downloaded from [Link] by University of Saskatchewan

Tip 2: Use a ‘backwards design’ problem that requires students to use all of the preparatory
when developing TBL courses and knowledge and their team’s brainpower to analyze, interpret,
and then commit to a choice or a decision. Further, you should
modules
avoid the temptation to ask a series of questions as a means of
With backwards design (Wiggins & McTighe 1998) the first ‘leading students through the thinking process.’ It is far better
question to ask yourself is, ‘What do I want my students to be to require them to make a difficult choice and let them work
able to DO by the end of this unit of study?’ Whether designing together to master the concepts and to discover and internalize
For personal use only.

a single TBL module for a unit of study, e.g. Starling’s Law and the relationships between them in the process of coming to a
cardiovascular physiology, or a series of modules that form the conclusion.
basis of an entire course, clarify what you want the students to Once you have decided what you want students to be able
be able to do by the end of the module or course. For to do and how you will assess whether or not they can do it,
example, a goal for a module in physiology/pharmacology the next two steps in backwards design are identifying what
focused on Starling’s Law would be for the students to be able content elements the class must master before they are ‘ready’
to apply their understanding of Starling’s law to accurately to solve the problem (i.e. the information that the students
interpret physiologic data from a case of congestive heart need to learn outside of class to be prepared for the module)
failure, explain how Starling’s Law governs which findings, and write the questions for the readiness assurance test (RAT)
predict which pharmacologic agent will affect specific (and do not call it a quiz – its purpose is readiness assurance
components of heart function. A traditional course in anatomy and you should emphasize its role by the terms you use in
would have many TBL modules, each one presenting a new talking about it).
clinical problem vignette linked to the cadaver dissection
component of the course and giving the students multiple
opportunities to learn the daily applicability of anatomy Tip 4: Have application exercises
knowledge for clinical practice. that promote both deep thinking
This single question is often the hardest one for instructors and engaged, content-focused
who are ‘content-driven’ to ask themselves. There is just so
discussion
much ‘content’ that we feel our students must know before
they can make use of it – but, TBL provides a way to have Over the years, we have come to realize that the single most
them master the content while they are applying it and get important aspect of successfully implementing TBL is what
feedback on how well they are ‘getting it’ as they go. your assignments require students to create. Whatever the
content, if you ask them to produce a lengthy document, they
will divide up the work which, in turn, will reduce learning
Tip 3: Make sure you organize the and, all too often, will result in negative feelings about their
module activities so that students peers and skepticism about working in a group. On the
can reach your learning goals and other hand, we have learned that, by using well-designed
you (and they) will know that they assignments, students will both learn from each other and
develop a great deal of confidence in the value of working in a
have done it
team.
After clarifying what you want your students to be able to do The key to designing effective assignments is ensuring
by the end of the module, the next step in backwards design is that what students are asked to do is characterized by 4 S’s at
creating a group application exercise. This should be a each of the stages in which they engage with the course
119

Team-Based Learning Institute Page 56 of 156

 D. X. Parmelee & L. K. Michaelsen

content – working alone, working within their team,

and working across teams (i.e. whole-class discussion). The
4 S’s are:

Significant Problem
For a successful group application exercise, select or create a
problem that the student can readily recognize as the kind of
problem that will be encountered in ‘real life,’ make it
Significant. In medical education, this is easy – there are an
infinite number of patient cases that are rich with data to be
interpreted, decisions to be made. But, there has to be a clear
link between the content that underlies the exercise and its
application. At the conclusion of the hypothetical module on
Starling’s law/physiology, you want to hear your students
talking about how the basic principles of the law are Figure 2. RAT scores.
applicable to understanding cardiac contractility in stress
situations and how to approach interventions. In addition,
Med Teach Downloaded from [Link] by University of Saskatchewan

the answers to these questions should never be discoverable in their respective answers, you can readily create a class
a text or article or lecture notes – they can only come from discussion that is far more informative to you and your
team members collaborating to figure them out. students than asking, ‘Somebody say what they think about
thus-and-such.’
We have also learned two lessons – sometimes by sad
Same Problem
experience – about the 4 S’s. One is that failing to do any one
With TBL, all of the small groups must be working on the of the 4 S’s substantially reduces both the intensity of class
Same Problem. If you assign different problems to different discussions and the resultant learning. The other is that, if you
For personal use only.

small groups, students are not accountable to each other fail to do any two of the 4 S’s, learning is minimal and pretty
because you lose the benefit of having any semblance of a much the only reason that students are willing to complete the
robust discussion (and learning!) between-group discussion of assignment is that it will have a negative impact on their grade.
the problem. Further, if you allow groups to choose their own
problem, they are not even accountable to you – unless you
are willing to do the research that you hope they would do.
Tip 5: Do not underestimate the
importance of the RAP
Specific Choice
The RAP is designed to link students’ advance preparation to
When your assignments require students to agree on a specific the group application exercises and provides a remarkable and
choice, the only way they can accomplish the task is by powerful opportunity for individual feedback and peer teach-
working together to critically appraise a situation, examine the ing within the teams. In addition, the RAP lets you (and the
existing evidence, and make a professional judgment. Further, students) know if you need to address gaps in their
the more specific the question, the better the learning. For understanding. If the content area is particularly difficult, e.g.
instance, if your module was about depression and pharma- autonomics, odds ratios and predictive values in critical
cologic interventions, a good question would be ‘Identify the appraisal, liver pathology, then the RAP should be separated
set of neurotransmitters that are affected by the best drug
in time from the group application exercise so that the
choice for this patient’ and not ‘What would be the best drug
instructor can give corrective feedback and/or provide addi-
for this patient’ because a more specific question requires a
tional input before they begin to tackle the group application
deeper analysis.
exercise. However, you do not have to cover everything – only
what you (and the students) know they need help with.
Simultaneous Report The RAP, when done well, unfailingly produces five
priceless outcomes even though it typically uses only a
You create an important ‘moment of truth’ when all the small
fraction of the overall class time (usually about 25–30%) for
groups are asked to post their responses to a question at the
any given unit of instruction. These are:
same time. Two things happen as soon as students realize that
the choice they will be making will be open to challenges from (1) Effective and efficient content coverage.
other groups. One is that, because of the potential of an ‘us (2) Development of real teams and students’ interpersonal
versus them’ situation, group cohesiveness increases. The and teamwork skills.
other is that students are far more engaged in the (3) Students gain an experience-based insight about the
within-groups discussion because they realize that they value of diverse input.
would not be able to hide if they do not ‘get it right.’ In (4) Development of students’ self-study and life-long
addition, by engaging students exploring how they arrived at learning skills.
120

Team-Based Learning Institute Page 57 of 156

 Twelve tips

(5) Class time during which you can provide the content effort working in their team. The instructor is accountable for
expertise to ensure that students develop critical providing students with the cognitive foundation they will
thinking skills. need to be ready to tackle the kinds of problems they will face
in medical practice and giving them opportunities to practice
In addition, data from the RAP provides data that definitively
developing their application skills.
answers the question of whether or not individuals are likely to
When TBL is fully employed, the vast majority of students
be held back by working in teams. Based on data from the past
are prepared, come to class, and engage each other in
23 years of using TBL (Michaelsen and Parmelee, unpub-
productive ways as they work together. As a result, even the
lished), teams will score higher than their own very best
students who start out with a skeptical attitude because of past
member 99.9þ% of the time1 and the most common outcome
negative experiences with learning groups will eventually
is that the worst team score will be higher than the highest
embrace TBL – ‘Finally, hard work as an individual and hard
individual score in an entire class.
work as a group pays off.’
Unfortunately, we have seen some instructors miss out on
As for the accountability of the instructor, some students will
part or all of these valuable outcomes because they have, for
inevitably start out with the impression that he/she is not
whatever reason, decided to: (1) skip either the individual or
‘teaching’ as in other classes, i.e. using lectures to state what will
the group component – or both; (2) use questions that are
be on the final exam, and, worse, we (the students) are having
merely designed see if the students did the reading, e.g. asking
to do all the work. Further, if the instructor is not following
‘picky’ or meaningless questions unrelated to the objectives of
through with his/her side of the bargain – doing a good job of:
the module; and (3) view this process as a way of getting
Med Teach Downloaded from [Link] by University of Saskatchewan

(1) providing students with the opportunity to practice using

another assessment for their course grade. The RAP is not just
well-designed applications assignments (i.e. using the 4 S’s) and
another ‘quiz’ and neither instructors nor students will reap its
(2) reminding students of the benefits that they are getting –
many potential benefits if it is treated as such.
then the doubts and the resentment are likely to persist.

Tip 6: Orient the class to why you

are using TBL and how it is different Tip 8: Providing a fair appeals
from previous experience they may process will inspire further learning
For personal use only.

have had with learning groups Inevitably, some students will disagree with your selection of a
best answer on a RAT question. They will do so on one of two
Most students will not have had a classroom experience like
bases: the question was written in such as way that they were
TBL. In fact, the majority of their experience with group work
confused or they feel you made an error in your interpretation
will have been struggling to complete poorly designed
of the content. The appeals process (Michaelsen 2008, p.24)
assignments that forced them into the uncomfortable position
provides the opportunity, preferably while they are still in
of having to choose between doing more than their fair share
class, to either re-write a question that they feel was poorly
or risk getting a bad grade and/or having to deal with difficult
written or articulate, in writing, why they feel their answer was
group members just to get anything done at all.
better, using references if appropriate. Accept appeals from a
These concerns are real and must be addressed or you will
team only; award credit to the appealing team(s) only and to
have a difficult time getting student buy-in unless students
the individual scores of the members of those teams.
understand both why you are using TBL and how TBL is
The appeals process provides a number of benefits. One is
designed to avoid the problems that they, all too often, have
that it motivates students to do a focused re-study of the exact
come to expect are a normal outcome from doing group work.
material that gave them the most trouble. Another is that, the
At a minimum, you need to outline your course objectives and
process of trying to put together a successful appeal requires to
provide an explanation of how they would be achieved in a
think deeply about both the specific ideas and the overall
traditionally taught course versus how you will achieve them
context within which they reside. Finally, students can often
by using TBL. Other suggestions to help them understand and
re-write your questions so that indeed they are better!
accept TBL include: (1) giving a practice RAT (many use the
course syllabus as the ‘subject matter’ for the test); (2)
engaging them in the process of determining the grading Tip 9: Peer evaluation is a
system for the course (Michaelsen et al. 2004) and, throughout
challenge to get going, but it
the course; and (3) reminding them about the benefits they are
experiencing along the way.
can enhance the accountability
of the process
Tip 7: Highlight accountability as the There are several ways to set up a peer evaluation process for
the course, and it may take some trial and error to find the one
cornerstone of TBL that fits well with your institution or course’s culture (Levine RE
The cornerstone of success of TBL is that the natural outcome 2008, Chapter 9). There are, however, numerous benefits from
of its processes is that individuals, teams, and the instructor are putting forth the effort. One of the most important is that, when
immediately and clearly accountable for behaving in ways that you use peer evaluations, students are accountable to the
promote learning. Students are accountable for coming to members of their team. Another is that a well-designed peer
class, preparing before they come, and investing time and evaluation process enables students to learn how to give
121

Team-Based Learning Institute Page 58 of 156

 D. X. Parmelee & L. K. Michaelsen

constructive feedback to one another and to gratefully receive the class when speaking – you will not need a roving
constructive feedback from peers – an invaluable competency microphone once they learn to be quiet when someone is
for future practice. speaking.

Tip 10: Be clear and focused with Conclusions

the advanced preparation We are grateful to have been invited to provide these 12 TIPS.
A criticism of TBL is that the instructor identifies the learning Over the past few years, we have provided many faculty
needs for the students, thereby robbing them of the opportunity development workshops and consultations, around the world,
to explore the potential domain of the content and make some to introduce medical educators to TBL and assist them with its
judgments about what they need to know. Based on past implementation in a variety of settings. In most cases, TBL has
experience, when you are specific about what you want them produced a positive transformation of the classroom experi-
to master before a TBL module, including posting action- ence for both the students and the instructor. Sometimes,
oriented objectives such as ‘Be able to articulate how dopamine however, we hear comments from faculty such as: ‘I tried it a
affects sodium channels at the receptor level,’ you invite them few times, but gave up because the students didn’t like it,’ or
to go beyond doing the minimum of preparation since that will ‘Does one have to use all the components? The GRAT sounds
only help for the individual part of the RAP. They learn quickly like a waste of time.’ Unfortunately, whenever we ask about
that for their team to be really successful in the group work, the details of a less-than-successful attempt, we almost always
Med Teach Downloaded from [Link] by University of Saskatchewan

they must master the advance assignment assiduously and learn that one or more of the components had been omitted or
devote additional effort to exploring the content domain. Tying altered substantially. The strategy has been well tested and
the TBL objectives to the course objectives is essential. works, but works best when all of the components are
included in the design and implementation.

Tip 11: Create the teams

thoughtfully
Declaration of interest: The authors report no conflicts of
We have three principles to guide the process of getting a class
For personal use only.

interest. The authors alone are responsible for the content and
into teams: (1) make the process transparent so all students
writing of the article.
know how they ended up in a particular team, even if the
process is totally random; (2) distribute what you define as
‘resources’ for a team as evenly as possible, for instance, a Notes on contributors
beginning class of medical students might have several
DEAN X. PARMELEE is the associate dean for Academic Affairs, and
students who have advanced degrees in one of the basic devotes most of his time to improving medical student education.
medical sciences, so you want to assign them to different LARRY K. MICHAELSEN is a professor of Management, an active teacher of
teams; and (3) strive for the teams to have a diverse undergraduate business school students, and both he and Dr Parmelee
composition, i.e. gender balance, rural or urban backgrounds, conduct workshops on TBL for faculty development at a variety of
science/nonscience majors. Letting a class know that teams institutions of higher learning internationally. He is also the David Ross
Boyd Professor Emeritus at the University of Oklahoma, a Carnegie Scholar,
that have diversity within, however defined, will have unique
a Fulbright Senior Scholar, and former editor of the Journal of Management
strengths to draw upon in the challenging modules ahead. Education.

Tip 12: Several low-budget ‘props’ Note

facilitate the implementation of a 1. Based on data from 6161 students in 1115 teams since
good module 1986–1114 team scores were higher than the score of their
own very best member.
One does not need to spend several thousands of Euros for the
latest audience response system or any high-definition tech-
nology to get a well-constructed TBL module to work. We
recommend using IFATTM response forms for the group References
readiness assurance because students will hover over the Fink D. 2003. Creating significant learning experiences: An integrated
scratch-off card, talk with each other, make eye contact, and approach to designing college courses. San Francisco, CA: Jossey-Bass.
Levine RE. 2008. Peer evaluation in team-based learning. In: Michaelsen LK,
be passionate about whether or not the correct answer is going
Parmelee D, McMahon K, Levine RE, co-editors. Team-based learning in
to emerge. They receive immediate feedback, let one another health professions education. Sterling, VA: Stylus Publishing. Chapter 9.
know things like ‘You were right! Next time make us listen to Michaelsen LK, Knight AB, Fink LD. 2004. Team-based learning: A
you!’ Prepare folders for each team, color code the compo- transformational use of small groups in college teaching. Sterling, VA:
nents to make the sequence of activities clear, collect Stylus Publishing.
Michaelsen LK, Parmelee D. Analysis of individual versus team scores on
everything so that you do not have to start ‘de nova’ every
readiness assurance tests, undergraduate and professional student
year – a good module is a treasure. Buy or build flagpoles to performance. Unpublished data.
demarcate the position of teams; laminate the lettered cards for Wiggins G, McTighe JH. 1998. Understanding by design. Merrill
simultaneous responses. Require students to stand and face Prentice Hall.
122

Team-Based Learning Institute Page 59 of 156

Team-Based Learning Institute Page 60 of 156
 Roberson, B., & Franchini, B. (2014). Effective task design
for the TBL classroom. Journal on Excellence in College
Teaching, 25(3&4), 275-302.

Effective Task Design for the TBL Classroom

Bill Roberson
Billie Franchini
University at Albany

Group and team tasks are the culminating outputs of student

learning in team and collaborative learning environments. How
they are conceived and designed, therefore, can directly deter-
mine the success of the pedagogical strategy. A key design issue
for creating effective tasks is how best to focus student knowl-
edge, observation, and analysis toward a concrete action that
makes thinking visible. Actions in the shape of clear decisions
applied to complex scenarios, within a restricted framework of
options, are most likely to channel student thinking toward
higher-level goals. The authors provide principles and examples
for designing group tasks in any discipline.

Introduction
Effective task design and management are at the heart of team-based
learning (TBL). Whether or not the Readiness Assurance Process (the TBL
process of testing students on their attempt to cover a unit of content on
their own) is successful in preparing students to apply what they know,
it is the collective decision making required by team tasks that truly fo-
cuses student learning, provides traction in the learning process, induces
team cohesion, and stimulates general student enthusiasm. If the tasks are
not carefully conceived and challenging in the right way, student focus
drifts, classroom energy falls off, and teams fail to cohere. For this reason,
task design should be a first concern for an instructor transitioning from
more traditional teaching to TBL. Effective design and implementation
of tasks can offset many problems, and can even carry to partial success
an otherwise flawed TBL implementation. The purpose of this article is to

275

Team-Based Learning Institute Page 61 of 156

276 Journal on Excellence in College Teaching

frame the challenge of task design conceptually, extract some principles

based on that conceptualization, and offer examples showing how the
principles can be put into practice in a range of disciplines.

Tasks Make Learning Visible

A central tenet of TBL is that student learning is driven through fre-
quent and, whenever possible, immediate feedback. In order for this to
happen, student learning and consequent use of that learning in their
thinking have to be made visible—to students themselves as well as to
the instructor. Students, therefore, need to be required to act frequently in
ways that generate consequences that provoke reflection and demonstrate
visibly their thinking. The more focused and concrete the action, the more
visible will be the thinking and the learning—and the more immediately
useful will be the feedback.
Recent findings in neuroscience, cognitive science and psychology can
help us visualize this key role of action in the learning process. Bransford
(2000), for example, emphasizes the essential difference between under-
standing and memorization, citing numerous studies showing that simple
rote learning does not lead to transfer of knowledge. Cognitive psychol-
ogist Willingham (2009) argues that “Memory is the residue of thought”
(p. 54), meaning that thoughts are made manifest by actions, and only
acting on information can transfer it from working memory to long-term
memory. The work of neuroscientist Zull (2002 and 2011) reinforces these
findings, showing that learning that has not been put into the service of
action tends to remain dormant and through disuse becomes less retriev-
able from storage in the brain’s neuronal networks.
Much of this work builds on earlier studies in psychology, namely
Kolb’s theory of experiential learning, depicted in Figure 1. This idea of
the learning cycle is a useful guide in thinking about the process we are
trying to foster in our students’ cognitive functioning. The conception of
learning as a cycle helps us to envision how our knowledge of the brain
can be translated into successful classroom practice. In Kolb’s description,
the experience of an action leads to observing and reflecting on its conse-
quences. This reflection is the first step in abstracting from the experience
a conceptual understanding of what happened and what it might have
meant. As abstract theorizing develops, opportunities for experimenta-
tion with the use of that knowledge should follow so that students can
put their abstract understanding to the test. It is this ongoing interplay
between abstract conceptualization and active, concrete experience that
creates the possibility of storing learning and applying it to new situations.

Team-Based Learning Institute Page 62 of 156

 Effective Task Design for the TBL Classroom 277

Figure 1
The Kolb Experiential Learning Cycle
(from McLeod, 2010)

In the college classroom, one of the instructor’s most important jobs

is to design and stage opportunities for students to undergo this cyclical
action-reflection-conceptualization-action process so that relevant infor-
mation and ideas become fully networked in the brain. A comparable
level of fully networked understanding is extremely difficult to build
through less-active means—by sitting through a lecture, for example, or
watching a video or reading a text. Our students need to construct their
own conceptual understanding within the framework of active individual
experience. Each action we ask them to take leads to reflection and greater
awareness, which, in turn, leads to receptiveness to new information,
integration of that information, and planning for new, more informed
actions. In essence, we are helping our students work toward becoming
more intentional and more expert in their thinking and actions, particu-
larly with respect to our discipline. The assigned tasks that induce these
actions drive learning. They, therefore, need to be integrally connected to
the larger, overarching strategy of the course and directly tied to course
learning goals.

Course Design, Task Design, and Disciplinary Thinking

More traditional, instructor-centric teaching practices tend to shape
courses and curricula around disciplinary content. Syllabi are routinely

Team-Based Learning Institute Page 63 of 156

278 Journal on Excellence in College Teaching

structured as sequences of topics that will be covered from week to week,

and often track to textbooks with similar patterns. This approach to con-
tent can sometimes be a barrier to deep learning, as it does not capture
the full scope of what it means to work and think within a discipline.
The signature of a discipline, whether in the humanities, sciences, social
sciences, or professional fields, is less its content (which might be shared
among several disciplines) than its actions. Historians are historians not
just because they deal with historical texts and artifacts, but because they
use historical resources to inform actions that are typical of historians, such
as reconstruction of a past event, evaluation of the influence of a particu-
lar person, and the like. Sociologists might (and often do) use those very
same historical resources to inform a different set of actions, such as in the
analysis of a contemporary sociological condition or the determination
of how a social injustice came to exist. An economist might use the same
resources yet again to inform her construction of a predictive model of
behavior in a given set of market conditions.
A bit further afield, but no less relevant, an epidemiologist, trying to
track the evolution of a virus over time, might have reason to explore
these same historical resources because they contain evidence of behavior
and circumstances related to the emergence of a pandemic. Specific infor-
mation (“content”) does not suffice to define a discipline. Disciplines are
more clearly defined by how those working within the discipline collect,
organize, assess, and use information.
The real difference, therefore, between novice and expert thinkers
in our disciplines is not determined by the amount of information they
have covered or even mastered, but rather by their relative ability to in-
teract with that information. Course and task design need to be pointing
students not toward simply knowing more, but ultimately toward more
refined, more expert ways of responding to and using information. If we
want our students to become more expert in our disciplines, we need to
structure their encounters with content in ways that change what they
can do with knowledge.

Implications for Task Design

The most clarifying action a student can take is to make a decision. Requir-
ing collective decision-making provides an opportunity for students to
practice the kind of thinking we want to promote in our courses and
disciplines and is the starting point for effective overall TBL course
design. A well-constructed decision-based task integrates components
of higher-order thinking: analysis of the particular situation to deter-

Team-Based Learning Institute Page 64 of 156

 Effective Task Design for the TBL Classroom 279

mine competing priorities and values; various lines of reasoning; use of

relevant concepts, principles, laws, or other abstractions at play in the
situation; reflective, critical thinking (Are we sure of these facts? Are we sure
we understand?); and, ultimately, a judgment that is expressed in a visible,
concrete action/outcome that can be evaluated. Effective team tasks point
students consistently toward making decisions that reveal reasoning and
understanding in service of a judgment. The judgment students make
ideally will replicate as much as possible the kinds of judgments made
by disciplinary thinkers.
In order to put students on this track, we first need to identify and
characterize the kinds of actions and decisions that thinkers in our
disciplines execute frequently. Then we can reverse engineer situations
where students practice doing these very things. In this way, we ensure
that students also practice using the targeted disciplinary content of the
course. Here are some key questions that can help us begin the process:
• What do people in your discipline do with the informa-
tion they collect and/or use? What kinds of problems
do they try to solve?
• What is characteristic about the way practitioners of
your discipline think—that is, how do they approach
and enter problem-solving? How do they reason?
• What kinds of judgments do experts in your discipline
have to make?
• What assumptions consistently inform their decisions
and other actions?
• What are the discipline-specific actions and types of
decisions that a successful student will be ready to carry
out as a result of your course?
Jotting down several items for each of these questions will help instruc-
tors characterize and eventually locate or invent the types of tasks that
will be relevant to the learning targets of their course. What follows are a
few basic examples of decision-making in various disciplines.
• Economics: Decide which patterns of buyer behavior can
be determined from a given set of consumer data.
• Sociology: Decide what might be the implications of a
new data set for understanding a specific social phe-
nomenon.

Team-Based Learning Institute Page 65 of 156

280 Journal on Excellence in College Teaching

• Philosophy: Decide whether a given action is just or

rational, according to specific criteria or values.
• Business: Decide which marketing strategy to use, given
background data and consumer circumstances.
• Literature: Decide what patterns an author has construct-
ed to influence reader perceptions.
• Writing and Rhetoric: Decide which evidence would work
best to support a given thesis.
• History: Decide which account of an historical event is
most convincing, given competing perspectives and
evidence.
• Biology: Decide (predict) which environmental condi-
tions will most alter an organism’s DNA.
• Chemistry: Decide (predict) how a given molecular
structure will be changed by contact with other specific
types of molecules.
• Math: Decide which variables are significant or which
calculation strategy will produce the most valid or ac-
curate result.
From this macro perspective, in which we identify globally what stu-
dents need to be doing daily in order to practice disciplinary thinking, we
are ready to move to the micro-level and look at more specific elements
of task design. The most successful TBL courses are those in which the
instructor maintains the macro-micro perspectival exchange throughout
the course. Keeping an eye trained on the macro while working on the
micro will also facilitate the selection of material and formats for team
tasks and other assignments. The daily, specific team tasks need to inform
and align with the bigger actions (such as major graded assignments)—
and vice versa.

Situating Team Tasks in a Learning Sequence

For tasks to be perceived as authentic and valuable learning opportu-
nities, students need a clear sense that they are serving the stated learning
goals and disciplinary thinking goals considered above. This is particularly
true when we want to challenge students at a high level, such as by asking
them to make decisions that they perceive to be above their current level

Team-Based Learning Institute Page 66 of 156

 Effective Task Design for the TBL Classroom 281

of expertise. If they do not value the kind of thinking we are asking them
to practice, they may be resistant to the challenge. In this context, tasks
serve various tactical purposes at different times.

Before the RAP: Use Naïve Tasks to Launch a Learning Sequence

A common source of pushback early in a TBL course is students’
mistaken belief (sometimes unintentionally reinforced by a “helpful”
instructor) that they cannot do the reading on their own. In the face of
resistance, many instructors will instinctually move toward one of two
problematic practices: (1) giving students a highly detailed reading guide
or set of questions to answer or (2) lecturing before the I-RAT. Either of
these will undermine the goals of TBL. One way to avert this situation is
the use of naïve tasks.
Naïve tasks occur at the very beginning of a learning sequence—even
before the reading assignment—and are designed to induce an attitude of
inquiry. When designed and managed appropriately, they serve to chal-
lenge students to test their preconceptions and practice their reasoning
before being exposed to the targeted unit of content. In this way, naïve
tasks serve to surface pre-existing errors in student thinking. More im-
portant, though, making and defending a decision before having access
to key information promotes the perception that the information, when it
is eventually provided, will be a valuable tool or resource. Consequently,
students will be more likely to undertake the reading assignment with
greater enthusiasm because the readings are no longer a mere requirement.
They are, instead, perceived by students as being useful for the purpose
of assessing and improving their own thinking.
Naïve tasks provoke curiosity and function as a kind of reading guide
without becoming a crutch that reinforces students’ learned helplessness
the way more direct instruction can do. Reading with a specific, self-cor-
rective purpose also replicates the way actual experts (and our brains
in general) approach and respond to new information. The naïve task
strategy therefore supports the long-term goal for students to begin hon-
ing their intuitions about thinking in the discipline. The example below
is a naïve task from a course in economics. It can be used to introduce
the fundamental concept of “elasticity” or as practice to develop deeper
understanding after a general conceptual introduction. This task can be
set up using a graph and a brief explanation of how the axes and curves
show schematically the supply and demand relation to price within any
given market. For example, Figure 2 illustrates elasticity as a concept used
for measuring how likely change in a given market factor (for instance,
quantity/supply) might influence another factor (for instance, demand/
price).

Team-Based Learning Institute Page 67 of 156

282 Journal on Excellence in College Teaching

Figure 2
Elasticity

 
P (Price)

Q   P  
P    

Q (Quantity)

Students are then asked to choose an answer to the following:

Which of the following will NOT cause a shift in the demand
curve for ice cream?
A. The government gives every family $500 tax rebates.
B. The price of frozen yogurt doubles.
C. There is report that milk products used to produce ice cream
have special health benefits.
D. The price increases by $1.
E. None of the above—these all cause shifts in demand.
(example supplied by Shawn Bushway, Criminal Justice, University
at Albany)

Team-Based Learning Institute Page 68 of 156

 Effective Task Design for the TBL Classroom 283

After students have proposed and debated possible answers to this

question with their teams in class, they are ready to tackle the reading,
which is a more systematic presentation of market forces. Because the naïve
task has already driven students to grapple with the concepts presented
in the reading, they will now read actively, with an eye toward the kinds
of judgments and decisions they will be able to make, once they have
understood the new information.
Here are some sample naïve tasks from other disciplines:
• History: Read this paragraph (from an unknown source).
In which decade do you think it was written? Why?
• Anatomy: Look at this photo of a liver. What does it sug-
gest about the health condition of the person it belongs
to? Why?
• Literature: Read this paragraph. Predict the actions and
fate of the character you see described, based on the
limited information provided (and be ready to say why.)
• Engineering: Look at this design of a bridge. In an earth-
quake, which element is most at risk of failure? Why?
• Computer Science: Look at this sequence of code. Which
series of actions is it designed to execute in the robot?
Why?
• Various disciplines: Read this specific claim/statement.
Which of the following theories does it appear to repre-
sent/support?
While naïve team tasks can be used at the very beginning of the RAP
before students have read, they can also be used during the “informed”
application task phase of a sequence. In the latter case, naïve tasks prepare
students for new concepts that build on those already encountered in the
core readings (discussed below).
Finding the appropriate level of difficulty for naive tasks is essential
to their success: The tasks need to require a real judgment and a concrete
decision based on that judgment rather than merely ask students to supply
or apply basic knowledge. By asking students to act in the face of “insuf-
ficient information,” naïve tasks validate the role of information when it
finally lands. In order to create room for information, a naïve task needs
to be difficult enough that most teams will struggle and likely arrive at
the wrong answer at first.

Team-Based Learning Institute Page 69 of 156

284 Journal on Excellence in College Teaching

Beyond the level of difficulty of the task, cultivating an atmosphere of

playfulness is essential to encourage teams to persist in the face of this
difficulty. Students have to feel an intrinsic reward for “playing along” and
even getting the wrong answer. Handling wrong answers is also a crucial
moment for instructors—we need to acknowledge their errors in thinking
while demonstrating that with more information, the challenge we have
presented is surmountable. Finally, a publicly reported team decision is
essential so that students are held accountable for their current thinking
(like experts and professionals) and have an opportunity to re-examine
their position in light of other students’ responses to the same challenge.
Because naive tasks are intended to induce reflection and surface com-
mon student misconceptions rather than evaluate students’ final level of
learning, and because students need encouragement to take risks in their
thinking, the stakes for naïve tasks should remain low. This means that
they will most likely be ungraded, or at most be good for bonus points, in
order to minimize the perceived cost of error. The psychological support
of the team is also a fundamental component of naïve tasks. The team
structure allows students to be less self-conscious about errors than when
they feel they are individually accountable.

After the RAP: “Informed” Tasks That Put Knowledge to Use

An essential difference between a traditional course and one designed
for TBL is the role of content. In a TBL course, acquisition of course con-
tent/knowledge is not the primary learning goal, but it is the vehicle for
students to practice specific ways of thinking and acting. “Informed”
tasks, as opposed to naïve ones, ask students to convert their reading,
understanding, and reasoning into judgments and clear decisions that
make the learning and thought process visible.
There are multiple levels of informed tasks, and one of the first challeng-
es facing new TBL adopters is creating lower-level tasks that require real
judgments and authentic decisions rather than simple plug-in responses.
It is important to keep in mind that the Readiness Assurance Process has
confirmed basic understanding, and this does not need to be repeated.
Tasks that aim too low and ask only for basic recall/recognition/rote
memorization create little opportunity for meaningful struggle. These
tasks will often lead the most diligent students on the team to dominate
the conversations because they can simply rely on their memory or su-
perior reading skills, and less diligent students will learn that they can
freeload. This will not only undercut intellectual development, but will
also compromise team cohesion.

Team-Based Learning Institute Page 70 of 156

 Effective Task Design for the TBL Classroom 285

Assessing basic understanding is typically best suited for individuals

(in a homework task, for example). However, if the instructor does decide
to review basic understanding of concepts using team tasks, these should
minimally ask students to interpret or translate ideas and information so as
to demonstrate understanding rather than recall. Lower-level application
tasks, which ask students to transfer conceptual knowledge to concrete
situations and specific examples, also can be used to review and/or con-
firm basic understanding.
Bloom’s Taxonomy (1956) is the best-known model for classifying
learning objectives by level of intellectual challenge. The simplified ver-
sion in Figure 3 is a useful distillation of the taxonomy into three basic
cognitive levels and suggests some types of tasks that will lead to actions
corresponding to each category.
What follows are some elaborated suggestions for framing tasks that
address skills at the various levels.

Knowledge/Comprehension (framed as interpretation, transfer, and simple

application):
• Rank the following statements from most to least effec-
tive in summarizing the author’s argument about X.
(interpretation)
• Assign the following new statement to one of the three
categories identified by the author. (transfer, simple
application)
• According to the chapter, which of the following (new
statements) would be an acceptable definition of X?
(interpretation)
• According to the reading, which of the following (new
items) would be the best example of concept X? (transfer,
simple application)
• Physics: According to the reading, which kind of stress is
most likely to be at work when force is applied at point
A in the following (new) diagram? (transfer, simple
application)
• History: Now that you know the definition of “dynasty”
from the readings, which of the following (new) exam-
ples from history is most representative of the concept?
(transfer, simple application)

Team-Based Learning Institute Page 71 of 156

286 Journal on Excellence in College Teaching

Figure 3
Bloom’s Taxonomy (Simplified)  

Synthesis/Evaluation • Predict consequences

• Predict patterns
• Make judgments

Application/Analysis • Find causes

• Find patterns
•Conduct comparisons

Knowledge/Comprehension • Recall information

• Restate accurately
• Translate into new language
 

• Social Work: Which theory covered in the reading pro-

vides the best explanation of what occurred in this (new)
case of child abuse? (transfer, simple application)
Analysis (framed as comparison, contrast, analytical differentiation):
• Which factor in the given list below would you weigh
most heavily in a diagnosis of X (a new case)?
• Which of the following theories (that you just read
about) would be most useful in predicting the outcomes
of this (new) process?
• Which of the following (new) statements is consistent/
not consistent with the writer’s perspective?
• Which of the following claims about X phenomenon
could be explained/defended/refuted by an application
of Y theory?
The highest-level tasks require more complex processing and use of
knowledge. They target broader judgments that reference multiple factors
and thereby call for expert-like decision-making:

Advanced Analysis, Synthesis, and Evaluation (framed as expert-like

judgments that integrate understanding for complex decisions):

Team-Based Learning Institute Page 72 of 156

 Effective Task Design for the TBL Classroom 287

• Rank the following strategies/recommendations /

explanations in terms of which would be the most ef-
fective, in light of the theories we just read about.
• Given the facts of this scenario, and the competing
priorities, decide upon which of the following recom-
mendations you would make first.
• Analyze this new data set: Based on the theories covered
in the reading, and given what you now know about X,
which of the following explanatory hypotheses has the
most credibility?
• Based on the facts as you now interpret them, evaluate
the relative truth of the following claims by ranking
them.
As most of these latter examples show, one reliable technique for
writing higher-level tasks is to think in terms of situations, scenarios and
cases that are typically encountered in the discipline. Brookfield (2011)
provides an effective overview, with examples, of “Scenario Analysis”
techniques, in Teaching for Critical Thinking. Scenarios allow you to embed
many variables that can be used to introduce multiple concepts, theories
and perspectives into students’ discussion, as well as to complicate the
task, if desired, through a mix of relevant factors and red herrings.

Promoting Critical Thinking Through Task Design

Critical thinking is a productive consequence of intellectual frustration.
It begins to occur at that moment where knowledge, insight, reasoning,
and other assets prove to be inadequate for addressing with complete
confidence the problem at hand: Students are forced to make a decision
that stretches them. This is the moment where they will finally adopt a
critical thinking attitude and ask themselves, “What are we really sure
of? Are we making the right assumptions? Are we overlooking something
because we are biased? Have we exhausted all possibilities? Do we have
access to any additional information? What does our best judgment tell
us? What are the potential consequences of any of our possible actions?
Which of those consequences are we most willing to accept?”
The emergence of critical thinking in the TBL classroom is closely inter-
woven with the building of team coherence. Team coherence and critical
thinking both develop when students are forced to consider, respect,
evaluate, and respond to the positions and ideas of other team members.
This rarely occurs when the task is open-ended, such as in a brainstorm

Team-Based Learning Institute Page 73 of 156

288 Journal on Excellence in College Teaching

or other “generate solutions” assignments. As long is as it is possible to

believe that “one idea is just as good as another—we don’t need to eval-
uate and prioritize,” many students (and most humans!) will shy away
from the hard work of real thinking.
The function of the collective decision task, therefore, is to place a
restrictive frame around the team’s action. This restriction forces the
team to evaluate, integrate and, if needed, respectfully discount a team
member’s inputs en route to a judgment and a focused decision. A sound
idea, a persuasive line of reasoning or a convincing argument will even-
tually emerge when it withstands the critique of all team members. As the
team’s coherence develops, so do the comfort, freedom and willingness
of individual members of the team to speak frankly about the value of
any other team member’s idea.
A secondary but nevertheless important dimension of this centripetal
pressure on teams is time limitation. As long as students have the impres-
sion that a decision can be deferred or deflected (“we don’t have enough
time, so we give up”; “we need more information, so we won’t respond”)
critical thinking will not readily occur. Time limits on tasks and the expec-
tation that reporting will happen, finished or not, are therefore essential.
Finally, a crucial element of the critical-thinking process is making
mistakes: if students are to develop an attitude of persistence in the face
of difficulty, they must become experienced in confronting and reflecting
candidly on the errors in their thinking. This means that it is essential for
teachers to balance the pressures created by forced decisions and time
limits with a healthy respect for honest, thoughtful mistakes. In fact, in-
structors must force students to make errors that will create opportunities
for careful consideration of where their prior knowledge and ways of
thinking are insufficient. Creating this atmosphere requires a mix of graded
and ungraded team tasks, careful attention to team-building, and strategic
debriefing of tasks to induce productive reflection. The 4-S principles of
task design are essential to fostering this environment.

Principles of Task Design:

Elaboration on Michaelsen’s 4 S’s
We begin this section by referencing the original framework for TBL task
design, conceived first by Michaelsen, Knight, and Fink (2004) as the 3-S’s,
then later revised by Michaelsen and Sweet (2008) to become the 4-S’s:
• Significant problem
• Specific choice

Team-Based Learning Institute Page 74 of 156

 Effective Task Design for the TBL Classroom 289

• Same problem
• Simultaneous report
The longer we have worked with these principles, the more relevant and
empowering they have proven to be. Each of the S’s captures a necessary
dimension of task design and management. “Significant problem” and
“specific choice” establish how the task will be drawn from content and
structured for student action. “Same problem” and “simultaneous report”
address how the task will be administered and managed. In the following
paragraphs, we seek to build out from these principles, by elaborating
on their original rationale and by supplying some examples of how they
can be operationalized.

1. Significant Problem: Selecting Content for a Task

What is truly problematic in your field and in the content you are
teaching? What is difficult to understand fully and to resolve? In order
for students to engage with your content at a high level, they have to be-
lieve that what they are struggling to do really matters. Tasks, therefore,
need to address questions that are compelling in your discipline. The best
tasks ask students to make judgments and decisions that parallel those of
experts exposed to similar (or, at least, parallel, analogous) circumstances,
conditions and information. A truly significant problem is, ideally, one where
the teams’ responses may not fully resolve the issue; they serve mainly as
the pretext and entry point for inquiry and reflection. In fact, the very best
problems (which may or may not be within the scope of your particular
course) point toward disagreements among experts in the field—problems
where different paths can lead to credible and defensible solutions.
When the problem is significant, real learning occurs during the
debriefing of the task. If the debriefing discussion ends shortly after stu-
dents show their answers, the challenge may not have been sufficiently
problematic or, therefore, truly significant. In the best of cases, there will
be substantial disagreement among the teams, but even when all teams
have chosen the “correct” or “best” answer, a truly significant problem can
still lead to a lively discussion in the debrief, as students will still need to
explain and justify their thought processes, which may vary across teams.
Tasks that can be accomplished by applying simple knowledge in a
single-step reasoning process to arrive at an answer are unlikely to chal-
lenge students meaningfully. Similarly, tasks that simply elicit an opinion,
impression, or personal perspective will fall short of the mark. Tasks that
allow students to stumble upon a correct answer without having engaged

Team-Based Learning Institute Page 75 of 156

290 Journal on Excellence in College Teaching

in a rigorous thought process are destined to be trivial. To be effective and

authentically significant a task has to lead students to a decision point that in-
vites—and may even demand—the question “Why?” “Why?” is the doorway
to course content and disciplinary thinking—and to meaningful inter-team
conversations.

2. Specific Choice: Delimiting Student Action

Our instincts sometimes tell us that the best way to lead students toward
a full exploration of multiple perspectives is to start discussions with a
wide-ranging question or set of questions that will open several possible
avenues of inquiry. Whenever we tell students to “discuss,” we envision
that they will use the collective wisdom of their group to converge toward
meaningful possibilities. The problem with this approach among relative
novices is that they often take the conversation in directions that may not
be highly productive. Contrary to our instincts, we need to shape and stage
student conversations around tasks that more carefully direct them toward
a productive outcome, a specific choice. Figure 4 uses the image of a tunnel
to communicate the dynamic of an effective discussion. At the outset of
the process is the frame that establishes the field of action. The format of
the question structures the discussion and sets expectations for how it
will be reported. At the other end of the process is the moment of public
accountability, in the form of the proposed solution (product or decision)
that emerges from the team discussion. Between these two moments, the
teams experience a sense of relative autonomy. They are free to exploit
any means at their disposal to find and evaluate all relevant possibilities
in the process of reaching the conclusion supported by all team members.
Tasks that direct students toward a specific choice do not stifle student
thinking but concentrate it so that feedback on the task can be directed
at specific, anticipated discoveries and realizations. Restricted decision
making allows the instructor to ensure the terms of the whole class
debrief. The forced compare-and-select approach means that students
will be engaged in very specific points of analysis during the team deci-
sion-making process. A broader-ranging discussion can follow during the
task debrief, after students have begun to sort through the possibilities
that the instructor has provided.
What follows is an example of how a typical discussion prompt be-
comes a TBL question. Consider a typical group discussion prompt (from
a course in sociology):
Discuss the factors that Karlsen, writing in The Devil in the Shape
of the Woman, argues are relevant in an accusation of witchcraft.
What seems to be important?”

Team-Based Learning Institute Page 76 of 156

 Effective Task Design for the TBL Classroom 291

Figure 4
The Shape of the Learning Process in a TBL Course

A TBL decision task prompt could read as follows:

Based on your reading in Carol Karlsen’s The Devil in the Shape
of the Woman, which of the following would she consider the
most relevant factor in an accusation of witchcraft?
A. Accuser’s concern with maleficium
B. Accuser’s gender
C. Accuser’s relationship with clergy
D. Accused’s relationship with clergy
E. Accused’s age

Here we have pointed the teams’ conversations to a limited set of

possibilities, and in doing so we have ensured that students will weigh
exactly the factors we want them to weigh. If there are other issues that
are important, we will have the opportunity to bring those out in the
debrief of the team answers.
The example above demonstrates one obvious strategy for creating
specific choice tasks: multiple-choice questions. Below are several other

Team-Based Learning Institute Page 77 of 156

292 Journal on Excellence in College Teaching

formats that can also lead to simultaneously reportable, focused choices.

Ranking: Rank the following solutions in order of their plausibility
(Debrief: Report highest or lowest).
Sorting: In the envelope on your table are strips of paper, each listing
a statement about X phenomenon. Sort them according to the 4 theories
we have been studying (Debrief: Report whole solutions on poster; or,
ask students to announce by show of cards how they categorized an
individual item).
Scoring: Read the following excerpt. On a scale of 1-4, assign a score
that indicates how successfully this writer has applied X principle.
Sequencing (chronological; procedural; logical; narrative): Place the follow-
ing events from American history in chronological order; or place the
following steps in the order that represents the most effective procedure
for solving X problem.
True/False: Evaluate the following statements and decide as a team
whether they are true or false. Be prepared to explain and defend your
team’s answers:
• Humans are more highly evolved than ants.
• Over time, species evolve into better or more highly
evolved species.
(example supplied by Kristina Spaulding, Psychology, Uni-
versity at Albany)
What does not belong? Look at this slide (not shown) that lists nine
consumer behaviors. With your team, select the five (or three, etc.) behav-
iors that research has shown to be most greatly affected by an economic
downturn.
Matching: Figure 5 provides an example of a task based on matching.

3. Same Problem: Strategic Task Administration

Same problem can be one of the least intuitive elements of 4-S design,
because it runs counter to many traditional beliefs about teaching. As
information in the disciplines continues to expand, we feel increasing
pressure to “cover” as much content as we possibly can in any given class
period. Well-meaning instructors may believe that one way to achieve this
coverage—and to remove ourselves from the center of our classrooms—is
to create situations where students “teach” each other. To achieve this,

Team-Based Learning Institute Page 78 of 156

 Figure 5
Matching

Match the claims in list 1 with the correct causal mechanisms in list 2.
   

List 1: Claim   List 2: Causal Mechanism  

A. Farm subsidies increase production.   1. Slaughterhouses have become fewer and
larger.  
B. Market concentration in food production 2. The policy lowers the cost per unit, which
and distribution leads to increased
subsidies.  
leads to an increase in demand for the good.  

C. High corn yields cause negative 3. HFCS lowers the cost of soda, which
externalities, such as pollution runoff.   increases consumption.  
D. E coli infections have increased 4. Fertilizers and pesticides increase
dramatically.   production per acre.  

Team-Based Learning Institute

E. Corn subsidies increase child obesity.   5. Many former executives served in the USDA
and FDA.  
Effective Task Design for the TBL Classroom

(Example supplied by David Rousseau, Rockefeller College of Public Affairs & Policy, University at
Albany)
   
 
293

Page 79 of 156
294 Journal on Excellence in College Teaching

we divvy up tasks, asking each group to be responsible for one element

of the content and then to share their findings with the rest of the class.
But when it comes time to report and “teach” the others, there is little
intrinsic motivation for students to care or listen. Rather than inspiring
curiosity about what the other groups have to say, the divide-and-conquer
approach actually quells it. Students are forced to sit through reports and
discussions that have no immediate relevance to them.
Students are interested in what their peers have to say when they
themselves have a stake in the conversation. If all teams are at work on
the same task, the learning moment will be the debriefing of team re-
sponses, which begins with comparison of those responses across teams.
When a team can see that “We were sure we were right, but our answer
is different from everyone else’s!” they are ready to listen to their peers
and participate in a learning conversation. Their egos and emotions are
engaged. They have an authentic desire to know: “How did you arrive at
that answer? What about X? Why didn’t you consider Y?”

4. Simultaneous Report
Now that all teams are working on the same task, the logic of a dra-
matic, simultaneous report becomes evident. It is useful for the instructor
to adopt a visualization method that works well consistently: cards, post-
ers, personal response systems (clickers), whiteboard “reveal,” or other
mechanism. Experience has convinced us that cards or other visual tools
work better than clickers for this purpose. While clickers can be used to
simultaneously report team decisions, they fail to provide the crucial sense
of immediacy and dramatic ownership that comes when students hold
up cards or sheets showing how they decided, vis à vis the other teams.
Aside from the theatrical flourish that brings energy to the classroom,
simultaneous report has a more fundamental function in the learning pro-
cess: public, highly visible accountability that levels the playing field for
all students in the room. Students need to see how their thinking compares
to that of others in order to reflect candidly and self-assess. If teams are
asked to report their responses sequentially, rather than simultaneously,
students can fall into the trap of self-deception: their ideas can conve-
niently and comfortably morph to those that belong to whichever group’s
report seems most convincing or most admired by the instructor. In this
case, the opportunity for real self-assessment is lost. Sequential reporting
also introduces the risk that students will begin off-task side conversations
and fail to pay attention to or participate in the whole class discussion.
Consistently creating tasks that allow for simultaneous report is a chal-

Team-Based Learning Institute Page 80 of 156

 Effective Task Design for the TBL Classroom 295

lenge for instructors new to TBL. In some cases (for example, multiple
choice questions), report-out strategies are relatively simple to devise.
With more complex tasks, a little more creativity is sometimes required.
For example, if students are asked to create a ranked list, a simultaneous
report can begin by asking teams to show (on a card, for example) their
top one or two—or bottom one or two—choices.

Beyond the 4-S’s: Other Principles of Task Design

4S+1: Focus Tasks on Concrete Actions
Too often, we initiate discussions with students by directly referencing
abstractions they have read about, such as definitions, systems, principles,
taxonomies. A prevailing assumption is that once students master the lan-
guage of a definition or schema, they will then be able to use those tools
in their thinking and decision making. When we begin with abstractions,
however, we frequently find that students can mimic understanding by
identifying or even reciting formal definitions but may not really grasp
the implications of what they are able to recognize—and even repeat
accurately.
Students’ passive familiarity with abstract concepts will be converted
to active understanding only when it is applied and tested at the level of
concrete, specific scenarios that evoke the abstractions without necessar-
ily citing them. The economics example above of teaching “elasticity” by
means of a question about the price of ice cream is a case in point. The
earlier in the process students can be confronted with specific situations,
the more quickly they will gain traction with the abstractions.
To illustrate further, let us consider a classic approach in which an
instructor asks students to check their understanding using a multiple
choice format and bases the task on statements written in language close
to that of the textbook.
Original question:
By what mechanism does dopamine cause behavior to
increase or strengthen?
A. Dopamine causes pleasure.
B. Dopamine motivates willingness to work for rein-
forcement.
C. Dopamine predicts the arrival of a reinforcer.
D. None of the above

Team-Based Learning Institute Page 81 of 156

296 Journal on Excellence in College Teaching

A student responding to this question is likely to recognize a correct an-

swer that echoes the language of the reading (“reinforce”), but familiarity
with the language does not indicate that students can apply the concept.
To do so, the task would need to be more concretely situated:

Revised question:
Sara finds that she cannot stop eating chocolate. Which
of the following explanations is the most credible?
A. It causes Sarah to feel pleasure.
B. It increases Sarah’s motivation to seek out and eat
chocolate.
C. It creates a sense of anticipation for something good
(chocolate).
D. None of the above
(example supplied by Kristina Spaulding, Psychology, Uni-
versity at Albany)
Students who can answer this question accurately are likely to have
demonstrated an understanding of how dopamine works, because they
cannot slide by with simply parroting textbook language.
What we know about the nature of learning is that students gain deeper
traction, faster, with course content if their first encounters with it include
concrete experiences framed by and informed by the abstractions. As we
move through a learning sequence or cycle, tasks may eventually become
more abstract, but students need to start with decisions that make real and
visible the significance and implications of targeted concepts.

4S+2: Worksheets Are for Individuals; Decisions Are for Teams

In the interests of efficiency, we may be tempted to present a sequence
of small tasks all at once. Designing tasks that lead students through a
complete thought process is an essential strategy, but giving teams sev-
eral tasks at the same time on a single handout or a worksheet will lead
to behaviors that TBL is specifically designed to prevent (for example, a
dominant student taking over or a “divide and conquer” approach).
Teams are effective when their tasks drive them to converge collective-
ly on a single decision. If we really want teams to work through a suite
of tasks, we will need to isolate each one as a separate decision, with

Team-Based Learning Institute Page 82 of 156

 Effective Task Design for the TBL Classroom 297

simultaneous report at each step of the way. In this case, they should be
scaffolded, one upon the other, each leading to decisions with greater
complexity and integration of learning.
If the suite of ideas cannot be represented as a sequence of discrete
team decisions, consider assigning the earlier parts of the sequence to
individuals to work on separately before assigning teams a decision-mak-
ing task. This approach works well in courses that require students to
practice quantitative calculations. Students work through the necessary
calculations individually, then convene as a group to make a broader,
more conceptual judgment that is based on the collective understanding
gained from individual work.

4S+3: Plan the Debrief When You Plan the Task

The design of a task is ultimately only as good as its execution and
management. If you have not anticipated what students’ responses to
the task will be, you may not be ready to debrief their decisions effec-
tively. What if everybody agrees or gets it right? What if everyone gets
it wrong? A task that adheres to 4-S design and works well on paper can
be completely derailed in the classroom by the unexpected. Having some
strategies in mind can help to avoid this problem.
Have a concrete plan for simultaneous report—and make sure not
to follow a simultaneous report with a sequential report of each team
explaining its answer. Cluster answers during the debrief: “I see that
several of you said ‘A.’ Team 2, what was your reasoning for ‘A’? Ok, did
any teams have a different reason for answer ‘A’? Team 4, you said ‘B’;
why?” While it is important to bring to the surface the different reasons
for why teams arrived at their answers, polling each team in sequence
undermines the purpose of simultaneous report. If every team gets the
correct or best answer, the debrief of team answers will proceed very dif-
ferently than a situation where there is a wide variation in answers. An
instructor must assess where a deeper analysis of multiple team answers
is required and where it is superfluous or repetitive.
Defer the reveal of a correct or best answer, if there is one, until you
have debriefed the teams’ responses—“as if” all responses are possible.
In some cases, you may even want to leave the problem unresolved, so you
can send students back into their teams (or back to the texts) to reconsider
their thinking via a new question. Once the instructor has stepped in and
offered the “correct” answer, meaningful discussion has ended because
the expert has spoken. There is an essential difference between asking stu-
dent teams how they arrived at an answer that might be right, and asking
them how they arrived at their answer if they already know it is wrong.

Team-Based Learning Institute Page 83 of 156

298 Journal on Excellence in College Teaching

Know where students are likely to struggle with a task, but be flexible
when your prediction isn’t on target. Out of respect for students, you
may have to let them go a ways down the wrong path before you redirect
them to more productive territory. As instructors, we are often made un-
comfortable by student errors, and we feel responsible for immediately
correcting them. However, intervening too early can diminish a team’s
sense of ownership of their own responses.

4S+4: Logistics and Management Matter

The problems of typical “group-work” are much more likely to arise
if students are unclear about what is expected of them. We have found
that some basic logistical strategies are useful in keeping teams focused
and engaged.
Give clear directions for each task in writing (ideally projected on the
classroom screen). If there is to be a series of tasks, show directions for
each separate step/sub-task on a separate slide. Keep directions visible
while students are working. This serves several purposes: First, writing
out each step of the directions in advance forces you to think through the
process in which you want students to engage; second, you don’t have to
remember all the steps in class because they are right in front of you; third,
teams are now able to work autonomously (and you don’t have to repeat
the directions individually to each of them). Students will quickly learn
that they, not the instructor, are responsible for keeping their teams on task.
Use time limits—and make them visible. If students feel that a con-
versation can continue ad infinitum, especially with a complex question,
they will defer making a decision (and spend a lot of time trying to con-
vince you that they can’t reach a decision). Requiring teams to produce an
answer—publicly—within a given time helps them maintain focus and
also sends the message that “we can’t” isn’t an option. To create an even
greater sense of urgency, always allow less time than you think they really
need to answer a given question. The energy that is created by a good task
can quickly be depleted by lag time when teams finish before time is up.
Finally, this is also a strategy for encouraging teams’ autonomy and ac-
countability. Rather than depending on you to remind them of how much
time remains to complete a task, students learn that they need to track
their team’s progress and arrive at answers in the given amount of time.
Practice team tasks from day one. There are several good reasons to
have students engaging with challenging team tasks from the first day of
class, but one of them is to get them accustomed to the level of autono-
my and accountability they will be facing in a TBL classroom. For many

Team-Based Learning Institute Page 84 of 156

 Effective Task Design for the TBL Classroom 299

students, this is an adjustment, and there will be some growing pains

associated with the process. Starting right away with content-driven,
meaningful tasks (even before you discuss the syllabus!) will help to
demonstrate not only why you have structured your course in the ways
you have but also how students are expected to handle the process. Naïve
tasks work especially well on Day One.

4S+5: Use Non-4 S Tasks Sparingly, but Strategically

A key function of 4 S design is building team cohesion. After teams
have begun to perform effectively (often around or after the midpoint of a
semester), you may find it possible to mix in tasks that ask for a more com-
plex product. You should continue using 4 S tasks frequently to continue
team building, but pushing teams to engage in more synthesis/creation
tasks may require more flexibility with task design. For this purpose, we
propose a few practices that, when teams are already functioning at a high
level, can work with some consistency. Note that many of these examples
still allow for and suggest using simultaneous report; keep this in mind
any time you diverge from a strict 4 S structure. Having the opportunity
to compare products across teams remains powerful, even when those
products are complex.

Limited Word Task: Teams are asked to distill a complex idea or set of
ideas into a single word or limited number of words (1, 2 or 3)
Example: Given the situation described in the case study
you just read, use 3 words to summarize the first actions
a therapist would need to address in responding to this
patient. When prompted, send a team member to the
board to write your 3 words.
Single Claim Task: Similar to the single word task, teams are asked to
summarize an argument in a single clause sentence/thesis.
Example: Read the paragraph on the handout and, as
a team, summarize its primary argument in a single
sentence. When prompted, send a team member to the
board to write your sentence.
Construct a Thesis: Teams are given a context and asked to take a stance
on an issue and construct a thesis statement that they would use to make
a written argument.

Team-Based Learning Institute Page 85 of 156

300 Journal on Excellence in College Teaching

Example: Using the example of one character in the

novel, write a thesis statement to defend or refute the
following claim: “In James Baldwin’s novel Go Tell it on
the Mountain, the Church is ultimately a positive force in
the characters’ lives because it provides an empowering
community and a place where individuals can express
themselves.” When prompted, send a team member to
the board to write your thesis.
Framing an Argument: Adapted from Bean’s (2001) frame paragraph
exercise, this is an expansion of the “construct a thesis” exercise where
in addition to creating their thesis, teams are asked to identify the sub-ar-
guments they would use to flesh out their argument.
Example: After you have created your thesis statement,
list four arguments (in the form of a topic sentence) in
support of your thesis statement with at least one spe-
cific piece of evidence from the text, with page number,
which you would use to support each.
Diagram or Image Task: Teams are asked to distill and represent a complex
set of relationships into a single image, diagram, or flow chart, which is
drawn on a large sheet of poster paper. Using a pre-established cue, all
the posters go up at the same moment for simultaneous report. Debriefing
can be traditional (instructor asks teams to explain their representation),
or can be adapted to practices where students comment on each other’s
works (e.g., gallery walk) using stickers or other tools.
Example: Design a flow chart predicting the sequence
of physical and mental actions of children solving the
following problem. . . .

Reports and Debriefs for Non-4-S Tasks

Tasks with more elaborated products may require some invention when
it comes to having teams report and compare their answers. Rather than
have students report sequentially, there are other strategies available for
reporting that retain the energy and focus of simultaneous reporting. One
of these is the technique known as the “Gallery Walk,” in which teams
write their products on large sheets of paper and attach them to the wall
in the manner of an art gallery. Students (either in teams or as individuals)
then pass around the room and record their evaluation or comments for
each product. Numbers or other mechanisms can be used to rank products

Team-Based Learning Institute Page 86 of 156

 Effective Task Design for the TBL Classroom 301

according to various criteria. In this way, the assessments of the products

can be reported simultaneously, for example:
• Hold up a card/number for the poster that represents
________ most clearly.
• Hold up a card for the poster that is most/least _______.
Another approach is for students to attach colored stickers to posters
according to given criteria. The reporting then follows from identifying
the posters with the most stickers of a given color. Other excellent tech-
niques for reporting and assessing complex team tasks, such as “Stacked
Transparencies,” “Hot Seat,” “Best Solution Tournament,” can be found
in Appendix 2 of Sibley and Ostafichuk’s Teamwork That Works: Guide to
Implementing Team-Based Learning (2013).

Conclusions
Effective task design can be daunting and time-consuming because it
requires a new perspective on both student activity and the content of
your course. For this reason, it is important to enter TBL with an attitude
of exploration and reflection: Tasks that “don’t work” are often very
valuable as they give you the opportunity to re-consider your goals and
your approach. Just as we advocate for creating a classroom atmosphere
where students come to recognize the role of errors in the learning pro-
cess, we believe that instructors must enter their own TBL courses with
the expectation that there is room to learn and grow.
Thinking analytically about what you expect a task to accomplish, the
kinds of thinking it is seeking to promote, how it is constructed to induce
student action, and the responses you expect from students—these are
not only crucial to success in the classroom, but are also key to becoming
more facile with the process of task design. After you have experimented
with different task structures, based on the principles and strategies dis-
cussed in this chapter, you will discover what works for your classroom,
your students, and your content. Experience will also help you hone your
instincts about where modifications will make tasks more successful.
Having just a few of these formats under your belt will ultimately make
task design more navigable with each successive implementation.

References
Bean, J. C. (2001). Engaging Ideas. San Francisco, CA: Jossey-Bass.
Bloom, B. J., & Krathwohl, D. R. (1956). Taxonomy of educational objectives:

Team-Based Learning Institute Page 87 of 156

302 Journal on Excellence in College Teaching

The classification of educational goals, by a committee of college and university

examiners. New York, NY: Longmans, Green.
Bransford, J. D., Brown, A. L., Cocking, R. R., Donovan, M. S., & Pellegri-
no, J. W. (2000). How people learn: Brain, mind, experience, and school
(expanded ed.). Washington, DC: National Academy Press.
Brookfield, S. D. (2011). Teaching for critical thinking. San Francisco, CA:
Jossey-Bass.
McLeod, S. A. (2010). Kolb’s Learning Styles and Experiential Learning
Cycle—simply psychology. Retrieved from [Link]
[Link]/[Link]
Michaelsen, L. K., & Knight, A. B. (2004). Creating effective assignments:
a key component of team-based learning. In L. K. Michaelsen, A. B.
Knight, & L. D. Fink (Eds.), Team-based learning: A transformative use of
small groups in higher education (pp. 51-72). Sterling, VA: Stylus.
Michaelsen, L. K., & Sweet, M. (2008). The essential elements of team-based
learning. New Directions for Teaching and Learning, 116, 7-27.
Sibley J., & Ostafichuk P. (Eds.). (2013). Teamwork that works: guide to im-
plementing team-based learning. Sterling, VA: Stylus.
Willingham, D. T. (2009). Why don’t students like school? San Francisco,
CA: Jossey-Bass.
Zull, J. T. (2002). The art of changing the brain. Sterling, VA: Stylus.
Zull, J. T. (2011). From brain to mind. Sterling, VA: Stylus.

Bill Roberson, Ph.D., directs the teaching and learning center that serves the Albany
campus of the State University of New York. A former faculty member and now faculty
developer, his career has focused on the integration of critical thinking into the university
classroom. To that end, he has been a practitioner of team-based learning since 2000,
and he has consulted with faculty at approximately 80 institutions in North and South
America and Europe on course design for critical thinking, active learning, assessment
of teaching, and the use of TBL to promote critical thinking. He has held positions in
faculty development at the University of North Carolina - Chapel Hill, Indiana Univer-
sity, and the University of Texas at El Paso, where he was founding executive director of
that university’s division for instructional technology, classroom design, digital media
production, and distance learning. He came to New York in 2006 to create the Institute
for Teaching, Learning, and Academic Leadership at the University at Albany, State
University of New York ([Link]). Billie Franchini, Ph.D., is assistant director
of the Institute for Teaching, Learning and Academic Leadership at the University at
Albany (SUNY). She entered faculty development after more than a decade of teaching
at the high school and university levels. She teaches both undergraduate and graduate
courses using TBL and has worked with scores of faculty to support them in implementing
TBL in their own courses. Examples and other evidence of her work can be found on the
Institute’s website, [Link].

Team-Based Learning Institute Page 88 of 156

Assigning Students to Groups for
Engineering Design Projects: A Comparison
of Five Methods
LT COL JAMES L. BRICKELL from a low Perry-level* to a relatively high Perry-level empha-
sizing team work, communication (oral and written), and basic
LT COL DAVID B. PORTER comprehensive planning tools for the design of Air Bases for
LT COL MICHAEL F. REYNOLDS the Air Force.
CAPT RICHARD D. COSGROVE The course is divided into three distinct blocks: (1) The
United States Air Force Academy framework for designing much of the base infrastructure...run-
Colorado Springs, CO way, taxiway and apron/pad designs, rigid and flexible pave-
ment designs, as well as utility systems requirements, (2) Base
Comprehensive Planning (BCP) and related socio-political
and environmental concerns...noise, Air Installation
I. INTRODUCTION Compatible Use Zones (AICUZ), and encroachment, and (3)
The fact that group learning can enhance education is well air base performance in a combat environment...force bed-
established. No less an authority than Albert Einstein1 argues down, pre-attack planning, and base recovery after attack.
persuasively for the essentially collaborative nature of human Each block culminates in a group project requiring the stu-
learning. Likewise, Harold Leavitt suggests that all organiza- dents’ synthesis of all the material covered in that block. The
tions must consider the influence of informal groups on critical course also has quizzes and examinations in order to test the
processes: students’ subject knowledge. Overall, 44% of a student’s final
‘‘The problem is not shall groups exist, but shall groups be grade was based on group effort and 56% on individual effort.
planned or not? If not, the individualized organizational gar- For the Fall, 1991 semester, there were 442 students
den will sprout groupy weeds all over the place.2” enrolled in 24 sections of CE 310. Within each section stu-
More recently, researchers such as Bruffee3, and Johnson dents were divided into work groups of approximately four stu-
and Johnson4 have explored the positive contributions groups dents. Each section had a maximum of 20 students. The course
can make to education in much greater detail. Porter5 asserts was taught by eight instructors.
that to be its best, education must become a team sport. The
Harvard Assessment Seminars found particularly strong sup-
port for the use of groups in higher education.6 III. EXPERIMENTAL DESIGN
How group assignments should be made to optimize stu- Our intention was to test whether grouping students
dent learning is far from clear. Numerous methods of group according to ability (as indicated by their incoming GPAs) or
assignments can be used, ranging from allowing the students to curricular interests (technical or nontechnical) affected student
select their own groups to the instructor making the group performance, attitudes, or efficiencies when compared with
assignments using a variety of different criteria. This paper allowing students to select their own groups. Each class, or
presents the results of an investigation into the effectiveness of section, was first grouped by interest: the “technical” group
group performance using five different methods of assigning were those students majoring in engineering or the basic sci-
students to work groups. The investigation took place at the ences, and the “nontechnical” group were those students
United States Air Force Academy during the Fall semester majoring in humanities or the social sciences. A third group
1991, and involved a core course taught by the Academy’s was identified, neither technical nor nontechnical, that consist-
Department of Civil Engineering, CE 310, “Air Base Design ed of students who were meeting the minimum graduation
and Performance.” requirements. This group (Bachelor of Science, BS) would fill
In Civil Engineering 310 students work on course projects a technical or nontechnical position based on the specific needs
in groups. Before this study, there were no standards for of a section to completely fill all groups. After the class was
assigning students to these groups; it was left to each individual separated by interest, the individuals were then rank-ordered
instructor’s discretion, wondering if certain methods of select- by GPA. The desired number of students in each group was
ing groups might be more effective than others. four. In most cases, each group had four students assigned,
however, for classes with less than 20 students, some
three-person teams were created.
II. COURSE BACKGROUND
*For a complete discussion of Perry Level, see: Culver, R.S. and J.T. Hackes.
CE 310 is a core curriculum requirement for all junior-level
Perry’s Model of Intellectual Development. Engineering Education: 221-226
cadets at the Academy. The course was designed to progress (December 1982)

July 1994 Journal of Engineering Education 259

Team-Based Learning Institute Page 89 of 156

 The five methods of group assignments are illustrated in were five sections for each method of assignment except for
Table 1. Method 1 involves assigning groups heterogeneous- method 1 (heterogeneous/ heterogeneous) which only had
ly/heterogeneously with respect to GPA and Interest, i.e., each four. The five groups in each class or section were named
group has both technical and nontechnical majors, and the Eagle (E), Falcon (F), Ice (I), Maverick (M), and Yankee (Y),
GPAs of all group members are different. Those groups and the groups were selected such that each group had similar
assigned by method 2 (heterogeneous/homogeneous) are com- average GPAs.
prised of only technical or nontechnical majors and the GPAs The instructors were informed that an experiment was
of all group members differ. For method 3 (homogeneous/het- under way; however, they were not informed as to how each of
erogeneous), each group again has both technical and nontech- their section’s groups were selected. Instructors teaching multi-
nical majors, but their GPAs are all similar. Method 4 (homo- ple sections were likely to have different grouping strategies
geneous/homogeneous) has groups assigned with either all used in each section. The students were not told they were a
technical or all nontechnical majors, and all group members part of an experiment. Group assignments were accomplished
have similar GPAs. Method 5 was the control; students select- by the principle investigators, and given to the instructors.
ed their own teams. Grading of the projects was standardized among instructors
All groups within a particular section were assigned using by providing detailed guidance to each instructor so as to make
the same method. The methods varied among sections. There the process as objective as possible. In addition, the Course
Director (the person responsible for administering the course)
spot-checked each instructor’s graded projects to ensure all
projects were graded similarly.

IV. DATA REDUCTION

The following information for each student was collected.
1. Name
2. Squadron
3. GPA
4. MPA (Military Performance Average)
5. Major
6. Type group (1,2,3,4,5)
7. Civ Engr 310 Grades
A. Group only average
Table 1. Methods of Group Assignments B. Individual effort only average
C. Overall average
8. Results of students critiques
A. Criteria
B. Instructor
C. Projects
D. Classmates
E. Course
9. Time Studies (The amount of time each group spent
accomplishing the projects after Block 1-the 1950s Project,
and Block 2-the 1990s project)
Items 1-5 and 8 were obtained from the Academy’s
Registrar, and items 6, 7 and 9 from data collected from the
course.
From the data, the following hypotheses were tested:
1. The method of group assignment affects individual and
group performance.
2. The method of group assignment affects students’ atti-
tudes toward the material, the course, the instructor or their
classmates.
3. The method of group assignment affects group efficiency.
Individual grades, group grades, and overall grades were
compared across assignment methods. Likewise, student atti-
tudes regarding criteria, instructor, projects, classmates, and
the course were compared by analyzing responses to an end of
course critique as shown in Table 2. The time required to
Table 2. Course Critique
complete both the 1950s and the 1990s projects were also

260 Journal of Engineering Education July 1994

Team-Based Learning Institute Page 90 of 156

compared across groups. A T-test7 was used to determine if are statistically (P<0.05)(7) different from the values for the
differences from the control (self-select) were statistically sig- control group (self-select).
nificant. An F-test7 was used to determine if there was evi-
dence that the variations between the control and other groups
VI. DISCUSSION OF RESULTS
were not equal. If there was evidence that variations were not
equal, an estimated T-statistic7 was used to determine if differ- The results of the grade study are shown in Figure 1. Once
ences from the control were statistically significant. again, the methods of assignment are as previously defined
with method 5, self-select as the control. For grades deter-
mined from individual effort, there were no statistically signifi-
V. RESULTS
cant differences between the control and any of the other
The results of this study are summarized in Table 3. The methods. Likewise for the overall total grades. For group
first number for each entry represents the mean value of the grades, though, students grades were significantly higher for
parameter. The number directly below the mean represents the methods 2 and 3, while there was no significant difference for
sample variance. The asterisks identify parameter values that methods 1 and 4 compared with the control. Also, from Table
3, it can be seen that method 2 has the lowest variance in
grades. The higher grades with lower variance indicates a bet-
ter, more consistent performance among groups as well as
across group members. The significantly higher variations for
methods 1 and 4 indicates a larger range of group perfor-
mance. It may be for method 2 that students of similar inter-
ests find it easier to communicate and work with each other.
Also, different GPAs may influence group organization with a
natural leader (high GPA) and those that are more likely to
prefer to be assigned tasks to complete their portion of the
project. This may also impact the group and allow it to per-
form more efficiently and effectively. Nonetheless, it is appro-
priate to conclude that group selection had only slight effects
on graded performance.
The results of the attitudes study are shown in Figure 2.
For criteria, responses for methods 1, 2 and 4 are statistically
significantly higher than the control, while there was no signif-
icant difference for method 3. The same results occurred
when considering the responses rating the instructor. The
only significant increase in ratings for the projects was for
method 2. The only significant increase in responses toward
classmates was shown by method 3. The responses for meth-
ods 1, 2 and 4 were significantly higher for the overall course
Table 3. Summary of Results rating. Although not all differences were significant, method

Figure [Link] Comparisons Among Methods of Group Figure 2. Attitude Comparisons Among Methods of Group
Assesment Assignment

July 1994 Journal of Engineering Education 261

Team-Based Learning Institute Page 91 of 156

5, the self-selected control, generally has the lowest ratings of
all methods tested across the five categories. In most cases, as
seen in Table 3, method 5 also yields the largest variances in
responses. It appears that for self-selected groups there is a
large range of attitudes about all aspects of the course, with an
average rating below that of the other methods of group
assignments. No direct measures were taken concerning how
students self-selected in method 5, but it is reasonable to
assume previous acquaintances and proximity were factors. Table 4. Instuctor Assignments by Method
Thus, self-selected groups may be more “social” entities than
other groups. This type of group may actually encourage dis- VIII. CAVEATS
content about all aspects of the course (including the instruc- 1. Results may be contingent on the nature/type of group
tor). The results of the Time Study are shown in Figure 3. tasks. Generalization to other courses should be done cau-
The only statistically significant differences from the control tiously.
occurred for method 4 for the 1950s project and for methods 2 2. The greatest advantage in the attitudes for methods 2
and 4 for the 1990s project. Students assigned by method 2 and 3 appears in the instructor category. Since this was a fac-
have the least amount of time invested in the projects, while tor that was supposed to be randomized across methods, it
those assigned by method 4 have the greatest amount of time. might reflect a nonrandom distribution of instructors. A
It is also interesting that most of the four selected groups took review of the instructor assignments resulted in the distribu-
a little longer on the first project but then appeared to increase tion shown in Table 4
their efficiency during the second project.

REFERENCES
VII. CONCLUSIONS
1. Einstein,A. Ideas and Opinions. Bonanza Books. New York
1. Appointed groups with a mixture of homogeneity and (1984).
heterogeneity perform better (earn higher group grades) when 2. Leavitt, H.J. Suppose We Took Groups Seriously... in Staw,
compared with self-selected groups. B. (ed.) Psychological Foundations of Organizational Behavior.
2. Allowing students to select their own groups results in Goodyear, Santa Monica, California (1977).
the poorest attitudes about the course, their instructors, the 3. Bruffee, K.A. The Art of Collaborative Learning. Change.
projects, their classmates, and other criteria. (Mar/Apr 1987), pp.42-47.
3. Method 2 of group assignments, heterogeneous with 4. Johnson, D.N. and R.T. Johnson, The Socialization and
respect to GPA and homogeneous with respect to interest, Achievement Crisis: Are Cooperative Learning Experiences the Solution?
appears to be the most effective method of group assignment in L. Brickman (ed.), Applied Social Psychology, Annual 4. Sage,
when considering: (1) group performance (group grades), (2) Beverly Hills, California (1983).
attitudes about the course and its administration, and (3) effi- 5. Porter, D.B. Total Quality Education. in Lam, K.d.; Watson,
ciency in the use of time for this particular course. F.D. and Schmidt, G.R. (eds.) Total Quality: A Textbook of Strategic
Quality Leadership and Planning. Air Academy Press, Colorado
Springs, Colorado (1991).
6. Light, R.L. The Harvard Assessment Seminars: Explorations
with Students and Faculty About Teaching, Learning and Student Life.
Howard University, Combrite, Massachusetts (1990).
7. Scheaffer, R.L. and McClare, J.T. Statistics for Engineers.
Duxbury Press, Boston, Massachusetts (1982).

Figure 3. Time Studies Comparisons Among Methods of

Group Assignment

262 Journal of Engineering Education July 1994

Team-Based Learning Institute Page 92 of 156

Building  your  first  TBL  Module  
Homework  Assignment    

 
 

Part  One:  Design  a  TBL  Module  

Your  task  is  to  design  one  TBL  module  (backwards  of  course!).  Use  the  following  questions  to  
guide  your  effort.    
 
1. D EFINE  MODULE  LEARNING  O UTCOMES    
 
What  will  your  students  will  be  newly  able  to  do  as  a  result  of  this  module  and  its  activities?    
 
Your  answer  to  this  question  will  be  your  module  learning  outcomes.  
Write  2-­‐3  outcomes  as  concrete  actions,  for  example:  
 
• In  a  political  science  course,  you  might  want  students  to  become  effective  at  
analyzing  and  assessing  the  factors  that  contribute  to  political  corruption.  
• In  a  construction  management  course,  you  might  want  students  to  become  skilled  
at  interpreting  drawings,  or  analyzing  a  project  plan  for  budgetary  purposes.  
• In  a  geology  class,  you  might  want  students  to  be  able  to  infer  from  specific  
geological  landscapes  the  probable  causes  and  history  of  their  formation.  

Support  Resources:  Writing  Learning  Outcomes  and  Bloom’s  Taxonomy  sections  of  
Creating  a  TBL  Module  Thread,  TBL  book  p.  18,  116-­‐118  

2. IDENTIFY  CONTENT  THAT  W ILL  BE  USED  IN  M ODULE  

Make  a  list  of  the  concepts,  ideas,  principles,  perspectives,  core  facts,  and  other  
information  that  students  need  in  order  to  be  ready  to  accomplish  the  learning  outcomes.  
These  are  normally  things  that  would  be  covered  in  the  reading  assignment.  
 

Team-Based Learning Institute Page 93 of 156

 3. BUILD  4S  A CTIVITIES  
Now  it  is  time  to  take  those  concrete  actions  described  in  the  learning  outcomes  and  turn  
them  into  4S  task/activities.  You  want  to  design  a  sequence  of  in-­‐class  4-­‐S  tasks  (at  least  
3)  that  your  students  will  do  in  order  to  get  some  practice  thinking  in  this  new  way  and  
developing  their  analysis  and  judgment  skills.  You  might  want  to  scaffold  the  tasks  from  
simple  to  more  complex.  Make  sure  these  are  challenging  and  require  more  than  simple  
recall  of  “correct”  information.  It’s  OK  if  there’s  a  best  or  “correct”  answer,  as  long  as  
there’s  enough  ambiguity,  complexity  and  doubt  for  deep  discussion.  
 

 
 
For   each   activity,   create   you   will   need   to   step-­‐by-­‐step   instructions   that   highlight   what  
students   will   need   to   do   to   complete   the   task.   Write   down   the   actual   instructions   and  
prompt/question  you  will  use  to  direct  students  through  the  task.    
 
Support  Resources:  Creating  a  TBL  Module  Thread,  TBL  book  p.114-­‐142,  186-­‐196  
 
 
4. CREATE  R EADINESS  A SSURANCE  TEST  
The   Learning   Outcomes   and   4S   tasks   create   the   major   shape   for   the   module.   Now   it   is  
time   to   identify   more   specifically   what   students   will   need   to   read   (or   watch,   if   you   use  
videos)   in   order   to   acquire   the   resources   and   tools   (information!)   needed   to   begin  
participating   in   the   activities   you   have   designed.   What   are   the   (best)   chapters,   articles,  
pages,  clips,  etc.  that  convey  the  targeted  ideas?  Make  a  list  of  these  items,  which  you  will  
eventually  give  to  students  as  their  reading  assignment.    
 

Team-Based Learning Institute Page 94 of 156

  
 
How   much   is   appropriate   varies,   but   keep   in   mind   that   you   want   students   to   read   for  
basic   awareness   and   understanding,   not   mastery.   The   reading   needs   to   be   rich   enough  
and  long  enough  to  provide  starting  content  for  several  activities  over  the  length  of  the  
module.    
 
Now   write   5-­‐10   multiple-­‐choice   questions,   based   on   the   reading,   that   focus   on   your  
target  content.  Like  the  readings,  the  test  is  about  first  exposures  and  ensuring  readiness,  
not   mastery.  
 
• Create  some  questions  at  the  level  of  recall  and  understanding  
• Create  some  questions  at  the  level  of  application  and  analysis  (comparison,  contrast,  
for  example)  
• Create  a  few  items  that  are  likely  to  generate  some  debate  and  discussion  

Supporting  Resources:  Writing  multiple  choice  questions  and  Bloom’s  Taxonomy  sections  
of  Creating  a  TBL  Module  Thread,  TBL  book  p.  74-­‐113  
 

5. C ONSIDER  HOW  YOU  WILL  EVALUATE  THE  LEARNING  

How   will   you   know   what   the   students   can   do   at   the   end   of   a   module   and   if   they   have  
achieved  the  learning  outcomes?  
 
There  are  many  possibilities  (select  at  least  one  for  this  assignment):  
 
• Individual  assignments  
• Traditional  individual  testing  –  quiz,  midterm,  final  examinations  
• Team  analysis  worksheets  completed  as  part  of  4S  task  

If  you  want  to  use  an  assignment,  write  down  the  actual  assignment  prompt.    
 
If  you  are  using  testing,  write  down  2-­‐3  of  the  actual  higher-­‐level  questions  that  you  will  
use  for  your  final  evaluation.  
 
If  you  are  going  to  have  teams  complete  a  worksheet  that  highlights  their  analysis,  write  
down   the   prompts   you   will   use   to   instruct   students   on   what   you   want   them   to   do   and  
record.  
 
 

Team-Based Learning Institute Page 95 of 156

Part  Two  -­‐  Prepare  a  micro-­‐summary  of  your  module  
You  will  use  this  summary  to  present  your  work  with  your  colleagues  and  get  feedback.  
 
• 1-­‐page  version  of  the  module  (ONE  SIDE  of  ONE  SHEET)  
 
• Teammates  will  read  this  as  a  way  of  getting  a  picture  of  what  your  plans  are.  
 
• Please  bring  5  copies  of  this  1-­‐pager.  Email  Jim  Sibley  ([Link]@[Link])  by  7  
am  Thursday  morning  if  you  want  copies  printed  for  you.    

Your  Micro-­‐Summary  should  include:  

 
1. A  few  action  oriented  learning  outcomes  
 
2. List  of  2-­‐3  key  concepts,  ideas  or  other  critical  information  that  students  will  need  
to  get  from  the  readings  and  a  description  of  the  reading/prep  assignment  
(sources)  
 
3. One  RAT  question    
 
4. One  in-­‐class  4-­‐S  activity  –  be  prepared  to  point  out  where  each  of  the  4S’s  is  
present  in  the  activity.  
 
5. Description  of  final  assessment  strategy  
 
6. Any  other  concern  or  issue  that  you  would  feedback  on.  
 

Team-Based Learning Institute Page 96 of 156

Creating  a  TBL  module  thread  
When  we  first  try  to  envision  a  TBL  module,  it  can  be  helpful  to  imagine  the  module,  as  a  
tapestry  of  learning.  Our  first  task  is  to  create  the  threads  that  the  tapestry  will  ultimately  
be  woven  from.  A  single  thread  will  link  a  Learning  Outcome,  to  a  4S  activity,  to  Readiness  
Assurance  questions,  finally  to  a  reading.  

The  Learning  Outcomes  defines  where  we  want  student  to  go,  the  careful  combination  of  a  
reading  and  Readiness  Assurance  questions  get  students  ready  to  apply  what  they  
abstractly  know,  and  the  powerful  4S  learning  tasks  then  lets  students  put  their  knowledge  
to  work  in  the  concrete  world  and  show  us  they  know.  

We  design  backwards  -­‐  first  by  defining  the  desired  Learning  Outcomes,  then  turning  our  
attention  to  the  creation  of  a  4S  task,  and  finally  selecting  the  readings  and  creating  the  
Readiness  Assurance  questions  to  ready  the  learners.  

Step  1-­‐3:  Create  Learning  Outcome  

Step  1:  Create  a  Learning  Outcome  
 
Define  Flood  Return  Period  

This  is  a  very  typical  content  focused,  low  Bloom’s  level  learning  outcome.    
Step  2:  Raise  the  Bloom’s  level  of  Learning  Outcome  
 
Explain  Flood  Return  Period  

In  this  step,  we  raise  the  Bloom’s  level.  See  page  6  of  this  section  for  different  verbs  you  
can  substitute  to  target  higher  Bloom’s  levels.  In  this  example,  the  change  has  moved  the  
learning  outcome  from  the  lowest  Bloom’s  level  –  Remember  -­‐  to  the  slightly  higher  
Understand  level.    
Step  3:  Make  Learning  Outcome  about  Action  
 
Demonstrate  understanding  of  Flood  Return  Period  

In  this  step,  we  change  the  learning  outcome  to  be  about  concrete  action  rather  than  
abstract  understanding.    

But  in  this  example  selecting  a  verb  like  “demonstrate”,  doesn’t  provide  any  information  
on  how  the  students  might  “demonstrate  understanding”.  It  is  time  to  think  about  how  to  
make  student  thinking  more  visible.  We  do  this  by  imaging  a  4S  task  that  will  give  students  
the  opportunity  to  use  what  they  know,  extend  their  knowledge,  and  finally  show  us  they  
know  (achieve  Learning  Outcome).  
We  now  want  to  think  about  our  discipline  and  the  kinds  of  questions  experts  are  routinely  
asked  to  make,  the  kinds  of  data  they  work  with,  the  inferences,  judgments,  and  decision  

  Page  1  

Team-Based Learning Institute Page 97 of 156

they  are  required  to  make.  These  will  ultimately  be  the  source  of  your  inspiration  of  what  
makes  a  great  4S  task.  

Step  4-­‐5:  Design  4S  Team  Tasks  

Step  4:  Select  a  4S  question  prompt  
 
It  is  worth  remembering  we  want  the  question  prompt  to  constrain  the  decision  space,  so  
the  reporting  discussion  is  more  focused  on  salient  issues.  Think  about  the  difference  that  
would  occur  when  we  imagine  the  report  conversation  for  what  would  be  the  best  thing  
to  do  in  this  situation  versus  the  more  diffuse  what  would  you  do  it  this  situation?    

Combining  the  identified  disciplinary  action  with  the  desired  Learning  Outcome,  it  is  time  
to  go  shopping  for  a  4S  question  prompt.  You  want  one  that  contains  a  superlative  (like  
best  above)  that  will  make  students  analyze,  discriminate,  and  finally  report  a  reasonable  
choice  or  course  of  action.  See  page  14  of  this  section  for  a  list  of  possible  4S  prompts.    
Step  5:  Use  template  to  create  4S  question  parts  
 
Once  we  have  a  question  prompt  in  mind,  we  can  start  completing  the  other  pieces  of  a  
complete  4S  question.  It  is  worth  noting  that  the  specific  detail  you  add  to  the  scenario  can  
guide  students  to  analyze  the  problem  a  certain  way  using  the  provided  detail  or  pointers  
to  data  sets.  Similarly,  the  mix  of  different  course  of  action/decisions  possibilities  can  have  
students  naturally  examine  the  situation  from  a  specific  set  of  perspectives  that  you  have  
intentionally  pointed  them  towards.  

  Page  2  

Team-Based Learning Institute Page 98 of 156

  
 
Step  6-­‐8:  Design  Readiness  Assurance  Sequence  
 
Step  6:  Determine  important  concepts  to  test  
 
Looking  at  the  4S  task  you  have  created,  it  is  time  to  think  about  what  concepts,  definitions,  
and  vocabulary  the  students  will  need  to  start  their  analysis.  Make  a  preliminary  list.  
Step  7:  Select  Appropriate  Reading  
 
Knowing  what  concepts,  definitions,  and  vocabulary  your  students  need  to  get  started  –  
find  a  concise  reading  that  has  sufficient  detail  (not  too  much)  and  is  at  an  appropriate  
reading  level  to  be  accessible  to  your  students.  
Step  8:  Create  a  variety  of  Readiness  Assurance  questions  
 
Time  to  write  a  few  multiple-­‐choice  questions  for  the  Readiness  Assurance  test.  Question  
should  be  a  mix  at  mostly  Bloom’s  levels  –  remember,  know,  and  some  light  application.  
See  the  Bloom’s  section  page  6  of  this  section  for  suggestions  of  possible  question  leaders.  
Select  a  few  leaders  and  complete  your  multiple-­‐choice  questions.  

   

  Page  3  

Team-Based Learning Institute Page 99 of 156

Creating  Learning  Outcomes  

BUILD  YOUR  FIRST  SIMPLE  LEARNING  OUTCOME  

Creating great learning outcomes is a cyclic, iterative process where you

revisit and refine as your course design proceeds. You create your first
provisional outcomes are then used to build other components of course
design – selecting teaching and learning activities and designing assessment
materials. As the other components begin to come into focus, you will
periodically revisit and refine your initial learning outcomes. At a very
simple view – writing a learning outcome can be as simple as attaching a
Bloom’s verb to a piece of a piece of content.

Consider: I want students to know about flood return periods, I could simply
add the Bloom’s verb “describe” to flood return period.

Describe + Flood Return Period = Learning Outcome

Learning Outcomes are directly focused on student achievement and become

more detailed by module end, on exactly what the students will be able to do.
Learning Outcomes often contain references to the knowledge, skills, and
judgment abilities you want your students to develop. Your initial Learning
Outcome statements are often the precursors to ideas for 4S Application
tasks.

  Page  4  

Team-Based Learning Institute Page 100 of 156

 Learning Outcomes are often much more rigorously constructed then this
simple approach. A model known as the ABCD model is often applied.

A – stands for audience – “The student will”

B - stands for behaviour – “write an explanation”
C – stands for condition – “in 30 minutes”
D – stands for degree – “with no mistakes”

For our purpose we can keep the learning outcomes simpler at the beginning.

HOW  TO  M AKE  LEARNING  OUTCOMES  THAT  ARE  GOOD  FOR  TBL  

When we start thinking about the 4S Application tasks, we want to try to write
Learning Outcomes that focus on more concrete actions rather than abstract
understanding. We are looking for concrete actions just like a discipline expert
takes. Good Learning Outcomes express how experts in your field or discipline
would use the course content to solve disciplinary problems. The more concrete
you can make the learning outcomes the easier it will be to develop 4S
Application tasks from them.

Sample Learning Outcomes for a statistic course: by the end of this course
students will be able to use their knowledge of statistical principles to:

• Complete a statistical analysis

• Select an appropriate sampling plan
• Develop a survey instrument and plan to gather information from a
specific population

Sample Learning Outcomes for a genetics counseling course: by the end of

this course students will be able to use their knowledge of genomics to:

• Interpret genome sequencing data

• Identify genetic markers with greatest risk of disease/abnormality
• Develop counseling plan to work with specific family issues

Sample Learning Outcomes for a business course: by the end of this course
students will be able to use their knowledge of marketing principles to…

• Conduct a market analyses

• Evaluate a marketing plan
• Select or Develop marketing techniques to reach specific populations of
clients

Sample Learning Outcomes for a history course: by the end of this course
students will be able to use their knowledge of early Canadian history to…

  Page  5  

Team-Based Learning Institute Page 101 of 156

 • Interpret written accounts of historical events in light of cultural
dynamics
• Assess (and estimate) the bias or orientation of a given author
• Develop arguments for current policies or political positions based on
historical context

Bloom’s  Taxonomy  
Benjamin  Bloom  helped  develop  the  Cognitive  domain  
taxonomy  of  educational  objectives  to  help  educators  prepare   Blooms  –  Cognitive  Domain    
examinations  and  other  assessment  materials  that  test  different  
levels  of  knowledge  .and  understanding.   Remembering

Bloom’s  can  be  effectively  used  to  create  test  items  that  test  
Understanding
different  levels  of  understanding,  since  the  different  levels  can   Applying
directly  be  mapped  to  specific  verbs.  These  verbs  can  both  be  
used  to  generate  learning  objectives  and  create  test  questions   Analyzing
that  correspond  to  Bloom’s  different  levels.  
Evaluating
COGNITIVE DOMAIN (LOWER LEVELS) Creating

REMEMBERING

Verbs: Recalling, defining, recognizing, listing, describing, retrieving, naming

Common Question Leaders:

• What is the definition of….

• What is the name of…
• What is the best description of…
• List the following….
• Why did….?
• How is…?
• Where is…?
• When did … happen?

Understanding

Verbs: Explaining ideas or concepts, interpreting, summarizing, paraphrasing,

classifying, explaining, locating, identifying, restating

• Common Question Leaders:

• How would you classify…?

• What facts or ideas best shows….?
• Interpret in your own words…?
• Which statement best supports…?

  Page  6  

Team-Based Learning Institute Page 102 of 156

 • How would you summarize…?
• What is the main idea of…?

Applying

Verbs: implementing, carrying out, using, executing, translate, employing,

illustrating

Common Question Leaders:

• What is the best first step?

• What is the most significant problem?
• What would be the worst thing to do?
• Would it be a mistake to…?
• What is the most common mistake?
• Which test would you order next?
• What is the most common diagnosis?
• How would you use…?
• How would you solve?
• What is the most logical order?
• What approach would you use..?
• What would result if….?
• What facts would you select to show…?

SOME  TIPS  FOR  USING  BLOOM ’S  FOR  TBL  

Write a variety of Low-level questions

• What did the text say? (Remembering)

• What did the text mean? (Understanding)
• How could you apply it? (Recognize an example of a concept)

Have a few Low-level questions that invite discussion

• Which statement is most accurate?

• Based on the theory that you read about, what is most likely to happen is we
apply X?
• Which of these items best represent the qualities/characteristics of X?

Have one or two Higher-level questions that invite discussion

• Based on what you have read about theory A, which of the strategies listed below
has the best chance of success, given the specified conditions (X, Y, Z)?

   

  Page  7  

Team-Based Learning Institute Page 103 of 156

GRONLUND’S  QUESTION  PROMPTS  

Illustrative knowledge questions

Knowledge of Terminology

• What word means the same as ________?

• Which statement best defines the term ________?
• In this sentence, what is the meaning of the word ________?

Knowledge of Specific Facts

• Where would you find ________?

• Who first discovered ________?
• What is the name of ________?

Knowledge of Conventions

• What is the correct form for ________?

• Which statement indicates correct usage of ________?
• Which of the following rules applies to ________?

Knowledge of Trends and Sequences

• Which of the following best describes the trend of ________?

• Which is the most important cause of ________?
• Which of the following indicates the proper order of ________?

Knowledge of Classifications and Categories

• What are the main types of ________?

• What are the major classifications of ________?
• What are the characteristics of ________?

Knowledge of Criteria

• Which of the following is a criterion for judging ________?

• What is the most important criterion for selecting ________?
• What criteria are used to classify ________?

Knowledge of Methodology

• What method is used for ________?

• What is the best way to ________?
• What would be the first step in making ________?

Knowledge of Principles and Generalizations

  Page  8  

Team-Based Learning Institute Page 104 of 156

 • Which statement best expresses the principle of ________?
• Which statement best summarizes the belief that ________?
• Which of the following principles best explains ________?

Knowledge of Theories and Structures

• Which statement is most consistent with the theory of ________?

• Which of the following best describes the structure of ________?
• What evidence best supports the theory of ________?

Illustrative comprehension and application questions

Comprehension Questions

• Which of the following is an example of ________?

• What is the main thought expressed by ________?
• What are the main differences between ________?
• What are the common characteristics of ________?
• Which of the following is another form of ________?
• Which of the following best explains ________?
• Which of the following best summarizes ________?
• Which of the following best illustrates ________?
• What do you predict would happen if ________?
• What trend do you predict in ________?

Application Questions

• Which of the following methods is best for ________?

• What steps should be followed in applying ________?
• Which situation would require the use of ________?
• Which principle would be best for solving ________?
• What procedure is best for improving ________?
• What procedure is best for constructing ________?
• What procedure is best for correcting ________?
• Which of the following is the best plan for ________?
• Which of the following provides the proper sequence for ________?
• What is the most probable effect of ________?

From: How to make Achievement Tests and Assessments - 5th edition by Norman
Gronlund

   

  Page  9  

Team-Based Learning Institute Page 105 of 156

Writing  RAP  Questions  
At  the  heart  of  the  Readiness  Assurance  Process  is  a  series  of  readings  and  multiple-­‐choice  
tests  that  cover  the  important  fundamental  knowledge  that  students  will  need  to  know  to  
begin  the  4S  application  activities.    
GETTING  READY  TO  W RITE  RAP  QUESTIONS  
Once  you  understand  what  the  culminating  student  performance  will  be,  you  turn  your  
attention  to  preparing  student  for  first  engagement  with  the  content  (reading  and  RAP)  
and  then  the  progression  of  4S  activities  that  leads  to  that  culminating  4S  performance.    

Identify  specific  knowledge  the  students  will  need  to  begin  effectively  engaging  with  the  4S  
activities.  This  is  not  everything  they  need  to  solve  every  activity  but  what  they  require  as  
an  entry  point  to  the  problem-­‐solving  conversation.  You  do  this  by  mapping  back  from  the  
4S  application  activity  to  important  foundational  knowledge  that  the  students  will  need  to  
be  successful.  When  you  are  clear  on  the  basic  knowledge  students  need  to  know,  you  are  
then  ready  to  select  appropriate  student  preparation  materials  and  construct  RAP  
questions.  

Select  appropriate  student  preparation  materials.  There  is  an  iterative  loop  as  you  
select/define/refine  the  concepts  to  be  initially  tested,  and  then  select  and  refine  the  
preparation  materials.  For  preparation  materials,  we  most  often  use  readings,  but  videos,  
lecture  recordings,  or  narrated  PowerPoint’s  can  work  well.  Over  the  years  we  have  
discovered  that  less  is  more  with  readings.  The  amount  of  readings  that  students  will  
tolerate  depends  on  the  particular  discipline  and  institutional  context.  Our  readings  are  
closer  to  25  pages  for  2  weeks,  which  is  down  from  our  original  75  pages  for  two  weeks.  
We  found  that  students  were  spending  a  short,  fixed  amount  of  time  completing  readings  
without  regard  for  complexity  and  length  of  readings.  Remember  the  Readiness  Assurance  
Process  is  not  trying  to  be  comprehensive.  It  is  just  giving  students  an  entry  point  to  the  
problem-­‐solving  conversation.  

One  aside  –  when  teachers  are  first  introduced  to  the  idea  of  the  flipped  classroom,  they  
are  often  concerned  on  how  to  cram  their  1  hour  lectures  into  a  10-­‐12  minute  videos.  This  
is  the  wrong  way  to  look  at  it.  The  preparation  materials  are  just  to  get  students  started.  It  
is  not  all  that  students  learn  in  a  module,  so  you  need  to  create  a  selective  subset  of  your  1  
hour  of  lecture  content  –  focusing  of  high  level  themes  and  must  know  basic  concepts  and  
definitions.  Students  will  be  motivated  to  learn  the  additional  content  to  solve  the  exciting  
4S  team  tasks.  
Develop  a  list  of  important  concepts  and  ideas  to  test  with  your  RAP  questions.  The  RAP  
question  coverage  doesn’t  need  to  be  comprehensive,  you  are  providing  students  the  
foundational  knowledge  and  understanding  they  need  to  begin  problem-­‐solving.    

   

  Page  10  

Team-Based Learning Institute Page 106 of 156

WRITING  MCQ  QUESTIONS  

Multiple-­‐Choice  questions  have  two  main  parts:  the  question  stem  or  leader,  and  the  
options  (which  include  a  correct  answer).  When  beginning  to  construct  a  multiple-­‐choice  
question,  write  the  stem  of  the  question  first.  A  well-­‐constructed  stem  is  a  stand-­‐alone  
question  that  could  be  answered  without  examining  the  options.  The  wording  of  the  stem  
and  the  verbs  it  contains  determines  the  overall  difficulty  of  the  question.  

Multiple-­‐Choice  Questions  have  a  reputation  for  only  testing  lower  level  skills  like  
knowledge  and  recall.  In  the  question  example  below  students  are  asked  the  difficult  task  -­‐  
to  select  the  citation  that  is  most  accurate.  All  citations  have  errors  and  the  students  are  
really  being  asked  to  “hypothesize”  which  errors  will  have  the  greatest  impact  on  the  
citations  effectiveness.    This  question  is  testing  at  a  very  high  “Blooms”  level.  Writing  
questions  at  higher  “Blooms”  level  is  difficult,  but  NOT  impossible.  
 

In  your  argument,  you  are  citing  a  number  of  cases  from  different  
courts.  This  is  the  first  time  you  cite  any  of  these  cases.  What  is  the  most  
accurate  citation  sentence  (use  your  citation  manual)?  

 
1. Wyman  v.  Newhouse,  93  F.2d  313,  315  (2d  Cir.  1937);  Henkel  Co.  v.  
Degremont,  136  F.R.D.  88,  94  (E.D.  Pa.  1991),  Willametz  v.  Susi,  54.  F.R.D.  363,  
465  (D.  Mass.  1972).  
 
2. Henkel  Co.  v.  Degremont,  136  F.R.D.  88,  94  (E.D.  Pa.  1991);  Willametz  v.  
Susi,  54.  F.R.D.  363,  465  (D.  Mass.  1972);  Wyman  v.  Newhouse,  93  F.2d  313,  
315  (2d  Cir.  1937).  
 
3. Willametz  v.  Susi,  54.  F.R.D.  363,  465  (D.  Mass.  1972);  Henkel  Co.  v.  
Degremont,  136  F.R.D.  88,  94  (E.D.  Pa.  1991);  Wyman  v.  Newhouse,  93  F.2d  
313,  315  (2d  Cir.  1937).    
 
4. Wyman  v.  Newhouse,  93  F.2d  313,  315  (2d  Cir.  1937),  Willametz  v.  Susi,  54.  
F.R.D.  363,  465  (D.  Mass.  1972),  Henkel  Co.  v.  Degremont,  136  F.R.D.  88,  94  
(E.D.  Pa.  1991).    
 
 

Have  a  peer  or  colleague  review  your  questions.  It  can  be  difficult  to  see  flaws  in  our  own  
questions,  when  we  have  spent  hours  writing  them.  A  fresh  set  of  eyes  can  help  us  catch  
many  errors.  There  is  nothing  more  uncomfortable  then  dashing  off  a  set  of  poorly  written  
questions,  rushing  to  class,  and  enduring  the  inevitable  student  backlash  and  discontent.  

   

  Page  11  

Team-Based Learning Institute Page 107 of 156

SOME  RULES  FOR  MCQ  QUESTION  WRITING  

For good question stems, consider following rules:

• Stems should be stand-alone questions.

• Stems should be grammatically complete.
• Negative stems should be used with caution.
• If a key word appears consistently in the options, try to move it to the stem.
• Word the stem such that one option is indisputably correct.

For creating good options, consider following rules:

• Make sure each incorrect option is plausible but clearly incorrect.

• Make sure that the correct answer (keyed response) is clearly the best.
• Avoid, if possible, using “all of the above”.
• Use “none of the above” with caution.
• Try to keep options similar lengths, since test-wise students will pick the longest
option if unsure (too long to be wrong).
• Make sure options are grammatically consistent with the stem (question leader)
and use parallelism.
• Make sure that numerical answers are placed in numerical order, either
ascending or descending.

Well-constructed multiple-choice questions are not easy to create. But the quality of
the multiple-choice questions you use in your Team Test can make or break the tone
of your class. Nothing is more uncomfortable than rushing poor questions to the
classroom and having to endure the inevitable student backlash. Good questions are
absolutely essential to our success, and putting in the effort to write good questions is
worth your time and attention.

Spend time reviewing and revising your questions. It can be very helpful to have a
colleague look at your questions. When we write them we are often too close to see all
the mistakes. Just like good writing is about good editing, good MCQ questions are
about reflection and revision

   

  Page  12  

Team-Based Learning Institute Page 108 of 156

QUESTION  LEVEL  M IX  ON  RAP  TESTS  

Write  your  RAP  multiple-­‐choice  questions  at  Bloom’s  Remember,  Understand,  and  light  
Application  level  of  difficulty.  This  is  not  about  testing  all  that  students  will  learn  in  the  
module,  but  instead  only  what  they  need  to  begin  effectively  problem-­‐solving  (4S  
Application  Activities).  It  is  important  to  pitch  the  RAT  at  the  right  level  to  encourage  
students  to  engage  deeply  but  not  so  difficult  that  they  lose  heart.  

The test should be a mix of approximately 20% remembering (did you do the
readings?), approximately 60% understanding (did you understand what you read?),
and finally, 20% application, The application questions can be in the form of “which
concept applies to this situation” (are you ready to use what you have read?). To use a
book analogy, you want to write these tests more at the table-of-contents level then at
the index level.

You can include a few simpler questions that just provide simple accountability that
the student has completed the readings. Try to ask about topics that students are
likely to interpret incorrectly. Test common misconceptions that might undermine
students’ ability to successfully begin problem-solving. You can ask which concept
applies to a given situation or scenario. You can focus on the relationship between
concepts; this is an efficient way to test two concepts at once.

   

  Page  13  

Team-Based Learning Institute Page 109 of 156

Designing  4S  Tasks  
During  a  4S  Application  task,  students  get  to  concretely  apply  what  they  have  abstractly  
learned  from  the  readings.  You  want  students  connecting  
abstract  concepts  from  the  readings  with  concrete  
experience  during  the  4S  team  Application  tasks.  Making  
connections  during  4S  team  tasks  is  important  to  consolidate  
student  learning.    
Helping  students  see  gaps  in  their  knowledge  motivates  the  
students’  look  up  what  they  don’t  know  and  then  
immediately  putting  that  knowledge  into  action  tests  and  
deepens  their  understanding.    
You  need  to  present  a  scenario  that  creates  the  context  in  
which  what  students  “know”  abstractly  (via  their  readings)  is  put  to  the  test  when  they  try  
to  “use”  it  in  concrete,  specific  case.  Your  job  is  to  find  or,  if  necessary,  fabricate  these  
scenarios.  
4S  EXTENDED  EXAMPLE    (from  Roberson)  

Students  in  Sociology  might  “know”  Maslow’s  hierarchy  of  human  needs,  and  could  score  
well  on  a  test  that  asked  them  to  recite  and  explain  it.  But  now  imagine  the  Application  
task  you  give  students,  based  on  their  initial  understanding  of  Maslow:    

You  are  a  social  worker  and  you  have  been  given  the  case  of  “Maria  from  Syria.”  
Given  your  understanding  of  Maslow’s  hierarchy,  look  at  these  data,  make  an  
assessment  of  her  situation,  and  decide  the  best  way  to  proceed  in  interacting  with  
her:  “Maria  comes  from  a  middle  class  family  (her  father  was  a  dentist)  in  a  small  
town  in  war-­‐torn  Syria.  She  immigrated  with  just  her  two  children  to  Canada  2  years  
ago,  and  came  to  Ontario.  She  now  works  long  hours  at  minimum  wage  as  a  
housekeeper.  She  recently  re-­‐married  and  currently  lives  with  her  abusive,  alcoholic  
husband.  One  of  her  children  has  health  problems...etc.”    
If  the  details  of  the  case  are  rich,  it  quickly  becomes  clear  to  students  that  Maria’s  
case  is  complicated,  and  that  Maslow’s  hierarchy,  while  it  is  a  useful  tool  to  help  
analyze  the  situation,  does  not  lead  to  an  easy  assessment  or  judgment.    
EXAMPLE  4S  PROMPTS  (superlatives  or  implied  superlatives  to  force  a  specific  choice)  

• A  patient  comes  into  emergency  with  the  following  symptoms...    

 
o What  is  the  first  thing  you  would  do?  And  why?  
o What  is  the  first  test  you  would  order?  And  why?  
o What  would  be  the  worst  thing  to  do?  And  why?  
 

  Page  14  

Team-Based Learning Institute Page 110 of 156

• Given  3  possible  programs  to  end  homelessness  in  your  city,  select  the  program  
that  is  the  best  and  will  likely  be  most  strongly  supported  by  local  agencies  and  
Civic  leaders?  (Michaelsen  and  Sweet)  
 
• What  is  the  most  relevant  theory  that  explains  the  behaviour  in  the  video?  
(Kubitz  and  Lightner)  
 
• Which  of  the  following  best  describes  the  opportunity  cost  of  coming  to  class  
today?  (Espey)  
 
• Which  of  the  following  should  the  University  do  to  best  increase  the  quality  of  
Undergraduate  education?  (Mahler)  
 
• Which  sampling  scenario  would  best  address  this  research  project?  (Mahler)  
 
• Given  three  valid  historical  interpretations  of  the  progressive  Movement,  discern  
which  best  describes  the  Progressives  revealed  in  our  manifesto?  (Restad)  
 
• In  Clarence  Page’s  op-­‐ed  piece  “The  Problem  With  Trashing  Liberty”’  where  does  
the  responsibility  for  a  safe  a  civil  society  lie?  Which  of  the  following  three  
philosophers  (X,  Y,  and  Z)  does  Clarence  Page  most  agree  with  on  these  fronts?  
(Roberson  and  Reimers)  
 
• What  of  the  following  passage  in  the  Bhagavad  Gita  best  illustrates  reflection  
about  the  nature  of  Krishna’s  divinity?  (Dubois)  
 
• Rank  how  useful  each  source  is  for  understanding  the  fears  of  the  Cold  War  era.  
(Restad)  
 
• Which  teacher  should  be  nominated  for  a  teaching  award?  (Croyle  and  Alfaro)  
 
• Which  indicator  (from  a  list  of  5  plausible  alternatives)  is  most  critical  to  making  
a  correct  diagnosis  in  this  case?  (Michaelsen  and  Sweet)  
 
• If  a  moving  vehicle  overloaded  this  bridge  structure,  which  component  would  
likely  fail  first?  
 
• You  are  making  a  home  assessment,  which  of  the  following  safety  hazards  would  
be  of  greatest  concern?  (Clark)  
 
• After  assessing  Mrs.  Randall’s  dining  room  what  would  be  your  first  
recommendation  to  protect  her  from  falls?  (Clark)  
 
• What  line  on  this  tax  form  would  pose  the  greatest  finical  risk  due  to  an  IRS  
audit?  (Michaelsen  and  Sweet)  
 

  Page  15  

Team-Based Learning Institute Page 111 of 156

 • Given  a  set  of  real  data,  which  of  the  following  advertising  claims  is  least  (or  
most)  supportable?  (Michaelsen  and  Sweet)  
 
• You  are  consulting  for  a  new  business  owner  who  wants  to  open  a  dry-­‐cleaning  
store  in  Norman,  Oklahoma.  Where  would  you  recommend  locating  a  new  dry-­‐
cleaning  business?    (Michaelsen)  
 

ANOTHER  PATHWAY  DESCRIPTION  OF  4S  DEVELOPMENT  PROCESS  

First,  you  may  need  to  make  your  original  Learning  Outcomes  more  CONCRETE.    

Next,  you  need  to  create  problem  scenarios/situations  where  students’  factual  knowledge  
(from  RAP  process)  is  useful,  but  maybe  insufficient  to  solve  the  problem  definitively.    

Next,  when  creating  these  scenarios  you  want  to  clarify  exactly  what  do  you  want  students  
to  be  doing.  

Evaluate/judge  something  (object,  product,  creation,  situation)?  

•
Analyze  or  diagnose  a  situation?  
•
Interpret  something  (text,  artifact,  data  set)?  
•
Solve  a  particular  type  of  messy  problem?  
•
 
Next,  identify  the  concrete  information/data  sets  the  students  will  work  with:    

• Texts  (such  as  cases,  descriptions,  excerpts  from  a  textbook,  writing  samples,  etc.)  
• Images  (visualizations,  diagrams,  videos,  etc.)  
• Data  (spreadsheets,  graphs,  charts,  etc.)  
• Objects  (products,  specimens,  etc.)  
 
Next,  you  need  to  pick  the  format  of  students’  action:  
 
• Will  they  compare?  
• Will  they  sort?  
• Will  they  rank?  
• Will  they  score?  
• Will  they  choose  the  best  course  of  action?  
• Will  they  distill  and  represent  in  a  written  format?  
 
Next,  determine  how  to  make  student  thinking/decisions  visible  so  it  can  be  represented  
in  a  simultaneous  report.  Can  their  answer  be  represented  with?  
a. Colour  Voting  Cards  
b. Single  Number  
c. Single  Letter  
d. Single  word  or  phrase      

  Page  16  

Team-Based Learning Institute Page 112 of 156

  
Sometimes  this  means  converting  a  complex  response  into  a  simple  response.  For  example,  
after  a  ranking  task,  ask  students  to  report  their  #1  choice,  rather  than  their  entire  ranking  
scheme.  If  you’ve  asked  students  to  compile  a  list,  ask  them  to  choose  the  MOST  critical  
item  on  their  list  and  report  it.  Every  task  needs  to  lead  to  a  moment  of  sharp  
differentiation:  “I  choose  this  over  that.”  Getting  the  students  to  this  moment  sets  up  
“WHY?”  as  the  teacher’s  entry  point  for  interactions  leading  to  student  analysis,  reflection,  
and  critical  thinking.  The  simultaneous  report  naturally  lets  teams  compare  their  decisions  
and  decision-­‐making  process  to  other  teams.  
Finally,  it  is  good  to  develop  a  facilitation  plan  for  debriefing  the  4S  Application  task,  to  
ensure  students  learn  the  most  they  can  from  the  task.  Debriefs  always  begins  by  asking  
ALL  teams  to  simultaneously  report  their  answers/decisions.  A  good  plan  provides  you  
with  a  way  to  organize  the  discussion  that  follows,  and  direct  students  into  a  dialogue  with  
each  other.  
Instructor:  “OK,  I  see  three  groups  said  “B”  and  two  groups  said  “C.”  Let’s  start  with  those  
of  you  who  said  “C.”  Please  explain  to  the  other  students  why  you  chose  this  answer?  

Later:  OK,  teams  who  said  B,  how  would  you  respond  to  them?  

Later  still:  Nobody  chose  A.  Why  did  you  discount  that  possibility?  

  Page  17  

Team-Based Learning Institute Page 113 of 156

Learning  Outcome  to  4S  Activity  Story  
Define  Flood  Return  Period  

This  is  a  very  typical  content  focused,  low  Bloom’s  level  learning  outcome.  Next  step  is  to  
raise  the  Bloom’s  level  (in  this  case  using  Bloom’s  Cognitive  Domain  Taxonomy).    

Explain  Flood  Return  Period  

The  change  has  moved  the  learning  outcome  from  the  lowest  Bloom’s  level  –  Remember  -­‐  
to  the  slightly  higher  -­‐  Understand  –  level.  In  the  next  step,  we  try  to  change  this  learning  
outcome  to  be  about  concrete  action  rather  than  just  abstract  understanding.    
Demonstrate  understanding  of  Flood  Return  Period  

This  is  often  the  first  attempt  at  making  the  learning  outcome  more  about  concrete  action.  
Unfortunately,  it  doesn’t  provide  any  information  on  how  the  students  might  “demonstrate  
understanding”.  In  the  next  step,  we  try  again  to  make  it  about  concrete  action  but  this  
time  so  student  understanding  is  put  to  use  in  a  visible  way.    

Students  will  predict  the  outcome  of  a  situation    

This  is  getting  better.  Students  are  using  their  knowledge  of  Flood  Return  Period  and  
applying  abstractly  to  a  concrete  situation.  This  is  key.  You  can  start  to  get  glimmers  of  
what  an  activity  might  look  like  where  students  show  you  that  they  know  (knowledge  in  
the  service  of  action).  But  what  is  missing  here  is  discrimination  and  judgment.    

Students  will  predict  the  most  likely  outcome  of  a  specific  situation    

We  now  have  discrimination  and  judgment  but  still  a  little  too  open  ended  to  have  
students  make  decisions  that  are  easily  comparable  and  drive  an  intense  reporting  
discussion  that  examines  only  the  salient  issues  that  need  to  be  considered  to  make  a  
“good”  judgment  and  decision  “in  this  case”.  Constraining  the  possible  outcomes  to  be  
considered  can  help  you  structure  the  analysis  and  discussion  so  very  specific  issues  are  
discussed  and  very  specific  analysis  is  done.  Lets  constrain  the  possible  choices!  

F INAL TRANSFORMATION TO 4S ACTIVITY

Which  of  these  outcomes  is  most  likely  given  this  situation  (using  your  
knowledge  of  flood  return  period)  

• Possible  Outcome  1  
• Possible  Outcome  2  
• Possible  Outcome  3  
• Possible  Outcome  4  
 

  Page  18  

Team-Based Learning Institute Page 114 of 156

We  have  transformed  a  lower  level  learning  outcome,  that  at  best  could  be  assessed  in  an  
examination,  into  a  powerful  classroom  activity  that  is  structured  to  lead  to  a  rich,  deep  
reporting  discussion.  
 
FINAL  ELABOARTED  VERSION  OF  THIS  4S  ACTIVITY  

 
 
You  are  head  of  Engineering  for  a  large  dam  project  on  the  Yellow  river  in  the  
Ningxai  province  of  China.  The  dam  is  to  be  located  in  the  Yiling  district  near  the  
exit  of  the  Ordos  Loop  section  of  the  river.  The  dam  is  to  be  located  at  34°49′46″N    
111°20′41″E.  T he  Y ellow  river  is  China’s  third  largest  river.  The  river  is  
characterized  by  extremely  high  silt  loads,  especially  in  spring  floods.  The  local  
bedrock  is  highly  fractured  gneiss.  The  dam  will  be  a  concrete  earthfill  hybrid  
design.  You  have  been  asked  to  determine  some  of  the  main  design  parameters,  
including  safety  related  question  like  what  flood  event  return  period  to  build  the  
dam  to  withstand.  
 
What  flood  return  period  would  you  recommend  the  dam  be  designed  to  
withstand?  
 
A) once  in  50  year  flood    
B) once  in  100  year  flood  
C) once  in  200  year  flood  
D) once  in  500  year  flood  
 

   

  Page  19  

Team-Based Learning Institute Page 115 of 156

USING  THE  4S  STRUCTURE  

Lets  examine  how  to  structure  problems  using  the  4S  framework  so  they  lead  to  
consistently  powerful  activities.  There  are  4  major  tenets  to  consider  when  structuring  a  
4S  activity.  
 
First,  we  should  use  the  kinds  of  questions/problems  and  problem  solving/analysis  
procedures  that  disciplinary  experts  are  routinely  asked  to  use/make.  Since  most  
disciplines  are  more  about  their  actions  rather  than  there  content.  Next  we  need  to  make  
problem  about  concrete  action  in  a  concrete  situation  with  real  consequences.  This  helps  
make  student  understanding  visible  to  both  the  teacher  and  student.  Then  we  need  to  
think  about  the  kinds  of  complex  analysis  that  will  required  of  students  to  
analyze/interpret  the  scenario  or  problem  statement.    Finally,  we  will  need  to  constrain  
choice  to  intensify  the  learning.  
 
T ENET 1: U SE E XPERT -LIKE D ISCIPLINARY P ROBLEMS
 
A  nice  feature  of  this  example  is  that  it  asks  the  kind  of  question  an  expert  would  need  
to  make.    
 
“Disciplines  are  more  clearly  defined  by  how  those  working  within  the  discipline  collect,  
organize,  assess,  and  use  information”  (Roberson  and  Franchini,  2014,  p.  278)  
 
“If  we  want  our  students  to  become  more  expert  in  our  disciplines,  we  need  to  structure  
their  encounters  with  content  in  ways  that  change  what  they  can  do  with  knowledge.”  
(Roberson  and  Franchini,  2014,  p.  278)  
 
T ENET 2: M AKE IT ABOUT C ONCRETE A CTION IN THE REAL WORLD
 
Students  need  to  use  their  understanding  (gained  in  the  pre-­‐readings,  lectures  or  
previous  activities)  to  make  expert-­‐like  concrete  decisions  that  will  have  very  concrete  
consequences.  You  want  to  design  concrete  scenarios  where  conceptual  and  abstract  
understanding  helps  students  make  better  decisions.  
 
The  quality  of  the  problem  ultimately  controls  the  effectiveness,  energy,  and  learning  
outcomes  of  an  activity.    
 
“Students,  therefore,  need  to  be  required  to  act  frequently  in  ways  that  generate  
consequences  that  provoke  reflection  and  demonstrate  visibly  their  thinking.  The  more  
focused  and  concrete  the  action,  the  more  visible  will  be  the  thinking  and  the  learning—
and  the  more  immediately  useful  will  be  the  feedback.”  (Roberson  and  Franchini,  2014,  p.  
276)  

  Page  20  

Team-Based Learning Institute Page 116 of 156

  
“Effective  team  tasks  point  students  consistently  toward  making  decisions  that  reveal  
reasoning  and  understanding  in  service  of  a  judgment.”  (Roberson  and  Franchini,  2014,  p.  
279)  
 
“What  we  know  about  the  nature  of  learning  is  that  students  gain  deeper  traction,  faster,  
with  course  content  if  their  first  encounters  with  it  include  concrete  experiences  framed  
by  and  informed  by  the  abstractions”  (Roberson  and  Franchini,  2014,  p.  296)  
 

TENET  3:  REQUIRE  COMPLEX  ANALYSIS,  DISCRIMINATION,  AND  JUDGEMENT    

 
Coming  up  with  a  good  and  defensible  solution  requires  the  integration  and  analysis  of  
many  different  factors  and  the  weighing  of  tradeoffs  (like  cost  vs.  safety).  There  are  a  
lot  of  things  for  the  teams  to  consider  in  determining  a  reasonable  course  of  action  and  
coming  up  with  a  reasonable  defense  for  their  final  decision.    
 
Possible  issues  that  need  to  be  considered:  
 
● How  big  are  the  flood  events?  
● Are  changing  climate  patterns  going  to  affect  the  size  and  frequency  of  flood  
events?  
● What  is  the  difference  in  cost  to  design  to  withstand  the  different  levels  of  flood  
events?  
● Are  there  unique  landscape  or  bedrock  concerns?  How  could  we  mitigate  them?  
● What  are  the  population  patterns  downstream?  
● How  would  downstream  populations  be  effected  by  a  failure  at  different  flood  
levels?  
● How  do  these  kind  of  dams  typically  fail?    
● Can  the  dam  be  constructed  to  fail  elegantly  and  reduced  the  threat  to  
downstream  populations  during  flood  events?  
 
“Scenarios  allow  you  to  embed  many  variables  that  can  be  used  to  introduce  multiple  
concepts,  theories  and  perspectives  into  students’  discussion,  as  well  as  to  complicate  the  
task,  if  desired,  through  a  mix  of  relevant  factors  and  red  herrings.”  (Roberson  and  
Franchini,  2014,  p.  287)  
 
TENET  4:  CONSTRAIN  CHOICE  TO  INTENSIFY  ANALYSIS  AND  DISCUSSION  

  Page  21  

Team-Based Learning Institute Page 117 of 156

 The  example  at  first  glance  looks  a  lot  like  a  multiple-­‐choice  question  and  many  
teachers  worry  that  constraining  choice  like  this  will  limit  the  depth  of  the  discussion.  
It  is  quite  the  opposite,  constrained  choices  focuses  student  energies  on  analysis  of  
specific  issues,  which  ultimately  helps  with  team  to  team  comparisons  that  allows  
students  to  see  how  their  thinking  contrasts  other  teams.  But  this  would  be  like  saying  
a  murder  trial  is  decided  by  a  two  option  multiple-­‐choice  question.  These  kinds  of  
constrained  choice  questions  are  potent  discussion  starters.  This  really  becomes  clear  
for  all  to  see  during  the  public  reporting  of  team  decisions.  
 
“The  function  of  the  collective  decision  task,  therefore,  is  to  place  a  restrictive  frame  
around  the  team’s  action.  This  restriction  forces  the  team  to  evaluate,  integrate  and,  if  
needed,  respectfully  discount  a  team  member’s  inputs  en  route  to  a  judgment  and  a  
focused  decision.”  (Roberson  and  Franchini,  2014,  p.  288)  
 
“Tasks  that  direct  students  toward  a  specific  choice  do  not  stifle  student  thinking  but  
concentrate  it  so  that  feedback  on  the  task  can  be  directed  at  specific,  anticipated  
discoveries  and  realizations.”    
(Roberson  and  Franchini,  2014,  p.  290)  
 
   

  Page  22  

Team-Based Learning Institute Page 118 of 156

 A  problem  that  is  significant  and  interesting  to  
the  students  –  not  a  toy  problem  

Complex  scenario  to  analyze  –  with  relevant  

information,  irrelevant  information,  missing  
information,  constraints,  trade-­‐offs  -­‐  that  require  
expert-­‐like  concrete  action  

Specific  question  statement  -­‐  that  often  uses  

superlatives  like  best  next  step,  worst  thing  to  do,  
most  appropriate  action,  most  likely  outcome,  
greatest  concern,  etc.  

Multiple  reasonable  courses  of  action  –  some  

more  reasonable  than  others  

  Page  23  

Team-Based Learning Institute Page 119 of 156

Team-Based Learning Institute Page 120 of 156
The National Teaching & Learning

Volume 17 Number 5 September 2008

First-Day Questions for the

• EDITOR’S NOTE, p. 3.
Learner-Centered Classroom
• First-Day Questions for the Gary A. Smith
Learner-Centered Classroom, University of New Mexico
Gary A. Smith, University of New
Mexico, p. 1. When we let
students know something about
“D o my students know why I
ask them to learn this way?”
I said it aloud while reading the
them, guiding them when they were
stuck and thrilled to contribute
additional insights that they re-
the machinery of how we’re trying teaching evaluations from the quested out of curiosity. My shorter
to teach them, they actually learn previous semester. I had taken the presentations focused on topics that
more and better. plunge; I had moved substantially they were struggling with. Grades
• LEARNING DIARY: Building away from an almost entirely lecture- went up significantly.
Cognitive Assemblies: An and-exam format in the second- So, why did these evaluations
Exercise in Course Design, semester geology course. Instead, reveal so many signs of dissatisfac-
Carolyn G. Shapiro-Shapin, Ph.D., many class sessions featured tion? Why had my evaluation scores
Grand Valley State University, students working in small groups to gone down while student achieve-
p. 5. Automotive redesign offers apply content, which they first ment had gone up? The reason
hints on streamlining conventional encountered largely through became clear as I read the written
assigned reading, on to authentic comments. The students were
thinking and teaching.
geological problems. displeased with the greater work.
• VIEWPOINT: Mission They were content to ignore
Statements Reconsidered, Confronting a Problem reading assignments, assuming that
Michael Harris, Kettering Each weekend, students com- I would lecture over the content
University and Roxanne Cullen, pleted an online assessment. that was important. They were
Ferris State University, p. 7. Although low-stakes scores were content not to review information
Doesn’t all the emphasis on provided for in-class and online and construct knowledge except by
learner-centered classes imply assignments, cramming
something for the way the whole these were the night
school operates? primarily before
formative exams.
• DEVELOPER’S DIARY: Are opportunities to However, in
You Organized and Prepared?, guide both the my class they
Ed Nuhfer, Director of Faculty students and me. needed to
Development, California State Class sessions come to class
University – Channel Islands, p. 9. had been hugely prepared to
Nuhfer has some good ideas about more energetic do in-class
linking merit pay to better and and fun, at least assignments
better teaching. for me. No with their
• AD REM . . .: Scout’s Motto: dozing students, blank stares, and peers. Moreover, they had to be
Be Prepared, Marilla Svinicki, constant glances toward the clock. reviewing and applying what they
University of Texas-Austin, p. 12. Students were doing things, and I had learned weekly for the online
was enjoying my interaction with assessments. There were a few
Maybe reading texts after class
makes more sense.

Team-Based Learning Institute Page 121 of 156

 12345678901234567890123456789012123
12345678901234567890123456789012123
12345678901234567890123456789012123
12345678901234567890123456789012123
12345678901234567890123456789012123
12345678901234567890123456789012123
comments about how much they better; I still didn’t like to eat some THE NATIONAL TEACHING
12345678901234567890123456789012123
had learned in the class, but for things. Wouldn’t it be the same for 12345678901234567890123456789012123
& LEARNING FORUM
12345678901234567890123456789012123
most students the cost seemed too the students? What I needed was a 12345678901234567890123456789012123
12345678901234567890123456789012123
high. Clearly, although I adopted way to engage them to see that how I 12345678901234567890123456789012123
Executive Editor:
12345678901234567890123456789012123
what I was convinced was a more taught the course mattered to them; 12345678901234567890123456789012123
James Rhem, Ph.D.
12345678901234567890123456789012123
successful pedagogy for my stu- that learning this way helped them
12345678901234567890123456789012123
2203 Regent Street, Suite B
12345678901234567890123456789012123
12345678901234567890123456789012123
Madison, WI 53726
dents, the learners were so used to accomplish goals that were impor- 12345678901234567890123456789012123
12345678901234567890123456789012123
Editorial Advisory Board
other approaches that they really tant to them. 12345678901234567890123456789012123
12345678901234567890123456789012123
did not know what I was doing and, 12345678901234567890123456789012123
Jonathan Fife, Director Emeritus
12345678901234567890123456789012123
The First-Day Questions 12345678901234567890123456789012123
ERIC Clearinghouse on Higher Education
more importantly, why I was doing it. 12345678901234567890123456789012123
12345678901234567890123456789012123
Gabriele Bauer, Director
Today, I realize that this is typical, On that anxious August day, I 12345678901234567890123456789012123
12345678901234567890123456789012123
Center for Teaching Effectiveness
but at the time, I did not know greeted the students and after a 12345678901234567890123456789012123
12345678901234567890123456789012123
University of Delaware
about resources (like Felder and minimum of predictable first day 12345678901234567890123456789012123
12345678901234567890123456789012123
Pat Hutchings, Vice President
12345678901234567890123456789012123
Brent, 1996) to show me how to stay review of the syllabus, I projected 12345678901234567890123456789012123
The Carnegie Foundation
on track. What should I do? this text on the screen: 12345678901234567890123456789012123
12345678901234567890123456789012123
for the Advancement of Teaching
12345678901234567890123456789012123
Getting Student Buy-In
“Thinking of what you want to get 12345678901234567890123456789012123
12345678901234567890123456789012123
Susan Kahn, Director
out of your college education and 12345678901234567890123456789012123
Urban Universities Portfolio Project
12345678901234567890123456789012123
As the next offering of the course this course, which of the following is 12345678901234567890123456789012123
Indiana University–Purdue University
12345678901234567890123456789012123
approached, I thought about what most important to you? 12345678901234567890123456789012123
Indianapolis
12345678901234567890123456789012123
to do differently. Sure, I could make 1. Acquiring information (facts, 12345678901234567890123456789012123
Wilbert McKeachie
12345678901234567890123456789012123
12345678901234567890123456789012123
Professor of Psychology, Emeritus
the workload more manageable— principles, concepts) 12345678901234567890123456789012123
12345678901234567890123456789012123
University of Michigan
focus more on the learning out- 2. Learning how to use informa- 12345678901234567890123456789012123
12345678901234567890123456789012123
Edward Neal, Director
comes by covering less content, tion and knowledge in new situa- 12345678901234567890123456789012123
12345678901234567890123456789012123
Center for Teaching and Learning
assigning more succinct and tions 12345678901234567890123456789012123
12345678901234567890123456789012123
University of North Carolina–Chapel Hill
purposeful reading, and giving 3. Developing lifelong learning 12345678901234567890123456789012123
12345678901234567890123456789012123
Christine Stanley
12345678901234567890123456789012123
clearer and shorter assignments skills.” 12345678901234567890123456789012123
Executive Associate Dean
12345678901234567890123456789012123
and assessments to provide oppor- It was a gamble—I had no idea 12345678901234567890123456789012123
College of Education & Human Development
12345678901234567890123456789012123
tunity for reflection. However, was how they would respond. I encour- 12345678901234567890123456789012123
Texas A&M University
not the real problem that the aged them to chat with their 12345678901234567890123456789012123
12345678901234567890123456789012123
R. Eugene Rice
students did not understand why I neighbors. Then I polled them.
12345678901234567890123456789012123
12345678901234567890123456789012123
Senior Scholar
12345678901234567890123456789012123
taught the course in this apparently Two hands went up in support of 12345678901234567890123456789012123
Program in Leadership & Change
12345678901234567890123456789012123
Antioch University
unfamiliar style? option number 1. Twenty-one 12345678901234567890123456789012123
12345678901234567890123456789012123
12345678901234567890123456789012123
Marilla Svinicki
I planned to start the first day hands rose in support of option 12345678901234567890123456789012123
with a summary of the research on number 2. The remaining 13 12345678901234567890123456789012123
Professor of Psychology
12345678901234567890123456789012123
University of Texas at Austin
12345678901234567890123456789012123
active and students selected 12345678901234567890123456789012123
12345678901234567890123456789012123
Editorial correspondence:
reflective the final option. 12345678901234567890123456789012123
12345678901234567890123456789012123
learning. That We talked about 12345678901234567890123456789012123
James Rhem
12345678901234567890123456789012123
2203 Regent Street, Suite B
Why had my evaluation each one, with 12345678901234567890123456789012123
would show 12345678901234567890123456789012123
Madison, WI 53726
12345678901234567890123456789012123
them that I knew advocates for each 12345678901234567890123456789012123
something about scores gone down while stating their case. 12345678901234567890123456789012123
Subscription information:
12345678901234567890123456789012123
12345678901234567890123456789012123
The National Teaching & Learning Forum
12345678901234567890123456789012123
teaching and student achievement The two students 12345678901234567890123456789012123
2203 Regent Street, Suite B
12345678901234567890123456789012123
not just about who favored 12345678901234567890123456789012123
Madison, WI 53726
geology and that had gone up? acquisition of 12345678901234567890123456789012123
12345678901234567890123456789012123
The National Teaching & Learning Forum
12345678901234567890123456789012123
they were really information 12345678901234567890123456789012123
(ISSN 1057-2880) is published six times
12345678901234567890123456789012123
taking a “state of revealed that it was 12345678901234567890123456789012123
during the academic year by James
12345678901234567890123456789012123
Rhem & Associates, LLC — December,
the art” course. not so much that 12345678901234567890123456789012123
12345678901234567890123456789012123
February, March, May, September,
Wouldn’t they then see that this option seemed most important 12345678901234567890123456789012123
12345678901234567890123456789012123
October.
everything I was having them do was but that it was most basic. They saw 12345678901234567890123456789012123
12345678901234567890123456789012123
One-year individual subscription: $59.
really for their own benefit as the list as hierarchical, and al- 12345678901234567890123456789012123
12345678901234567890123456789012123
Periodicals postage paid at Madison, WI
learners? though they thought application of 12345678901234567890123456789012123
12345678901234567890123456789012123
12345678901234567890123456789012123
Postmaster: Send change of address to:
As I outlined what I would say knowledge was more important 12345678901234567890123456789012123
12345678901234567890123456789012123
The National Teaching & Learning Forum
and what graphs I would show, I than memorization, they felt that 12345678901234567890123456789012123
12345678901234567890123456789012123
2203 Regent Street, Suite B
could not help but think that I was they had to master the factual 12345678901234567890123456789012123
12345678901234567890123456789012123
Madison, WI 53726
really missing the point. What I was information before they could use 12345678901234567890123456789012123
12345678901234567890123456789012123
Copyright © 2008
12345678901234567890123456789012123
planning somehow brought back it. This led to further discussion of 12345678901234567890123456789012123
James Rhem & Associates, LLC
12345678901234567890123456789012123
from childhood memory the why learners needed to meet all 12345678901234567890123456789012123
12345678901234567890123456789012123
DUPLICATION BY PHOTOCOPYING
admonition of my parents to eat my three of these goals, even if we each 12345678901234567890123456789012123
12345678901234567890123456789012123
OR OTHER MEANS IS STRICTLY
vegetables. Just because it was good held one as being more important 12345678901234567890123456789012123
FORBIDDEN.
12345678901234567890123456789012123
12345678901234567890123456789012123
for me didn’t make the food taste than the other two. 12345678901234567890123456789012123
12345678901234567890123456789012123
[Link]
12345678901234567890123456789012123
September
12345678901234567890123456789012123
12345678901234567890123456789012123
12345678901234567890123456789012123
2 THE NATIONAL TEACHING & LEARNING FORUM Vol. 17, No. 5 2008

Team-Based Learning Institute Page 122 of 156

 Now I had them where I wanted that were most important to them. Students desire to accomplish the
them. I projected a new question on Rather than me lecturing over the educational goals that come from
the screen: assigned reading and leaving them deep learning. One colleague,
“All three of these goals are to fend for themselves on home- instructing a course of 100 students
clearly important. However, let’s work, they were going to come to in a non-majors, general-education
think for a moment of how best
to accomplish these goals.
Learning is not a spectator
sport—it takes work; that The results that semester were dramatically different.
includes work in the classroom
and work that you do outside of Not only was the active classroom fostering better
the classroom. So, of these
three goals, which do you think
learning performance on exams and other
you can make headway on outside assignments but also my teaching evaluations rose to
of class by your own reading and
studying, and which do you their highest levels.
think would be best achieved
in class working with your
classmates and me?” class having read, and sometimes science class, has experienced
The polling was nearly unani- struggled with, the text. The situations where not a single
mous that acquiring information problem-solving that would apply student chose acquisition of factual
was the easiest to do alone and that the reading content and develop knowledge as their prime learning
the other two goals seemed more logical hypothesis statements and objective. More importantly, all
complicated and would profit from testing would take place in class. By students prepare at some level for
peer and instructor influence. This, completing these assignments and most classes, participate enthusiasti-
then, led to a discussion of how to the online assessments, they would cally during in-class activities, and
pursue goals 2 and 3. These goals always know whether they were complete the assessments while
are not achieved by reading or mastering both the content knowl- acknowledging, sometimes with a
listening to a lecturer—students edge and its application and reminder, that these learner-
must actively do things in order to relationship to how scientists know centered opportunities closely
learn. Students learn best (Davis, about the natural world. Moreover, I
1993) when they take an active role: would be continually reviewing
• When they discuss what they their progress, too, working with
are reading individuals where they showed lack
• When they practice what they of mastery and going back to
are learning material when most of the class
• When they apply practices and showed evidence of confusion and
ideas. misconception.
The Impact
The results that semester were
dramatically different. Not only was
They were content to the active classroom fostering better
learning performance on exams
cram the night before and other assignments but also my
exams. teaching evaluations rose to their
highest levels. Students actually match their
complained if they thought I was own objectives.
lecturing too much. I have since The impact of the first-day
The need for active learning in used the first-day questions in every questions to engage students with
class in order to reach their goals course I teach and at all levels for their learning is further enhanced
leads students to accept that they three years. The strategy has been by asking students to assess their
have to read and otherwise prepare shared with colleagues through preferred learning styles. By
before coming to class by making first faculty development workshops and administering a free, online
contact with content on their own. I frequently hear back about their learning style inventory (e.g.,
This discussion became a segue experiences. Some use classroom Felder-Silverman Index of Learn-
to explaining how the course was response systems (clickers) to ing Styles, [Link]
structured—that it was about their maintain anonymity during polling. unity/lockers/users/f/felder/
learning and achieving the goals The results are very consistent: public/[Link]) and spend-

Vol. 17, No. 5 2008 THE NATIONAL TEACHING & LEARNING FORUM 3

Team-Based Learning Institute Page 123 of 156

 And all it takes from time to time
is a good idea or a new thought or a
Pass this pull-out card along to a

new twist on an old thought to add

ing a little time in class during the

friend you think would enjoy the

early weeks discussing the results,

new vigor to your teaching life.

the good news

intelligence and variety of the

newsletters get read

students commonly encounter for

Journals get filed;

the first time that (a) they have
particular learning style prefer-

Share
ences, and (b) that different people
learn in different ways. This
exercise enhances the first-day
questions by connecting active and
reflective learning strategies in the
course to the way the students learn,

Forum.
and it supports the use of a diverse
toolbox of learning activities in
order to assist a diverse group of
learners to succeed.

B203
Students may not have much
experience with active learning or
expectations placed upon them for

SHIP TO (please print):

The National Teaching & Learning Forum
their learning. The first-day

Money-back guarantee if you are not completely satisfied.

question activity is important for
getting student buy-in to why active-
learning strategies are used, and to
the partnership responsibilities of
instructor and student. Without this

City/State/Zip
introductory dialogue, the expecta-

)
Name/Title
tions of coming to class prepared,

Institution

)
Country
Address

Phone (
working with peers in class, and

Dept.

Fax (
completing frequent assessments of
Call: 608 • 255 • 4469
Fax: 509 • 267 • 1146

learning may be foreign to students

❏ I want to take advantage of bulk subscription savings.

and seem like too much work
compared to listening to lectures
($55 in Canada and elsewhere)

and regurgitating facts on exams.

However, once students acknowl-

Wouldn’t your entire

campus benefit from
SPECIAL PROMOTIONAL PRICE

a Site License to the

six-issue subscriptions.

edge the linkage between their

goals and the implemented
Yes, enter my subscription

learning methods, they have a new

❏ 1 year $49 in the United States

FORUM?
appreciation for why learner-
centered instructors do what they
do and learners come to value these
methods so long as they are used
effectively and they can measure
(Qty.)

their own learning.

Please begin

Contact:
❏ New

Gary A. Smith
Office of Support for Effective Teaching
MSC05 3400
University of New Mexico
Albuquerque, NM 87131
Telephone: (505) 277-2348
Subscription

(add $8.00 per subscription outside the United

E-mail: gsmith@[Link]
Bulk Discount Offer

Price Per
Domain*

$1,000.00
Site License Offer
Price per

$250.00
$500.00
$750.00
1,000+

References:
$42.50
$30.00
$27.50
$25.00
$23.25
$22.00

• Davis, B. G. 1993. Tools for Teaching.

San Francisco: Jossey-Bass.
• Felder, R. M., and Brent, R. 1996.
Subscriptions

No. of Full-

“Navigating the Bumpy Road to Student-

501 - 1,000

Centered Instruction.” College Teaching,

101 - 500
Faculty

v. 44, no. 2, 43-47.

No. of

1 - 100
20 - 49
50 - 74

Time
4 - 19
2-3

4 THE NATIONAL TEACHING & LEARNING FORUM Vol. 17, No. 5 2008

Team-Based Learning Institute Page 124 of 156

Extending  Smith’s  First  Day  Questions  
Helping  student  buy-­‐in  to  the  flipped  classroom  

 
You  can  introduce  students  to  TBL  and  why  you  are  using  it,  with  this  mock  Application  Activity  
based  on  Gary  Smith’s  activity  from  the  National  Teaching  and  Learning  Forum  Newsletter  article  
First-­‐Day  Questions  for  the  Learner-­‐Centered  Classroom  (NTLF  newsletter,  2008).    The  article  asks  
the  reader:  “Thinking  of  what  you  want  to  get  out  of  your  college  education  and  this  course,  which  
of  the  following  is  most  important  to  you?”  
1 Acquiring  information  (facts,  principles,  concepts)  
2 Learning  how  to  use  information  and  knowledge  in  new  situations  
3 Developing  life-­‐long  learning  skills  
The  teachers  will  give  time  for  intra-­‐team  discussion  leading  to  a  team  decision.  Then  the  teacher  
will  pass  out  the  TBL  voting  cards  and  ask  the  teams  to  simultaneously  report  by  holding  up  the  
card  that  corresponds  to  their  team’s  decision.  Then  you  can  facilitate  a  full  class  discussion  
contrasting  the  various  team  decisions.  This  activity  both  shows  the  students  the  mechanics  of  the  
Application  Activity  process  and  clearly  surfaces  differing  student  beliefs  on  what  good  classroom  
learning  should  look  like.  There  is  a  wonderful  way  to  extend  this  activity  (Smith,  2013).  At  the  
end  of  the  activity  students  are  asked  to  revisit  the  items  on  the  list  and  consider  which  of  the  
items  would  be  better  achieved  in  class  and  which  items  could  be  achieved  through  individual  
study.  They  will  quickly  zero  in  on  “acquiring  information”  as  something  they  could  do  on  their  
own.  You  can  then  revisit  the  format  of  TBL  and  show  them  that  is  exactly  how  TBL  is  structured,  
you  acquire  some  information  on  your  own  and  then  come  to  class  where  we  can  work  on  higher  
order  goals  like  application  and  life  long  learning  skills.  
 
References  
Smith,   G.   (2008).   First-­‐Day   Questions   for   the   Learner-­‐Centered   Classroom.   National   Teaching   and  
Learning  Forum  Newsletter.  Retrieved  from  [Link]  
 
Smith,   G.   (2013,   November)   Selling   Active   Learning   to   Faculty   Requires   a   Student   Purchase,   Too.    
Session  presented  at  38th  Annual  POD  Meeting,  Pittsburgh,  PA.  

Team-Based Learning Institute Page 125 of 156

Readiness  Assurance  Process  
Classroom  Logistics  

The   RAP   is   like   any   other   classroom   activity,   where   preparation   can   help   to   make   sure   the  
process  runs  smoothly.  
 
Pre-­‐‑Class  Preparation  
 
Many  teachers  use  team  folders.  Team  folders  are  
preloaded   with   the   test   question   sheets,   some  
kind  of  answer  recording  forms  (maybe  Scantron  
in  larger  classes),  and  an  appeals  form.  Folders  let  
you   simplify   the   handling   of   materials.   In   large  
classes,   we   ask   a   team   representative   to   come   to  
the  front  of  the  class  to  pick  up  and  drop  off  their  
team’s   folder,   so   the   teacher   remains   at   the   front  
of   the   class.   Creating   these   folders   for   each   team  
not   only   simplifies   getting   materials   to   and   from  
the  teams;  it  sends  the  important  message  to  your  
students   that   you   have   taken   time   to   be  
organized.  
 
Timing  
 
The   typical   RAP   takes   50-­‐‑70   minutes   for   a   20-­‐‑question   test.   In   shorter   classes,   teachers   will  
often   shorten   the   RAP   test.   For   our   50-­‐‑minute   classes,   we   often   give   12-­‐‑15   questions   in   50  
minutes;   this   gives   us   time   to   complete   the   entire   five-­‐‑stage   RAP   process.   Many   TBL  
practitioners  now  recommend  even  shorter  10  question  RAPs,  since  student  are  often  eager  to  
get  to  4S  problem-­‐‑solving  main  events.  
 
Class  Start  
 
We  start  by  announce  there  will  be  a  RAP  and  how  much  time  students  will  be  given  for  the  
iRAT   and   tRAT.   A   general   rule   of   thumb   is   three   to   five   minutes   for   both   folder   distribution  
and  time  for  students  to  get  their  names  on  the  answer  forms,  then  one  minute  per  question  on  
the  iRAT,  and  slightly  longer  for  tRAT  (1.5  minutes  per  question).    How  long  you  need  to  give  
is  ultimately  controlled  by  the  difficulty  of  the  questions.  Stray  on  the  side  of  making  questions  
easier,  rather  than  harder.  Hard  or  tricky  questions  can  quickly  burn  up  student  goodwill.  Be  
careful.  
 
The  iRAT  (Individual  Readiness  Assurance  Test)  
 
To   begin   the   Individual   Readiness   Assurance   Test   (iRAT),   we   ask   students   to   put   away   any  
notes  or  other  reading  materials.  We  then  ask  one  representative  from  each  team  to  come  to  the  
front   of   the   class   to   pick   up   their   team’s   folder.   Teams   are   not   to   open   their   folders   until   all  

Team-Based Learning Institute Page 126 of 156

teams  are  reseated.  Next,  we  ask  the  teams  to  open  their  folders,  distribute  the  tests,  and  begin.  
While   the   students   are   completing   the   iRAT,   we   circulate   around   the   room   and   clarify   any  
difficulties  that  students  may  have  understanding  the  questions.    
 
Once  the  allotted  iRAT  time  has  elapsed,  students  are  asked  to  collect  their  team’s  individual  
answer   sheets   and   send   one   representative   to   the   front   of   the   room   to   exchange   the   answer  
sheets  for  their  IF-­‐‑AT  card.  Students  are  reminded  to  hang  on  to  their  question  sheets  for  the  
team  Readiness  Assurance  Test  (tRAT).    
 
Students   who   are   absent   on   the   day   of   a   RAP   typically   receive   a   zero   for   both   the   iRAT   and  
tRAT   unless   they   have   satisfied   some   other   predetermined   course   requirements.   These  
requirements  could  be  to  make  plans  with  the  teacher  in  advance  to  take  the  iRAT  separately,  
to   provide   a   medical   note   in   order   to   be   excused   from   the   RAP   and   have   the   tests   excluded  
from   grading,   or   to   require   a   signed   note   from   their   teammates   with   permission   to   share  the  
team  score.  In  this  latter  case,  if  a  student  has  been  a  prepared  and  consistent  contributor  to  the  
team,  most  teammates  are  happy  to  share  their  team  score  with  the  absent  student.  
 
The  tRAT  (Team-­‐‑Readiness  Assurance  Test)  
 
Before  starting,  we  remind  students  of  any  decremental  scoring  scheme  we  might  use  with  the  
IF-­‐‑AT   cards.   On   a   four-­‐‑option   IF-­‐‑AT   card   (A-­‐‑D),   we   often   award   four   points   for   a   correct  
answer   on   the   first   scratch,   two   points   for   the   second   scratch,   one   point   for   the   third   scratch  
and   zero   if   they   needed   to   scratch   all   four   possible   answers.   Different   teachers   use   different  
decremental   scales.   Whatever   scale   you   use,   the  
important   thing   is   that   you   are   rewarding   the  
students   for   continuing   to   discuss   the   question  
seriously.  Otherwise,  after  one  incorrect  scratch,  you  
would   just   rub   the   rest   off   and   reveal   all   the   other  
answers   and   a   valuable   learning   opportunity   would  
be  lost.    
 
During   the   tRAT,   we   circulate   around   the   room  
monitoring  the  students’  progress.  If  we  notice  that  a  
large  number  of  teams  are  finished  before  the  allotted  
time   is   up,   we   will   ask   the   whole   class   who   needs  
more  time.  If  only  a  few  teams  need  more  time,  then  
we   will   often   announce   that   there   are   perhaps   two  
minutes  left  (known  as  the  2  minute  rule).  
 
You   write   the   question   numbers   on   the   board   and  
ask  teams  to  put  check  marks  beside  the  question  or  
questions  they  would  like  clarified.  This  is  actually  a  very  good  option,  since  some  concepts  are  
clarified   enough   during   the   tRAT   that   students   do   not   need   further   clarification.   If   you   only  
work   from   the   tally   report   of   the   individual   tests,   you   may   end   up   talking   about   topics   that  
were  already  resolved  during  the  team  test.  

Team-Based Learning Institute Page 127 of 156

  
Teachers   often   record   the   teams’   average   iRAT   scores   and   the   range   of   tRAT   scores   on   the  
chalkboard.  This  achieves  two  goals.  First,  it  shows  students  the  value  of  working  as  a  team,  as  
tRAT   scores   are   usually   10-­‐‑20%   higher   than   the   average   iRAT   score.   Secondly,   it   motivates  
teams  to  perform  better  than  their  peers  by  creating  friendly  competition.  
 
The  Appeal  Process  
 
At  end  of  the  team  test  (tRAT),  teams  are  encouraged  to  appeal  questions  that  they  got  wrong.  
Scholarly  appeal  arguments  can  be  generated  by  any  team  and  are  written  on  the  appeal  form  
included   in   each   team’s   folder.   The   teacher   makes   it   clear   that   the   appeals   will   only   be  
considered  outside  of  class  time  and  that  the  results  will  be  announced  at  the  next  class.  Some  
student   will   try   to   open   a   conversation   about   why   a   particular   answer   should   be   considered  
correct;   you   can   shut   the   conversation   down   by   simply   asking   the   team   to   complete   that  
appeals  form  and  you  will  look  at  it  after  class.  You  need  to  establish  a  rule  of  when  appeals  
are   due.   Some   teachers   insist   by   end   of   class;   other   teachers   insist   by   end   of   day,   often  
submitted  by  email.  One  way  to  ensure  that  all  team  members  have  contributed  to  the  appeal  
is  to  have  them  sign  a  statement  of  collaboration  at  the  bottom  of  the  appeal  form.  
 
Teacher  Clarification/Mini-­‐‑Lecture  
 
At   the   end   of   the   testing   and   appeal   phases,  
teachers  respond  to  items  identified  in  the  RAP  
as   still   challenging   to   students.   You   must   not  
review   the   test   question   by   question,   but   only  
review   the   questions   and   concepts   that   remain  
problematic   for   the   students.   Students   like   the  
mini-­‐‑lecture   since   they   know   it   won’t   be   too  
long  and  the  teacher  is  talking  about  something  
they  know  they  don’t  know.  
 
Ending  Class  Well  
 
Students   are   asked   to   place   all   question   sheets   and   the   IF-­‐‑AT   form   in   their   team   folder   for  
collection.  We  often  get  team  members  to  sign  the  back  of  the  IF-­‐‑AT  card,  as  this  can  simplify  
requests  made  by  absent  students  who  still  want  credit.  We  remind  students  that  all  question  
sheets  must  be  returned  or  the  whole  team  will  receive  a  penalty,  usually  a  mark  of  zero.  We  
mark   the   folders   with   the   number   of   students   in   each   team;   this   lets   us   easily   check   that   all  
question   sheets   are   returned.   A   team   representative   is   then   asked   to   bring   the   team   folder   to  
the  front  of  the  room.  
 
Read  more  in  Getting  Started  with  Team-­‐‑Based  Learning  –  page  74-­‐‑113  
 
 

Team-Based Learning Institute Page 128 of 156

4S  Activity  Facilitation  
Classroom  Logistics  

“The design of a task is ultimately only as good as its execution and management”
(Roberson and Franchini, 2014, p. 297).

Many people new to TBL worry about facilitating the discussion after simultaneous reporting.
Although we all have much to learn about facilitation, the good news is the 4S structure of TBL
activities lead discussions that are simpler to facilitate than open general discussions.

P LANNING  AN  A PPLICATION  A CTIVITY  

The organization of an effective TBL Application Activity parallels that of any effective teaching
activity; it needs a well-thought-out structure with a beginning, middle, and end. There are
many frameworks used to describe such structures, but they generally have the same elements
grouped in different ways. I will use one of the simpler top-level frameworks to describe
structure: Set, Body, Close. Feel free to adapt your personal favourite if you have one.

Set, Body, Close is highly adaptable for almost any size group and any duration. It aligns,
respectively, with the beginning, middle, and end of an activity or lecture. The Set portion sets
the stage and primes the learners for what is to come; most importantly, it establishes the tone,
conveys why the topic is important, and outlines the objectives. The Body is the core of the
class; in a conventional lecture, this would be where the teacher delivers content, while in a
TBL class, this is the Application Activity (including teams working on the exercise as well as
the discussion and debriefing that follow). Finally, the Close wraps everything up. It
summarizes the class or activity, emphasizes what has been accomplished, and relates the
outcomes to the objectives from the Set.

Team-Based Learning Institute Page 129 of 156

FACILITATING  SIMULTANEOUS  REPORTING  
F IRST  REPORT  THE  PATTERNS  YOU  SEE  

The first thing you do, following simultaneous report, is announce the patterns you see...I see
mostly A's and C's with a few D's - then you begin asking teams why they made their decision -
going team to team and building a reasoned argument together with your students.

P ROBE  STUDENT  THINKING  

When teams commit to their positions

and publically report it, the instructor
can then facilitate the report by simply
going team to team asking “why did
your team pick A” going to another
team “why did you pick B”. You work
the room going side to side and
making sure to engage all parts of the
room. Try to resist talking to teams
next to you. This can become a
conversation between the instructor
and one team and not a conversation between all teams. One trick is to ask teams across the
room to add their two cents. This helps the reporting conversation stay between students and
not between instructor and one team of students. You can ask a team to tell you about their
team’s deliberations – what did they talk about, how did you decide…What is nice about this is
you are only asking them to recount the conversation, not what is the right answer.

Team-Based Learning Institute Page 130 of 156

H ELP  STUDENTS  BUILD  A  WELL  CONSIDERED  ARGUMENT  

When discussion energy begins to wane you can begin to ask “why didn’t you pick…” or “was
there a second choice that your team considered?”…”why do you think someone might pick that
choice?” These are the same kinds of questions that you need to use to play devil’s advocate if
all the teams agree.

It can be helpful to remember the components of a well-reasoned argument (Toulin et al, 1984)

1. CLAIM is made
2. GROUNDS (facts, evidence) are offered
3. WARRANT - connects claim to grounds
4. BACKING – a theoretical or experimental foundation for warrant is shown
5. Appropriate QUALIFIERS are used to temper claim (some, most, many, few)
6. Possible REBUTTALS are considered

Remembering the pieces of a well reasoned argument can help you push the students to deeper
engagement, analysis. and argumentation.

C LOSING  WELL  –  SO  WHAT  HAS  BEEN  LEARNED  CAN  BE  REINFORCED  

Another important consideration is to close the discussion well. You want to make sure
students get reminded of the important take-aways, the assumptions examined, and the
inferences that needed to be made. You can summarize or even better have the students
paraphrase a summary of the discussion. Reflective one minute paper can be used to great
effect here. You can simply ask students to quickly individually list the “3 most important
points” or “2 remaining concerns” or “a context where it might not be applicable.” A nice finish
to this activity is to have teams compile these points into a team consensus worksheet. However
you do it, not closing activities well robs them of some of their value.

Read  more  in  Getting  Started  with  Team-­‐‑Based  Learning  –  page  114-­‐‑142  

Team-Based Learning Institute Page 131 of 156

Team-Based Learning Institute Page 132 of 156
Marie Thomas
Professor of Psychology
California State University San Marcos
 
TBL  Module:  Categorical  Data  and  Chi-­‐Square  
 
Table  of  contents  
• Introduction  (p.  1)  
• Reading  assignment  and  learning  outcomes  (p.  1)  
• Readiness  assurance  test  (pp.  3-­‐4)  
• Plan  of  class  (p.  2)  
• Introductory  application  activity  (pp.  5-­‐6)  
• Ending  application  activity  (p.  7-­‐8)  
 
Introduction  
This  module  is  one  I  use  in  my  advanced  psychological  statistics  course.    The  name  of  the  course  is  
deceptive;  it  is  actually  intermediate  level.    I  discovered  early  on  that  students  often  remembered  little  
from  their  introductory  course  so  my  class  has  evolved  to  the  point  where  I  spend  the  first  half  of  the  
semester  reviewing  introductory  statistics  topics  at  a  deeper  level;  I  then  move  to  new  material  after  the  
midterm.  
 
Most  of  the  topics  in  this  module  are  covered  in  introductory  statistics  classes  and  the  concepts  are  
quite  elementary.    Yet  I  find  that  students  have  a  great  deal  of  trouble  understanding  these  statistical  
techniques.    It  may  be  because  it’s  often  the  last  topic  covered  in  the  introductory  course  and  
instructors  may  not  have  enough  time  to  do  it  justice.    Whatever  the  reason,  this  module  requires  more  
repetition  of  concepts  than  would  seem  necessary  or  appropriate  for  a  class  at  this  level.    But  I  like  the  
application  activities,  as  do  the  students  who  find  them  very  useful.    This  module  typically  requires  4  to  
5  hours  of  class  time.  
 
Reading  assignment  and  learning  outcomes  
I  use  a  textbook  for  this  course  and,  in  the  first  half  of  the  semester,  cover  approximately  one  chapter  
per  week.    The  reading  assignment  for  this  module  is  Ch.  6  in  David  Howell’s (2013) Statistical
methods for psychology (8th Ed.). Belmont, CA: Thomson.  
 
I  create  a  “reading  guide”  for  each  module  which  is,  essentially,  the  learning  outcomes.    I  don’t  expect  
my  students  to  understand  the  entire  chapter  without  some  help.    I  highlight  the  outcomes  that  will  be  
covered  by  RAT  questions,  but  students  understand  that  for  the  midterm/final  they  are  responsible  for  
understanding  all  of  the  listed  concepts.  
 
• I  can  describe  the  characteristics  of  the  chi-­‐square  distribution.    
• I  can  define  the  terms  expected  and  observed  frequencies.    
• I  can  perform  a  chi-­‐square  goodness-­‐of-­‐fit  test  by  hand  and  using  statistical  software,  and  can  
interpret  the  results.  
• I  can  perform  chi-­‐square  contingency  table  analysis  (test  of  independence)  by  hand  and  using  
statistical  software,  and  can  interpret  the  results.  
• I  can  describe  the  problems  associated  with  small  expected  frequencies.  
• I  can  describe  the  assumptions  associated  with  chi-­‐square.    

1  
 

Team-Based Learning Institute Page 133 of 156

• I  can  describe  one  way  to  handle  dependent  or  repeated  measurements  using  chi-­‐square.  
• I  can  explain  and  demonstrate  how  to  calculate  d-­‐family  and  r-­‐family  measures  of  effect  size.  
• I  can  explain  and  demonstrate  how  to  calculate  kappa.  
 

Plan  of  class  

• iRAT  and  tRAT  
• Introductory  application  activity:  What  kind  of  chi-­‐square?  (see  p.  5)  
• Mini-­‐lecture:  Introduction  to  chi-­‐square  
• Challenge!    Teams  are  presented  with  data  we  used  earlier  in  the  semester  for  an  activity  on  
distributions  and  hypothesis  testing.    The  data  are  most  appropriately  analyzed  using  chi-­‐square.    I  
give  the  teams  a  few  minutes  to  see  if  they  can  figure  out  what  kind  of  chi-­‐square  to  use,  and  return  
to  the  problem  later.  
• Mini-­‐lecture:  Chi-­‐square  goodness  of  fit  test  
• Students  work  on  a  goodness  of  fit  test  problem  (done  by  hand)  within  their  teams;  we  discuss  the  
results  as  a  class.  
• Mini-­‐lecture:  Chi-­‐square  test  of  independence  
• Students  work  on  the  challenge  problem  (done  by  hand)  within  their  teams;  we  discuss  the  results  
as  a  class.  
• Mini-­‐lecture:  Assumptions  of  chi-­‐square  
• Mini-­‐lecture:  Effect  size  measures  
• Students  work  on  effect  size  problems  (done  by  hand)  within  their  teams;  we  discuss  the  results  as  a  
class.  
• Students  work  on  two  problems  using  the  computer  and  SPSS  
• Final  team  activity:  “Meditation  increases  compassionate  responses  to  suffering”  (see  pp.  7-­‐8)  

   

2  
 

Team-Based Learning Institute Page 134 of 156

Readiness  Assurance  Test  
1. In a chi-square test, the variables are:
a.   interval  level.  
b.   rank-­‐order  (ordinal).  
c.   ratio  scale.  
d.   categorical  (nominal).  
 
2.   The  values  in  a  chi-­‐square  distribution  are  always  greater  than  0  and:  
a.   less  than  1.  
b.   are  normally  distributed.  
c.   can  be  quite  large.  
d.   are  negatively  skewed.  
 
3.   The  main  idea  of  a  chi-­‐square  test  is  that  you:  
a.   test  the  estimated  degree  of  fit  (proportion  of  variance  accounted  for)  of  one  variable  to  the  
other  variable.  
b.   test  how  well  the  pattern  of  observed  frequencies  fits  some  expected  pattern  of  frequencies.  
c.   compare  the  estimated  population  variances,  to  see  if  they  vary  from  each  other  more  than  by  
chance.  
d.   compare  the  estimated  population  means,  to  see  if  they  vary  from  each  other  more  than  by  
chance.  
 
4.    The  degrees  of  freedom  for  the  chi-­‐square  goodness  of  fit  test  are  the:  
a.   mean  number  of  individuals  per  category,  minus  1.  
b.   number  of  categories  minus  1.  
c.   mean  number  of  individuals  per  category,  minus  the  number  of  categories.  
d.   total  number  of  individuals,  minus  the  number  of  categories.  
 
5.   In  a  chi-­‐square  goodness  of  fit  test,  the  null  hypothesis  is  that  the:  
a.   number  of  people  in  one  category  is  no  greater  than  the  number  of  people  in  the  other  
categories.  
b.   variances  of  the  populations  of  categories  are  the  same.  
c.   means  of  the  populations  of  categories  are  the  same.  
d.   observed  proportion  of  people  over  categories  does  not  depart  from  what  is  expected  by  
chance.  
 
6.   In  chi-­‐square,  the  expected  frequencies  are  the  frequencies  we  would  expect  if  the:  
  a.   null  hypothesis  is  true.  
b.   null  hypothesis  is  false.  
c.   research  hypothesis  is  true.  
d.   research  hypothesis  is  false.  
 
7.   Contingency  table  analysis  is  sometimes  called  the  chi-­‐square  test  of  independence.    This  is  because,  
under  the  null  hypothesis,  the:  
a.   variables  are  not  related.  
b.   variables  in  the  table  should  never  be  compared.  
c.   observed  frequencies  are  larger  than  the  expected  frequencies.  
d.   distribution  of  one  variable  varies  over  different  levels  of  the  other.  

3  
 

Team-Based Learning Institute Page 135 of 156

  
8.   The  degrees  of  freedom  for  a  contingency  table  analysis  is  the:    
  a.   total  number  of  category  levels  minus  1.  
b.   number  of  categories  minus  1.  
c.   number  of  participants  minus  the  number  of  cells.  
d.   number  of  columns  minus  1,  times  the  number  of  rows  minus  1.  
 
9.   One  advantage  of  the  chi-­‐square  test  over  most  other  inferential  statistical  procedures  is  that  it:  
a.   can  use  the  distributions  of  any  other  statistical  procedure  as  a  comparison.  
b.   has  minimal  assumptions  about  populations.  
c.   can  be  easily  applied  to  repeated-­‐measures  designs.  
d.   does  not  require  as  many  participants.  
 
10.   Which  of  the  following  would  NOT  be  allowed  in  an  ordinary  application  of  the  chi-­‐square  test?  
a.   The  number  of  successes  of  an  advertisement  is  compared  for  three  different  groups  of  people  
to  see  if  the  number  of  successes  is  equal  for  the  three  groups.  
b.   Tall  and  short  people  are  compared  on  which  religion  they  belong  to.  
c.   The  number  of  people  of  different  ethnicities  working  in  a  particular  company  is  compared  to  
the  proportions  of  those  ethnicities  in  the  general  public.  
d.   Students'  preference  for  studying  while  sitting  or  lying  down  is  assessed  as  sophomores  and  
then  again  as  seniors.  
 

   

4  
 

Team-Based Learning Institute Page 136 of 156

Introductory  application  activity  
For  this  activity,  each  team  is  given  a  sheet  with  the  following  scenarios  and  response  options,  as  well  as  
a  small  whiteboard  and  marker.    Teams  are  given  a  couple  of  minutes  to  read  each  scenario  and  there  is  
a  race  to  see  which  team  will  hold  up  its  whiteboard  with  an  answer  first.    When  there  is  disagreement,  
teams  are  asked  to  defend  their  answer.    This  activity  generates  a  great  deal  of  energy  and  gets  students  
ready  to  explore  topics  in  greater  depth.  
 
What  kind  of  chi-­‐square?  
 
1.   A  gym  wanted  to  assess  member  satisfaction  with  a  new  fitness  program.    They  compared  members  
who  had  joined  the  fitness  program  to  members  who  had  not  joined  the  fitness  program  by  asking  
them  whether  they  were  satisfied  or  not  satisfied  with  their  weight  loss  in  the  last  six  months.  They  
hypothesized  that  members  who  had  joined  the  fitness  program  were  more  likely  to  be  satisfied  
with  their  weight  loss.  
a.   Chi-­‐square  goodness  of  fit  test  
b.   Contingency  table  analysis  (chi-­‐square  test  of  independence)  
c.   This  can’t  be  analyzed  using  a  chi-­‐square!!  
 
2.   A  pharmaceutical  company  wants  to  determine  whether  a  sleeping  pill  is  effective.  They  randomly  
assign  individuals  either  to  take  a  sleeping  pill  or  to  take  a  placebo.  They  compare  the  amount  of  
time  participants  are  asleep.  They  hypothesize  the  sleeping  pill  group  will  sleep  longer  than  the  
placebo  group.    
a.   Chi-­‐square  goodness  of  fit  test  
b.   Contingency  table  analysis  (chi-­‐square  test  of  independence)  
c.   This  can’t  be  analyzed  using  a  chi-­‐square!!  
 
3.   In  a  poll  200  residents  in  a  small  town  were  asked  whether  they  supported  the  construction  of  a  
new  mall.  They  responded  whether  they  did,  they  didn’t,  or  whether  they  had  no  preference.      
a.   Chi-­‐square  goodness  of  fit  test  
b.   Contingency  table  analysis  (chi-­‐square  test  of  independence)  
c.   This  can’t  be  analyzed  using  a  chi-­‐square!!  
 
4.   A  company  that  stocks  a  vending  machine  wants  to  optimize  sales.  The  company  offers  five  different  
beverage  options  and  wants  to  know  whether  some  beverages  are  more  popular  than  others.      To  
determine  this,  the  numbers  of  beverages  of  each  kind  sold  in  one  week  are  counted.  
a.   Chi-­‐square  goodness  of  fit  test  
b.   Contingency  table  analysis  (chi-­‐square  test  of  independence)  
c.   This  can’t  be  analyzed  using  a  chi-­‐square!!  
 
 
 

5  
 

Team-Based Learning Institute Page 137 of 156

5.   An  ad  agency  wants  to  see  if  a  set  of  complimentary  pens  they’ve  designed  are  gender  neutral.  They  
place  both  their  green  and  yellow  pens  in  a  can  and  keep  track  of  how  many  males  and  females  pick  
up  each  color  of  pen.  
a.   Chi-­‐square  goodness  of  fit  test  
b.   Contingency  table  analysis  (chi-­‐square  test  of  independence)  
c.   This  can’t  be  analyzed  using  a  chi-­‐square!!  
 
6.    A  researcher  is  testing  how  effective  a  list  of  words  is  at  causing  subjects  to  recall  a  specific  word  
that  was  not  listed.  The  researcher  collects  data  from  males  and  females  during  their  freshman  year,  
and  then  again  during  their  sophomore  year  and  then  compares  each  person's  data  from  the  two  
years.  
a.   Chi-­‐square  goodness  of  fit  test  
b.   Contingency  table  analysis  (chi-­‐square  test  of  independence)  
c.   This  can’t  be  analyzed  using  a  chi-­‐square!!  
   

6  
 

Team-Based Learning Institute Page 138 of 156

Final  application  activity  
 
For  this  activity,  students  are  given  the  introduction  and  method  section  of  the  following  article  (see  file  
meditation_compassion)  :      Condon,  P.,  Desbordes,  G.,  Miller,  W.  B.,  &  DeSteno,  D.  (2013).  Meditation  
increases  compassionate  responses  to  suffering.  Psychological  Science,  20,  1-­‐3.      
 
After  reading  these  short  sections,  students  complete  a  worksheet  as  a  team,  which  I  collected  at  the  
end  of  class.    This  article  works  perfectly  for  two  reasons.    First,  it  only  reports  and  discusses  two  
statistics,  both  of  which  appear  in  this  module  (chi-­‐square  and  odds  ratios).    Second,  after  taking  a  
contemplative  pedagogy  workshop  in  August,  I  integrated  mindfulness  techniques  through  the  entire  
semester  of  this  particular  class,  so  students  were  primed  to  be  interested  in  the  topic.  
 
While  this  is  not  a  traditional  4S  activity,  it  pulled  several  pieces  of  the  module  and  course,  in  general,  
together  and  was  a  good  ending  to  the  topic.  
 
 

 
 

7  
 

Team-Based Learning Institute Page 139 of 156

Team:  

 
Meditation  increases  compassionate  responses  to  suffering  
 
1.   Read  the  introduction  and  method  sections  of  the  article.  
 
2.   What  is  the  research  hypothesis  (is  there  more  than  one  hypothesis)?  
 
 
 
3.   What  is  the  null  hypothesis?  
 
 
 
4.   The  authors  did  not  find  a  difference  in  behavior  as  a  function  of  meditation  protocol;  therefore,  the  
analysis  uses  data  that  collapsed  participants  in  the  two  meditation  conditions.    The  results  of  the  
study  are  below.  
 
  Meditation   Waiting-­‐list   Total  
training   control  
No  help   10     16       26  
Help   10        3   13  
Total   20   19   39  
 
a.   What  type  of  analysis  should  be  done  on  these  data?    Perform  the  analysis;  report  and  interpret  
your  result  below.  
 
 
 
 
b.   Calculate  an  r  measure  of  effect  size;  report  and  interpret  the  result.  
 
 
 
 
c.   Calculate  the  odds  ratio  (a  d  measure  of  effect  size);  report  and  interpret  the  result.  
 
 
 
 
5.   What  are  you  overall  conclusions  about  this  study?  
 
 
 
 
6.   Do  you  notice  anything  interesting  about  the  data?    What  questions  are  you  left  with?  

8  
 

Team-Based Learning Institute Page 140 of 156

Rick Goedde
Director of Management Studies
Associate Professor
Department of Economics
St. Olaf College
Northfield, MN 55057
goedde@[Link]

TBL Module - Developing & Testing Your Own Stock Screen

Context for this module

This module is taken from an introductory undergraduate course in investments and has student teams
create their own stock screen using a computer database containing financial information for
approximately 9,000 publicly traded companies. Students must develop criteria to screen the database for
high potential stocks as of a given past date and then test to see how these stocks would have done after
the historical purchase date. Stock screening is one of six modules in the course. The other five are
equity analysis, fixed income securities, options/futures, mutual funds/exchange-traded funds, and
portfolio theory.

1. What are the goals and objectives of this lesson?

This module is useful for students interested in careers in investments, especially as an equity analyst.
Stock screening is a valuable starting point in stock selection, but it is important to emphasize the need for
further fundamental analysis of each stock. After completing the module, students should be able to:
a. Identify stock characteristics that have historically led to high-performance.
b. Identify screening criteria that are invalid or illogical.
c. Optimize a back-tested stock screen.

2. Give a brief description of the pre-class preparation.

In addition to assigned reading for this module, students must also individually work through a tutorial
that I developed to help them understand how to screen and back-test a screen using the stock database.
Their work with the tutorial is not graded, but the students understand that doing well on the RAT and the
team application activity is dependent upon working through the tutorial carefully. The tutorial is
included in this document at the end of the instructions for the team application activity.

3. Show the readiness assurance tool (RAT).

Below is a ten-question RAT with the answer key.

4. Show the application exercises and how the answers will be managed by the teacher/facilitator of the
TBL session.

The instructions for the team application activity are at the end of this document. Below is a summary of
how this activity adheres to the four Ss:

Significant problem: This activity provides students with hands-on experience screening a stock database.
They not only learn the technique for screening and back-testing the screen to optimize it, but they also
get to work with the type of data that practitioners use. This problem is also significant in that students
could test endless combinations of criteria in order to optimize their screen. There is no correct answer,

Team-Based Learning Institute Page 141 of 156

only the highest return-to-risk ratio in the class and comparisons to what other teams have done in
previous semesters.

Same problem: All teams are trying to optimize a stock screen using the same database so that they can
compare results (screening criteria) to the other teams.

Specific choice: Each team will be reporting the screening criteria which they hope will lead to the
highest return-to-risk ratio of any team in the class.

Simultaneous report: After using their computers to develop and test their screening criteria, each team
writes its best set of criteria on an anonymous poster to hand in to the instructor at the deadline. The
instructor tapes these posters to the classroom walls for a gallery walk. Teams are trying to identify
invalid and illogical criteria, and they are voting for the screen (other than their own) which they believe
will result in the highest return-to-risk ratio. A classroom discussion follows where teams have to justify
their choice and explain any invalid or illogical criteria they found.

5. Describe any other aspects of the module or the course you find relevant (e.g. course content, peer
evaluation, grading, appeals process, facilitator notes.)

Appeals Process: Students can appeal an incorrect answer on the tRAT. I use the appeals process as
explained in the Michaelsen, Knight, and Fink book (2004).

Peer Evaluation: Students do peer evaluation twice during the 13-week semester, once during Week 5 and
then again at the end of the course. The Week 5 evaluation is not included in the course grade, but is
designed to give each student performance feedback from the team and to give each student practice in
providing constructive criticism. Students answer two questions for each teammate and him or herself:
What has been Teammate X’s most significant contribution to the team?
In what way could Teammate X most improve his or her contribution to the team?
This feedback is given to each student without revealing the name of the evaluator. Students lose points
if they do not hand in an evaluation for each teammate, or if they do not make a serious attempt at
providing constructive criticism. There is no quantitative part to this first peer evaluation.

At the end of the course, students answer the same two questions plus a question on if and how the
teammate has improved between the two evaluations. In addition each student must divide up a point
total (10 points times the number of teammates, excluding the evaluator) among the teammates with the
requirement that the same number of points cannot be given to every teammate (at a minimum, one
teammate must receive a 9 while another one receives an 11). These points are included in the final
course grade.

Team-Based Learning Institute Page 142 of 156

Mgt. Studies 380: Investments RAT #X: Developing & Testing Your Own Stock Screen
Prof. Rick Goedde

1. The primary purpose of creating a stock screen from a stock database is

A. to remove bias and emotion from the stock selection process.
B. to identify stocks characteristics which have historically proven to be successful.
C. to provide a starting point for further analysis of the stocks on the screened list.
D. to gain an appreciation for the fact that one or two excellent stocks can overcome a number of
mediocre or poor stocks.

2. The Stock Investor Pro database includes pre-defined stock screens based on the screening principles
of well-known investors. Which one of the following statements about these screens is FALSE?
A. The creators of the database developed the screens by reading books about the investor’s style.
B. These pre-defined screens have been tested over many years, so the resulting stocks should be
purchased without further analysis in order to remove bias from the selection process.
C. The screens are tested over many years by running them monthly and adjusting the stock holdings
each month.
D. Cash dividends are not included in the returns from testing the pre-defined screens.

3. Which of the following statements best summarizes O’Shaughnessy’s Cornerstone Growth Strategy?
A. Select stocks with strong earnings growth, but don’t pay too much for them (low price-to-
sales ratio).
B. Select values stocks (low price-to-sales ratio) of large companies (market capitalization >
$150).
C. Select a well-diversified portfolio (50 companies) of large company (market capitalization
> $150) value stocks (low price-to-sales ratio).
D. Select stocks that the market likes (rising price), but don’t pay too much for them (low
price-to-sales ratio).

4. Related to screening a stock database, which one of the following statements is FALSE?
A. Back-testing means that you assume you are at some date in history with no knowledge of
the future. Screen the database as of that date and see how the selected stocks would
have done in the future.
B. Survivor bias results from the fact that a stock database includes companies that existed in
the past, but went out of business.
C. Including a minimum liquidity criterion in a screen is important because there are many
stocks in the database that cannot be bought or sold at a fair price.
D. All valuation criteria must include the purchase price of the stock.

5. The annual rate of return of a portfolio can be calculated by adding up the annual return for each
stocks in the portfolio and dividing by the number of stocks in the portfolio. This simple method can be
used only if a certain assumption is true. Which one of the following assumptions must be true?
A. All of the stocks are purchased at the same time.
B. None of the stocks can have a negative annual return.
C. An equal dollar amount is invested in each stock.
D. All of the stocks are sold at the same time

Team-Based Learning Institute Page 143 of 156

6. Maximizing the historical annualized rate of return through back-testing would seem to be best way to
measure the success of a stock screen because investors want the highest return possible. Why is it
preferable to try to maximize annual return divided by risk (standard deviation of returns) instead?
A. Screens that have significant losses on their way to high returns might cause the investor
to sell out of fear before the high returns can be realized.
B. Risk and return are positively correlated. Finding high risk stock screens is one way to
maximize return.
C. Using return divided by risk is required to make apples-to-apples comparisons between
stock screens.
D. Including risk in the formula allows one to compare results to published statistics from the
Securities and Exchange Commission (SEC).

7. Which of the following statements best summarizes the “ten-bagger concept?”

A. A stock (not purchased on margin) can only lose 100% of its cost, but there is no limit to
how much it can gain (200%, 500%, 1000%, etc.).
B. A low-priced stock that increases in value by ten times its cost or more will make almost
any stock portfolio profitable.
C. Screening a stock database is the highest-probability way to identify a stock that multiplies
in value.
D. Stocks under $10 per share have the best chance of gaining significantly in value.

8. Assume that you are using the Stock Investor Pro database to test a stock screen as of the date, M13.
One of the following criteria for this test is invalid. Which one is it?
A. P13/P25-1 < .40
B. Sales Y1 > Sales Y2
C. P13/(Sales Y2/Shares Average Y2) < 1.5
D. PE-Average Y3 < 20

9. Assume that you are using the Stock Investor Pro database to test a stock screen as of the date, M13.
All of the following criteria are valid. Which one is the LEAST logical?
A. P13 < 5
B. P13*Price-Volume M013/21 > 1000
C. Return on equity Y2 > 15
D. P13/EPS-Continuing Y2 > 90

10. Assume that the back-testing of your screen results in the following four annual returns: 10%, -20%,
40%, and 30%. Which of the following formulas correctly calculates the return-to-risk ratio in Excel?
A. =((.1*-.2*.4*.3)^(1/4))/STDEV(.1,-.2,.4,.3)
B. =((1.1*.8*1.4*1.3)^(1/4)-1)/STDEV(1.1,.8,1.4,1.3)
C. =((1.1*.8*1.4*1.3)^(1/4)-1)/STDEV(.1,-.2,.4,.3)
D. =((.1*-.2*.4*.3)^(1/4))/STDEV(1.1,.8,1.4,1.3)

Answer Key: 1C 2B 3D 4B 5C 6A 7A 8B 9D 10C

Team-Based Learning Institute Page 144 of 156

Mgt. Studies 380: Investments
Prof. Rick Goedde

Developing & Testing Your Own Stock Screen

Overview & Project Objectives

Your team will develop criteria for a stock screen using the Stock Investor Pro (SI) database and then test
the screen over four years. The goal is to maximize the return-to-risk ratio (average annual portfolio
return over the four years divided by the standard deviation of those returns). This project involves
reporting only your best screening criteria to the class. Stock screening is a valuable starting point in
stock selection, but it is important to emphasize the need for further fundamental analysis of each
screened stock. After completing this project, you should be able to:
d. Identify stock characteristics that have historically led to high-performance.
e. Identify screening criteria that are invalid or illogical.
f. Optimize a back-tested stock screen.

Detailed Instructions

1. Each person on your team should work through the tutorial starting on page 3 of this document. We
will discuss this tutorial in class. Feel free to e-mail questions to me, call or visit during office hours (or
set up an appointment for a different time), or ask questions about the tutorial in class. Working through
this tutorial will help you to do well on the RAT and the team application activity itself.

2. Use the SI database to develop a stock selection screen. Your goal is to maximize the return-to-risk
ratio. You must have exactly five stocks in your portfolio each year. This means that one of your criteria
must have a variable minimum or maximum parameter to get exactly five stocks.

3. Calculate your portfolio return for each of the following four years:

a. April 30, 1998 (M049) to April 30, 1999 (M037)

b. April 30, 1999 (M037) to April 30, 2000 (M025)
c. April 30, 2000 (M025) to April 30, 2001 (M013)
d. April 30, 2001 (M013) to May 3, 2002 (M001)

Over the first two years, the S&P 500 index rose 31 percent (a bull market). Over the last two years, the
index fell 26 percent (a bear market). This will allow you to test your system over very different market
conditions.

4. One of the criteria in your screen must insure that daily dollar trading volume is $100,000 or greater.
Specifically, [Price M049]*[Price-Volume M049]/21>=100 for the first year. You must change the
month number for the three subsequent years. This criterion cannot be the variable one to get to five
stocks. Liquidity is so important in buying stocks that this criterion is required.

5. Be sure to avoid using data which didn’t exist at the time the screen would have been run. The screen
is run four times: M049, M037, M025 and M013. This means that you will have to create more custom
data fields than if you were running the screen as of the last date in the database. Violating this
requirement is a fatal flaw in doing stock screening research and invalidates your results.

Team-Based Learning Institute Page 145 of 156

6. Be sure to show the detailed formula for any custom data field you create. The reader has no other
way of knowing what you did.

7. Growth rates can only be three years long because there are only three years of data in the database
prior to M049 (April 30, 1998). You can’t use four years of data at M037 just because it exists – you
must be consistent between years.

8. KEEP IT SIMPLE!!! Remember that you must test your screen over four years, and that it will
involve trial-and-error to come up with the best criteria. Reduce your workload by keeping it simple.
You need some way to identify “good companies” (at least one “quality” criterion), and some other way
to make sure that they are reasonably priced (one “valuation” criterion). Quality and valuation are the
two key underlying concepts of most successful screens. You should also make sure that the stocks have
adequate liquidity (the average dollar amount that is traded in a day or month must be high enough).
Some advice: Lower priced companies tend to do better because of the ten-bagger concept.

9. Rank the five stocks for each year by purchase price from lowest to highest. Create a view which
includes the company name, industry, the purchase price, the return over the next year, and all other data
fields which you screened for. Export the view to Excel. Be sure to give a descriptive label to all column
headings (For example, don’t label a column as Udef6.) Calculate the portfolio return for each year
(average of the five stocks), the compounded average of the four annual returns, and the standard
deviation of the four annual returns. Finally, divide the four-year compounded average return by the
standard deviation to get the return-to-risk ratio. It is this ratio that you are trying to maximize.

10. You will be writing the criteria for your best screen on poster paper that I give you for display to the
entire class. Do not put your team name, or any other identifying information on it. Each team will
analyze the screen of the other teams, identify any invalid or illogical criteria, and vote for the screen
most likely to succeed (highest return-to-risk ratio).

Team-Based Learning Institute Page 146 of 156

 Tutorial to Test a Stock Screen

Introduction

The purpose of this tutorial is to give you some experience using a stock database to test the performance
(rate of return) of a stock screen over a two-year period. The idea is to go back in time and assume today
is a particular historical date, and that you have no knowledge of the future beyond that date. As of that
date, you enter specific criteria into the database and screen for the best five stocks out of 8,875 stocks in
the Stock Investor Pro database. You can then check the five stocks to see how they did in the “future”.

In this system, the investor owns stocks for one year (April 30 to April 30) and then reruns the screen to
find stocks to buy for the next year and so on. You will analyze the system over the following two
periods: April 30, 2000 to April 30, 2001 and April 30, 2001 to May 3, 2002 (the database you are
working with ends on May 3, 2002).

Detailed Instructions (First review the Joys of Stock Picking exercise)

Create six custom data fields – three data fields for each year. Click on the Custom Field Editor (triangle
icon) and maximize the screen. If you need help, use the Contents part of the Help system to learn about
creating your own custom data fields. The three fields for 2001-02 are:

a. [Price M013]/([Sales Y2]/[Shares Average Y2])

b. [Price M013]*[Price-Volume M013]/21
c. [Price M001]/[Price M013]-1

Note: Once you have created the data fields for 2001-02, the easiest way to create the other year is
to backspace month and year numbers out and type in the new numbers. You then save the field
with a new name. This is quicker than creating the custom fields from scratch.

Price M001 is the closing price of the stock on May 3, 2002, the last date in the database. Price M002 is
March 29, 2002 and so forth, counting months backwards. The earliest price in the database is Price
M120, or May 31, 1992. Sales Y2 are sales for the calendar year 2000 (the first year before the April 30,
2001 buy date), Y3 is 1999, etc.

Field a. is called the price-to-sales ratios, equaling the price on April 30 divided by the latest fiscal year
sales per share. This is a measure of value used to insure that only reasonably priced stocks are
purchased. A price-to-sales ratio of .35 can be thought of as paying $.35 for $1.00 of sales per share.
Your other data fields will use Sales and Shares Average from Year 3 (Y3).

Field b. is an estimate of the average daily dollar trading volume of the stock during April, 2001. This is
the amount of stock (in dollars, not shares) that exchanged hands on an average day. High volume is
desirable because it is easier to get transactions executed, and usually means a smaller bid-ask spread.
Dividing by 21 converts a monthly number to daily because there are 21 trading days in a month on
average.

Field c. is “the future”. It is calculating the one-year rate of return (April 30 to April 30) for the stock.
You will calculate the five-stock portfolio rate of return from April 30, 2000 to April 30, 2001 and from
April 30, 2001 to May 3, 2002.

Note: Be careful when counting months back from M001 (May 3, 2002). As a check figure, the
earliest price you need for this assignment is M025 (April 30, 2000).

Team-Based Learning Institute Page 147 of 156

Create a screen for each year to find five high-potential stocks out of the 8,875 stocks in the database.
Click on the Screen Editor (funnel icon) and maximize the screen. If you need help, use the Contents part
of the Help system to learn about creating screens. The screen has five criteria (you will have to change
the variables for each of the two years, but the following list is for 2001-02):

a. Exchange Not equal Over the counter [Find it in Company Information]

b. [Price M013] >= 1 ($1.00)
c. [Price M013] <= 3 ($3.00)
d. [Price M013]*[Price--Volume M013]/21 >= 500 ($500,000)
e. [Price M013]/([Sales Y2]/[Shares Average Y2]) <= .35

Use the custom fields you created for Criteria d. and e. Click on How Many to see how many stocks
fulfill all five criteria. Adjust the .35 in Criterion e. up or down until you have exactly five stocks. The
500 represents $500,000 of stock changing hands on average each day during April, 2001. Save as 2001
Screen. Click Apply.

Rank the five stocks by purchase price from lowest to highest. Click on Tools in the upper left corner of
the screen, then Rank, then the + sign before Prices – Monthly Close, then Price M013 and Ascending,
then OK.

Create a view for each year so that your computer screen will show only the fields (columns) that you’re
interested in, rather than every field available. Click on the View Editor (eye glasses) and maximize the
screen. If you need help, use the Contents part of the Help system to learn about creating views. The
view has seven fields (you will have to change all of the fields except Company name and Industry to
correspond to 2000-01, but the following list is for 2001-02):

a. Company name (find it under Company Information)

b. Industry (Company Information)
c. [Price M013] (Prices – Monthly Close)
d. [Price M001]/[Price M013]-1 (Custom Fields)
e. [Price M013]*[Price-Volume M013]/21 (Custom Fields)
f. [Price M013]/([Sales Y2]/[Shares Average Y2]) (Custom Fields)
g. Sales Y2 (Income Statement – Annual)

Save as 2001 View. In the upper right corner of the screen, choose 2001 View from the drop-down menu.
Export the view to Excel (refer to Step 6 in the Introductory Exercise for SI if you forgot how to export).

Find the average (mean) of Column D for both years. Note that Column D is the one-year rate of return
for the stocks you selected. If you put equal dollars into each of the five stocks, the average of Column D
would be the rate of return for the portfolio.

Assume that you also ran the above screen for 1998-99 and 1999-2000, and got 30% and 50% returns for
the portfolio, respectively. Calculate the average annual compounded rate of return for the portfolio over
the four-year period. Use Excel and the following formula =((1+w)*(1+x)*(1+y)*(1+z))^(1/4)-1, where
the four variables are the portfolio rates of return for the four years. Note that we are ignoring any
dividends paid on these stocks. You should get an answer of approximately 111%.

Use =STDEV( ) to calculate the standard deviation of the four rates of return. Divide the average return
by the standard deviation to get a ratio – the percentage return for one percent of standard deviation, or
return-to-risk ratio. You should get an answer of approximately .8.

Team-Based Learning Institute Page 148 of 156

TBL Module Exemplar in Mechanical Design

Authour: Dr. Peter Ostafichuk, [Link], P. Eng

Department of Mechanical Engineering, University of British Columbia

Context

• 3rd year undergraduate course on mechanical design with an emphasis on analysis tools used to
design functioning components
• Meetings:
o 2 x 50-minute class / week (1 section)
o 1 x 2-hr tutorial (with TA) (3 sections)
• Students
o 140
o Teams of 5 (some 6), formed from like disciplines for scheduling but otherwise random
• Module topics
o 0 Review (2 weeks)
o 1 Fracture (2 weeks)
o 2 Fatigue (2 weeks)
o 3 Shafts (2 weeks)
o 4 Welding (2 weeks)
o 5 FEA computer modelling (2 weeks)
o Course review (1 week)
• Tiered lecture theatre

Grade Distribution
• 15% RAP (7.5% iRAT and 7.5% tRAT*)
• Assignments (20% team*, 10% individual for peer assessment tasks on peerScholar)
• Midterm 20%
• Final exam 35%

*team items are multiplied by a peer evaluation score, based on 3 iPeer evaluation. Mean peer
evaluation is 100 in a team – some are above, some are below.

Readings

Modules 1-5 each have assigned pre-readings from the course text. The text is large and information-
dense. Each module topic above is a chapter of approximately 40-60 pages. Reading guides are used to
focus on key elements and to make the readings more digestible and less daunting. Readings are
divided into three categories:
• Required: the primary source of material for RAP quizzes (15-20 pages)
• Beneficial: additional materials to support the required readings (10-15 pages)
• Supplementary: nice-to-know material, not required for the course (balance of chapter)

The expected time commitment for the readings is approximately 1-hr every two weeks.

Example readings are provided.

Team-Based Learning Institute Page 149 of 156

Readiness Assurance Test

• 15 questions, A-E multiple choice

o iRAT on scantron
o tRAT on IF-AT
o Test is protected (not revealed or available outside of RAP)
• Prepared in folders
o Blue “Individual” folder pre-loaded with 6 scantrons and 6 question books (even for
teams of 5)
o Red “Team” folder pre-loaded with 1 IF-AT and 1 appeal form
• Support during test: 2 TAs

• Process:
1. Announcement (e.g. put away books and phones)
 Hint: speak about why no phones; project a “no cell phone” graphic
2. Distribute Individual folders
3. Individual test: ~1 minute per question + 3 minutes (18-20 minutes)
4. Students return Scantrons to Individual folder (team holds onto all question booklets)
5. Trade Individual folders for Team folders
6. Team test: same timing as Individual; 1 TA invigilates, 1 TA scores scantrons, I assist
with sorting scantron forms while keeping an eye on the class
7. Review individual performance (scantron report) once scoring is done and teams are
working
8. Teams hand in folder with IF-AT, 6 question booklets, appeal form
9. TA counts question booklets to ensure 6 per team (automatic 0 on iRAT and tRAT
otherwise… teams are warned and this is written right on the test!);
 hint: rip the corners of question booklets as they are counted
10. TA separates any used appeals forms and puts a fresh blank one back in the folder
11. Based on iRAT performance (summarized by scanner software) I address any areas of
concern
• Tips:
o iRAT scores are lower than tRAT scores because the team outperforms its strongest
members; also, if you use IF-ATs with a 4-2-1-0 scoring scheme, there are multiple
chances for marks on a tRAT. I normally see 70-80% iRAT average, and 90-95% tRAT
average
o Remind students that the purpose of the RAP process is to get them ready, including
providing you and them feedback on areas of strength and weakness; not a “test”
o If you want to reuse questions (I suggest you do), you need to be firm:
 Explain why you are being strict
 Make sure no exam booklets leave the room (count booklets returned by team,
as described above)
 Make sure no one has a phone out or takes notes (automatic 0 and loss of IF-AT)
o Look at the summary statistics (if you use a scanner to score the iRATs)
 Consider mean score for each question; I aim for some a mix of easy (quick
confidence-builders) and challenging (good for team discussion)
 Consider discrimination index (or point biserial). It measures whether students
who did well overall did well on a question. Scrutinize and revise questions
where this is near 0 (no correlation) or negative (weak students on the test
performed better on that question)

Examples questions are provided.

Team-Based Learning Institute Page 150 of 156

4S Application Tasks

Application tasks range from

simple in-class multiple choice
questions to large, complex
out-of-class assignments.

Simple, in-class question 1.

This would be a short (5-

minute) in-class activity. While
coloured cards could be used
here, I would often go for
something like clickers or a
particular pose or gesture (i.e.
point in a specific direction for
each letter) just to keep things
moving quickly.

Simple in-class question 2.

Here’s an example that builds

off of the last. There are some
subtleties for students where
the thinking from the previous
question does not apply here.

Hint: if you show of hands, coloured cards, or some other voting method, look for the team that
reports their answer last or changes their answer and call on them first in the discussion.

Next page: an assignment example from a related course used with a gallery walk. It takes
students several weeks to complete. By identifying a performance metric (in this case, cost), it
is possible for teams to compare their project merit (after several weeks of work) with
someone else’s project. The reporting can also be done online (e.g. Google form), or low tech
(see sticky note continuum example on next page), and then teams with the best design “on
the hot seat” can show their solution on the doc cam for scrutiny with the goal for the class to
identify the best design that meets all requirements.

Team-Based Learning Institute Page 151 of 156

Another Assignment Example

Included is an extended team assignment (on the redesign of an amusement park ride) that was peer
assessed on peerScholar. It is a messy and ill-defined problem that forces students to make (and
defend) many assumptions. The purpose here was less about the 4 Ss and in class discussion, but
more about developing the skill of reviewing someone else’s work and giving constructive feedback.
It hit the S’s of significant problem, same problem, and specific choice). In peerScholar, the
assignment took place in three stages:

1. Teams worked together to interpret the problem and propose a solution. Each individual
uploaded their team’s solution to peerScholar.
2. In peerScholar, each individual was randomly assigned two other team’s assignments to
assess. They did this using 4 different criteria related to the appropriateness of the
assumptions and analysis and the quality of the final design proposed.
3. In peerScholar, each individual was responsible for reviewing the feedback they received,
discussing it with their team, and assessing the quality of the assessment they received.

Team-Based Learning Institute Page 152 of 156

 MECH 326 Reading Guide

Module 1: Fracture

Overview
The readings outlined in this guide are intended to prepare a foundation of knowledge
and skills that will be used in the MECH 326 classes and tutorials. You will have a short
multiple choice Readiness Assurance Process (RAP) Quiz on this material. The quiz will
primarily test your ability to recall this material; the emphasis is not to test your ability to
apply or use the material (that will come later).

The readings are divided into three categories:

 Required: the primary source of material for the Readiness Assurance Process
(RAP) Quiz. Each student is expected to complete the required readings.
□ Beneficial: additional analyses, derivations, explanations and examples to
provide in-depth understanding of the course material. These readings help
develop a more complete understanding of course concepts necessary for the
tutorials, exercises, exams, and design projects. It your choice whether or not
you do the beneficial readings.
- Supplementary: good information for any mechanical designer to know.
Useful for the course, but not part of tutorials or exams.

All readings are drawn from the course text: Budynas, R.G. and Nisbett, J.K., Shigley’s
Mechanical Engineering Design, McGraw-Hill. Section number references are to the 10th
Edition. The 9th Edition sections are the same, with the exception that an additional section
appears in 5-13.

Readings

Section Title Notes

A short introduction to the main topics of the
 5 -
chapter
Additional information to help orient you to the
 5-1 Static strength
purpose of the chapter topic
Review material that is helpful information for the
 5-2 Stress concentration introduction to fracture mechanics section (5-12)
and for later in the course.
A short overview of the various static failure
□ 5-3 Failure theories
theories; we won’t be using these too much
Maximum-Shear-Stress One of two failure theories for ductile materials
- 5-4 Theory for Ductile you should have seen before; we won’t be using
Materials this too much

Team-Based Learning Institute Page 153 of 156

 While technically not required for our work on
Distortion-Energy
fracture, this failure theory is important for other
/□ 5-5 Theory for Ductile
work in the course and so is required reading.
Materials
(Example 5-1 is beneficial but not required)
5-6 to Additional ductile and You are welcome to review this material, but it
-
5-11 brittle failure theories will not appear in the course
This is the bulk of the material we will be using in
Introduction to
 5-12 MECH 326 and it consists of an introduction to the
Fracture Mechanics
growth of cracks in mechanical elements.
A summary of the key equations presented in the
Important Design chapter. Not bad to skim over, but most of the
- 5-13
Equations theories represented are not part of the assigned
readings.

Team-Based Learning Institute Page 154 of 156

 Sample MECH 326 RAP Quiz

1 Instructions
This test is closed-book. Electronic devices and aids must be put away. Standard exam policies apply.
Part 1: Individual – for each of the following questions, mark your response in pencil on the computer
score card. When time is up, place your score cards in your individual folder and hold onto this booklet.
Part 2: Team – once you receive your team folder, work as a team and mark your responses by scratching
the appropriate boxes on the IF-AT (scratch) cards.
Reveal a  in 1 scratch for 4 pts, 2 scratches for 2 pts, 3 scratches for 1 pt, 4 or 5 scratches 0 pts.
Record your total score and team name in the space at the top of the card. You can appeal any question
using the included form. Place your IF-AT card, all exam booklets, and your appeal form in your team
folder and hand it in. You must return all booklets – your entire team will receive a score of zero on
both the individual and team portions of the RAP quiz if any booklets are missing from the folder.

2 RAP Quiz Questions

Choose the best response - a given question may have more than one choice that is correct but marks will
only be given for the best answer.

1. What is fracture toughness?

a) The area under the stress-strain curve for the material near a crack
b) The local yield strength in the material at a crack tip
c) The local fracture strength in the material at a crack tip
d) The critical stress intensity factor
e) The ratio of the critical stress intensity to the stress intensity factor

2. In practice, the theoretical stress concentration factor, Kt, is often ignored (set to unity) under
which condition(s)? (choose a single best response)
a) The material is ductile
b) The material is brittle
c) The material is ductile and the loading is cyclical
d) The material is ductile and the loading is static
e) The material is brittle and the loading is axial
𝟏𝟏
3. The von Mises stress is given by 𝝈𝝈′ = �(𝝈𝝈𝟏𝟏 − 𝝈𝝈𝟐𝟐 )𝟐𝟐 + (𝝈𝝈𝟐𝟐 − 𝝈𝝈𝟑𝟑 )𝟐𝟐 + (𝝈𝝈𝟑𝟑 − 𝝈𝝈𝟏𝟏 )𝟐𝟐 . Why does
√𝟐𝟐
shear stress not appear in this equation?
a) The expression above is based on principal stresses
b) Shear stresses do not cause distortion of an element
c) Shear stresses average to zero on a small element
d) For cases where von Mises is used, only hydrostatic forces matter
e) The 𝜎𝜎′ expression above is for normal stress only – there is a separate equation for 𝜏𝜏′

Team-Based Learning Institute Page 155 of 156

 MECH 326 Assignment Example (Abridged)

Scenario
With your newfound expertise in fracture
mechanics, your team has been approached
to assist with the preliminary redesign of a
particular amusement park ride. 1 The ride
is identical to “The Beast” at Playland at the
PNE (see figure to the right). Up to 20
riders sit in chairs at the end of a long
swinging and spinning pendulum.

The client is looking to change the

aesthetic of the design by replacing Beam
A (see lower figure) with the most slender
beam possible. Your task is to recommend
the beam cross-section and material that
still achieves a safety factor of at least 10.
The client is worried about the potential for
fracture, so they like to know what type
and size of crack they need to be able to
identify in their visual inspections

The client indicates this is part of a larger

feasibility study to look the viability of a
redesign of the ride, but they do not have
much in the way of specifications or details
for you. They mention you should take a
look at the videos of this ride online;
otherwise, you have not been given a lot to
work from in this project. They expect you
to make and justify assumptions and
approximations as you feel are
appropriate.

Deliverable
Prepare a brief two-part report for the client to outline your work and your recommendation. Include
a body suitable for a general engineering audience, and an appendix suitable for an expert audience.
The body should not exceed three pages and should be suitable for engineers familiar with the basic
concepts of mechanics of materials, but not necessarily the details found in MECH 326. Your appendix
should provide sufficient analysis to support your design. Through peerScholar, you will assess other
teams’ designs and they will assess yours.

1 Disclaimer: As much as I believe in all of you and know you will make great engineers, and as much as I’m doing my
best to help you to learn fracture mechanics, if someone approaches you and asks you to design an amusement park
ride, please say no for now – leave it to the experts!

Team-Based Learning Institute Page 156 of 156