Chapter 5

New Assessments and Environments

for Knowledge Building

Marlene Scardamalia, John Bransford, Bob Kozma, and Edys Quellmalz

Abstract This chapter proposes a framework for integrating two different approaches
to twenty-first century skills: “working backward from goals” and “emergence of
new competencies.” Working backward from goals has been the mainstay of educa-
tional assessment and objectives-based instruction. The other approach is based on
the premise that breakthroughs in education to address twenty-first century needs
require not only targeting recognized objectives but also enabling the discovery of
new objectives—particularly capabilities and challenges that emerge from efforts to
engage students in authentic knowledge creation. Accordingly, the focus of this chapter
is on what are called “knowledge building environments.” These are environments
in which the core work is the production of new knowledge, artifacts, and ideas of
value to the community—the same as in mature knowledge-creating organizations.
They bring out things students are able to do that are obscured by current learning
environments and assessments.
At the heart of this chapter is a set of developmental sequences leading from entry-
level capabilities to the abilities that characterize members of high-performing
knowledge-creating teams. These are based on findings from organization science and
the learning sciences, including competencies that have already been demonstrated by
students in knowledge-building environments. The same sources have been mined for
principles of learning and development relevant to these progressions.

M. Scardamalia (*)
University of Toronto, Canada
e-mail: [email protected]
J. Bransford
University of Washington, Seattle
B. Kozma
Kozmalone Consulting
E. Quellmalz
WestEd, San Francisco, California

P. Griffin et al. (eds.), Assessment and Teaching of 21st Century Skills, 231
DOI 10.1007/978-94-007-2324-5_5, © Springer Science+Business Media B.V. 2012
232 M. Scardamalia et al.

Knowledge Societies and the Need for Educational Reform

There is general agreement that the much-heralded “knowledge society” (Drucker

1994, 1968; Bell 1973; Toffler 1990) will have profound effects on educational,
cultural, health, and financial institutions, and create an ever-increasing need for
lifelong learning and innovation. This need for innovation is emphasized by the
shift from manufacturing-based to knowledge-based economies, with the health and
wealth of nations tied to the innovative capacity of its citizens and organizations.
Furthermore, Thomas Homer-Dixon (2000) points out that problems such as global
climate change, terrorism, information glut, antibiotic-resistant diseases, and the
global financial crisis create an ingenuity gap: a critical gap between our need for
ideas to solve complex problems and the actual supply of those ideas. More and more,
prosperity—if not survival—will depend on innovation and the creation of new
knowledge.
Citizens with little or poor education are particularly vulnerable. As David and
Foray (2003) emphasize, disparities in productivity and growth of various countries
have far less to do with their natural resources than with their capacity for creating
new knowledge and ideas: “The ‘need to innovate’ is growing stronger as innovation
comes closer to being the sole means to survive and prosper in highly competitive
and globalized economies” (p. 22).
The call to action that launched this project, entitled Transforming Education:
Assessing and Teaching 21st Century Skills (2009) stresses the need for systemic
education reform to address the new challenges that confront us:
The structure of global economy today looks very different than it did at the beginning of
the 20th century, due in large part to advances in information and communications tech-
nologies (ICT). The economy of leading countries is now based more on the manufacture
and delivery of information products and services than on the manufacture of material
goods. Even many aspects of the manufacturing of material goods are strongly dependent
on innovative uses of technologies. The start of the twenty-first century also has witnessed
significant social trends in which people access, use, and create information and knowledge
very differently than they did in previous decades, again due in many ways to the ubiquitous
availability of ICT. These trends have significant implications for education. Yet most edu-
cational systems operate much as they did at the beginning of the 20th century and ICT use
is far from ubiquitous. Significant reform is needed in education, world-wide, to respond to
and shape global trends in support of both economic and social development (p.1).

According to one popular scenario, the introduction of technological advances

into education will democratize knowledge and the opportunities associated with it.
This may be too “romantic” a view, however. The current project is based on the
assumption, shared by many (Laferrière 2001; Raizen 1997; Law 2006), that there is
little reason to believe that technology combined with good intentions will be enough
to make the kinds of changes that need to happen. To address these challenges, edu-
cation reform must be systemic, not just technological. Systemic reform requires
close ties between research-based innovation and practice (e.g., Bransford and
Schwartz 2009), and assessment of progress, in order to create the know-how for
knowledge-age education and workplace productivity. It also requires the alignment
5 New Assessments and Environments for Knowledge Building 233

of organizational learning, policy, and the other components of the system (Bransford
et al. 2000; Darling-Hammond 1997, 2000). As the call to action indicates:
Systemic education reform is needed that includes curriculum, pedagogy, teacher training,
and school organization. Reform is particularly needed in education assessment. . . . Existing
models of assessment typically fail to measure the skills, knowledge, attitudes and charac-
teristics of self-directed and collaborative learning that are increasingly important for our
global economy and the fast-changing world (p.1).

Trilling and Fadel (2009) in their book 21st Century Skills: Learning for Life in
Our Times talk of “shifting-systems-in-sync.” In order to judge different approaches
to assessment, it is necessary to view them within the larger context of system
dynamics in education. Traditionally, testing has played a part in a system that tends
to stabilize at a level of mediocre performance and to be difficult to change. The
system itself is well recognized and gives us such phenomena as the “mile wide,
inch deep” curriculum, which no one advocates and yet which shows amazing
persistence. Inputs to the system include standards, arrived at by consensus of
educators and experts, tests geared to the standards, textbooks and other educational
material geared to the standards and the tests, responses of learners to the curriculum
(often manifested as failure to meet standards), responses of teachers, and pressures
from parents (often focused on desire for their children to perform well on tests).
These various elements interact until a state is reached that minimizes tensions
between them. The typical result is standards that represent what tests are able to
measure, teachers are comfortably able to teach, and students are comfortably able
to learn. Efforts to introduce change may come from various sources, including new
tests, but the system as a whole tends to nullify such efforts. This change-nullifying
system has been well recognized by education leaders and has led to calls for “systemic
reform.” On balance, then, a traditional objectives- and test-driven approach is not a
promising way to go about revolutionizing education or bringing it into the twenty-
first century.
What are the alternatives? How People Learn (2000) and related publications
from the National Academies Press have attempted to frame alternatives grounded
in knowledge about brain, cognitive, and social development and embodying break-
through results from experiments in the learning sciences. A rough summary of
what sets these approaches apart from the one described above is elaborated below,
including several examples that highlight the emergence of new competencies. In
essence, instead of starting only with standards arrived at by consensus of stake-
holders, these examples suggest the power of starting with what young learners are
able to do under optimal conditions (Fischer & Bidell 1997; Vygotsky 1962/1934).
The challenge then is to instantiate those conditions more widely, observe what new
capabilities emerge, and work toward establishing conditions and environments that
support “deep dives” into the curriculum (Fadel 2008). As the work proceeds, the
goal is to create increasingly powerful environments to democratize student
accomplishments and to keep the door open to further extensions of “the limits of
the possible.” This open-ended approach accordingly calls for assessments that are
concurrent, embedded, and transformative, as we elaborate below. These assessments
234 M. Scardamalia et al.

must be maximally useful to teachers and students so that they are empowered to
achieve new heights. Formative assessment thus takes on a new meaning. It is integral
to the learning process and connects communities (Earl 2003; Earl and Katz 2006).
Instead of using it to narrow the gap between present performance and some targeted
outcome, it is used to increase the distance between present performance and what
has gone before, opening the door for exceeding targeted outcomes. It is additionally
used to create increasingly effective knowledge-building environments that sustain
such work and produce greater change over time.
In twenty-first century schools and other educational settings, knowledge and
technological innovation will be inextricably related, as is currently the case in
many knowledge-creating organizations, which provide models for high-level
twenty-first century skills in action and the knowledge-building environments that
support them. Once information and communication technology (ICT) becomes
integral to the day-to-day, moment-to-moment workings of schools, organizations,
and communities, a broad range of possibilities for extending and improving designs
for knowledge-building environments and assessments follow. Accordingly, the
goals for this chapter are to:
• Generate an analytic framework for analyzing environments and assessments
that characterize and support knowledge-creating organizations and the knowledge-
building environments that sustain them;
• Apply this framework to a set of environments and assessments in order to
highlight models, possibilities, and variations in the extent to which they engage
students in or prepare them for work in knowledge-creating organizations;
• Derive technological and methodological implications of assessment reform;
• Propose an approach to research that extends our understanding of knowledge-
building environments and the needs and opportunities for promoting twenty-first
century skills.
We start by discussing two concepts that underlie our whole treatment of assessment
and teaching of twenty-first century skills: knowledge-creating organizations and
knowledge-building environments.

Knowledge-Creating Organizations

A popular saying is that the future is here now; it’s simply unevenly distributed.
Knowledge-creating organizations are examples; they are companies, organizations,
associations, and communities that have the creation, evaluation, and application of
knowledge either as their main function or as an essential enabler of their main
functions. Examples include research institutes, highly innovative companies,
professional communities (medicine, architecture, law, etc.), design studios, and
media production houses.
Creating new knowledge entails expectation and the means to go beyond current
practice. Its goals are emergent, which means that they are formed and modified in
5 New Assessments and Environments for Knowledge Building 235

the course of pursuing them. If computer design had not been characterized by
emergent goals, computers would still be merely very fast calculating machines.
Emergent outcomes cannot be traced back to subskills or subgoals, because they
come about through self-organization—structure that arises from interactions
among simpler elements that do not themselves foreshadow the structure. Color is a
classic example of emergence; individual molecules do not have any color, but
through self-organizing processes, molecular structures arise that do have color.
System concepts are similarly applied to explaining the evolution of complex
anatomical structures (Dawkins 1996) and to accounting for creativity (Simonton
1999)—one of the widely recognized twenty-first century skills. Creative work and
adaptive expertise (Hatano and Inagaki 1986) alike are characterized by emergent
goals. This makes them especially relevant to twenty-first century skills. The message
here is not that “anything goes” and standards and visions should be abandoned.
Instead, the message is that high standards and policies that support them must
continually be “on the table” as something to be evaluated and exceeded, and that
processes for innovation need to be supported, celebrated, assessed, and shared.
In a study by Barth (2009), “Over two-thirds of employers said that high school
graduates were ‘deficient’ in problem solving and critical thinking.” The importance
of this point is highlighted by a survey in which about 3,000 graduates of the
University of Washington, 5–10 years after graduation, rated the importance of
various abilities they actually used in their work (Gillmore 1998). The top-ranked
abilities were (1) defining and solving problems, (2) locating information needed to
help make decisions or solve problems, (3) working and/or learning independently,
(4) speaking effectively, and (5) working effectively with modern technology,
especially computers. These were the abilities rated highest by graduates from all
the major fields. Regardless of the students’ field of study, these skills outranked
knowledge and abilities specific to their field. They correspond fairly closely to
items that appear on twenty-first century skill lists generated by business people and
educators. Accordingly, it seems evident that they represent something important in
contemporary work life, although precisely what they do represent is a question yet
to be addressed.
The fact that so much of the pressure for teaching twenty-first century skills is
coming from business people has naturally provoked some resistance among edu-
cators. Their main objections are to the effect that education should not be reduced
to job training and that the private sector should not be dictating educational pri-
orities. These are legitimate concerns, but they can be answered in straightfor-
ward ways:
• Teaching twenty-first century skills is a far cry from job training. It amounts to
developing abilities believed to be of very broad application, not shaped to any
particular kind of job. Indeed, as The North American Council for Online
Learning and the Partnership for twenty-first Century Skills state (2006):
“All citizens and workers in the twenty-first century must be able to think
analytically and solve problems if they are to be successful—whether they are
entry-level employees or high-level professionals” (p.7).
236 M. Scardamalia et al.

• Employability is an important consideration for today’s students. Contrasting the

changes taking place today with those of the Industrial Revolution, Peter Drucker
(2003) has pointed out that very little relearning was required for a farm worker
to become a factory worker, but that extensive learning and relearning is required
for a factory worker to become a knowledge worker—learning that is best started
in childhood.
• Crawford (2006) has questioned the emphasis on skills in the processing of
abstract information. It is not expected that everyone will become what Reich
(1991) called “symbolic analysts,” but symbolic analysis and the use of technology
for carrying it out are becoming increasingly essential for otherwise “manual”
occupations (Leonard-Barton 1995).
• Well-accepted educational values require that whatever is done to promote
twenty-first century skills should not be confined to the élite. It must be inclusive,
foster equal participation, address issues of citizenship and multiculturalism, and
provide for deliberative governance (Hearn and Rooney 2008; Robinson and
Stern 1997; Trevinarus 1994, 2002).
• Increasing the level of knowledge-related skills is not only important for the
managers and developers in an organization but also for empowering workers at
all levels “to assume more responsibilities and solve problems themselves” (U.S.
Department of Commerce et al. 1999, p.1).
• It is not assumed that modern corporations, research laboratories, design studios,
and the like represent ideal models for education to emulate. There is probably
as much to be learned from studying their shortcomings as from studying their
successes. What they do represent, which is valuable for education systems, are
social organizations that function to produce knowledge rather than merely to
transfer and apply it. Thus they offer insight into a level of constructivism deeper
than that characteristic of even the more active kinds of school learning
(Scardamalia and Bereiter 2003).
The previous bullet point returns us to the theme of knowledge building and
emergence. Instead of taking at face value the twenty-first century skills identified
by committees of educators and business people, we might start by considering
what constitutes knowledge creation at its best and what traits, abilities, and envi-
ronments enable it. It is characteristic of “soft” skills of all kinds (of which twenty-
first century skills are a subset) that everyone already possesses them to some degree
(unlike “hard” skills, such as solving simultaneous equations and tooth filling,
which may be totally lacking in the untrained). Thus for each skill identified as
relevant to knowledge creation, we may establish a continuum running from the
skill level almost everyone may be assumed to have, up to a level sufficient for
engaging in creative knowledge work. The skills and competencies required for
productive work in innovative organizations and professions provide a foundation
for designing environments, practices, and formative assessments to help schools
and education systems meet twenty-first century expectations (Trevinarus 1994,
2002; Wiggins and McTighe 2006; Anderson 2006).
5 New Assessments and Environments for Knowledge Building 237

Knowledge-Building Environments

The term “knowledge building” appears in approximately a million web docu-

ments. Sampling documents with a business orientation suggests the term is used as
a synonym for “knowledge creation,” roughly equivalent to concepts such as collec-
tive intelligence, intellectual capital, knowledge work, and innovation. Sampling
education documents suggests it is used more as a synonym for “constructivist
learning” (Wilson 1996), with rough equivalence to concepts such as active learning,
discovery learning, and inquiry- and project-based learning.
The term “knowledge building” was originally introduced into the educational
literature in 1989 (Bereiter and Scardamalia 1989, p.388) and had its basis in studies
of expertise and innovation, summarized in the book Surpassing Ourselves: An
Inquiry Into the Nature and Implications of Expertise (Bereiter and Scardamalia
1993). The phrase “progressive problem solving” was used to denote the process by
which experts become experts and continue to develop their expertise (in contrast to
becoming experienced nonexperts)—through investing their surplus cognitive
resources in tackling problems at higher levels. The same basic idea, applied to
knowledge building, took the form of a contrast between shallow and deep construc-
tivism. If we imagine a line with shallow constructivism at one end and deep con-
structivism at the other, much of what is called “constructivist learning” in schools
would be located toward the shallow end. Take for example the ubiquitous school
“project” in which different project members assemble information that is then
compiled in a multimedia presentation. One longtime observer of the school scene
described this as using a computer to make a scrapbook. Knowledge building, with
its focus on knowledge creation, would be located at the opposite end, aiming for
the deepest levels of work with ideas, leading to emergence of new ideas and con-
tinued efforts to improve them (Scardamalia and Bereiter 2003).
Over the history of thought, the idea of knowledge as a human construction is
relatively new. Designing environments to support knowledge creation is newer yet.
Schools were not built for that purpose, and to this day many would claim they
should not or could not. Yet universal access to the process whereby new knowledge
is created arguably depends on bringing knowledge-building environments into
schools.
In brief, we use the term “knowledge-building environments” to refer to contexts
supportive of the emergence and further development of new ideas—knowledge
creation in organizations of all kinds. Conceptually, economically, and technologi-
cally, it may be necessary to connect currently distinct environments for creative
work with ideas (e.g., knowledgeware) to those for learning (e.g., courseware, tutorials,
simulations), so as to encourage their integration and easy movement between these
different and essential aspects of mature knowledge work. What would a more inte-
grative approach look like? We say more about that below. For now, we elaborate
the concept of knowledge-building environments by focusing on features that favor
the emergence of new skills.
238 M. Scardamalia et al.

Knowledge-building environments provide special support for creative work

with ideas, so that ideas may grow from nascent form to something of greater
consequence than could have been imagined before. Improved ideas emerge as they
are generated in multiple and varied contexts and are entered into communal spaces.
Within these more public spaces, collaborators as well as competitors can elaborate,
critique, reframe, link, re-position, create higher-order structures, explore and devise
uses for ideas, and in other ways work creatively with them. It is through such
sustained and varied engagement that ideas, like colorless molecules, acquire new
properties through structural organization. In line with this emergence perspective,
a knowledge-building approach considers the “promisingness” of an idea, recognizing
that through new combinations and sustained work, something brilliant might
emerge. In creative knowledge work it is important both to avoid wasting resources
on unpromising ideas and to guard against killing off ideas that have promise. As
the designer of a program for forest conservation remarked in response to criticisms
of the plan, “an imperfect program which can be improved is better than none at all”
(“Saving the rainforest: REDD or dead?” 2009).
In summary, a knowledge-building environment, virtual or otherwise, is one that
enhances collaborative efforts to create and continually improve ideas. It exploits
the potential of collaborative knowledge work by situating ideas in a communal
workspace where others can criticize or contribute to their improvement. In these
collaborative open contexts, discourse that is democratic and directed toward idea
advancement compounds the value of ideas, so that collective achievement exceeds
individual contributions. A local knowledge-building community gains strength as
it connects to a broader one. The local community not only draws upon, but also
affords participation in, the larger one, with possibilities for symmetrical advances
of knowledge. A successful knowledge-building environment will bring innovation
closer to the central work of an organization. It is an environment in which members
are continually contributing to and enhancing the shared intellectual resources of
the organization. Each advance precipitates another, so that at both the individual
and group level, there is continual movement beyond current understanding and
capacity. Emergence becomes a way of life, different from but both more productive
and more personally satisfying than a life restricted to following known paths to
known goals. Innovation, as Peter Drucker (1985, p.151) put it, becomes “part and
parcel of the ordinary, the norm, if not routine.”

New Goals and Methods to Support the Emergence of New Skills

Advocates for the adoption of twenty-first century skills generally look for this to
have an overall transformative effect on the schools. However, the nature and extent
of this envisaged transformation can range from conservative to fundamental, as
suggested by the following three levels:
1. Additive change. Change is expected to result from the addition of new skill
objectives, new curriculum content (nanotechnology, environmental studies,
5 New Assessments and Environments for Knowledge Building 239

cross-cultural studies, systems theory, technology studies, etc.), and new technology.
Changes to existing curricula will be needed to make room for additions.
2. Assimilative change. Instead of treating work on twenty-first century skills as an
add-on, existing curricula and teaching methods are modified to place greater
emphasis on critical thinking, problem solving, collaboration, and so forth. This
is the most widely recommended approach and reflects lessons learned from
the disappointing results of a previous wave of “higher-order thinking skills”
instruction that took the additive approach (Bereiter 1984).
3. Systemic change. Instead of incorporating new elements into a system that retains
its nineteenth century structure, schools are transformed into twenty-first century
organizations. Toward this end we present a case for schools to operate as
knowledge-creating organizations. The envisaged educational change is not
limited to schools, however. Knowledge creation by young people can and often
does take place in out-of-school contexts.
The present authors clearly favor systemic change but recognize that the realities
of public education often mean that assimilative change, and in many cases additive
change, is as far as a school system will go in adapting to twenty-first century oppor-
tunities and needs. Accordingly, approaches to teaching and assessing twenty-first
century skills need to be applicable and potentially transformative at any of the three
levels. That said, however, we suggest that countries whose schools are transformed
into knowledge-creating organizations may gain a tremendous advantage over those
that struggle to incorporate knowledge-age education into industrial-age curricula
and structures.
Two general strategies are applicable to pursuing the practical goals of advancing
twenty-first century skills, and we argue that both are important and need to be used
in a complementary fashion. One is the approach of working backward from goals.
The other is one that, for reasons that will become evident, we call an emergence
approach.
“Working backward from goals” to construct a system of subgoals and a path
leading from an initial state to the goal is one of the main strategies identified in
Newell and Simon’s classic study of problem solving (1972). It will be recognized
as the most frequently recommended way of designing instruction. As applied to
educational assessment, it comprises a variety of techniques, all of which depend on
a clearly formulated goal, the antecedents of which can be identified and separately
tested. Although working backward is a strategy of demonstrable value in cases
where goals are clear, it has two drawbacks in the case of twenty-first century skills.
Most twenty-first century skills are “soft” skills, which means among other things
that there is an inevitable vagueness and subjectivity in regard to goals, which there-
fore makes “working backward” not nearly so well structured as in the case of
“hard” skills (such as the ability to execute particular algebraic operations). A more
serious difficulty, however, is that working backward from goals provides no basis
for discovering or inventing new goals—and if twenty-first century education is to
be more than a tiresome replication of the 1970s “higher-order skills” movement, it
has to be responsive to potential expansions of the range of what’s possible.
240 M. Scardamalia et al.

As noted earlier, in the context of teaching and testing twenty-first century skills,
“working backwards from goals” needs to be complemented by a working-forward
approach growing out of what has been called the “systems revolution” (Ackoff
1974). Self-organization and emergence are key ideas in a systems approach to a
vast range of problems. An “emergence” approach, when closely tied to educational
experimentation, allows for the identification of new goals based on the discovered
capabilities of learners. The observation that, in advance of any instruction in rational
numbers, children possess an intuitive grasp of proportionality in some contexts led
to formulation of a new goal (rational number sense) and development of a new
teaching approach that reversed the traditional sequence of topics (Moss 2005).
Results suggest that both the traditional goals (mastering appropriate algorithms)
and the path to achieving them (starting by introducing rational numbers through
models that connect children’s whole number arithmetic) were misconceived, even
though they were almost universally accepted. If that can happen even on such a
well-traveled road as the teaching of arithmetic, we must consider how much riskier
exclusive reliance on a working-backward approach might be to the largely untried
teaching of twenty-first century skills. But the drawback of the emergence approach,
of course, is that there is no guarantee that a path can be found to the emergent goal.
Invention is required at every step, with all its attendant uncertainties.
Two concrete examples may help clarify the nature of an “emergence” approach
and its benefits. The first example expands on the previously cited work of Moss
(2005). The second example, drawn from work on scientific literacy, points to a
potentially major twenty-first century skill that has gone unrecognized in the
top-down and “working-backward” approaches that have dominated mainstream
thinking about twenty-first century skills.
1. Beyond rational number skills to proportional thinking. Failure to master
rational numbers is endemic and has been the subject of much research. Much of
the difficulty, it appeared, is that students transferred their well-learned whole
number arithmetic to fractions and thus failed to grasp the essential idea of
proportionality, or the idea that fractions are numbers in their own right. The
standard way of introducing fractions, via countable parts of a whole, was seen
as reinforcing this tendency. Joan Moss and Robbie Case observed, however, that
children already possessed an idea of proportionality, which they could demon-
strate when asked to pour liquid into two different-sized beakers so that one was
as full as the other. Once proportional reasoning was recognized as a realistic
goal for mathematics teaching, “working backwards” could then be applied to
devising ways of moving toward that goal. Moss (2005) developed a whole envi-
ronment of artifacts and activities the purpose of which was to engage students
in thinking proportionally. Instead of introducing fractions as the starting point
for work on rational numbers, Moss and Case started with percentages, as being
more closely related to spontaneous understanding (consider the bars on computer
screens that register what percent of a task has been completed). In final assess-
ments, students in grades 5 and 6 outperformed educated adults. Another name
for proportional thinking is rational number sense. Greeno (1991) characterized
number sense as knowing one’s way around in a numerical domain, analogous to
5 New Assessments and Environments for Knowledge Building 241

knowing one’s way around in a geographical area. It is not something that is

directly taught but rather something that emerges from experience in crossing
and recrossing a domain in different directions and with different purposes.
It is assessable, but it is not specifiable in the way that hard skills are. And, quite
obviously, proportional thinking or rational number sense is a more fundamental
and more skill-enhancing outcome than mastering (or not quite mastering) a
number of rational number algorithms.
2. Beyond “scientific method” to theory building. The second example of an
emergence approach, more directly related to twenty-first century skills, comes
from work on theory building. Broadly conceived, creative knowledge work of
all kinds—planning, inventing, and so forth—is theory building. Even the Wright
Brothers, known to the world as exceptionally clever tinkerers, were explicitly
engaged in theory building at the same time they were engaged in building an
airplane (Bereiter 2009). Ability to construct, test, and improve theory-like
knowledge structures could therefore rate as a top-level twenty-first century skill.
It does not appear on twenty-first century skill lists, however, possibly because it
is not readily described in skill terms and because little is known about what
students are capable of in this respect. Expert opinion has suggested that work on
theory building should wait until high school (Smith and Wenk 2006) and that
the learning progression should start with hypothesis testing and control of vari-
ables (Kuhn et al. 1992; Schauble et al. 1995). Instructional results from this
approach have not been encouraging with respect to scientific literacy, and there
have been many efforts to find new approaches (Carey et al. 1989; Carey and
Smith 1993; Honda 1994; Smith et al. 2000), with further confirmation of the
conventional expert wisdom that theory building is beyond the capacity of young
students. When free to pursue problems of understanding on their own initiative,
however, students were observed to engage spontaneously in a good deal of theo-
rizing (Scardamalia and Bereiter 2006). A small experiment was carried out in
which grade 4 students in a class where knowledge building was the norm were
compared with similar students who had followed a more traditional inquiry
approach (Bereiter and Scardamalia 2009). In the knowledge-building class there
was no explicit teaching of “scientific method” and no carrying out of pre-specified
experiments. Instead, the students were supported in creating, exploring, and
considering theories from multiple perspectives. Results showed significantly
higher levels of theoretical work and scientific literacy and superior scientific
writing for the emergent goals approach (Bereiter and Scardamalia 2009; Chuy
et al. 2009). Theory building, it turns out, is not only possible in 10- to 12-year-
olds but also at even earlier ages. A kindergarten teacher in the same school
learned of the findings and thought her students might have relevant, untapped
capacities. She asked them to generate theories about why some trees in their
schoolyard had no new leaves in the early spring while other trees did. The children
not only generated a number of reasonable explanations but also connected these
with supportive facts. It would seem, therefore, that theory building could justifi-
ably gain a place among the twenty-first century skills to be developed and tested
from early childhood onward. Work by Shutt et al. (2011) also supports this
point of view.
242 M. Scardamalia et al.

In a later section, on technology for supporting the emergence of new competencies

(pp. 237 ff.), we discuss the specific forms of support that have enabled the
achievement of exceptional levels of proportional reasoning and theory development.
As the preceding examples suggest, discovering new goals is not simply a matter of
turning students loose in an environment and waiting to see what happens.
Discovering new goals is an aspect of scientific discovery, and rarely is such discovery
accidental. People know in a general way what they are looking for, and particular
moves may be carefully calculated, but this process as a whole has to be structured
so as to allow room for unexpected insights. When Darwin set sail on the beagle, he
did not know he was about to explain the origin of species, but he was not merely a
collector of curious specimens, either.
Most current school reform efforts, whether involving new management structures
or the introduction of new standards and curricula, are additive as far as their
treatment of twenty-first century skills is concerned. Changes are based on conser-
vative practices and templates drawn from instruction in traditional subjects. More
transformative change requires goals and methods to be considered anew. Education
for twenty-first century skills may in fact have no “tried and true” methods to draw
on, so riskier approaches are needed. It would be difficult to get excited about
twenty-first century education reform were it nothing more than extending existing
goals to more demanding performance levels. It should, of course, include such
goals—performance demands are indeed likely to rise, and there will, no doubt,
continue to be students who need help in meeting even today’s modest standards.
But anything that deserves the name of education for the twenty-first century needs
new kinds of objectives, not simply higher standards for existing ones.
In the following sections, we examine twenty-first century skills as they are
being enacted in knowledge-creating organizations. We focus on what is involved
in the knowledge creation being carried out by experts actually working in these
organizations, providing a sharpened focus for “working backward” to identify
methods and goals that might apply to schools, while allowing us to go beyond the
identification of the desirable traits and skills that are viewed by employers wishing
to hire people for knowledge work. We then consider the knowledge-building
environments that support work in knowledge-creating organizations, followed by
examining learning and assessment theory. In the section on specific investiga-
tions, we propose investigations within an emergence framework, using findings
from the working-backward approach to test transfer and generalization effects so
as to achieve a best-of-both-worlds synthesis of working backward and emergence
of new competencies.

Characteristics of Knowledge-Creating Organizations

How do businesses succeed in a knowledge economy? How are knowledge-intensive

firms organized and how do they function? How are jobs different in a knowledge
economy? And what kinds of skills are needed?
5 New Assessments and Environments for Knowledge Building 243

Industry- or firm-level studies in the USA (Stiroh 2003), the U.K. (Borghans
and ter Weel 2001; Dickerson and Green 2004; Crespi and Pianta 2008), Canada
(Gera and Gu 2004; Zohgi et al. 2007), France (Askenazy et al. 2001; Maurin and
Thesmar 2004), Finland (Leiponen 2005), Japan (Nonaka and Takeuchi 1995),
and Switzerland (Arvanitis 2005) have found many similar results—a major factor
in the success of highly productive, innovative firms is the use of ICT (UNESCO
2005). Of course, productivity and innovation increases did not come merely with
the introduction of new technologies. Rather, technology use must be associated
with a pattern of mutually reinforcing organizational structures, business practices,
and employee skills that work together as a coherent system. Also, organizational
structures have become flatter, decision making has become decentralized, informa-
tion is widely shared, workers form project teams within and across organizations,
and work arrangements are flexible. These changes in organizational structures and
practices have been enabled by the application of ICT for communication, informa-
tion sharing, and simulation of business processes. For example, a U.S. Census
Bureau study (Black and Lynch 2003) found significant firm-level productivity
increases associated with changes in business practices that included reengineering,
regular employee meetings, the use of self-managed teams, up-skilling of employees,
and the use of computers by front-line workers. In Canada, Zohgi et al. (2007) found
a strong positive relationship between both information sharing and decentralized
decision making and a company’s innovativeness. Recent studies of firms (Pilat 2004;
Gera and Gu 2004) found significant productivity gains when ICT investments were
accompanied by other organizational changes, such as new strategies, new business
processes and practices, and new organizational structures. Murphy (2002) found
productivity gains when the use of ICT was accompanied by changes in production
processes (quality management, lean production, business reengineering), manage-
ment approaches (teamwork, training, flexible work, and compensation), and external
relations (outsourcing, customer relations, networking).
These changes in organizational structure and business practices have resulted in
corresponding changes in the hiring practices of companies and the skills needed by
workers. A study of labor tasks in workplaces found that, commencing in the 1970s,
routine cognitive and manual tasks in the U.S. economy declined and nonroutine
analytic and interactive tasks grew (Autor et al. 2003). This finding was particularly
pronounced for rapidly computerizing industries. The study found that, as ICT is
taken up by a firm, computers substitute for workers who perform routine physical
and cognitive tasks but they complement workers who perform nonroutine problem-
solving tasks. Similar results were found in the U.K. and the Netherlands (Borghans
and ter Weel 2001; Dickerson and Green 2004), France (Maurin and Thesmar 2004)
and Canada (Gera and Gu 2004).
Because repetitive, predictable tasks are readily automated, computerization of
the workplace has raised the demand for problem-solving and communications
tasks, such as responding to discrepancies, improving production processes, and
coordinating and managing the activities of others. In a survey of U.K. firms,
Dickerson and Green (2004) found an increased demand for technical know-how
and for skills in high-level communication, planning, client communication,
244 M. Scardamalia et al.

horizontal communication, problem solving, and checking. Meanwhile, there was a

decreased demand for physical skills. The net effect of these changes is that companies
in the USA, the UK, and other advanced economies (Lisbon Council 2007) are hiring
workers with a higher skill set. It is also interesting that many of these skills
(e.g., communication, collaboration, flexibility) are often referred to as “soft skills,”
yet are some of the most important for success and some of the most difficult to help
people develop to high levels of refinement.
The creation of knowledge as a social product (Scardamalia and Bereiter 2003,
2006) is a major part of that higher skill set. It requires collective responsibility for
accomplishments, and it is something that scientists, scholars, and employees of
highly innovative companies do for a living (Nonaka and Takeuchi 1995). An interesting
example is the design of Boeing 787 aircraft, built by nearly 5,000 engineers
(not counting production workers) from around the world. The design and engineering
work takes place simultaneously at multiple sites, over a long period of time, and
yet all the parts ultimately fit nicely together (Gates 2005). In collaborative, creative
endeavors of this nature, team members need to understand the top-level goal and
share responsibility for the interrelated network of ideas, subgoals, and designs,
with success dependent on all members rather than concentrated in the leader.
They share responsibility for establishing effective procedures, for assigning and
completing practical tasks, for understanding and facilitating team dynamics
(Gloor 2006), for remaining cognitively on top of activities and ideas as they unfold
(Leonard-Barton 1995), and for the process as a whole. As issues emerge, they col-
lectively shape the next steps, build on each other’s strengths, and improve their
ideas and designs. Members create the cultural capital of their organization as they
refine the “knowledge space” and products that represent their collective work.
Of course this work includes timelines, specified goals, and deadlines. The idea
of collective responsibility is not to ignore such aspects but to engage participants
in setting deadlines, taking responsibility for achieving them, and redefining goals
and schedules as necessary. It also requires a commitment to working in public
spaces, making one’s thinking and processes explicit and available, and entering
artifacts into the shared knowledge space to advance the state of knowledge of
the community. If everyone is doing the same thing (as is often the case in schools),
the redundant, repetitive work interferes with productivity. The shared problem
space needs to grow, based on shared goals and helpful, diverse contributions from
all members.
This cluster of changes—organizational structure, business practices, and
more-complex employee tasks and skills—is particularly pronounced for knowledge-
intensive, knowledge-creating organizations. Probably the most intensive knowledge-
creating organizations are research laboratories. Current research in the sociology
and anthropology of science has focused on two aspects of the work of scientists:
the distributed nature of scientific work over time, resources, and place and the
moment-by-moment coordination of instruments, representations, and discourse as
scientists construct meaning from the results of their research.
In contemporary science, creating new knowledge requires the coordination of
activities through time and across space to assemble methods, tools, and theories,
5 New Assessments and Environments for Knowledge Building 245

building on previous findings to conduct new research and generate new knowledge
(Fujimura 1992). To achieve this spatial and temporal coordination, scientists
develop technological and social systems that support the movement of specialized
scientific objects, like ideas, data, sketches, and diagrams, across this distributed
network. This coordination within and across organizations and across time, place,
and objects was apparent in Kozma’s study (Kozma et al. 2000; Kozma 2003) of
chemists in a pharmaceutical company. Here the synthetic products of one group
were frequently the starting materials of another group, as activities related to the
creation of a new drug were distributed across laboratories, chemists with different
specializations, and equipment with different purposes. This coordination was
maintained, in part by standardized procedures and in part by attaching labels with
diagrams of chemical structures to the vials as they moved from lab to lab.
The laboratory is where the moment-by-moment work of science is done, much
of it centered on instruments and representations. In their collaborative activities,
scientists talk and represent visually their ideas to one another in supportive physical
spaces (Ochs et al. 1996). The indexical properties of these physical spaces and
representations are essential for the ways that scientists collaborate and establish
shared meaning (Goodwin and Goodwin 1996; Hall and Stevens 1995; Suchman
and Trigg 1993). In their discourse, scientists make references to the specific features
of diagrams and data visualizations as they coordinate these representations to
understand the products of their work (Kozma et al. 2000; Kozma 2003). The features
of these representations are often used as warrants for competing claims about their
finding, as scientists try to adjudicate their different interpretations.
These research findings on the practices, organizational structures, and needs of
innovative, knowledge-creating organizations have significant implications for the
practices and organizational structures of environments needed to support the acqui-
sition of twenty-first century skills and for finding productive connections between
in- and out-of-school learning environments. Knowledge-creating organizations
rank high on all of the twenty-first century skills listed in various documents and
articles (for example, The Partnership for 21st century skills 2009; Binkley et al.
2009; Johnson 2009). Consequently, an analysis of knowledge-creating organiza-
tions additionally provides high-end benchmarks and models to guide the design
and implementation of modern assessment. For example, the literature on how
distributed teams have managed to successfully produce more and better outputs
helps to operationalize concepts such as collaboration, group problem solving, use
of ICT, and so on. Also relevant are the social, material, and technological practices
and organizational structures in which members of knowledge-creating organiza-
tions operate.
Table 5.1 maps in condensed form the characteristics of knowledge-creating
organizations onto the twenty-first century skills presented in Chap. 1. Our goal is
to align these different perspectives and, as elaborated below, provide an analytic
framework for educational environments and assessments to identify those most in
keeping with characteristics of knowledge-creating organizations.
There are major differences between twenty-first century skills as they figure
in school curricula and the skills manifested in knowledge-creating organizations.
246 M. Scardamalia et al.

Table 5.1 Twenty-first century skills as experienced in knowledge-creating organizations

Twenty-first century skills Experience in knowledge-creating organizations
Creativity and innovation Work on unsolved problems; generate theories and models, take
risks, etc.; pursue promising ideas and plans
Communication Knowledge building/progressive discourse aimed at advancing the
state of the field; discourse to achieve a more inclusive,
higher-order analysis; open community knowledge spaces
encourage peer-to-peer and extended interactions
Collaboration/teamwork Collective or shared intelligence emerges from collaboration and
competition of many individuals and aims to enhance the social
pool of existing knowledge. Team members aim to achieve a
focus and threshold for productive interaction and work with
networked ICT. Advances in community knowledge are prized,
over-and-above individual success, while enabling each
participant to contribute to that success
Information literacy/ Going beyond given information; constructive use of and contribution
research to knowledge resources to identify and expand the social pool of
improvable ideas, with research integral to efforts to advance
knowledge resources and information
Critical thinking, High-level thinking skills exercised in the course of authentic
problem solving, knowledge work; the bar for accomplishments is continually
and decision making raised through self-initiated problem finding and attunement to
promising ideas; participants are engaged in complex problems
and systems thinking
Citizenship—local and Citizens feel part of a knowledge-creating civilization and aim to
global contribute to a global enterprise; team members value diverse
perspectives, build shared, interconnected knowledge spanning
formal and informal settings, exercise leadership, and support
inclusive rights
ICT literacy ICT integrated into the daily workings of the organization; shared
community spaces built and continually improved by partici-
pants, with connection to organizations and resources
worldwide
Life and career skills Engagement in continuous, “lifelong,” and “life-wide” learning
opportunities; self-identification as a knowledge creator,
regardless of life circumstance or context
Learning to learn/ Students and workers are able to take charge at the highest,
metacognition executive levels; assessment is integral to the operation of the
organization, requiring social as well as individual
metacognition
Personal and social Team members build on and improve the knowledge assets of the
responsibility—incl. community as a whole, with appreciation of cultural dynamics
cultural competence that will allow the ideas to be used and improved to serve and
benefit a multicultural, multilingual, changing society

In schools the skills are frequently treated separately, each having its own learning
progression, curriculum, and assessment. In knowledge-creating organizations
different facets of work related to these skills represent a complex system, with the
skills so intertwined that any effort to separate them in contexts of use would
undercut the dynamic that gives them meaning.
5 New Assessments and Environments for Knowledge Building 247

Characteristics of Knowledge-Building Environments

Knowledge-building environments represent complex systems that support emergent

outcomes. They are places that, like knowledge-creating organizations, produce
public knowledge—knowledge that does not just reside in the minds of individuals
but that is available to others to build on and improve. Public knowledge develops
through discourse, in which declarative statements play a necessary role, as do models,
theories, and artifacts that are available to the community as a whole. Having stu-
dents become active agents in knowledge construction is an important theme in the
literature on school reform and knowledge-building processes (Engle and Conant
2002; Herrenkohl and Guerra 1998; Lamon et al. 1996; Lehrer et al. 2000; Paavola
and Hakkarainen 2005; Tabak and Baumgartner 2004). Of particular interest in this
regard is collective cognitive responsibility, the requirement to take responsibility
for the state of public knowledge (Scardamalia 2002).
As the Boeing example suggests, networked, communal knowledge spaces
are at the heart of work in knowledge-creating organizations. Accordingly, the
work of participants has an “out-in-the-world” existence. The intellectual life of the
community—objectified as theories, inventions, models, plans, and the like—is
accessible, in tangible form. In the business world, this is referred to as the organiza-
tion’s corporate knowledge; in the knowledge-building literature, it is referred to as
“community knowledge” (Scardamalia 2002). This community knowledge space is
typically absent from classrooms, making it hard for students’ ideas to be objecti-
fied, shared, examined, improved, synthesized, and used as “thinking devices”
(Wertsch 1998) so as to enable further advances. It also makes assessment difficult
because students’ ideas are neither explicit nor in tangible form. In contrast, the
commitment to work in open, shared spaces not only renders ideas as objects of
discussion and improvement but opens the door for concurrent, embedded, and
transformative assessment, as we elaborate below. In turn, these communities
can sustain work at the high end of twenty-first century skills, as identified in
Table 5.1.

Group Learning

Group learning and group cognition may well become the dominant themes of
technology in the next quarter-century, just as collaborative learning was in the
previous one (Stahl 2006). Group learning is learning by groups, which is not the
same as learning in groups or individual learning through social processes. The term
learning organization (Senge 1990) reflects this emphasis on the organization itself
operating as a knowledge-advancing entity and reflects the larger societal interest in
knowledge creation. Knowledge building is a group phenomenon, even when
contributions come from identifiable individuals. Members are responsible for the
production of public knowledge that is of value to a community. Again, this maps
directly onto the Boeing example presented above. The community may be a
248 M. Scardamalia et al.

research or design group or the world at large, or it may be a group of learners—in

which case it is important to distinguish individual learning from the group’s
knowledge-building accomplishments. Neither one can be reliably inferred from
the other, although the interaction between the two is vital and deserving of study in
its own right. We return to this issue in the final sections of this chapter.
In a knowledge-building group, the crucial assessment questions are about the
group’s achievements in advancing the state of knowledge—comparable to the
“state of the art” reviews common in the disciplines and professions. Self-assessment
by a knowledge-building group can be valuable both for helping the group progress
and for individual learning (Lee et al. 2006). External assessment can serve the
purposes of troubleshooting and management. Evidence available suggests that
such an approach increases individual learning, not just group learning, because
the group needs each individual’s contribution; thus there is social pressure to
perform (e.g., Barron 2003). However, this is a finding much in need of replication
and extended study.

Knowledge-Building Developmental Trajectory

Building on the characteristics of knowledge-creating organizations and what we

know about learning, we can begin to specify the characteristics of knowledge-
building environments and the implications they have for educational practices.
Table 5.2 is an elaboration of Table 5.1 and provides a developmental framework for

Table 5.2 Developmental trajectory for knowledge-creating environments

Characteristics of knowledge-creating organizations
Twenty-first century skills Entry level High
Creativity and innovation Internalize given information; Work on unsolved problems;
beliefs/actions based on generate theories and models,
the assumption that take risks, etc.; pursue promising
someone else has the ideas and plans
answer or knows the truth
Communication Social chitchat; discourse that Discourse aimed at advancing the
aims to get everyone to state of the field and at achieving
some predetermined point; a more inclusive, higher-order
limited context for analysis; open spaces encourage
peer-to-peer or extended peer-to-peer and extended
interactions interactions
Collaboration/teamwork Small group work: divided Shared intelligence from collaboration
responsibility to create a and competition enhances
finished product; the existing knowledge. Individuals
whole is the sum of its interact productively and work
parts, not greater than that with networked ICT. Advances in
sum community knowledge are prized
over individual success, while
enabling each to contribute to it
(continued)
5 New Assessments and Environments for Knowledge Building 249

Table 5.2 (continued)

Characteristics of knowledge-creating organizations
Twenty-first century skills Entry level High
Information Inquiry: question-answer, Collaborative expansion of social
literacy/research through finding and pool of improvable ideas, with
compiling information; research integral to efforts to
variable testing research advance knowledge
Critical thinking, Meaningful activities are High-level thinking skills exercised
problem solving, and designed by the director, in authentic knowledge work; the
decision making teacher, or curriculum bar for accomplishments is
designer; learners work on continually raised by participants
predetermined tasks set by as they engage in complex
others problems and systems thinking
Citizenship—local and Support of organization and Citizens feel part of a knowledge-
global community behavioral creating civilization and aim to
norms; “doing one’s best”; contribute to a global enterprise;
personal rights they value diverse perspectives,
build shared knowledge in formal
and informal settings, exercise
leadership, and support inclusive
rights
ICT literacy Familiarity with and ability to ICT integrated into organization’s
use common applications daily work; shared community
and web resources and spaces built and continually
facilities improved by participants, with
connection worldwide
Life and career skills Personal career goals Engagement in continuous, “life-
consistent with individual long,” and “life-wide” learning
characteristics; realistic opportunities; self-identification
assessment of require- as a knowledge creator, regardless
ments and probabilities of of life circumstance or context
achieving career goals
Learning to learn/ Students and workers provide Students and workers are able to take
metacognition input to the organization, charge at the highest, executive
but the high-level levels; assessment is integral to
processes are under the the operation of the organization,
control of someone else requiring social as well as
individual metacognition
Personal and social Individual responsibility; Team members build on and improve
responsibility—incl. local context the knowledge assets of the
cultural competence community, with appreciating
cultural dynamics that allow the
ideas to be used and improved for
benefit of multicultural, multilin-
gual, changing society

analyzing learning environments. For each twenty-first century skill, the table suggests
a continuum running from the entry-level characteristics that may be expected
of students who have had no prior engagement in knowledge building to a
level characteristic of productive participants in a knowledge-creating enterprise.
250 M. Scardamalia et al.

Fig. 5.1 Centrality of deep

disciplinary knowledge to all
knowledge work

The continuum is an “emergence” continuum—a developmental trajectory from

active or constructivist learning as the entry point, to complex systems of interactiv-
ity and knowledge work that enable the generation of new knowledge, the capacity
to exceed standards, and the drive to go beyond best practice at the high end.
In the section on needed research, we propose experiments to develop this
scheme, including additional points along the continuum, to indicate how designing
environments with sights set on the high-end of the scale can facilitate the advancement
of any school, any teacher along these lines.

Advancing Domain Knowledge and Twenty-First Century

Skills in Parallel

Twenty-first century skills—often labeled “soft” or “generic” skills—have been

widely recognized as central to innovative capacity and hence as vital for success
in a twenty-first century global economy. Although twenty-first century skills are
recognized in recent curriculum standards, the main emphasis in standards and
assessments is on “hard” skills in language and mathematics as well as “hard” factual
knowledge. There is a concern that attention given to “soft” skills will detract from
efforts to improve the skills and subject-matter knowledge for which the schools are
held accountable. The consensus among researchers in the learning sciences is that
these two are not in conflict (Bransford et al. 2000; Darling-Hammond et al. 2008);
their interdependence is suggested in Fig. 5.1. In formal education beyond the most
basic “3 Rs” level, hard skills are generally treated as a part of domain knowledge.
Ability to solve quadratic equations, for instance, is part of algebraic domain
knowledge. Hence, as modeled in Fig. 5.1, domain knowledge and hard skills are
combined to constitute the focus of formal education, while a common set of soft
skills surrounds expertise in all domains.
5 New Assessments and Environments for Knowledge Building 251

Making twenty-first century skills universally accessible, rather than the province
of knowledge élites, requires that the environments that support knowledge creation
be made accessible to all. From the emergence perspective, the challenge is to shift
to environments that take advantage of what comes naturally to students across the
full range of twenty-first century skills (idea production, questioning, communica-
tion, problem solving, and so forth) and engage them in the kinds of environments
for sustained idea development that are now the province of knowledge élites. These
knowledge-building environments that score at the high end of all the developmen-
tal continua identified in Table 5.2 increase innovative capacity through engagement
in a knowledge-building process—the production of public knowledge of value to
others so that processes of collective responsibility for knowledge advancement can
take hold (Scardamalia and Bereiter 2003). That is how idea improvement, leading
to deep disciplinary knowledge, gets to the center of the enterprise, with twenty-first
century skills inseparable and serving as enablers.
Comparative research and design experimentation are needed to add substan-
tially to the knowledge base on relations between inquiry and knowledge-building
activities and the meeting of traditional achievement objectives. The research and
design experiments proposed in the final section should help address these issues
through use of formative assessment, combined with other assessments, selected to
evaluate advances in both “hard” and “soft” skills, and the changes over time that
are supported through work in information-rich, knowledge-building environments.
The proposition to be tested is: Collective responsibility for idea improvement in
environments that engage all students in knowledge advancement should result in
advances in domain knowledge in parallel with advances in twenty-first century
skills. This argument is in line with that set forth by Willingham (2008): “Deep
understanding requires knowing the facts AND knowing how they fit together,
seeing the whole.”
This notion that deep understanding or domain expertise and twenty-first century
skills are inextricably related has led many to argue that there is not much new in
twenty-first century skills—deep understanding has always required domain under-
standing and collaboration, information literacy, research, innovation, metacognition,
and so forth. In other words, twenty-first century skills have been “components
of human progress throughout history, from the development of early tools, to
agricultural advancements, to the invention of vaccines, to land and sea exploration”
(Rotherham and Willingham 2009).
But is it then also true that there are no new skills and abilities required to address
the needs of today’s knowledge economy? One defensible answer is that the skills
are not new but that their place among educational priorities is new. According to
Rotherham and Willingham, “What’s actually new is the extent to which changes in
our economy and the world mean that collective and individual success depends on
having such skills. … If we are to have a more equitable and effective public education
system, skills that have been the province of the few must become universal.”
“What’s new today is the degree to which economic competitiveness and educa-
tional equity mean these skills can no longer be the province of the few” (Rotherham
2008). Bereiter and Scardamalia (2006) have argued, however, that “there is in fact
252 M. Scardamalia et al.

one previously unrecognized ability requirement that lies at the very heart of the
knowledge economy. It is the ability to work creatively with knowledge per se.”
Creative work with knowledge—with conceptual artifacts (Bereiter 2002)—must
advance along with work with material artifacts. Knowledge work binds hard and
soft skills together.
The deep interconnectedness of hard and soft skills has important implications
for assessment, as does the commitment to individual contributions to collective
works. As Csapó et al. state in Chapter 4 of this book, “how a domain is practiced,
taught, and learned impacts how it should be assessed… the real promise of technol-
ogy in education lies in its potential to facilitate fundamental, qualitative changes in
the nature of teaching and learning” (Panel on Educational Technology of the
President’s Committee of Advisors on Science and Technology 1997, p.33).
Domains in which it is most important to include technology in the assessment of
twenty-first century skills include, according to Csapó and colleagues, those in
which technology is so central to the definition of the skill that removing it would
render the definition meaningless (e.g., the domain of computer programming),
those in which higher levels of performance depend on technology tools, and those
that support collaboration, knowledge building, and the social interactions critical
for knowledge creation. We would argue that to make knowledge building and
knowledge creation broadly accessible, technological support for knowledge build-
ing also needs to be broadly accessible (e.g., see also Svihla et al. (2009)).
Assessment of “soft” skills is inherently more difficult than assessing the “hard”
skills that figure prominently in educational standards. Assessing knowledge-
creation processes may be even harder. Nonetheless, this core capability should be
further enhanced and clarified through programs of research and design that aim to
demonstrate that the processes that underlie knowledge creation also underlie deep
understanding; knowledge-building environments promote both. We return to these
ideas below.

Advancing Literacy and Closing Gaps

Among the skills needed for life in the knowledge age, literacy is perhaps the most
crucial. Without the ability to extract and contribute useful information from
complex texts, graphics, and other knowledge representations, one is in effect barred
from knowledge work. Print literacy (as with other literacies) has both hard-skill
and soft-skill components; e.g., in reading, fluent word recognition is a testable
hard skill, whereas reading comprehension and critical reading are important soft
skills. Soft-skill components of reading are mandated and tested, but traditional
schooling typically deals with them through often ineffectual “practice makes perfect”
approaches.
Although there are diverse approaches to literacy education, most of them treat it
as an objective to be pursued through learning activities that have literacy as their
main purpose. For the most part, with school-based reading, motivation comes from
5 New Assessments and Environments for Knowledge Building 253

the level of interest in the reading material itself. Consequently, the unmotivated
reader, who is frequently one for whom the decoding of print is not fluent, is a
persistent problem (Gaskin 2005). During the past decade, however, new approaches
have developed in which the focus is not on literacy as such but on collaborative
inquiry, where the primary motivation for reading is solving shared problems of
understanding. Effects on literacy have been as great as or greater than those of
programs that emphasize literacy for its own sake (Brown and Campione 1996; Sun
et al. 2008, 2010). Work in Knowledge Forum technology, specially developed to
support knowledge building, has provided evidence of significant literacy gains
through ICT (Scardamalia et al. 1992; Sun et al. 2008, 2010). Whereas literacy-
focused programs typically engage students with reading material at or below their
grade level, students pursuing self- and group-directed inquiry frequently seek out
material that is above their grade level in difficulty, thus stretching their comprehen-
sion skills and vocabularies beyond those normally developed. Rather than treating
literacy as a prerequisite for knowledge work, it becomes possible to treat knowl-
edge work as the preferred medium for developing the literacies that support it, with
student engagement involving a full range of media objects, so as to support multi-
literacies. This approach raises major research issues, which we return to in the final
section of this chapter.

Knowledge-Building Analytic Framework

We have developed a knowledge-building analytic framework to advance the two

goals presented in the introduction to this chapter, to:
• Derive an analytic framework for analyzing environments and assessments that
characterize and support knowledge-creating organizations and the knowledge-
building environments that sustain them
• Apply this framework to a set of environments and assessments to better under-
stand models, possibilities, and variations in the extent for which they engage
students in knowledge-creating organizations or prepare them for work in them
In the “Annex” at the end of this chapter we have included a template that can
serve as a scoring scheme to apply to a broad range of environments and assessments,
making it possible to characterize strengths and weaknesses of knowledge-building
environments and assessments. The scheme is the same as presented above, in
Table 5.1. It is simply set up in the “Annex” as a scoring scheme to encourage users
to assess specific environments and compare scores by different assessors of the same
environment. Users have reported that it is a helpful instrument for reflection on key
aspects of the environment analyzed, and becomes increasingly beneficial once they
have a chance to view and discuss ratings of the same environment by different raters.
The discussion of rationales for different ratings facilitates understanding of the
dimensions and functions associated with knowledge-creating organizations. Graduate
students studying in the field of knowledge creation tended to rate environments lower
254 M. Scardamalia et al.

than the proponents of those environments (see Table 5.3 and Fig. 5.6, second section
of the “Annex”), but not much can be made of this, as the sample is very small. We
offer the template to foster the sort of conversation that may be engendered through
analysis of a developmental framework related to characteristics of a knowledge-cre-
ating organization.

Knowledge-Building and Learning Theories

An important question is how competencies that foster work in a knowledge society

relate to modern theories of learning. For example, how does an emphasis on knowl-
edge building fit the “How People Learn” framework, shown in Fig. 5.2, which has
been used by a National Academy of Science committee to organize what is known
about learning and teaching (National Research Council 2000). The framework
highlights a set of four lenses that can be used to analyze learning environments,
ranging across homes, community centers, classrooms, schools, and higher levels of

Fig. 5.2 The “How People Learn” framework (Adapted from How People Learn–National
Research Council, 2000)
5 New Assessments and Environments for Knowledge Building 255

educational organization. The components of the framework involve a focus on four

areas that need to be flexibly balanced, depending on current goals and needs. Each
area of the framework is accompanied by a set of questions that are useful for
exploring the design of learning opportunities, particularly those that support
knowledge building.
1. Knowledge centered: What needs to be taught to meet the changing needs of
people and societies? (Answering this question is fundamental to this entire
project.)
2. Learner centered: How can new information be connected with learners’ existing
beliefs, values, interests, skills, and knowledge so that they learn with understanding
and can flexibly use what they know?
3. Community centered: How can we develop communities of learners that value
excellence as people work together to build new knowledge for the common
good? And how can we broaden our sense of community and explore opportunities
for learning that connect activities in and outside schools?
4. Assessment centered: How can we develop frequent and useful opportunities for
students, teachers, school systems, and nations to assess the progress they are
making toward twenty-first century skills?

Knowledge Centered

As discussed above, the world has changed and different kinds of skills and
knowledge are required for successful and productive lives in the twenty-first
century. Many of the skills identified above are not tied directly to traditional subject
domains, such as the sciences, mathematics, or history—all these, of course, will
continue to be important in the twenty-first century. Work by contributors to this
series of chapters suggests that constant questioning about what people need to learn
is one of the most important activities for our future.

Expertise and Knowledge Organization

More than ever before, experts’ knowledge must be more than a list of disconnected
facts and must be organized around the important ideas of current and expanding
disciplines. This organization of knowledge must help experts know when, why,
and how aspects of their vast repertoire of knowledge and skills are relevant to
any particular situation (see Bransford et al. 2000). Knowledge organization especially
affects the ways that information is retrieved and used. For example, we know that
experts notice features of problems and situations that may escape the attention of
novices (e.g., see Chase and Simon 1973; Chi et al. 1981; de Groot 1965). They
therefore “start problem solving at a higher place” than novices (de Groot 1965).
Knowledge building suggests that learning must include the desire and ability to notice
256 M. Scardamalia et al.

new connections and anomalies and to actively seek ways to resolve disconnects by
restructuring what they know and generating new, domain-bridging ideas.
Generative knowledge building must also be structured to transcend the problem
that current courses and curriculum guidelines are often organized in ways that fail
to develop the kinds of connected knowledge structures that support activities such
as effective reasoning and problem solving. For example, texts that present lists of
topics and facts in a manner that has been described as “a mile wide and an inch
deep” (e.g., see Bransford et al. 2000) are very different from those that focus on the
“enduring ideas of a discipline” (Wiske 1998; Wilson 1999). However, a focus on
knowledge building goes beyond attempts to simply improve learning materials and
seeks to help learners develop the vision and habits of mind to develop their own
abilities to refine, synthesize, and integrate.

Adaptive Expertise

An especially important focus on knowledge building separates “routine experts”

from “adaptive experts” (e.g., Hatano and Inagaki 1986; Hatano and Osuro 2003).
Both routine experts and adaptive experts continue to learn throughout their
lifetimes. Routine experts develop a core set of skills that they apply throughout
their lives with greater and greater efficiency. In contrast, adaptive experts are much
more likely to change their core skills and continually expand the breadth and depth
of their expertise. This restructuring of core ideas, beliefs, and skills may reduce
their efficiency in the short run but make them more flexible in the long run. These
processes of restructuring often have emotional consequences that accompany real-
izations that cherished beliefs and practices need to be changed. Research by Anders
Ericsson and colleagues (2009) shows that a major factor in developing expertise is
to resist plateaus—in part by continually moving out of one’s comfort and engaging
in “deliberate practice.” This analysis of expertise highlights the need for unlearning
as well as learning, and for the kinds of social collaboration that are often invisible
when we see write-ups of “experts” in the research literature or the media (e.g., see
Bransford and Schwartz 1999).
This research has implications for the design of environments to support knowl-
edge building. First, an emphasis on building a deep understanding of key ideas is
important. This serves as the basis for organizing facts that would otherwise depend
on sheer memorization. Second, understanding with respect to the adaptability of
knowledge structures highlights the need to support processes of review and
reflection.

Learner Centered

The learner-centered lens of the How People Learn framework overlaps with the
knowledge-centered lens, but specifically reminds us to think about learners rather
5 New Assessments and Environments for Knowledge Building 257

than only about subject matter. Many educators deal with issues of understanding
learners in ways that allow them to engage in culturally responsive teaching (e.g.,
Banks et al. 2007). This includes learning to build on people’s strengths rather than
simply seeing weaknesses (e.g., Moll 1986a, b), and helping people learn to “find
their strengths” when confronted with new knowledge building challenges. Several
important aspects of being learner centered are discussed below.

Understanding the Constructive Nature of Knowing

The constructive nature of knowing grew out of the work of Swiss psychologist Jean
Piaget. Piaget used two key terms to characterize this constructive nature: assimilation
and accommodation. In Piaget’s terms, learners assimilate when they incorporate
new knowledge into existing knowledge structures. In contrast, they accommodate
if they change a core belief or concept when confronted with evidence that prompts
such as change.
Studies by Vosniadou and Brewer illustrate assimilation in the context of young
children’s thinking about the earth. They worked with children who believed that
the earth is flat (because this fit their experiences) and attempted to help them under-
stand that, in fact, it is spherical. When told it is round, children often pictured the
earth as a pancake rather than as a sphere (Vosniadou and Brewer 1989). If they
were then told that it is round like a sphere, they interpreted the new information
about a spherical earth within their flat-earth view by picturing a pancake-like flat
surface inside or on top of a sphere, with humans standing on top of the pancake.
The model of the earth that they had developed—and that helped them explain how
they could stand or walk upon its surface—did not fit the model of a spherical earth.
Everything the children heard was incorporated into their preexisting views.
The problem of assimilation is relevant not only for young children but also for
learners of all ages. For example, college students have often developed beliefs
about physical and biological phenomena that fit their experiences but do not fit
scientific accounts of these phenomena. These preconceptions must be addressed in
order for them to change their beliefs (e.g., Confrey 1990; Mestre 1994; Minstrell
1989; Redish 1996). Creating situations that support accommodation is a significant
challenge for teachers and designers of learning environments—especially when
knowledge building is involved.

Connecting to Students’ Previous Experiences

Ideally, what is taught in school builds upon and connects with students’ previous
experiences, but this is not always the case. A number of researchers have explored
the benefits of increasing the learner centeredness of teaching by actively searching
for “funds of knowledge” in students’ homes and communities that can act as
bridges for helping them learn in school (e.g., Lee 1992; Moll 1986a, b; Moses
1994). Examples include helping students see how the carpentry skills of their
258 M. Scardamalia et al.

parents relate to geometry, how activities like riding the subway can provide a context
for understanding algebra, and how everyday language patterns used outside of
school often represent highly sophisticated forms of language use that may be taught
in literature classes as an academic subject but have not been linked to students’
out-of-school activities. Work by Bell and colleagues specifically links activities
in homes and communities with work in schools (e.g., Bell et al. 2009; Tzou and
Bell 2010).

Learner Centeredness, Metacognition, and Basic Cognitive Processes

Being learner centered also involves an awareness of some basic cognitive

processes that influence learning for everybody. “Metacognition” is the field of
psychology that can be used to help people learn about the cognitive processes that
underlie their own abilities to learn and solve problems. Several cognitive processes
are particularly important.

Attention and Fluency

Learning about attention is an important part of becoming a metacognitive learner.

For example, there are important constraints on how much we can pay attention to
at any particular point in time. The amount of attention that we need to devote to a
task depends on how experienced and efficient we are at doing it. When learning to
read, for example, the effortful allocation of attention to pronouncing words can
make it difficult to also attend to the meaning of what one is reading. The attentional
demands that accompany attempts to learn anything new mean that all learners must
go through a period of “klutziness” as they attempt to acquire new skills and knowledge.
Whether people persist or bail out during these “klutz” phases depends in part on their
assumptions about their own abilities. Some people may decide “I’m not good at this”
and give up trying before they have a chance to learn effectively (e.g., Dweck 1986).
Wertime (1979) notes that an important part of being learner centered is to help
students learn to persist in the face of difficulty by increasing their “courage spans.”
Technology presents challenges of “multitasking,” and many students feel that
this does not hurt their performance. They can be helped to test this idea for them-
selves by listening to a lesson with full attention versus listening to one while also
multitasking. This is an effective way to help students discover their own abilities
and limits rather than simply be forced to comply with “no computers can be on in
this class.”

Transfer

Learning about ourselves as learners also involves thinking about issues of transfer—
of learning in ways that allow us to solve novel problems that we may encounter later.
5 New Assessments and Environments for Knowledge Building 259

The mere memorization of information is usually not sufficient to support transfer.

Learning with understanding typically enhances the experience (e.g., NRC 2000).
An important goal for transfer is cognitive flexibility (e.g., Spiro et al. 1991). Experts
possess cognitive flexibility when they can evaluate problems and other types of
cases in their fields of expertise from many conceptual points of view, seeing
multiple possible interpretations and perspectives. Wiggins and McTighe (1997)
argue that understanding complex issues involves being able to explain them in
more than one way. Spiro et al. (1991) argue that the inability to construct multiple
interpretations in analyzing real-world cases can result from instruction that
oversimplifies complicated subject matter.

Motivation

Helping students learn to identify what motivates them is also an important part of
being learner centered that contributes strongly to knowledge building. Researchers
have explored differences between extrinsic motivators (grades, money, candy, etc.)
and intrinsic motivators (wanting to learn something because it is relevant to what
truly interests you). Both kinds of motivation can be combined; for example, we can
be intrinsically interested in learning about some topics and interested in receiving
extrinsic rewards as well (e.g., praise for doing well, a consultanting fee). However,
some people argue that too much of an emphasis on extrinsic rewards can under-
mine intrinsic motivation because people get too used to the external rewards and
stop working when they are removed (e.g., Robinson and Stern 1997).
There appear to be important differences between factors that are initially moti-
vating (the assumption that learning to skateboard seems interesting), and factors
that sustain our motivation in the face of difficulty (“hmm, this skateboarding is
harder to learn than it looked”). The social motivation support of peers, parents, and
others is an especially important feature that helps people persist in the face of
difficulties. It is also important to be provided with challenges that are just the right
level of difficulty—not so easy that they are boring and not so difficult that they are
frustrating. Creating the right kinds of “just manageable difficulties” for each student
in a classroom constitutes one of the major challenges and requires expert juggling
acts. Explorations of the literature on motivation can be found in Deci and Ryan
(1985), Dweck (1986) and Stipek (2002).

Agency

An emphasis on knowledge building especially highlights an important aspect of

metacognition and motivation that involves the need for people to develop socially
responsive agency. That is, students must learn to make their own choices, experi-
ence the social consequences that arise from them, and revise their strategies when
necessary. This is a progressive process of moving from the situation in which the
teacher makes decisions about student learning to one where students are increas-
ingly responsible for their own learning activities.
260 M. Scardamalia et al.

An example involves a recent set of studies on science kits for middle school
students (Shutt et al. 2009). They involve hands-on activities such as working with
and studying (without harming them) fish, isopods, and a variety of other creatures.
Throughout the course of the year, the goal is to develop a sense of key variables
(e.g., range of temperatures, ranges of acidity, etc.) that affect the life of all species.
As originally developed, the science work is extremely teacher directed; the hypotheses
to be tested and the methods to be used, such as determining whether isopods desire
moist or dry soil, are specified by the teacher. Redesigning these teaching situations
has been found to give much more agency to the students. They are given a terrarium
and told that their task (working in groups) is to keep their organisms (e.g., isopods)
alive. To be successful, they have to choose what questions to ask, how to run the
studies, how to do the kind of background research (via technology when needed),
and so forth. The initial findings (more precise data will be available soon) show
that the sense of agency is very important to students and they take their work very
seriously. This kind of activity can hopefully strengthen other skills such as global
sensitivity since the students all do their work with the well-being of others (even
though they are nonhumans) foremost in their minds.

Community Centered

The preceding discussion explored a number of issues relevant to being knowledge

centered and learner centered. The community centered aspect of the How People
Learn framework is also related to being knowledge and learner centered, but it
focuses special attention on the social, material, and temporal nature of learning.

The Social Aspects of Learning

The social aspects of learning often include the norms and modes of operation of
any community that we belong to or are joining. For example, some classrooms
represent communities where it is safe to ask questions and say, “I don’t understand
this, can you explain it in a different way?” Others follow the norm of, “Don’t get
caught not knowing something.” A number of studies suggest that—in order to
be successful—learning communities should provide people with a feeling that
members matter to each other and to the group, and a shared belief that members’
needs will be met through their commitment to be together (Alexopoulou and Driver
1996; Bateman et al. 1998). Many schools are very impersonal places, and this can
affect the degree to which people feel part of, or alienated from, important commu-
nities of professionals and peers.
Concerns that many schools are impersonal and need to be smaller in order to be
more learner and community centered can also be misinterpreted as simply being an
argument for helping students feel good about themselves. This is very important,
of course, but more is involved as well. More includes searching for “funds of
5 New Assessments and Environments for Knowledge Building 261

knowledge” in students’ lives and communities that can be built upon to enhance
their motivation and learning. The more we know about people, the better we can
communicate with them and hence help them (and us) learn. And the more they
know about one another, the better they can communicate as a community.
The importance of creating and sustaining learning communities can be traced to
Vygotsky’s theory in which culture and human interaction represent central devel-
opmental processes. Vygotsky focused on the intersection between individuals and
society through his concept of the zone of proximal development (ZPD)—the dis-
tance between the actual developmental level as determined by independent prob-
lem solving and the level of potential development as determined through problem
solving under adult guidance, or in collaboration with more capable peers (Vygotsky,
1962/1934). What a child can perform today with assistance, she will be able to
perform tomorrow independently, thus preparing her for entry into a new and more
demanding collaboration. The emphasis here is on the ways learners draw on each
other for ideas and resources that support or scaffold their own learning.

The Material Aspects of Learning

Vygotsky also emphasized the ways in which material resources, such as tools and
technologies, change the nature of tasks and the cognitive skills that are required to
perform them. This is particularly important in the twenty-first century, not only
because of the ways in which technologies have changed the nature of task and
work in the world outside of schools but because students increasingly use a wide
range of technologies in their everyday lives and bring these technologies with them
into schools. Often teachers do not take advantage of these technologies or use the
skills and experiences that students bring with them as a way to increase students’
knowledge of school subjects or further develop their twenty-first century skills.
Learning and assessment are far different if students have access to a range of tech-
nological tools, digital resources, and social support than if they learn or are assessed
without access to these resources; while the real world of work and students’ social
environments are filled with these tools and resources, they can be effectively built
into the learning environment (Erstad 2008).

The Temporal Aspects of Learning

At a broader level, being community centered also means reaching beyond the walls
of the schools in order to connect with students’ out-of-school experiences, including
experiences in their homes.
Figure 5.3, from the LIFE Center, illustrates the approximate time spent in formal
(school) and informal (out-of-school) environments. A great deal of learning goes
on outside of school (Banks et al. 2007), but often teachers do not know how to
connect these kinds of experiences to school learning. Earlier we discussed the idea
of searching for “funds of knowledge” that exist in communities and can be built
262 M. Scardamalia et al.

Fig. 5.3 Time spent in formal and informal learning across a typical lifespan. Estimated time spent
in school and informal learning environments. Note: This diagram shows the relative percentage of
their waking hours that people across the lifespan spend in formal educational environments and
other activities. The calculations were made on the best available statistics for a whole year based
on how much time people at different points across the lifespan spend in formal instructional envi-
ronments. (Reproduced with permission of The LIFE Center.) (The LIFE Center’s Lifelong and
Lifewide Diagram by LIFE Center is licensed under a Creative Commons Attribution-Share Alike

3.0 United States License . (LIFE Center: Stevens et al. 2005) LIFE
Center (2005). “The LIFE Center’s Lifelong and Lifewide Diagram”. This diagram was originally
conceived by Reed Stevens and John Bransford to represent the range of learning environments
being studied at the Learning in Informal and Formal Environments (LIFE) Center (http://life-slc.
org). Graphic design, documentation, and calculations were conducted by Reed Stevens, with key
assistance from Anne Stevens (graphic design) and Nathan Parham (calculations)

upon so as to help students succeed. The challenge is to help students build strong
social networks within a classroom, within a school, and between classrooms and
in- and out-of-school contexts.

Assessment Centered

We’ve discussed learning centered on knowledge, learner, and community; now we

turn to assessment-centered learning. It is easy to assume that assessment simply
involves giving tests to students and grading them. Theories of learning suggest
roles for assessment that involve much more than simply making up tests and
giving grades.
5 New Assessments and Environments for Knowledge Building 263

First, teachers need to ask what they are assessing. This requires aligning their
assessment criteria with the goals for their students (part of being knowledge centered)
and the “readiness” of students in their classroom (learner and community centered).
Assessing memorization (e.g., of properties of veins and arteries) is different from
assessing whether students are understanding why veins and arteries have various
properties. Similarly, assessing whether students can answer questions about life
cycles (of frogs, for example) is different from assessing whether they will sponta-
neously retrieve this information when attempting to solve problems.
At the most general level, issues of what to assess relate to the issue of what
students need to know and be able to do in order to have fulfilling lives once they
graduate. Because of rapid changes in society, this is an issue that constantly needs
to be reconsidered. Debates about standardized tests include concerns that they may
“tip” teaching in a direction that is counterproductive for students because some
teachers spend most of their time teaching to the tests while the tests do not assess
the range of skills, knowledge, and attitudes needed for successful and productive
lives in the twenty-first century.

Different Kinds and Purposes of Assessment

An especially important aspect of the assessment-centered lens in the How People

Learn framework is its emphasis on different kinds of assessments for different
purposes. When most people think about assessments, they think about summative
assessments. These include unit exams at the end of a unit, standardized tests at the
end of the year, and final exams at the end of a course. Summative assessments
come in all forms: multiple choice tests, essays, presentations by students, and so
forth. These assessments are very important as an accountability mechanism for
schools, teachers, and students. Often they reveal important information that the
teachers wish they had seen earlier. This is why formative assessments are important.
These are used for the purpose of improving teaching and learning. They involve
making students’ thinking visible as they progress through the course, giving them
feedback about their thinking, and providing opportunities to revise.

Assessment and Theories of Transfer

It is also important for teachers to understand ways in which assessment practices

relate to theories of transfer. Consider summative assessments, for example. We all
want to make sure that these provide an indication of students’ ability to do some-
thing other than simply “take tests.” Ideally, our assessments are predictive of
students’ performance in everyday settings once they leave the classroom.
One way to look at this issue is to view tests as attempts to predict students’
abilities to transfer from classroom to everyday settings. Different ways of thinking
about transfer have important implications for thinking about assessment. Central to
traditional approaches to transfer is a “direct application” theory and a dominant
264 M. Scardamalia et al.

methodology that Bransford and Schwartz (1999) call “sequestered problem

solving” (SPS). Just as juries are often sequestered in order to protect them from
possible exposure to “contaminating” information, subjects in experiments are
sequestered during tests of transfer. There are no opportunities for them to demon-
strate their abilities to learn to solve new problems by seeking help from other
resources, such as texts or colleagues, or by trying things out, receiving feedback,
and getting opportunities to revise. Accompanying the SPS paradigm is a theory
that characterizes transfer as the ability to directly apply one’s previous learning to
a new setting or problem. We call this the direct application (DA) theory of transfer.
Some argue that the SPS methodology and the accompanying DA theory of transfer
are responsible for much of the pessimism about evidence for transfer (Bransford
and Schwartz 1999).
An alternative view that acknowledges the validity of these perspectives also
broadens the conception of transfer by including an emphasis on people’s “preparation
for future learning” (PFL). Here, the focus shifts to assessments of people’s abilities
to learn in knowledge-rich environments. When organizations hire new employees,
they don’t expect them to have learned everything they need for successful adaptation.
They want people who can learn, and they expect them to make use of resources
(e.g., texts, computer programs, and colleagues) to facilitate this learning. The better
prepared they are for future learning, the greater the transfer (in terms of speed and/
or quality of new learning). Examples of ways to “prepare students for future learning”
are explored in Schwartz and Bransford (1998), Bransford and Schwartz (1999) and
Spiro et al. (1987).
The sole use of static assessments may mask the learning gains of many students,
as well as masking the learning advantages that various kinds of educational
experiences provide (Bransford and Schwartz 1999). Linking work on summative
assessment to theories of transfer may help us overcome the limitations of many
existing tests. Examples of SPS versus PFL assessments of learning and transfer are
discussed in Bransford and Schwartz (1999).

Implications for Assessment Reform

Two distinct approaches to the design of environments and assessment have been
described. One involves working backward from goals to construct a system of
subgoals and learning progressions from an initial state to the goal. The second
approach involves emergent goals that are not fixed in advance but take shape as
learning and thinking proceed. We have indicated the trade-offs associated with
both the working-backward and emergence approaches, and below, after reviewing
assessment challenges related to twenty-first century skills, we specify the research
needed, depending on what one sets out to pursue. In the additive model the “twenty-
first century skills” curriculum is added to the traditional curriculum, although often
the goal is more in line with assimilative efforts to merge skill and content elements
or to piggyback one upon the other. The problem, exacerbated if each twenty-first
5 New Assessments and Environments for Knowledge Building 265

century skill is treated separately, is that the current “mile wide, inch deep” curriculum
will grow miles wider and shallower, with the twenty-first century skills curriculum
taking valuable time away from traditional skills. The goal of the transformational
model is to effect a deeper integration of domain understanding with twenty-first
century skills. The rationale, elaborated in the section on the parallel advance of
domain knowledge and twenty-first century skills, is that if a deep understanding of
domain knowledge is achieved through exercising twenty-first century skills, the
result will be enhanced understanding in the domain, as well as advances in twenty-
first century skills. That is the guiding principle underlying the knowledge-building
approach. The knowledge-building analytic framework, described in the “Annex,”
helps those wishing to engage in this transformation to consider progress along its
multiple dimensions. Since these dimensions represent a complex interactive sys-
tem, treating them separately may prove more frustrating than helpful. Fortunately,
this also means that tackling one dimension is likely to lead to advances along sev-
eral of them. The implication for assessment is that we must anticipate and measure
generalization effects. We elaborate possibilities for design experiments to integrate
working-backward and emergence models in the section on specific investigations.
But first we discuss a broader set of issues regarding assessment challenges and
twenty-first century skills.

Assessment Challenges and Twenty-First Century Skills

The quest for evidence-based assessment of twenty-first century skills is hindered

by many factors. First, there are huge variations in formal and informal learning
environments and the kinds of assessment that are possible in them. Second, the
knowledge and skills that deal with the media and technologies used within a
domain need to be distinguished from domain-specific knowledge and skills
(Bennett et al. 2007; Quellmalz and Kozma 2003). Third, methods for designing
twenty-first century assessments and for documenting their technical quality have
not been widely used (Quellmalz and Haertel 2008). Fourth, assessments need to be
coherent across levels of educational systems (Quellmalz and Pellegrino 2009;
Pellegrino et al. 2001). Coherence must start with agreement on the definition of
twenty-first century skills and their component knowledge and techniques. Moreover,
the design of international-, national-, state-, and classroom-level tests must be
clarified and aligned, otherwise assessments at different levels will not be balanced
and inferences about student performance will be compromised.
Evidence-centered design (Messick 1994; Mislevy and Haertel 2006) links
twenty-first century skills to the task features and reports of evidence that characterize
student performance and progress. In the sections immediately following, we
describe how evidence-centered design can be used to develop formative assessments
that are embedded in learning environments and that link these formative assessments
to large-scale, summative assessments.
266 M. Scardamalia et al.

Cognitively Principled, Evidence-Centered Assessment Design

As described above, research on the development of expertise in many domains has

indicated that individuals proficient in a domain have large, organized, interconnected
knowledge structures and well-honed domain-specific problem-solving strategies
(Bransford et al. 2000). The design of assessments, therefore, should aim to mea-
sure both the extent and connectivity of students’ growing knowledge structures
and problem-solving strategies (Pellegrino et al. 2001; Glaser 1991). For example,
in the domain of science, core knowledge structures are represented in models of the
world built by scientists (Hestenes et al. 1992; Stewart and Golubitsky 1992).
Technologies are seen as tools that support model-based reasoning by automating
and augmenting performance on cognitively complex tasks (Norman 1993; Raizen
1997; Raizen et al. 1995).
The NRC report, Knowing What Students Know, presents advances in measurement
science that support the integration of cognitive research findings into systematic
test design frameworks. As a brief overview, evidence-centered assessment design
involves relating the learning to be assessed, as specified in a student model, to a
task model that specifies features of the task and questions that would elicit observa-
tions of learning, and to an evidence model that specifies the student responses and
scores that serve as evidence of proficiency (Messick 1994; Mislevy et al. 2003;
Pellegrino et al. 2001). These components provide a structure for designing assess-
ments of valued twenty-first century skills and also for evaluating the state of current
assessment practices. Evidence-centered design (Messick 1994; Mislevy and Haertel
2006) can be used to design formative assessments and link these to large-scale,
summative assessments.

The Role of Domain Knowledge

An issue for large-scale twenty-first century assessments is the role of knowledge

about topics and contexts in a discipline or specialization that is required to accom-
plish tasks and technology-based items. Large-scale assessments of twenty-first
century skills cannot assume that all students will have learned a particular
academic content. Fortunately, assessments of twenty-first century skills within
learning environments can identify the content knowledge within which they will
be situated. In academic subjects, current assessments of problem-solving and
critical-thinking skills, if they are directly assessed and reported at all, are typically
reported as components of subject-matter achievement (i.e., math problem solving,
science inquiry), not as distinct twenty-first century skills. In addition, in core school
subjects as well as informal settings, students may use common or advanced tech-
nologies, but their technology proficiencies tend not to be tested or reported.
Therefore, to assess and report progress on twenty-first century skills, the design of
assessments of students’ performance relevant to them must specify the knowledge
and skills to be tested and reported for each skill (see Chap. 2); either crosscutting
processes such as problem solving or communication, or their ability to use
5 New Assessments and Environments for Knowledge Building 267

technologies in a range of academic and practical problems. An important feature

of knowledge-building environments and the assessments of ICT skills within
them will be to test not only the use of ICT tools, simple and advanced, but also
the learners’ skill in using a range of ICT tools to extend and build their knowledge
and strategies for increasingly more complex tasks. In addition, learners’ adaptive
expertise, their ability to transfer their existing knowledge and strategies to novel
problems, will need to include direct assessment of their ability to learn and apply
new technologies.

Assessments Embedded in Technology-Rich Environments

The design of assessments must begin by specifying their purposes and intended
uses (AERA/APA/NCME 1999). These specifications then lead to validity questions
such as “Does the assessment support the inferences and actions based on it?” The
two conventional distinctions are between summative and formative purposes. As
indicated earlier, summative assessments are administered at the end of an interven-
tion, or a unit within it, so as to judge whether goals have been met. Formative
assessments are administered during interventions to inform learners and instruc-
tors, giving time for midcourse corrections. A recent definition proposed in the USA
by the Formative Assessment for Students and Teachers (FAST) state collaborative,
supported by the Council of Chief State School Officers, is that “Formative assess-
ment is a process used by teachers and students during instruction that provides
feedback to adjust ongoing teaching and learning to improve students’ achievement
of intended instructional outcomes.” According to the FAST definition, formative
assessment is not an instrument but the process of using information about progress
toward a goal to improve learning. Important attributes of formative assessments are
that the outcomes are intended and clearly specified in advance, the methods are
deliberately planned, the evidence of learning is used by teachers and students, and
adjustments occur during instruction. Attributes of effective FAST formative assess-
ment include: clearly articulated learning progressions; learning goals and criteria
for success that are clearly identified and communicated to students; evidence-based
descriptive feedback; self and peer assessment; and collaboration of students and
teachers in working toward learning goals. Formative assessments of twenty-first
century skills, therefore, would specify the twenty-first century outcomes and
systematic methods for monitoring progress and providing feedback, as well as
clear criteria for success. Formative assessments for twenty-first century skills
could be employed for all the twenty-first century skills in all kinds of learning
environments.
This FAST prescription of formative function of twenty-first century assessments
is quite different from the use of embedded assessments to validate large-scale
assessment results, or to augment the evidence that could be collected in a one-time,
on-demand test. A third function of embedded assessments can be to collect detailed
information about processes and progress for research purposes, and to begin to
create a more coherent integration of formative and summative assessment.
268 M. Scardamalia et al.

What Evidence Will Be Sought?

Within an evidence-centered design assessment framework, broad twenty-first

century skills, such as problem solving or communication, need to be further dis-
sected into component targets for assessment. Problem-solving targets in mathe-
matics might involve planning solution strategies or evaluating solutions. In
science, problem solving might involve targets such as planning investigations or
interpreting data in visualizations (Quellmalz and Kozma 2003). In literature,
problem solving may involve analyses of Shakespeare plays, looking for recur-
ring symbolism related to the plot. Problem-solving targets to assess in a practical
situation might involve selecting a green technology, such a wind turbine, and
analyzing its potential environmental impacts. The assessment targets for twenty-
first century problem-solving skills will be at a more general level for applications
across domains and situations. Problem-solving assessment tasks will need to rep-
resent structured problems with known solutions as well as problems with multi-
ple solutions. In domain-centered learning environments, assessment tasks will go
beyond the repetition of previously performed experiments to open-ended tasks
permitting numbers of appropriate methods for eliciting evidence of how well
learners plan, conduct, and interpret evidence in solving a problem or achieving
a goal.
Evidence-centered assessment design requires that embedded assessments artic-
ulate the qualitative or quantitative information that would document achievement
of each twenty-first century skill and its component targets. For formative assess-
ments, a crucial feature is that the evidence and criteria be understandable and useable
by teachers and students. For example, self and peer assessment are key features of
effective formative assessment. Such activities are already familiar in classes that
use peer review of drafts of compositions or peer critiques of presentations. In the
workplace, peer review is a hallmark of professional publications.
While common Internet and productivity tools are often integrated across
contexts and disciplines, the “tools of the trade” differ between humanities, sci-
ences, and social sciences, and so on, as well as between postsecondary learning
environments, the workplace, and the professions. In primary and secondary for-
mal schooling, common Internet and productivity tools are often integrated across
contexts and disciplines. Once again, the knowledge and skills will need to be
specified and further decomposed as they apply to different learning environments.
Evidence of achievement will also need to be specified in ways that are shareable
with learners and teachers. Thus embedded assessments of use of specific tech-
nologies will vary according to the context and domains emphasized. Nonetheless,
new assessment possibilities are opening up through efforts to create tools that are
useable across domains and that link domain-specific environments with more
general environments.
Twenty-first century skills are difficult to assess with timed, on-demand large-
scale tests, and typically better monitored over time within learning environments.
For example, creativity and innovation can be assessed in relation to how learners
have gone beyond what was specified in learning activities. Collaboration with
5 New Assessments and Environments for Knowledge Building 269

present and virtual peers and experts can be monitored throughout formation of teams,
integration of contributions, and feedback to reflect on the effectiveness of the team
processes and the achievement of goals.

Design of Assessments to Elicit Evidence of Twenty-First Century Skills

Systematic, direct assessment of twenty-first century skills in classrooms is rare.

Although students may be taught to use common and advanced tools, teachers tend
not to have specific standards for twenty-first century skills for students to meet nor
testing methods to gather evidence of student skill in using the technologies. In
either formal or informal learning environments, teachers are typically left on their
own to figure out how to integrate technology into their curricula or into informal
learning activities. The state of integration of the assessment of twenty-first century
skills into learning activities remains in its infancy.
Assessment must be designed to elicit evidence of learning related to each assess-
ment target. Research on effective formative assessment describes types of formal
and informal observations of learning, from questions to and from learners, to
examinations of work in progress, and evaluations of work products. However, these
observations should be planned for in advance with the criteria for success laid out
and shared with learners. For example, systematic observations of groups during
collaboration activities can be structured so as to record the types and quality of
interactions. These observations can be summarized and reviewed with groups and
individuals.
The twenty-first century skills integrate learners’ use of a range of technolo-
gies over the variety of contexts and domains in the learning environments. Central
to the twenty-first century skills is the learner’s ability to select and use appropri-
ate technologies during processes such as innovation, communication, collabora-
tion, problem solving, and citizenship. Technologies offer many possibilities for
designing richer, deeper, wider-ranging learning activities and assessments.
Possibilities for technology-supported reform of learning environments and
assessments include:
• Provision of authentic, rich, dynamic environments
• Access to collections of information sources and expertise
• Use of formal and informal forms of collaboration and social networking
• Presentation of phenomena, difficult or impossible to observe and manipulate in
classrooms
• Examples of temporal, causal, dynamic relationships “in action”
• Allowing multiple representations of stimuli and their simultaneous interactions
(e.g., data generated during a process)
• The use of overlays of representations, symbols
• Student manipulations/investigations, multiple trials
• Student control of pacing, replay, revision
• Making student thinking and reasoning processes visible
270 M. Scardamalia et al.

• Capturing student responses during activities (e.g., research, design, problem

solving)
• Allowing the use of simulations of a range of tools (Internet, productivity, domain
based)
Below, in the section on assessment and the knowledge-building developmental
trajectory, we extend this list. But first we introduce the notion of an assessment
profile and elaborate on the potential for new environments and assessments to
inform and be informed by large-scale assessments.

Assessment Profile

The purpose of the knowledge-building analytic framework, (see “Annex”) is to deter-

mine the extent to which an educational environment is moving toward a knowledge-
creating enterprise, in line with the developmental trajectories defined in Table 5.2. The
assumption underlying this framework is that educational environments, not only stu-
dents, should be evaluated. But of course the work of students must also be analyzed,
and for this purpose these dimensions need to be translated into measures of individual
and group performance. We propose such work as part of a necessary program of
research. But for now we offer six dimensions of assessment to support use and cover-
age of all manner of assessments to measure twenty-first century skills, across all class-
rooms, so as to ensure quality assessments and to guide instructional practices.
Alignment between assessments and twenty-first century skills. Some assessment
instruments may not assess or support one or more of the twenty-first century skills,
so it is helpful, for each target twenty-first century skill, to determine if there is
(1) full, (2) partial, or (3) no alignment.
Purpose and intended use of assessments. Assessment data, tasks, and items may
serve as (1) formative assessments, so students and instructors can monitor learning
and adjust instruction as it proceeds; (2) summative evidence of end-of-instruction
achievements; or (3) project evaluation or research, not shared with learners and
instructors. For each twenty-first century skill, it is worth tracking its purpose on
each of these purposes.
Construct representation. Assessment tasks and items can sometimes produce evi-
dence about only portions of the targeted constructs, desired knowledge or skills.
For example, if the target is systems knowledge, components or simple interactions
may be tested rather than dynamic, emergent behaviors. Or basic facts or steps may
be tested rather than higher-level, integrated knowledge and skills. When constructs
are only partially tested, important components may not be fully represented. For
each twenty-first century skill, it should be determined whether available evidence
represents (1) the construct, (2) part of the construct, or (3) none of the construct.
Integration into learning activities. Assessments in learning environments may
be integrated into ongoing activities to a greater or lesser extent. Integrated,
ongoing assessments can gather evidence of learning throughout their activities.
Interim assessments less directly linked to ongoing activities may be periodically
5 New Assessments and Environments for Knowledge Building 271

administered as checks. Or, decontextualized, external assessments can be dropped

in. Thus it is helpful, for each twenty-first century skill, to determine the extent to
which tasks and item responses (1) are fully integrated into learning activities; (2) are
assessed afterward, separately from learning activities; or (3) are not assessed.
Feasibility. Assessments in learning environments may also differ in the feasibility of
their use. They may be easily completed and interpreted by learners and instructors or
need access to technologies that may be permanently available, or only periodically.
Thus it needs to be determined whether the assessment is (1) easily used, with
minimal or no support; (2) possible to use, but requiring ongoing support; or (3)
complex, requiring specialized methods and support.
Technical quality. The assessments may require levels of expertise to administer
and score that are beyond the training of many instructors. Technical quality evidence
would include not only confirmation that the assessments provide credible informa-
tion for their intended uses in the environments (e.g., formative or summative), but
also that the interpretations of observations and evidence are reliable across instruc-
tors and environments. Thus it is important to clarify if technical quality is (1) fully
or (2) only partially established.

Connecting Learning Environments and Formative Assessments

to Large-Scale Tests

Currently, there are different, often competing, approaches to assessing twenty-first

century skills. One approach focuses on assessment of technology, such as the
International Computer Driving License and technology proficiency tests in some
states in the USA. These tests measure the facts and procedures needed to operate
common Internet and productivity tools, while the content or the academic or
applied problem and context are deliberately chosen to be familiar background
knowledge (Venezky and Davis 2002; Crawford and Toyama 2002). The cognitive
processes addressed in twenty-first century skills, such as problem solving, com-
munication, collaboration, innovation, and digital citizenship, are not targeted by
such tests of technology operations.
In a second approach, twenty-first century skills emphasize learning with
technology by presenting test problems and items that integrate measurement of
technology operations in terms of strategic use of technology tools to solve problems
with subject-matter knowledge and processes, by way of carefully designed sets of
tasks and items related to complex academic and real world problems.
In a third approach, the testing is implemented by technology. Assessments by
technology simply use technical infrastructures to deliver and score tests that are
designed to measure other content and skills, in subjects such as mathematics and
reading. These test designs aim to reduce or eliminate the demands of the technology,
treating it as a construct of no relevance. Equivalence of paper-based and technology-
based forms is the goal here. Technology-based tests are increasing rapidly in
large-scale state, national, and international testing, where technology is being
embraced as a way of reducing the costs and logistics of assessment functions, such
as test delivery, scoring, and reporting. Technology-based tests typically assume
272 M. Scardamalia et al.

that supporting technology tools such as calculators or word processors are irrelevant
to the content constructs being tested and, therefore, are not to be measured separately.
Since these types of testing programs seek comparability of paper and online tests,
the tests tend to present static stimuli and use traditional constructed-response and
selected-response item formats. For the most part, these conventional, online tests
remain limited to measuring knowledge and skills that can be easily assessed on
paper. Consequently, they do not take advantage of technologies that can measure
more complex knowledge structures and the extended inquiry and problem solving
included in the twenty-first century ICT skills described in the Assessment and
Teaching of twenty-first Century Skills project and reported in Chap. 2 (Csapó 2007;
Quellmalz and Pellegrino 2009). In short, a technology-delivered and scored test of
traditional subjects is not an assessment of twenty-first century ICT skills and should
not be taken as one. Twenty-first century skills assessments will not use technology
just to support assessment functions such as delivery and scoring, but will also focus
on measuring the application of twenty-first century skills while using technology.
Large-scale assessments of twenty-first century skills could provide models of
assessments to embed in learning environments, but current large-scale tests do not
address the range of twenty-first century skills in ways that would advance knowl-
edge-building environments. In the USA, the new 2012 Framework for Technological
Literacy for the National Assessment of Educational Progress sets out three major
assessment areas: technology and society, design and systems, and information
communication technologies (see naeptech2012.org). Technological literacy in the
framework blends understanding of the effects of technology on society, twenty-
first century skills, and technology design. The 2012 assessment will present a range
of long and short scenario-based tasks designed to assess knowledge and skills in
the three areas. In the USA, assessments of twenty-first century skills and techno-
logical literacy are required for all students by grade 8. However, state tests or
school reports are considered sufficient to meet this requirement, and school reports
may be based on teacher reports that, in turn, can be based on questionnaires or
rubrics that students use in ICT-supported projects. Most teachers do not have access
to classroom assessments of twenty-first century skills, or professional development
opportunities to construct their own tests. Moreover, the lack of technical quality of
teacher-made and commercially developed classroom assessments is well docu-
mented (Wilson and Sloane 2000). Even more of a problem is the lack of clarity for
teachers on how to monitor student progress on the development of twenty-first
century skills, not only the use of the tools, but ways to think and reason with them.
Teachers need formative assessment tools for these purposes.

Concurrent, Embedded and Transformative Assessment

of Knowledge Building

In line with the emergence approach as well as the knowledge-creation imperative

to continually go beyond what is currently viewed as best practice, we describe new
forms of data from classroom environments that make it possible to provide richer,
5 New Assessments and Environments for Knowledge Building 273

more comprehensive, and more readily available accounts of student performance

than are possible through traditional testing. They require new, powerful knowledge-
building environments of the sort discussed above.
In the preceding sections we have discussed embedded, formative, and summative
assessment; now we add the concepts of concurrent and transformative assessment.
Concurrent assessment means that the assessment is available instantaneously. The
challenge is effective design of feedback that informs high-level processes as well
as more straightforward procedures. Transformative means that the evaluation is not
simply an account of past performance, pointing to the next immediate steps, but
also provides indication of ways individuals and teams can tackle broader problems
and situate their work in relation to that of other team members and teams, within
and outside the school walls.
When student discourse is central to the operation of the community, with
members contributing to shared, public knowledge spaces, and building on each
other’s ideas, new forms of assessment make it possible to enrich the community’s
work and enable concurrent and transformative assessment. The discourse to be
analyzed may include online as well as face-to-face interactions, recorded through
video or conferencing software and transcribed. Profiles of student work can be
generated easily from such data. Even at this early stage, there is a great deal of
excitement among the researchers, teachers, and students who have pilot-tested
these tools in their classrooms. Teachers and students alike readily see their advan-
tage and generate ideas for improving them.
Data are generated automatically from student discourse and artifacts, and as
suggested below, the tools can be used to identify patterns and support continual
improvement in practice and student achievement. A substantial part of the chal-
lenge in advancing concurrent, embedded, and transformative assessment will be
avoiding pitfalls while taking advantage of substantial new opportunities.
Contributions. A contribution tool can provide measures of the number of notes
created, the nature of entries (based on keywords, media type, etc.), an overview
of the content areas participants worked in, and so forth. Contributions related to
a specific problem can be traced, thus making it possible to start investigating
individual and group problem solving. The teacher can use the tool during each
session or immediately afterward to determine how productive each student has
been (e.g., how many notes were read, created, or modified). Such information
helps the teacher to direct attention to students who may need more support or
instruction, and helps them identify barriers that are preventing students from
participating fully in the knowledge-building community. Students can use the
tools, if the teacher enables their access, to see where they are in the class distribu-
tion (no names are shown).
“Thinking Types” or scaffolds to support twenty-first century skills. Scaffolds can be
built on the basis of theory-driven accounts of advanced knowledge processes (see
the section on technology to support emergence of new skills). Computer-mediated
and customizable scaffold supports 1604 (e.g., “my problem solution,” “my theory,”)
allow teachers and students to use scaffolds and rubrics flexibly and for students to tag
their notes according to thinking type (Andrade 2000; Chuy et al. 2009; Law and
274 M. Scardamalia et al.

Wong 2003; Lai and Law 2006). By identifying the twenty-first century skill they are
engaged in (problem solving, theory development, research, decision making, etc.),
students become more cognizant of these skills. And once text is tagged, searching by
scaffolds makes it easy for students and teachers to find, discuss, and evaluate exam-
ples. Formative assessment tools can be used to provide feedback on patterns of use
and to help extend students’ repertoires.
Use of new media and multiliteracies. Students can contribute notes representing
different modalities and media, such as text, images, data tables, graphs, models,
video, audio, and so forth. Results suggest that growth in textual and graphical
literacy is an important by-product of work in media-rich knowledge-building envi-
ronments (Sun et al. 2008; Gan et al. 2007).
Vocabulary. A vocabulary tool can provide profiles for individuals and groups,
including the rate of new word use, use of selected words from curriculum guide-
lines (or from any set of words), and so on. It is also easy to look at the growth of
vocabulary in comparison to external measures or benchmarks, such as grade-level
lists. Thus teachers can determine if important concepts are entering the students’
productive vocabularies, the extent of their use of words at or above grade level,
their growth in vocabulary based on terms at different levels in the curriculum
guidelines, and so on. Information about the complexity and quality of notes can
also give the teacher direction as to the type of instruction the class may need. Early,
informal use of these vocabulary tools suggests that students enjoy seeing the growth
in their vocabulary, and begin to experiment with new words that have been used by
others in the class.
Writing. Measures of writing start with basic indicators (e.g., total and unique
words, mean sentence length). There are many sophisticated tools already developed,
and open-source arrangements will make it increasingly easy to link discourse and
writing environments.
Meta-perspectives. A brainstorming tool (Nunes et al. 2003) can be used to foster
students’ metacognitive thinking about specific skills and support students in the
exercise of creativity, leadership, and collaboration. Tools can also be built to allow
students to tag notes containing questions asked but not answered, claims made
with no evidence, etc. Once tagged, visualization tools can bring to the forefront of
the knowledge space ideas needing extra work.
Semantic analysis. This tool makes it possible to work in many and flexible ways
with the meaning of the discourse. A semantic-overlap facility extracts key words or
phrases from user-selected texts and shows overlapping terms. One application of this
tool is to examine overlapping terms between a participant’s discourse and discourse
generated by experts or in curriculum guidelines. Other applications include exami-
nation of overlapping terms between texts of two participants or between a student
text and an assigned reading. A semantic field visualization provides graphical dis-
plays of the overlapping terms by employing techniques from latent semantic analy-
sis (Teplovs 2008). For example, a benchmark can be identified (an encyclopedia
5 New Assessments and Environments for Knowledge Building 275

Day 1 Day 3 Day 10

Semantic Space of Benchmark Semantic Space of Benchmark Semantic Space of Benchmark

Semantic Semantic Semantic

Space of Student Discourse Space of Student Discourse Space of Student Discourse

Fig. 5.4 Semantic field visualization of a classroom over 10 days (Adapted from Teplovs 2008)

entry, a curriculum guideline or standard, etc.). The tool can show the overlap between
the students’ discourse and the benchmark over successive days, as the visualization
in Fig. 5.4 suggests.
Social network analysis. Social network analysis tools display the social rela-
tionships among participants based on patterns of behavior (e.g., who read/refer-
enced/built on whose note). A social network analysis tool can help teachers to
better understand who the central participants are in the knowledge-building dis-
course and to see whether existing social relationships are limiting the community’s
work or influencing it positively. The tool draws the teacher’s attention to children
who are on the periphery and makes it more likely that these children will receive
the support they may need to be more integral to the work of the class.
Increasing levels of responsibility for advancing collective knowledge is facili-
tated when student contributions to classroom work are represented in a communal
knowledge space. Below are graphics generated from the social network analysis
tool to give some sense of how it is possible to uncover classroom practices associ-
ated with advances in student performance—practices that would be impossible to
uncover without use of communal discourse spaces. The work reported in Fig. 5.5
(Zhang et al. 2007, 2009) is from a grade 4 classroom studying optics. The teacher
and students worked together to create classroom practices conducive to sustained
knowledge building. Social network analysis and independently generated qualita-
tive analyses were used to assess online participatory patterns and knowledge
advances, focusing on indicators of collective cognitive responsibility.
The social network graphs generated by the social network analysis tool indicate
increasingly effective procedures for advancing student knowledge corresponding to
the following social organizations: (a) year 1—fixed, small-groups; (b) year 2—inter-
active small groups working together throughout their knowledge work; and (c) year
3—opportunistic collaboration, with small teams forming and disbanding under the
276 M. Scardamalia et al.

Fig. 5.5 The emergent process of knowledge building over 3 years (This 3-year account, from the
perspective of the social network analysis tool, is described in detail in Zhang et al. 2009)
5 New Assessments and Environments for Knowledge Building 277

volition of community members, based on emergent goals that arose as they

addressed their shared, top-level goal of refining their knowledge of optics. The
third-year model maps most directly onto the organic and distributed social structure
in real-world knowledge-creating organizations. Among the three designs, the
opportunistic-collaboration model resulted in the highest level of collective cogni-
tive responsibility, knowledge advances, and dynamic diffusion of information. This
3-year account, as shown from the perspective of the social network analysis tool, is
shown in the following figure – Fig. 5.5 (see Zhang et al. 2009 for details).
In these graphs a node represents a group member. A line between two nodes
denotes a note linking relation between two members, indicating that one member
has built on or referred to a note by the other. The direction and frequency of such
connections are represented by the arrow and value on the line. The more informa-
tion flow a member carries, the more centrally he/she is displayed in a network.
Tools such as those presented above allow teachers and students to visualize their
work in new ways. They can be applied to discourse on any topic, at the group as
well as individual level. There are endless possibilities for reconstructing knowl-
edge spaces to bring different issues and concerns into perspective and to show
change over time. This work is in its infancy and Web 2.0/3.0 developments will
greatly enhance it.

Assessment, Open Knowledge Resources, and Development

of Knowledge Building

The need for developmental frameworks, definitions, and models can be seen
throughout the Assessment and Teaching of Twenty-First Century Skills project.
This is evident in the discussion of frameworks (Chap. 2), the argument for the need
to identify learning progressions to describe pathways that learners are likely to follow
toward the mastery of a domain (Chap. 3), and the discussion of item development
(Chap. 4). We hope to contribute to these efforts through identifying developmental
progressions grounded in the theory and practices of knowledge-creating organiza-
tions. We argue that all citizens should have the opportunity to participate in
knowledge-building environments that fully integrate twenty-first century skills and
move them along the developmental trajectories set out earlier in Table 5.2. The tools
we describe above can help accomplish this by charting progress and addressing
design principles in new ways.
Design principles for knowledge-building environments include: (a) empowering
users and transferring greater levels of agency and collective responsibility to them;
(b) viewing assessment as integral to efforts to advance knowledge and identify
problems as work proceeds; (c) enabling users to customize tools and request changes
so that the environments are powerful enough to be embedded in the day-to-day
workings of the organization; (d) supporting the community in self-directed rigor-
ous assessment so that there is opportunity for the community’s work to exceed,
rather than simply meet expectations of external assessors; (e) incorporating
278 M. Scardamalia et al.

standards and benchmarks into the process so that they are entered into the public
workspace in digitized form and become objects of discourse that can be annotated,
built on, linked to ongoing work, and risen above; (f) supporting inclusive design, so
there is a way in for all participants; this challenge brings with it special technologi-
cal challenges (Trevinarus 1994, 2002); (g) providing a public design space to sup-
port discourse around all media (graphics, video, audio, text, etc.) with links to all
knowledge-rich and domain-specific learning environments; and (h) encouraging
openness in knowledge work. Once these requirements are met, participants are
engaged with ICT in meaningful, interactive contexts, with reading and writing part
of their expressive work across all areas of the school curriculum. They can then
make extensive use of the forms of support that prove so helpful in knowledge-cre-
ating organizations—connections with other committed knowledge workers and
world-class knowledge resources.
Combining ICT-enabled discourse environments and open resources sets the
stage for breakthroughs in charting and enhancing development in knowledge-
building environments. For example, student discourse environments can be linked
to powerful simulation, tutorial, intelligent tutoring, and other domain-specific tools
(Quellmalz and Haertel 2008; Tucker (2009); http://www.ascd.org/publications/
educational_leadership/nov09/vol67/num03/The_Next_Generation_of_Testing.
aspx; http://oli.web.cmu.edu/openlearning/initiative). It is then possible to combine
the benefits of these different tools and promote interactions surrounding their use.
As explained in The Open Learning Initiative, Carnegie Mellon University, it is possi-
ble to build assessment “into every instructional activity and use the data from those
embedded assessments to drive powerful feedback loops for continuous evaluation
and improvement.” Assessments from these tutorials, simulations, games, etc., can
complement those described in the section on open-source software and program-
ming interfaces and, combined with interoperability of applications, allow us to
further break down the barriers between various environments and assessments that
have traditionally been separate and disconnected, so as to search and compile
information across them. Open resources make it possible to assemble information
on learning progressions, benchmarks, and learning modules. Curriki is an example
of a web site where the community shares and collaborates on free and open-source
curricula (http://www.curriki.org/). Creative Commons licenses further expand
access to information to be shared and built upon, bringing an expanded concept of
intellectual property.
These open resources, combined with data from discourse environments, make
it possible to build student portfolios, based on classroom work and all the web-
accessible information created from in- or out-of-school uses of simulations,
games, etc., across topics and applications (dealing with ethical issues presents a
different, significant challenge). Extended student portfolios will allow us to chart
student progress in relation to various and changing developmental benchmarks,
as well as to foster development through formative feedback. For example, “near-
est neighbor” searches, based on student semantic spaces, can identify other peo-
ple, in the same class or globally, as well as local or global resources, working
with similar content. Connections can then be made, just in time, any time, to meet
5 New Assessments and Environments for Knowledge Building 279

both teacher and student needs. This support can help the class as a whole to oper-
ate as a twenty-first century organization, as well as supporting individual student
achievement.
We envision worldwide teams of users (Katz et al. 2009) and developers taking
advantage of new data-mining possibilities, intelligent web applications, semantic
analysis, machine learning, natural language processing, and other new developments
to advance the state of the art in education.

Technology to Support Emergence of New Competencies

Two recent books discuss in depth the effects that new technologies can have in
shifting education on to a new basis for the twenty-first century. One is Rethinking
Education in the Age of Technology: The Digital Revolution and Schooling in
America (Collins and Halverson 2009). Collins and Halverson argue that new tech-
nologies create learning opportunities that challenge traditional schools. They envi-
sion a future in which technology enables people of all ages to pursue learning on
their own terms. Figure 5.3 above indicates that more time by far is spent in out-of-
school contexts, across the entire lifespan. If these become primary contexts for
learning, tasks designed especially for school will pale by comparison in their
impact on education. The second book is The World Is Open: How Web Technology
Is Revolutionizing Education (Bonk 2009). Bonk explains ways in which technolo-
gies have opened up the education world to anyone, anywhere. He discusses trends
such as web searching, open courseware, real-time mobility, portals, and so forth
that will impact learning in the twenty-first century. These technologies are not
envisaged as a cafeteria line for students to proceed along and pick and choose
(which, unfortunately, seems to have been the formative concept in many instruc-
tional support systems); instead, they are envisaged as constituting an environment
supportive of a more fully engaged community of learners, more open to the world’s
cognitive and emotional riches.
These ideas are in line with our earlier discussions of the emergence of new
competencies and open resources. Rather than simply extrapolating from existing
goals or expert-identified objectives, new goals can emerge from the capacities that
students demonstrate in supportive environments—such as the capacities for pro-
portional reasoning and theory building revealed in the examples cited. Both these
experimental approaches have, in fact, made use of computer-supported knowledge-
building environments that provide support for the creation of public knowledge
(Moss and Beatty 2006; Messina and Reeve 2006). Among the technical affordances
serving this purpose are “thinking types” or scaffolds, described above, “rise-above”
notes that serve the purposes of synthesis and the creation of higher-order represen-
tations of ideas, and graphical backgrounds for creating multiple representations
and organizing ideas (Scardamalia and Bereiter 2006).
In the theory-building work elaborated above, scaffolds supported theory build-
ing. The “theory supports” included the following phrases: “My theory,” “I need
280 M. Scardamalia et al.

to understand,” “Evidence for my theory,” “Putting our knowledge together,”

“A better theory.” To use these scaffolds, students simply need to click on one of
these phrases, arrayed on a panel to the left of their writing space, and a text field
containing the phrase is copied into their text at the appropriate point. Text added
by the student is automatically tagged according to the scaffold name. This simple
support has increased the use of these phrases in student writing and, results sug-
gest, has enhanced the high-level knowledge processes they represent. In the
Knowledge Forum environment, used in the theory-building example, scaffolds are
customizable, so these discourse supports can easily be changed to fit any twenty-
first century goal. (They can also be used after the fact, to mark up text already
written.) These scaffolds foster metacognitive awareness, as students use them to
characterize their discourse. The scaffold supports also serve as search parameters,
further encouraging their use and allowing students and teachers easily to search
their communal knowledge space so as to determine what different theories there
are in the database, what evidence is used to defend them, the nature of theories
that are considered to be improvements on earlier theories, and so forth. And it is
quite easy with these tagged “thinking types” to build formative assessments to
enhance student development. For example, it is possible to create profiles of stu-
dent or group activity, to find whether students and the class are generating lots of
theories but providing no evidence—or perhaps they are providing evidence but
cannot put their ideas together to generate an improved theory. Patterns of use
make it possible to detect underrepresented knowledge processes and to inform and
advance such work.
An important role for technology is to support individuals in constructive contribu-
tions to the group. The scaffolds help. At the group level the essential question is: Has
the public knowledge shared by a group progressed—to what extent has this knowl-
edge emerged from a group process as opposed to being merely an aggregation of
individual products? Web 3.0 “semantic web” developments treat ideas or meanings
rather than simply words as the units of primary interest. Some educational evaluation
tools have already taken advantage of these advances (Teplovs 2008) and we can look
forward to further developments that align more powerful web technology with edu-
cational needs for working in a knowledge-creating culture. We elaborate on these
ideas in the section on technological and methodological advances to support the
development of twenty-first century skills.
Although findings from the emergence approach are limited, they suggest that
students demonstrate advances across a broad range of twenty-first century skills
(Chuy et al. 2009; Gan et al. 2007; Sun et al. 2008, 2010), and that an emergence
approach may contribute genuinely new discoveries to inform large-scale assess-
ment. Positive results of an emergence approach also suggest that defining and
operationalizing twenty-first century skills one-by-one, while important for mea-
surement purposes, may not be the best basis for designing educational activity.
As technology blurs the line between in- and out-of-school contexts, and knowl-
edge becomes a social product situated in open worlds, the need for environments
and formative assessment that span educational contexts and support “community
knowledge” and group or “collective intelligence” will become increasingly
important.
5 New Assessments and Environments for Knowledge Building 281

Necessary Research

This section identifies important areas of research and development related to the
overall goal of developing new assessments and environments for twenty-first
century knowledge building. We start with research and development to improve
formative assessments in current learning environments and then move on to studies
and advances in formative assessment likely to transform schools into the image of
knowledge-creating organizations.

Analysis of Twenty-First Century Skills in Current

Learning Environments

A research program on reforming the assessment of twenty-first skills would

benefit from greater understanding of twenty-first century skills as represented
in current learning environments. Projects could be selected to represent various
learning environments, and assessments would focus on twenty-first century
skills frameworks and developmental trajectories. We anticipate that all of the
learning environments will show limits in the extent to which they address
twenty-first century skills, and this analysis could provide important informa-
tion for evidence-centered initiatives to promote these skills.
The second phase of the study would analyze the technical quality of the projects’
assessments and their utility for providing formative evidence during instruction.
Using the evidence-centered design framework, we anticipate that there will be
weak links between assessments of twenty-first century skills, learning tasks used to
elicit those skills, and the evidence that teachers and students can use to understand
development of the skills.
A third phase of the study would involve the creation of evidence-centered
classroom assessment systems with representative projects to address all or many
of the twenty-first century skills. Technical quality data would be collected about
their reliability and validity for classroom formative purposes. In addition, the
designs of the formative twenty-first century assessments would be linked to the
more compressed, constrained designs of the large-scale, summative twenty-first
century assessment tasks being designed by all ATC21S working groups.
Classroom formative assessments would be embedded in the learning activities,
provide evidence of ongoing learning processes related to twenty-first century
skills, such as problem solving, collaboration, and communication, and would
provide rich, deep, frequent streams of evidence to be used by learners and instruc-
tors during their learning activities to monitor and support their progress. For
example, in domain-centered learning environments, such rich, embedded forma-
tive assessment would be made possible by digital capture of student processes
during domain-specific learning activities such as information research, use of
simulations, and network analyses. The study would examine the formative utility
and technical quality of the assessments and the value they had added to interim
282 M. Scardamalia et al.

benchmark summative assessments and to even more distal large-scale state,

national, and international assessments. The research on the design of quality for-
mative assessments for the full range of twenty-first century skills that could be
embedded in projects in each of the different learning environments would serve
as models for reforming and transforming twenty-first century formative assess-
ments in learning environments.

Social and Technological Innovations for An Inclusive

Knowledge-Building Society

The goals currently being promoted for twenty-first century skill development are,
as previously noted, based mainly on expert and stakeholder analysis of goals. In
this section we propose design experiments that complement this top-down
approach to goal identification with a bottom-up approach based on the capacities,
limitations, and problems that learners reveal when they are actually engaged in
knowledge-creating work. The first step in mounting such research is to identify or
establish schools able to operate as knowledge-creating organizations—given, as
Laferrière and Gervais (2008) suggest—that at this point it may be difficult to locate
schools able to take on such work. The proposed research has the dual purpose of (a)
discovering previously unrecognized skill goals and (b) developing ways of assess-
ing these emergent skills through minimally intrusive instruments.
Sites thus engaged, willing to take on an ambitious new research agenda, and
equipped with appropriate technology, could then support a broad-based research
and development effort aimed at addressing questions related to knowledge prac-
tices and outcomes. At a policy level we would begin to collect data and evidence
to address issues that are dividing educators. For example, many educators favor
those curriculum procedures and processes that are well defined and have a step-
by-step character—but knowledge creation is not an orderly step-by-step process.
Knowledge creators go where their ideas take them. How can the challenge of
engaging students in more self-directed and creative work with ideas be reconciled
with the classroom routines and activity structures that many educators feel to be
essential for teachers, students, and curriculum coverage? How does self-organization,
an important component of knowledge creation, actually combine with intentional
development of ideas at the process level? How are promising ideas worthy of
further development sorted out from the large pool of ideas students often gener-
ate? How can “pooling of ignorance” be avoided?
“Pooling of ignorance” is a problem that looms large in discussions about open
discourse environments for naïve learners. Although “making thinking visible” is
one of the advantages claimed for constructivist computer environments, it can
increase the chances of “pooling ignorance” and spreading “wrong” ideas. Teachers,
accordingly, are tempted to exert editorial control over what ideas get made public
in student inquiry; and students, for their part, may learn that it’s better to put for-
ward authoritative ideas, rather than their own. Research is needed, first to deter-
mine whether “pooling ignorance” is a real or only an imagined problem, and
5 New Assessments and Environments for Knowledge Building 283

second—if it does prove to be real—to carry out design research to find a construc-
tive way to deal with this dilemma.
Concurrent, embedded, and transformative assessments need to be geared to
demonstrations of new ways around old problems. We can then collectively test the
notion that formative assessments, built into the dynamics of the community, will
allow for a level of self-correction and a focus on high-level goals unparalleled in
most educational contexts.

Challenges Related to Complex Interventions

Brown (1992), Collins et al. (2004) and Frederiksen and Collins (1989) discuss
theoretical and methodological challenges in creating complex interventions and
the problems of narrow measures. They stress the need for design experiments as a
way to carry out formative research for testing and refining educational designs
based on theoretical principles derived from prior research. It is an approach of
“progressive refinement.” As Collins et al. (2004) explain, design experimentation
involves putting a first version of a design into the world to see how it works. Then, the
design is constantly revised based on experience… Because design experiments are set in
learning environments, there are many variables that cannot be controlled. Instead, design
researchers try to optimize as much of the design as possible and to observe carefully how
the different elements are working out. (p.18)

Chapter 3 raises a number of methodological issues regarding assessment of

twenty-first century skills. The proposed research could contribute to progress on
each of the issues raised there: (a) Distinguishing the role of context from that of the
underlying cognitive construct—the experiment would allow us to find examples of
the construct across different national and domain contexts; (b) new types of items that
are enabled by computers and networks—the network we propose would implement
new designs and explore uses of new item types; (c) new technologies and new ways
of thinking to gain more information from the classroom without overwhelming the
classroom with more assessments—we propose to engage a network of international,
multilingual, cross-domain centers to explore issues and determine how concurrent,
embedded, and transformative assessments might begin to save teachers’ time; (d)
right mix of crowd wisdom and traditional validity—”crowd wisdom” and traditional
procedures can easily be combined in the environments we propose; (e) information
and data availability and usefulness—we can directly explore what it takes to trans-
late data into feedback to drive knowledge advancement; and (f) assessments for
twenty-first century skills that are activators of students’ own learning—through the
use of scaffolds, adaptive recommender systems, stealth assessments, visualizations,
and so on, we can explore assessments that facilitate students’ own learning.

Specific Investigations Within the Emergent Competencies Framework

We propose that an international network of pilot sites be established, both to coop-

erate in the multifaceted design research described below and to collaborate with
284 M. Scardamalia et al.

local researchers in creating and testing new designs tailored to their own conditions
and needs. A given site may collaborate in all or a subset of the specific investiga-
tions, but in any event the data they produce will be available for addressing the full
range of research questions that arise within the network. The following, therefore,
should be regarded as an initial specification, subject to modification and expansion.
Charting developmental pathways with respect to twenty-first century skills. As indi-
cated in the sections on embedded assessment and technology to support the emer-
gence of new skills, computer-based scaffolds can be used to support the development
of twenty-first century skills and formative assessments related to their use. An inten-
sive program of research to develop each skill would allow us to determine what stu-
dents at various ages are able and not able to do related to various twenty-first century
skills, with and without supports for knowledge creation. We would then be in a better
position to elaborate the developmental progressions set out in Table 5.2.
Demonstrating that knowledge-building pedagogy saves educational time rather
than adding additional, separate skills to an already crowded curriculum. Currently,
learning basic skills and creating new knowledge are thought by many to be com-
petitors for school time. In knowledge-building environments, students are reading,
writing, producing varied media forms, and using mathematics to solve problems—
not as isolated curriculum goals but through meaningful interactions aimed at
advancing their understanding in all areas of the curriculum. Rather than treating
literacy as a prerequisite for knowledge work, it becomes possible to treat knowl-
edge work as the preferred medium for developing multiliteracies. Early results
indicate that there are gains in subject-matter learning, multiliteracies, and a broad
range of twenty-first century skills. These results need to be replicated and
extended.
Testing new technologies, methods, and generalization effects. The international
network of pilot sites would serve as a test bed for new tools and formative assess-
ments. In line with replication studies, research reported by Williams (2009) sug-
gests that effective collaboration accelerates attainments in other areas. This
“generalization effect” fits with our claim that, although defining and operational-
izing twenty-first century skills one-by-one may be important for measurement pur-
poses, educational activities will be better shaped by a more global conception of
collaborative work with complex goals. Accordingly, we propose to study relation-
ships between work in targeted areas and then expand into areas not targeted. For
instance, we may develop measures of collaborative problem solving, our target
skill, and then examine its relationship with collaborative learning, communication,
and other twenty-first century skills. We would at the same time measure outcomes
on an appropriate achievement variable relevant to the subject matter of the target
skill. Thus we would test generalization effects related to the overall goal of educat-
ing students for a knowledge-creating culture.
Creating inclusive designs for knowledge building. It is important to find ways for
all students to contribute to the community knowledge space, and to chart advances
for each individual as well as for the group as a whole. Students can enter into the
discourse through their favorite medium (text, graphics, video, audio notes) and
5 New Assessments and Environments for Knowledge Building 285

perspective, which should help. Results show advances for both boys and girls,
rather than the traditional finding in which girls outperform boys in literacy skills.
This suggests that boys lag in traditional literacy programs because they are not
rewarding or engaging, whereas progressive inquiry both rewards and engages. New
designs to support students with disabilities will be an essential addition to environ-
ments to support inclusive knowledge building
Exploring multilingual, multiliteracy, multicultural issues. Our proposed research
would engage international teams; thus it would be possible to explore the use
of multilingual spaces and possibilities for creating multicultural environments.
More generally, the proposed research would make it possible to explore issues of a
knowledge-building society that can only be addressed through a global enterprise.
Administering common tests and questionnaires. While there is currently evidence
that high-level knowledge work of the sort identified in Table 5.1 for knowledge-
creating organizations can be integrated with schooling, starting no later than the
middle elementary grades (Zhang et al. 2009), data are needed to support the claim
that knowledge building is feasible across a broad range of ages, SES contexts,
teachers, and so forth, and that students are more motivated in knowledge-building
environments than in traditional environments. To maximize knowledge gains from
separate experiments, it will be important to standardize on assessment tools, instru-
ments, and data formats. Through directed assessment efforts, it will be possible to
identify parameters and practices that enable knowledge building (Law et al. 2002).
Identifying practices that can be incorporated into classrooms consistent with those
in knowledge-creating organizations. By embedding practices from knowledge-
creating organizations into classrooms, we can begin to determine what is required
to enable schools to operate as knowledge-creating organizations and to design pro-
fessional development to foster such practices. Data on classroom processes should
also allow us to refine the developmental trajectory set out in Table 5.2, and build
assessments for charting advances at the individual, group, and environment levels.
Demonstrating how a broader systems perspective might inform large-scale, on-
demand, summative assessment. We have discussed the distinction between a “work-
ing-backward” and “emergence” approach to advance twenty-first century skills and
connections between knowledge-building environments, formative assessments, and
large-scale assessment. Within the emergence approach, connections between stu-
dent work and formative and summative assessment can be enriched in important
ways. For example, as described above, scaffolds can be built into the environments
to encourage students to tag “thinking types.” As a result, thinking is made explicit
and analytic tools can then be used to assess patterns and help to inform next steps.
With students more knowledgeably and intentionally connected to the achievement
of the outcomes to be assessed, they can become more active players in the process.
In addition to intentionally working to increase their understanding relative to various
learning progressions and benchmarks, they are positioned to comment on these and
exceed them. As in knowledge-creating organizations, participants are aware of the
standards to be exceeded. As an example, toward the end of student work in a unit of
study, a teacher, published relevant curriculum standards in the students’ electronic
286 M. Scardamalia et al.

workspaces so they could comment on these standards and on how their work stood
up in light of them. The students noted many ways in which their work addressed the
standards, and also important advances they had made that were not represented in
the standards. We daresay that productive dialogues between those tested and those
designing tests could prove valuable to both parties. Semantic analysis tools open up
additional possibilities for an emergence framework to inform large-scale assess-
ments. It is possible to create the “benchmark corpus” (the semantic field from any
desired compilation of curriculum or assessment material), the “student corpus” (the
semantic field from any desired compilation of student-generated texts such as the
first third of their entries in a domain versus the last third), and the “class corpus”
(the semantic field from all members of the class, first third versus last third), and so
forth. Semantic analysis and other data-mining techniques can then be used to track
and inform progress, with indication of semantic spaces underrepresented in either
the student or benchmark corpus, and changes over time.
Classroom discourse, captured in the form of extensive e-portfolios, can be used
to predict performance on large-scale summative assessments and then, through for-
mative feedback, increase student performance. Thus results can be tied back to per-
formance evaluations and support continual improvement. Teachers, students, and
parents all benefit, as they can easily and quickly monitor growth to inform progress.
This opens the possibility for unprecedented levels of accountability and progress.

Technological and Methodological Advances

to Support Skills Development

Technological advances, especially those associated with Web 2.0 and Web 3.0
developments, provide many new opportunities for interoperability of environments
for developing domain knowledge and supporting student discourse in those
domains. Through coherent media-rich online environments, it is possible to bring
ideas to the center and support concurrent, embedded, and transformative assessment.
As indicated above, it is now possible to build a broad range of formative assess-
ments that will enrich classroom work greatly.
A key characteristic of Web 2.0 is that users are no longer merely consumers of
information but rather active creators of information that is widely accessible by oth-
ers. The concomitant emergence of online communities, such as MySpace, LinkedIn,
Flickr, and Facebook, has led, ironically and yet unsurprisingly, to a focus on individu-
als and their roles in these communities as reflected, for example, in the practice of
counting “friends” to determine connectedness. There has been considerable interest
in characterizing the nature of social networks, with social network analysis employed
to detect patterns of social interactions in large communities. Web 3.0 designs repre-
sent a significant shift to encoding semantic information in ways that make it possible
for computers to deduce relationships among pieces of information. In a Web 3.0
world the relationships and dynamics among ideas are at least as important as those
among users. As a way of understanding such relationships, we can develop an ana-
logue of social network analysis—idea network analysis. This is especially important
5 New Assessments and Environments for Knowledge Building 287

for knowledge-building environments where the concern is social interactions that

enable idea improvement (see Teplovs 2008). Idea network analysis offers a means of
describing relationships among ideas, much as social network analysis describes the
relationships among actors. Visualizations of idea networks, with related metrics such
as network density, will allow us to characterize changes in social patterns and ideas
over time. The demanding conceptual and research challenge, therefore, is to under-
stand and support the social dynamics that lead to knowledge advancement.
Through additional design work, aimed at integrating discourse environments,
online knowledge resources, and formative and summative assessments, we can
greatly extend where and how learning might occur and be assessed. By tracking the
semantics of participant discourses, online curriculum material, test items, texts of
experts in the field, and so on, we can map one discourse or corpus onto another and
track the growth of ideas. With collaborative online discourse integral to the opera-
tion of knowledge-building communities, we can further enhance formative assess-
ments so as to encourage participants to seek new learning opportunities and a
broader range of experts.
Effectively designed environments should make it possible to develop communication,
collaboration (teamwork), information literacy, critical thinking, ICT literacy,
and so forth in parallel—a reflection of how things work in knowledge-creating
organizations.

Annex: Knowledge-Building Analytic Framework

Template for Analyzing Environments and Assessments

1. DESCRIBE AN ENVIRONMENT AND/OR ASSESSMENT AS IT CURRENTLY EXISTS.
(Use as much space as you need)
2. INDICATE WHETHER THE EXAMPLE FITS PRIMARILY INTO AN ADDITIVE OR
TRANSFORMATIVE MODEL OF SCHOOL REFORM. TO PROVIDE THIS EVALUATION,
YOU SIMPLY NEED TO ASSIGN A SCORE FROM 1 (definitely additive) to 10 (definitely
transformative), AND PROVIDE A BRIEF RATIONALE. NOTE: Score = 1 (the goal is additive
if the environment, or assessment presented is designed to add a task or activity to school work
that remains little changed in overall structure, other than through the addition of this new task,
project, environment or assessment); Score = 10 (the goal is transformative if the environments
or assessment alters conditions of schooling in a substantial way, so students become encultur-
ated into a knowledge-creating organization that is supported by a knowledge-building
environment integral to the operation of the community).

SCORE _______

RATIONALE FOR SCORE: (Use as much space as you need)

3. PLEASE USE THE FOLLOWING EVALUATION FORM TO ASSESS THE
CHARACTERISTICS OF THE ENVIRONMENT AND/OR ASSESSMENT IN ITS
CURRENT FORM
Twenty-first century Characteristics of knowledge-creating organizations: a continuum that
skill (from Chap. 2) maps onto twenty-first century skills
1 5 10
288 M. Scardamalia et al.

Creativity and SCORE FROM 1 (internalize given information; beliefs/actions based

innovation on the assumption that someone else has the answer or knows the truth)
to 10 (work on unsolved problems; generate theories and models, take
risks, etc; pursue promising ideas and plans)

SCORE_______

RATIONALE FOR YOUR SCORE:

(Use as much space as you need)
DO YOU SEE A WAY TO IMPROVE YOUR ENVIRONMENT OR
ASSESSMENT ALONG THIS DIMENSION? IF SO, PLEASE
PROVIDE A BRIEF ACCOUNT OF HOW YOU MIGHT DO THAT,
OR HOW THE IDEAS IN THIS WORKING PAPER MIGHT HELP.
(Use as much space as you need)
Communication SCORE FROM 1 (social chitchat; discourse that aims to get everyone to
some predetermined point; limited context for peer-to-peer or extended
interactions) to 10 (knowledge building/progressive discourse aimed at
advancing the state of the field; discourse to achieve a more inclusive,
higher-order analysis; open community knowledge spaces encourage
peer-to-peer and extended interactions)

SCORE_______
RATIONALE FOR YOUR SCORE:
(Use as much space as you need)
DO YOU SEE A WAY TO IMPROVE YOUR ENVIRONMENT OR
ASSESSMENT ALONG THIS DIMENSION? IF SO, PLEASE
PROVIDE A BRIEF ACCOUNT OF HOW YOU MIGHT DO THAT,
OR HOW THE IDEAS IN THIS WORKING PAPER MIGHT HELP.
(Use as much space as you need)
Collaboration/ SCORE FROM 1 (small group work—divided responsibility to create
teamwork a finished product; the whole is the sum of its parts, not greater than
that sum) to 10 (collective or shared intelligence emerges from
collaboration and competition of many individuals and aims to
enhance the social pool of existing knowledge. Team members aim to
achieve a focus and threshold for productive interaction and work with
networked ICT. Advances in community knowledge are prized,
over-and-above individual success, while enabling each participant
to contribute to that success)

SCORE_______
RATIONALE FOR YOUR SCORE:
(Use as much space as you need)
DO YOU SEE A WAY TO IMPROVE YOUR ENVIRONMENT
OR ASSESSMENT ALONG THIS DIMENSION? IF SO, PLEASE
PROVIDE A BRIEF ACCOUNT OF HOW YOU MIGHT DO
THAT, OR HOW THE IDEAS IN THIS WORKING PAPER
MIGHT HELP.
(Use as much space as you need)
5 New Assessments and Environments for Knowledge Building 289

Information Literacy/ SCORE FROM 1 (inquiry: question-answer, through finding and

research compiling information; variable testing research) to 10 (going beyond
given information; constructive use of and contribution to knowledge
resources to identify and expand the social pool of improvable ideas,
with research integral to efforts to advance knowledge resources and
information)

SCORE_______
RATIONALE FOR YOUR SCORE:
(Use as much space as you need)
DO YOU SEE A WAY TO IMPROVE YOUR ENVIRONMENT
OR ASSESSMENT ALONG THIS DIMENSION? IF SO, PLEASE
PROVIDE A BRIEF ACCOUNT OF HOW YOU MIGHT DO
THAT, OR HOW THE IDEAS IN THIS WORKING PAPER
MIGHT HELP.
(Use as much space as you need)
Critical thinking, SCORE FROM 1 (meaningful activities are designed by the director/
problem solving and teacher/curriculum designer; learners work on predetermined tasks set
decision-making by others.) to 10 (high-level thinking skills exercised in the course of
authentic knowledge work; the bar for accomplishments is continually
raised through self-initiated problem finding and attunement to
promising ideas; participants are engaged in complex problems and
systems thinking)

SCORE_______
RATIONALE FOR YOUR SCORE:
(Use as much space as you need)
DO YOU SEE A WAY TO IMPROVE YOUR ENVIRONMENT
OR ASSESSMENT ALONG THIS DIMENSION? IF SO, PLEASE
PROVIDE A BRIEF ACCOUNT OF HOW YOU MIGHT DO
THAT, OR HOW THE IDEAS IN THIS WORKING PAPER
MIGHT HELP.
(Use as much space as you need)
Citizenship—local SCORE FROM 1 (support of organization and community behavioral
and global norms; “doing one’s best”; personal rights) to 10 (citizens feel part of a
knowledge-creating civilization and aim to contribute to a global
enterprise; team members value diverse perspectives, build shared,
interconnected knowledge spanning formal and informal settings,
exercise leadership, and support inclusive rights)

SCORE_______
RATIONALE FOR YOUR SCORE:
(Use as much space as you need)
DO YOU SEE A WAY TO IMPROVE YOUR ENVIRONMENT OR
ASSESSMENT ALONG THIS DIMENSION? IF SO, PLEASE
PROVIDE A BRIEF ACCOUNT OF HOW YOU MIGHT DO THAT,
OR HOW THE IDEAS IN THIS WORKING PAPER MIGHT HELP.
(Use as much space as you need)
290 M. Scardamalia et al.

ICT literacy SCORE FROM 1 (familiarity with and ability to use common applications
and web resources and facilities) to 10 (ICT integrated into the daily
workings of the organization; shared community spaces built and
continually improved by participants, with connection to organizations
and resources worldwide)

SCORE_______
RATIONALE FOR YOUR SCORE:
(Use as much space as you need)
DO YOU SEE A WAY TO IMPROVE YOUR ENVIRONMENT
OR ASSESSMENT ALONG THIS DIMENSION? IF SO, PLEASE
PROVIDE A BRIEF ACCOUNT OF HOW YOU MIGHT DO
THAT, OR HOW THE IDEAS IN THIS WORKING PAPER
MIGHT HELP.
(Use as much space as you need)
Life and career skills SCORE FROM 1 (personal career goals consistent with individual
characteristics; realistic assessment of requirements and probabilities of
achieving career goals) to 10 (engagement in continuous, “lifelong”
and “life-wide” learning opportunities; self-identification as a knowledge
creator, regardless of life circumstance or context)

SCORE_______
RATIONALE FOR YOUR SCORE:
(Use as much space as you need)
DO YOU SEE A WAY TO IMPROVE YOUR ENVIRONMENT
OR ASSESSMENT ALONG THIS DIMENSION? IF SO, PLEASE
PROVIDE A BRIEF ACCOUNT OF HOW YOU MIGHT DO
THAT, OR HOW THE IDEAS IN THIS WORKING PAPER
MIGHT HELP.
(Use as much space as you need)
Learning to learn/ SCORE FROM 1 (students and workers provide input to the organization,
meta-cognition but the high-level processes are under the control of someone else) to 10
(students and workers are able to take charge at the highest, executive
levels; assessment is integral to the operation of the organization,
requiring social as well as individual metacognition)

SCORE_______
RATIONALE FOR YOUR SCORE:
(Use as much space as you need)
DO YOU SEE A WAY TO IMPROVE YOUR ENVIRONMENT OR
ASSESSMENT ALONG THIS DIMENSION? IF SO, PLEASE
PROVIDE A BRIEF ACCOUNT OF HOW YOU MIGHT DO THAT,
OR HOW THE IDEAS IN THIS WORKING PAPER MIGHT HELP.
(Use as much space as you need)
5 New Assessments and Environments for Knowledge Building 291

Personal and social SCORE FROM 1 (individual responsibility; local context) to 10 (team
responsibility—incl. members build on and improve the knowledge assets of the community
cultural competence as a whole, with appreciation of cultural dynamics that will allow the
ideas to be used and improved to serve and benefit a multicultural,
multilingual, changing society)

SCORE_______
RATIONALE FOR YOUR SCORE:
(Use as much space as you need)
DO YOU SEE A WAY TO IMPROVE YOUR ENVIRONMENT OR
ASSESSMENT ALONG THIS DIMENSION? IF SO, PLEASE
PROVIDE A BRIEF ACCOUNT OF HOW YOU MIGHT DO THAT,
OR HOW THE IDEAS IN THIS WORKING PAPER MIGHT HELP.

(Use as much space as you need)

Table 5.3 Ratings of environments and assessments

ATC21S (N = 7) Grad students (N = 11)
Twenty-first century skills Mean SD Max Min Mean SD Max Min
Creativity 7.57 1.81 10 4 5.73 2.53 9 2
Communication 8.00 1.29 9 6 5.50 3.46 9 1
Collaboration 7.86 1.35 9 5 5.59 3.23 9 1
Information literacy 7.57 2.15 9 4 5.55 2.50 10 2
Critical thinking 7.14 1.86 9 4 6.27 3.07 10 2
Citizenship 7.14 2.91 9 2 4.50 2.52 8 1
ICT literacy 7.71 2.69 10 2 4.27 3.10 10 1
Life/career skills 7.57 2.51 9 3 5.86 2.79 10 1
Meta-cognition 8.00 2.00 10 4 4.32 1.95 7 1
Responsibility 7.71 2.21 9 4 4.00 2.76 8 1

Results Obtained by Means of Analytic Templates

Table 5.3 provides descriptive statistics of the ratings of environments and assessments
selected by (a) Assessment and Teaching of twenty-first Century Skills project
(ATC21S) volunteers versus those selected by (b) graduate students.
Figure 5.6 provides a graphical representation of the ratings of environments and
assessments selected by (a) Assessment and Teaching of Twenty-First Century
Skills (ATC21S) volunteers versus those selected by (b) graduate students, as listed
in Table 5.3.
292 M. Scardamalia et al.

10.0
9.0
8.0
7.0
6.0
Scores

5.0
4.0 ATC21S
3.0
Grad Students
2.0
1.0
0.0

lls

on

y
y

cy
n

y
p
n

ng

ilit
om tivit

ac
tio

hi
io

ki
ra

iti
ki

ib
ns
at

rs
er
ra

gn
te
a

in

ns
ic

ze

lit
bo
re

ee
i

th

co
l
un

po
C

T
iti
la

ar
al

a-
io

IC
C

es
ol

ic

/c
at

et
C

R
rit

fe
C

rm

M
C

Li
fo
In

21st century skills

Fig. 5.6 Ratings of environments and assessments

References

Ackoff, R. L. (1974). The systems revolution. Long Range Planning, 7, 2–20.

Alexopoulou, E., & Driver, R. (1996). Small group discussion in physics: Peer interaction modes
in pairs and fours. Journal of Research in Science Teaching, 33(10), 1099–1114.
American Educational Research Association, American Psychological Association, & National
Council on Measurement in Education (AERA, APA, NCME). (1999). Standards for educational
and psychological testing. Washington, DC: AERA.
Anderson, C. (2006). The long tail: Why the future of business is selling less of more. New York:
Hyperion.
Andrade, H. (2000). Using rubrics to promote thinking and learning. Educational Leadership,
57(5), 13–18.
Arvanitis, S. (2005). Computerization, workplace organization, skilled labour and firm productivity:
Evidence for the Swiss business sector. Economics of Innovation and New Technology, Taylor
and Francis Journals, 14(4), 225–249.
Askenazy, P., Caroli, E., & Marcus, V. (2001). New organizational practices and working conditions:
Evidence from France in the 1990’s. CEPREMAP Working Papers 0106. Downloaded on
October 4, 2009, from http://www.cepremap.cnrs.fr/couv_orange/co0106.pdf.
ATC21S – Assessment & Teaching of 21st century skills. (2009). Transforming education: assessing
and teaching 21st century skills [Assessment Call to Action]. Retrieve from http://atc21s.org/
wp-content/uploads/2011/04/Cisco-Intel-Microsoft-Assessment-Call-to-Action.pdf.
Autor, D., Levy, F., & Munane, R. (2003). The skill content of recent technological change: An
empirical exploration. Quarterly Journal of Economics, 118(4), 1279–1334.
Banks, J. A., Au, K. A., Ball, A. F., Bell, P., Gordon, E., Gutierrez, K. D., Brice Heath, S., Lee, C.
D., Lee, Y., Mahiri, J., Suad Nasir, N., Valdes, G., & Zhou, M. (2007). Learning in and out of
school in diverse environments: Life-long, life-wide, and life-deep. http://www.life-slc.org/
Barron, B. J. (2003). When smart groups fail. The Journal of the Learning Sciences, 12(3),
307–35.
5 New Assessments and Environments for Knowledge Building 293

Barth, P. (2009). What do we mean by 21st century skills? American School Board Journal.
Retrieved on October 8, 2009, from http://www.asbj.com/MainMenuCategory/Archive/2009/
October/What-Do-We-Mean-by-21st-Century-Skills.aspx
Bateman, H. V., Goldman, S. R., Newbrough, J. R., & Bransford, J. D. (1998). Students’ sense of
community in constructivist/collaborative learning environments. Proceedings of the Twentieth
Annual Meeting of the Cognitive Science Society (pp. 126–131). Mahwah: Lawrence Erlbaum.
Bell, D. (1973). The coming of post-industrial society: A venture in social forecasting. New York:
Basic Books.
Bell, P., Lewenstein, B., Shouse, A. W., & Feder, M. A. (Eds.). (2009). Learning science in informal
environments: People, places, and pursuits. Washington, DC: National Academies Press.
Bennett, R. E., Persky, H., Weiss, A., & Jenkins, F. (2007). Problem solving in technology rich
environments: A report from the NAEP technology-based assessment project, Research and
Development Series (NCES 2007–466). U.S. Department of Education, National Center for
Educational Statistics. Washington, DC: U.S. Government Printing Office.
Bereiter, C. (1984). How to keep thinking skills from going the way of all frills. Educational
Leadership, 42(1), 75–77.
Bereiter, C. (2002). Education and mind in the knowledge age. Mahwah: Lawrence Erlbaum
Associates.
Bereiter, C. (2009). Innovation in the absence of principled knowledge: The case of the Wright
Brothers. Creativity and Innovation Management, 18(3), 234–241.
Bereiter, C., & Scardamalia, M. (1989). Intentional learning as a goal of instruction. In L. B.
Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser
(pp. 361–392). Hillsdale: Lawrence Erlbaum Associates.
Bereiter, C., & Scardamalia, M. (1993). Surpassing ourselves: An inquiry into the nature and
implications of expertise. Chicago and La Salle: Open Court.
Bereiter, C., & Scardamalia, M. (2006). Education for the knowledge age: Design-centred models
of teaching and instruction. In P. A. Alexander & P. H. Winne (Eds.), Handbook of educational
psychology (2nd ed., pp. 695–713). Mahwah: Lawrence Erlbaum Associates.
Bereiter, C., & Scardamalia, M. (2009). Teaching how science really works. Education Canada,
49(1), 14–17.
Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., & Rumble, M. (2009). Developing 21st
century skills and assessments. White Paper from the Assessment and Learning of 21st Century
Skills Project.
Black, S. E., & Lynch, L. M. (2003). What’s driving the new economy: The benefits of workplace
innovation. The Economic Journal, 114, 97–116.
Bonk, C. J. (2009). The world is open: How web technology is revolutionizing education. San
Francisco: Jossey-Bass.
Borghans, L., & ter Weel, B. (2001). Computers, skills and wages. Maastricht: MERIT.
Bransford, J. D., & Schwartz, D. (1999). Rethinking transfer: A simple proposal with multiple
implications. In A. Iran-Nejad & P. D. Pearson (Eds.), Review of research in education (Vol. 24,
pp. 61–100). Washington, DC: American Educational Research Association.
Bransford, J. D., & Schwartz, D. (2009). It takes expertise to make expertise: Some thoughts about
how and why. In K. A. Ericsson (Ed.), Development of professional expertise: Toward mea-
surement of expert performance and design of optimal learning environments (pp. 432–448).
New York: Cambridge University Press.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How people learn: Brain, mind, experience,
and school. Washington, DC: National Academy Press.
Bransford, J., Mosborg, S., Copland, M. A., Honig, M. A., Nelson, H. G. Gawel, D., Phillips, R. S.,
& Vye, N. (2009). Adaptive people and adaptive systems: Issues of learning and design. In
A. Hargreaves, A. Lieberman, M. Fullan, & D. Hopkins (Eds.), Second International Handbook
of Educational Change. Springer International Handbooks of Education, (Vol. 23, pp. 825–
856). Dordrecht: Springer.
Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating
complex interventions. Journal of the Learning Sciences, 2(2), 141–178.
294 M. Scardamalia et al.

Brown, A. L., & Campione, J. C. (1996). Psychological theory and design of innovative learning
environments: On procedures, principles, and systems. In L. Schauble & R. Glaser (Eds.),
Innovations in learning: New environments for education (pp. 289–325). Mahwah: Lawrence
Erlbaum Associates.
Carey, S., & Smith, C. (1993). On understanding the nature of scientific knowledge. Educational
Psychologist, 28(3), 235–251.
Carey, S., Evans, R., Honda, M., Jay, E., & Unger, C. (1989). An experiment is when You Try It
and See if It works”: A study of junior high school Students’ understanding of the construction
of scientific knowledge. International Journal of Science Education, 11(5), 514–529.
Chase, W. G., & Simon, H. A. (1973). Perception in chess. Cognitive Psychology, 1, 33–81.
Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics
problems by experts and novices. Cognitive Science, 5, 121–152.
Chuy, M., Scardamalia, M., & Bereiter, C. (2009, August). Knowledge building and writing
development. Paper presented at the Association for Teacher Education in Europe Conference
(ATEE), Palma de Mallorca, Spain.
Collins, A., & Halverson, R. (2009). Rethinking education in the age of technology: The digital
revolution and schooling in America. New York: Teachers College Press.
Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological
issues. The Journal of the Learning Sciences, 13(1), 15–42.
Confrey, J. (1990). A review of research on student conceptions in mathematics, science programming.
Review of Research in Education 16, 3–55, C.B. Cazden, ed. Washington, DC: American
Educational Research Association.
Council, L. (2007). Skills for the future. Brussels: Lisbon Council.
Crawford, M. B. (2006). Shop class as soulcraft. The New Atlantis, 13, 7–24. Retrieved on
October 10, 2009, from http://www.thenewatlantis.com/docLib/20090526_TNA13Crawford
2009.pdf.
Crawford, V. M., & Toyama, Y. (2002). WorldWatcher looking at the environment curriculum:
Final external evaluation report. Menlo Park: SRI International.
Crespi, F., & Pianta, M. (2008). Demand and innovation in productivity growth. International
Review of Applied Economics, 22(6), 655–672.
Csapó, B. (2007). Research into learning to learn through the assessment of quality and organization
of learning outcomes. The Curriculum Journal, 18(2), 195–210.
Darling-Hammond, L. (1997). The right to learn: A blueprint for creating schools that work. San
Francisco: Jossey-Bass.
Darling-Hammond, L. (2000). Teacher quality and student achievement: A review of state policy
evidence. Education Policy Analysis Archives, 8(1).
Darling-Hammond, L., Barron, B., Pearson, P. D., Schoenfeld, A. H., Stage, E. K., Zimmerman, T.
D., Cervetti, G. N., & Tilson, J. L. (2008). Powerful learning: What we know about teaching
for understanding. San Francisco: Jossey-Bass.
David, P. A., & Foray, D. (2003). Economic fundamentals of the knowledge society. Policy Futures
in Education, 1(1), 20–49.
Dawkins, R. (1996). The blind watchmaker (Why the evidence of evolution reveals a universe
without design). New York: W. W. Norton.
de Groot, A. D. (1965). Thought and choice in chess. New York: Basic Books.
Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behaviour.
New York: Plenum.
Dickerson, A., & Green, F. (2004). The growth and valuation of generic skills. Oxford Economic
Papers, 56, 371–406.
Drucker, P. F. (1968). The age of discontinuity: Guidelines to our changing society. New York:
Harper & Row.
Drucker, P. (1985). Innovation and entrepreneurship: Practice and principles. New York: Harper
and Row.
Drucker, P. F. (1994, November). The age of social transformation. Atlantic Monthly, pp. 53–80.
5 New Assessments and Environments for Knowledge Building 295

Drucker, P. F. (2003). A functioning society: Selection from sixty-five years of writing on community,
society, and polity. New Brunswick: Transaction Publishers.
Dweck, C. S. (1986). Motivational processes affecting learning. American Psychologist, 41,
1040–1048.
Earl, L. M. (2003). Assessment as learning. Using classroom assessment to maximize student
learning. Thousand Oaks, CA: Corwin Press.
Earl, L. M., & Katz, S. (2006). Leading schools in a data-rich world: Harnessing data for school
improvement. Thousand Oaks: Corwin Press.
Engle, R. A., & Conant, F. R. (2002). Guiding principles for fostering productive disciplinary
engagement: Explaining an emergent argument in a community of learners classroom.
Cognition and Instruction, 20(4), 399–483.
Ericsson, K. A. (Ed.). (2009). Development of professional expertise. Toward measurement of
expert performance and design of optimal learning environments. New York, NY: Cambridge
University Press.
Erstad, O. (2008). Trajectories of remixing—Digital literacies, media production and schooling.
In C. Lankshear & M. Knobel (Eds.), Digital literacies. Concepts, policies and practices
(pp. 177–202). New York: Peter Lang.
Fadel, C. (2008, Summer). Deep dives in 21st century curriculum (pp. 3–5). Retrieved on
June 10, 2010, from http://mascd.schoolwires.net/1731106417449990/lib/1731106417449990/
Summer%202008/June%20Perspectives.Deep%20Dives.2008.pdf.
Fischer, K. W., & Bidell, T. R. (1997). Dynamic development of psychological structures in action
and thought. In R. M. Lerner (Ed.) & W. Damon (Series Ed.), Handbook of child psychology:
Vol. 1. Theoretical models of human development (5th ed., pp. 467–561). New York: Wiley.
Frederiksen, J. R., & Collins, A. (1989). A system approach to educational testing. Educational
Researcher, 18(9), 27–32.
Fujimura, J. (1992). Crafting science: Standardized packages, boundary objects, and translation.
In A. Pickering (Ed.), Science as practice and culture. Chicago: University of Chicago
Press.
Gan, Y. C., Scardamalia, M., Hong, H.-Y., & Zhang, J. (2007). Making thinking visible: Growth in
graphical literacy, Grades 3 and 4. In C. Chinn, G. Erkens, & S. Puntambekar (Eds.), Proceed-
ings of the International Conference on Computer Supported Collaborative Learning 2007
(pp. 206–208). Rutgers, The State University of New Jersey, Newark.
Gaskin, I. W. (2005). Success with struggling readers: The Benchmark School approach.
New York: Guilford.
Gates, D. (2005). Boeing 787: Parts from around world will be swiftly integrated. The Seattle
Times, September 11, 2005.
Gera, S., & Gu, W. (2004). The effect of organizational innovation and information technology on
firm performance. International Productivity Monitor, 9, 37–51.
Gillmore, G. M. (1998, December). Importance of specific skills five and ten years after gradua-
tion. OEA Research Report 98–11. Seattle: University of Washington Office of Educational
Assessment. Retrieved May 12, 2004, from http://www.washington.edu/oea/9811.htm.
Glaser, R. (1991). Expertise and assessment. In M. Wittrock & E. Baker (Eds.), Testing and
cognition (pp. 17–30). Englewood Cliffs, NJ: Prentice-Hall.
Gloor, P. A. (2006). Swarm creativity: Competitive advantage through collaborative innovation
networks. Oxford: Oxford University Press.
Goodwin, C., & Goodwin, M. H. (1996). Seeing as a situated activity: Formulating planes. In
Y. Engeström & D. Middleton (Eds.), Cognition and communication at work (pp. 61–95).
Cambridge: Cambridge University Press.
Greeno, J. G. (1991). Number sense as situated knowing in a conceptual domain. Journal for
Research in Mathematics Education, 22, 170–218.
Hall, R., & Stevens, R. (1995). Making space: A comparison of mathematical work in school and
professional design practices. In S. L. Star (Ed.), The cultures of computing (pp. 118–145).
London: Basil Blackwell.
296 M. Scardamalia et al.

Hatano, G., & Inagaki, K. (1986). Two courses of expertise. In H. Stevenson, J. Azuma, & K. Hakuta
(Eds.), Child development and education in Japan (pp. 262–272). New York: W. H. Freeman.
Hatano, G., & Osuro, Y. (2003). Commentary: Reconceptualizing school learning using insight
from expertise research. Educational Researcher, 32, 26–29.
Hearn, G., & Rooney, D. (Eds.). (2008). Knowledge policy. Challenges for the 21st century.
Northampton: Edward Elgar Publishing, Inc.
Herrenkohl, L. R., & Guerra, M. R. (1998). Participant structures, scientific discourse, and student
engagement in fourth grade. Cognition and Instruction, 16, 433–475.
Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. Physics Teacher, 30,
141–158.
Homer-Dixon, T. (2000). The ingenuity gap. New York: Knopf.
Honda, M. (1994). Linguistic inquiry in the science classroom: “It is science, but it’s not like a
science problem in a book.”Cambridge: MIT Working Papers in Linguistics.
Johnson, P. (2009). The 21st century skills movement. Educational Leadership, 67(1), 11–11.
Katz, S., Earl, L. M., & Jaafar, S. B. (2009). Building and connecting learning communities:
The power of networks for school improvement. Thousand Oaks: Corwin Press.
Kozma, R. B. (2003). Material and social affordances of multiple representations for science
understanding. Learning Instruction, 13(2), 205–226.
Kozma, R. B., Chin, E., Russell, J., & Marx, N. (2000). The role of representations and tools in the
chemistry laboratory and their implications for chemistry learning. Journal of the Learning
Sciences, 9(3), 105–144.
Kuhn, D., Schauble, L., & Garcia-Mila, M. (1992). Cross-domain development of scientific
reasoning. Cognition and Instruction, 9, 285–327.
Laferrière, T. (2001). Collaborative teaching and education reform in a networked world. In M.
Moll (Ed.), But it’s only a tool! The politics of technology and education reform (pp. 65–88).
Ottawa: Canadian Teachers Federation and Canadian Centre for Policy Alternative.
Laferrière, T., & Gervais, F. (2008). Communities of practice across learning institutions. In C.
Kimble, P. Hildreth, & I. Bourdon (Eds.), Communities of Practice: Creating Learning
Environments for Educators, Vol. 2 (pp. 179–197). Charlotte: Information Age Publishing Inc.
Lai, M., & Law, N. (2006). Peer Scaffolding of Knowledge Building through Collaboration of
Groups with Differential Learning Experiences. Journal of Educational Computing Research,
35(2), 121–142.
Lamon, M., Secules, T., Petrosino, A. J., Hackett, R., Bransford, J. D., & Goldman, S. R. (1996).
Schools for thought: Overview of the project and lessons learned from one of the sites. In L.
Schauble & R. Glaser (Eds.), Innovation in learning: New environments for education (pp.
243–288). Hillsdale: Lawrence Erlbaum.
Law, N. (2006). Leveraging technology for educational reform and pedagogical innovation:
Policies and practices in Hong Kong and Singapore. Research and Practice in Technology
Education and Learning, 1(2), 163–170.
Law, N., & Wong, E. (2003). Developmental trajectory in knowledge building: An investigation.
In B. Wasson, S. Ludvigsen & U. Hoppe (Eds.), Designing for change in networked learning
environments (pp.57–66). Dordrecht:: Kluwer Academic Publishers.
Law, N., Lee, Y., & Chow, A. (2002). Practice characteristics that lead to “21st century learning
outcomes”. Journal of Computer Assisted Learning, 18(4), 415–426.
Lee, C. D. (1992). Literacy, cultural diversity, and instruction. Education and Urban Society, 24,
279–291.
Lee, E. Y. C., Chan, C. K. K., & van Aalst, J. (2006). Students assessing their own collaborative
knowledge building. International Journal of Computer-Supported Collaborative Learning,
1, 277–307.
Lehrer, R., Carpenter, S., Schauble, L., & Putz, A. (2000). Designing classrooms that support
inquiry. In R. Minstrell & E. Van Zee (Eds.), Inquiring into inquiry learning and teaching in
science (pp. 80–99). Reston: American Association for the Advancement of Science.
Leiponen, A. (2005). Organization of knowledge and innovation: The case of Finnish business
services. Industry and Innovation, 12(2), 185–203.
5 New Assessments and Environments for Knowledge Building 297

Leonard-Barton, D. (1995). Wellsprings of knowledge: Building and sustaining the sources

of innovation. Boston: Harvard Business School Press.
Maurin, E., & Thesmar, D. (2004). Changes in the functional structure of firms and the demand for
skill. Journal of Labour Economics, 22(3), 639–644.
Messick, S. (1994). The interplay of evidence and consequences in the validation of performance
assessments. Educational Researcher, 32, 13–23.
Messina, R., & Reeve, R. (2006). Knowledge building in elementary science. In K. Leithwood,
P. McAdie, N. Bascia, & A. Rodrigue (Eds.), Teaching for deep understanding: What every
educator should know (pp. 110–115). Thousand Oaks: Corwin Press.
Mestre, J. P. (1994). Cognitive aspects of learning and teaching science. In S. J. Fitzsimmons, & L.
C. Kerpelman (Eds.), Teacher enhancement for elementary and secondary science and
mathematics: Status, issues, and problems. (pp.3–1—3–53). NSF 94–80, Arlington: National
Science Foundation.
Minstrell, J. (1989). Teaching science for understanding. In L. Resnick & L. Klopfer (Eds.),
Toward the thinking curriculum: Current cognitive research. 1989 Yearbook of the Association
for Supervision and Curriculum Development (pp. 129–149). Washington, DC: Association for
Supervision and Curriculum Development.
Mislevy, R. J., & Haertel, G. D. (2006). Implications of evidence-centred design for educational
testing. Educational Measurement: Issues and Practice, 25(4), 6–20.
Mislevy, R. J., Chudowsky, N., Draney, K., Fried, R., Gaffney, T., Haertel, G., Hafter, A., Hamel,
L., Kennedy, C., Long, K., Morrison, A. L., Murphy, R., Pena, P., Quellmalz, E., Rosenquist,
A., Songer, N., Schank, P., Wenk, A., & Wilson, M. (2003). Design patterns for assessing
science inquiry (PADI Technical Report 1). Menlo Park: SRI International, Center for
Technology in Learning.
Moll, L. C. (1986a). Creating strategic learning environments for students: A community-based
approach. Paper presented at the S.I.G. Language Development Invited Symposium Literacy
and Schooling, Annual Meeting of the American Educational Research Association, San
Francisco.
Moll, L. C. (1986b). Writing as a communication: Creating strategic learning environments for
students. Theory into Practice, 25, 102–108.
Moses, R. P. (1994). The struggle for citizenship and math/sciences literacy. Journal of
Mathematical Behaviour, 13, 107–111.
Moss, J. (2005). Pipes, tubes, and beakers: Teaching rational number. In J. Bransford & S. Donovan
(Eds.), How children learn: History, science and mathematics in the classroom (pp. 309–350).
Washington, DC: National Academies Press.
Moss, J., & Beatty, R. (2006). Knowledge building in mathematics: Supporting collaborative
learning in pattern problems. International Journal of Computer Supported Collaborative
Learning, 1(4), 441–465.
Murphy, M. (2002). Organizational change and firm performance. OECD Working Papers.
Downloaded on October 3, 2009 from http://puck.sourceoecd.org/vl=18659355/cl=20/nw=1/
rpsv/workingpapers/18151965/wp_5lgsjhvj7m41.htm.
National Research Council (2000). How people learn: Brain, mind, experience, and school.
Expanded version; J. D. Bransford, A. L. Brown, & R. R. Cocking (Eds.). Washington, DC:
National Academy Press.
Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs: Prentice-Hall.
Nonaka, I., & Takeuchi, H. (1995). The knowledge creating company: How Japanese companies
create the dynamics of innovation. New York: Oxford University Press.
Norman, D. A. (1993). Things that make us smart. Reading: Addison-Wesley Publishing Company.
Nunes, C. A. A., Nunes, M. M. R., & Davis, C. (2003). Assessing the inaccessible: Metacognition
and attitudes. Assessment in Education, 10(3), 375–388.
Ochs, E., Gonzales, P., & Jacoby, S. (1996). “When I come down I’m in the domain state”:
Grammar and graphic representation in the interpretive activity of physicists. In E. Ochs, E. A.
Schegloff, & S. Thompson (Eds.), Interaction and grammar (pp. 328–369). New York:
Cambridge University Press.
298 M. Scardamalia et al.

Paavola, S., & Hakkarainen, K. (2005). The knowledge creation metaphor—An emergent episte-
mological approach to learning. Science and Education, 14, 535–557.
Panel on Educational Technology of the President’s Committee of Advisors on Science and
Technology (1997, March). Report to the President on the use of technology to strengthen K-12
education in the United States. Retrieved on December 1, 2009, from http://www.ostp.gov/
PCAST/k-12ed.html.
Partnership for 21st Century Skills. (2009). Retrieved on October 1, 2009, from http://
www.21stcenturyskills.org/
Pellegrino, J., Chudowsky, N., & Glaser, R. (2001). Knowing what students know: The science
and design of educational assessment. Washington, DC: National Academy Press.
Pilat, D. (2004). The economic impact of ICT: A European perspective. Paper presented at a
conference on IT Innovation, Tokyo.
Quellmalz, E. S., & Haertel, G. D. (2008). Assessing new literacies in science and mathematics.
In D. J. Leu Jr., J. Coiro, M. Knowbel, & C. Lankshear (Eds.), Handbook of research on new
literacies. Mahwah: Erlbaum.
Quellmalz, E. S., & Kozma, R. (2003). Designing assessments of learning with technology.
Assessment in Education, 10(3), 389–407.
Quellmalz, E. S., & Pellegrino, J. W. (2009). Technology and testing. Science, 323, 75–79.
Raizen, S. A. (1997). Making way for technology education. Journal of Science Education and
Technology, 6(1), 59–70.
Raizen, S. A., Sellwood, P., Todd, R. D., & Vickers, M. (1995). Technology education in the
classroom: Understanding the designed world. San Francisco: Jossey-Bass.
Redish, E. F. (1996). Discipline-specific science education and educational research: The case of
physics. Paper prepared for the Committee on Developments in the Science of Learning, for the
Sciences of Science Learning: an Interdisciplinary Discussion.
Reich, R. B. (1991). The work of nations: Preparing ourselves for 21st century capitalism.
New York: A.A. Knopf.
Robinson, A. G., & Stern, S. (1997). Corporate creativity. How innovation and improvement
actually happen. San Francisco: Berrett-Koehler Publishers, Inc.
Rotherham, A. J. (2008). 21st-century skills are not a new education trend but could be a fad.
Retrieve October 8, 2009, from http://www.usnews.com/articles/opinion/2008/12/15/21st-
century-skills-are-not-a-new-education-trend-but-could-be-a-fad.html
Rotherham, A. J., & Willingham, D. (2009). 21st Century skills: The challenges ahead. Educational
Leadership, 67(1), 16–21.
Saving the rainforest: REDD or dead? (2009). Retrieved on December 19, 2009, from http://
edition.cnn.com/2009/WORLD/europe/12/18/un.redd.program.rainforests/index.html
Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge.
In B. Smith (Ed.), Liberal education in a knowledge society (pp. 67–98). Chicago: Open
Court.
Scardamalia, M., & Bereiter, C. (2003). Knowledge building. In Encyclopedia of education (2nd
ed., pp. 1370–1373). New York: Macmillan Reference.
Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology.
In K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 97–118). New York:
Cambridge University Press.
Scardamalia, M., Bereiter, C., Brett, C., Burtis, P. J., Calhoun, C., & Smith Lea, N. (1992).
Educational applications of a networked communal database. Interactive Learning
Environments, 2(1), 45–71.
Schauble, L., Glaser, R., Duschl, R. A., Shulze, S., & John, J. (1995). Students’ understanding of
the objectives and procedures of experimentation in the science classroom. Journal of the
Learning Sciences, 4, 131–166.
Schwartz, D. L., & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16(4),
475–522.
Senge, P. M. (1990). The fifth discipline. London: Century Business.
5 New Assessments and Environments for Knowledge Building 299

Shutt, K., Phillips, R., Van Horne, K., Vye, N., & Bransford, J. B. (2009). Developing science
inquiry skills with challenge-based, student-directed learning. Seattle: Presentation to the LIFE
Center: Learning in Informal and Formal Environments, University of Washington.
Shutt, K., Vye, N., & Bransford, J. D. (2011, April). The role of agency and authenticity in argu-
mentation during science inquiry. Paper presented at the annual meeting of the National
Association for Research in Science Teaching, Orlando, FL.
Simonton, D. K. (1999). Origins of genius: Darwinian perspectives on creativity. New York:
Oxford University Press.
Smith, C. L., & Wenk, L. (2006). Relations among three aspects of first-year college Students’
epistemologies of science. Journal of Research in Science Teaching, 43(8), 747–785.
Smith, C. L., Maclin, D., Houghton, C., & Hennessey, M. G. (2000). Sixth-grade Students’
epistemologies of science: The impact of school science experiences on epistemological devel-
opment. Cognition and Instruction, 18(3), 349–422.
Spiro, R. J., Vispoel, W. L., Schmitz, J., Samarapungavan, A., & Boeger, A. (1987). Knowledge
acquisition for application: Cognitive flexibility and transfer in complex content domains. In B.
C. Britton & S. Glynn (Eds.), Executive control processes in reading (pp. 177–199). Hillsdale:
Lawrence Erlbaum Associates.
Spiro, R. J., Feltovich, P. L., Jackson, M. J., & Coulson, R. L. (1991). Cognitive flexibility,
constructivism, and hypertext: Random access instruction for advanced knowledge acquisition
in ill-structured domains. Educational Technology, 31(5), 24–33.
Stahl, G. (2006). Group cognition: Computer support for building collaborative knowledge.
Cambridge: MIT Press.
Stewart, I., & Golubitsky, M. (1992). Fearful symmetry: Is God a geometer? Oxford: Blackwell
Publishers.
Stipek, D. (2002). Motivation to learn: Integrating theory and practice (4th ed.). Needham Heights:
Allyn and Bacon.
Stiroh, K. J. (2003). Growth and innovation in the new economy. In D. Jones (Ed.), New economy
handbook (pp. 723–751). San Diego/London: Elsevier/Academic Press.
Suchman, L. A., & Trigg, R. H. (1993). Artificial intelligence as craftwork. In S. Chaiklin &
J. Lave (Eds.), Understanding practice: Perspectives on activity and context (pp. 144–178).
New York: Cambridge University Press.
Sun, Y., Zhang, J., & Scardamalia, M. (2008). Knowledge building and vocabulary growth over
two years, Grades 3 and 4. Instructional Science. doi:10.1007/s11251-008-9082-5.
Sun, Y., Zhang, J., & Scardamalia, M. (2010). Developing deep understanding and literacy while
addressing a gender-based literacy gap. Canadian Journal of Learning and Technology 36(1).
Published online at http://www.cjlt.ca/index.php/cjlt/article/view/576
Svihla, V., Vye, N. J., Brown, M., Philips, R., Gawel, D., & Bransford, J. D. (2009). Interactive
learning assessments for the 21st century. Education Canada, 49(3), 44–47.
Tabak, I., & Baumgartner, E. (2004). The teacher as partner: Exploring participant structures,
symmetry, and identity work in scaffolding. Cognition and Instruction, 22(4), 393–429.
Teplovs, C. (2008). The knowledge space visualizer: A tool for visualizing online discourse. In
G. Kanselaar, V. Jonker, P. A. Kirschner, & F. J. Prins (Eds.), Proceedings of the International
Conference of the Learning Sciences 2008: Cre8 a learning world. Utrecht: International
Society of the Learning.
The North American Council for Online Learning & the Partnership for 21st Century Skills.
(2006). Virtual Schools and 21st Century Skills. Retrieved on October 8, 2009, from http://
www.inacol.org/research/docs/NACOL_21CenturySkills.pdf
Toffler, A. (1990). Power shift. Knowledge, wealth, and violence at the edge of the 21st century.
New York: Bantam Books.
Trevinarus, J. (1994). Virtual reality technologies and people with disabilities. Presence:
Teleoperators and Virtual Environments, 3(3), 201–207.
Trevinarus, J. (2002). Making yourself at home—Portable personal access preferences. In
K. Miesenberger, J. Klaus, & W. Zagler (Eds.), Proceedings of the 8th International Conference
on Computers Helping People with Special Needs (pp. 643–648). London: Springer.
300 M. Scardamalia et al.

Trilling, B., & Fadel, C. (2009). 21st Century skills: Learning for life in our times. San Francisco:
Jossey-Bass.
Tucker, B. (2009). The Next Generation of Testing. Retrieved on December 10, 2009, from http://
www.ascd.org/publications/educational_leadership/nov09/vol67/num03/The_Next_
Generation_of_Testing.aspx.
Tzou, C., & Bell, P. (2010). Micros and me: Leveraging students’ cultural repertoires of practice
around microbiology and health in the redesign of a commercially available science kit. Paper
presented at the meeting of the American Educational Research Association, Denver.
U.S. Department of Commerce, U.S. Department of Education, U.S. Department of Labour,
National Institute of Literacy, and the Small Business Administration (1999). Report retrieved
on October 8, 2009, from http://www.inpathways.net/_ACRNA/21stjobs.pdf
UNESCO. (2005). Towards knowledge societies. Paris: United Nations Educational, Scientific,
and Cultural Organization.
Venezky, R. L., & Davis, C. (2002). “Quo Vademus? The Transformation of Schooling in a
Networked World.” Version 8c. OECD Centre for Educational Research and Innovation, Paris.
http://www.oecd.org/dataoecd/48/20/2073054.pdf.
Vosniadou, S., & Brewer, W. F. (1989). The concept of the Earth’s shape: A study of conceptual
change in childhood. Unpublished paper. Center for the Study of Reading, University of
Illinois, Champaign.
Vygotsky, L. S. (1962). Thought and language. (E. Hanfmann & G. Vakar,Trans.). Cambridge,
MA: MIT Press (Original work published in 1934).
Wertime, R. (1979). Students’ problems and “courage spans. In J. Lockhead & J. Clements (Eds.),
Cognitive process instruction. Philadelphia: The Franklin Institute Press.
Wertsch, J. (1998). Mind as action. New York: Oxford University Press.
Wiggins, G. P., & McTighe, J. (1997). Understanding by Design. Alexandria: Association for
Supervision and Curriculum Development.
Wiggins, G. P., & McTighe, J. (2006). Examining the teaching life. Educational Leadership, 63,
26–29.
Williams, S. M. (2009). The impact of collaborative, Scaffolded Learning in K-12 Schools:
A Meta-Analysis. Report commissioned to The Metiri Group, by Cisco Systems.
Willingham, D. (2008, December 1). Education for the 21st century: Balancing content knowledge
with skills. Message posted to http://www.britannica.com/blogs/2008/12/schooling-for-the-
21st-century-balancing-content-knowledge-with-skills/
Wilson, B. G. (Ed.). (1996). Constructivist learning environments: Case studies in instructional
design. Englewood Cliffs, New Jersey: Educational Technology Publications, Inc.
Wilson, E. O. (1999). Consilience: The Unity of Knowledge. London: Vintage Books.
Wilson, M., & Sloane, K. (2000). From principles to practice: An embedded assessment system.
Applied Measurement in Education, 13(2), 181–208.
Wiske, M. S. (1998). What is teaching for understanding? In M. S. Wiske (Ed.), Teaching for
understanding: Linking research with practice (pp. 61–86). San Francisco: Jossey-Bass
Publishers.
Zhang, J., Scardamalia, M., Lamon, M., Messina, R., & Reeve, R. (2007). Socio-cognitive dynamics
of knowledge building in the work of nine- and ten-year-olds. Educational Technology Research
and Development, 55(2), 117–145.
Zhang, J., Scardamalia, M., Reeve, R., & Messina, R. (2009). Designs for collective cognitive
responsibility in knowledge building communities. The Journal of the Learning Sciences, 18,
7–44.
Zohgi, C., Mohr, R., & Meyer, P. (2007). Workplace organization and innovation (Working Paper
405). Washington, DC: Bureau of Labour Statistics.