TYPES OF MATERIALS EVALUATION Tomlinson and Masuhara (2004, p.

7) proposed the following criteria

There are many different types of materials evaluation. It is possible to for evaluating criteria:
apply the basic principles of materials evaluation to all types of evaluation, a. Is each question an evaluation question?
but it is not possible to make generalizations about procedures which apply b. Does each question only ask one question?
to all types. Evaluations differ, for example, in purpose, in personnel, in c. Is each question answerable?
formality and in timing. You might do an evaluation in order to help a d. Is each question free of dogma?
publisher to make decisions about publication, to help yourself in e. Is each question reliable in the sense that other evaluators would
developing materials for publication, to select a textbook, to write a review interpret it in the same way?
for a journal or as part of a research project. As an evaluator you might be a Tomlinson (2012) reports these criteria and gives examples from the many
learner, a teacher, an editor, a researcher, a Director of Studies or an checklists in the literature of evaluation criteria which their use exposes as
Inspector of English. You might be doing a mental evaluation in a inadequate in terms of specificity, clarity, answerability, validity and
bookshop, filling in a short questionnaire in class or doing a rigorous, generalizability
empirical analysis of data elicited from a large sample of users of the
materials. You might be doing your evaluation before the materials are used 2. Whilst-use evaluation
while they are being used or after they have been used. In order to conduct
an effective evaluation, you need to apply your principles of evaluation to This involves measuring the value of materials while using them or while
the contextual circumstances of your evaluation in order to determine the observing them being used. It can be more objective and reliable than pre-
most reliable and effective procedures. use evaluation as it makes use of measurement rather than prediction.
However, it is limited to measuring what is observable (e.g. ‘Are the
Three types of Evaluation instructions clear to the learners?’) and cannot claim to measure what is
happening in the learners’ brains. It can measure short-term memory
1. Pre-use evaluation through observing learner performance on exercises but it cannot measure
durable and effective learning because of the delayed effect of instruction.
Pre-use evaluation involves making predictions about the potential value of It is therefore very useful but dangerous too, as teachers and observers can
materials for their users. It can be context-free, as in a review of materials be misled by whether the activities seem to work or not. Exactly what can
for a journal, context influenced as in a review of draft materials for a be measured in a whilst-use evaluation is controversial, but I would include
publisher with target users in mind or context-dependent, as when a teacher the following:
selects a coursebook for use with her particular class. Often pre-use  Clarity of instructions
evaluation is impressionistic and consists of a teacher flicking through a  Clarity of layout
book to gain a quick impression of its potential value (publishers are well  Comprehensibility of texts
aware of this procedure and sometimes place attractive illustrations in the  Credibility of tasks
top righthand corner of the right-hand page in order to influence the flicker  Achievability of tasks
in a positive way). Even a review for a publisher or journal, and an  Achievement of performance objectives
evaluation for a ministry of education is often ‘fundamentally a subjective,  Potential for localization
rule of thumb activity’ (Sheldon, 1988, p. 245) and often mistakes are  Practicality of the materials
made. Making an evaluation criterion-referenced can reduce (but not  Teachability of the materials
remove) subjectivity and can certainly help to make an evaluation more  Flexibility of the materials
principled, rigorous, systematic and reliable. This is especially true if more  Appeal of the materials
than two evaluators conduct the evaluation independently and then average  Motivating power of the materials
their conclusions. For example, in the review of eight adult EFL courses  Impact of the materials
conducted by Tomlinson et al. (2001), the four evaluators devised one-  Effectiveness in facilitating short-term learning
hundred-and-thirty-three criteria together and then used them independently
and in isolation to evaluate the eight courses before pooling their data and Most of the above can be estimated during an open-ended, impressionistic
averaging their scores. Even then, though, the reviewers admitted that, ‘the observation of materials in use but greater reliability can be achieved by
same review, conducted by a different team of reviewers, would almost focusing on one criterion at a time and by using pre-prepared instruments of
certainly have produced a different set of results’ (p. 82). measurement. For example, oral participation in an activity can be
measured by recording the incidence and duration of each student’s oral
Making use of a checklist of criteria has become popular in materials contribution, potential for localization can be estimated by noting the times
evaluations and certain checklists from the literature have been frequently the teacher or a student refers to the location of learning while using the
made use of in evaluations (e.g. Cunningsworth (1984, 1995), Skierso materials and even motivation can be estimated by noting such features as
(1991), Brown (1997), Gearing, (1999)). The problem though is that no set student eye focus, proximity to the materials, time on task and facial
of criteria is applicable to all situations and, as Byrd (2001) says, it is animation. Whilst-use evaluation receives very little attention in the
important that there is a fit between the materials and the curriculum, literature, but Jolly and Bolitho (2011) describe interesting case studies of
students and teachers. Matthews (1985), Cunningsworth (1995) and how student comment and feedback during lessons provided useful
Tomlinson (2012) have also stressed the importance of relating evaluation evaluation of materials, which led to improvements being made in the
criteria to what is known about the context of learning and Makundan and materials during and after the lessons. Also Tomlinson and Masuhara
Ahour (2010) in their review of 48 evaluation checklists were critical of (2010) report materials development projects in which whilst-use
most checklists for being too context bound to be generalizable. Makundan evaluation was made use of.
and Ahour (2010) proposed that a framework for generating flexible criteria
would be more useful than detailed and inflexible checklists (a proposition 3. Post-use Evaluation
also made by Ellis (2011) and stressed and demonstrated by Tomlinson Post-use evaluation is probably the most valuable (but least administered)
(2003b)). type of evaluation as it can measure the actual effects of the materials on
the users. It can measure the short-term effect as regards motivation,
Other researchers who have proposed and exemplified frameworks for impact, achievability, instant learning, etc., and it can measure the long-
generating evaluation criteria include: term effect as regards durable learning and application. It can answer such
 McGrath (2002), who suggests a procedure involving materials important questions as:
analysis followed by first glance evaluation, user feedback and  What do the learners know which they did not know before starting to
evaluation using context-specific checklists. use the materials?
 Riazi (2003), who suggests, surveying the teaching/learning situation,  What do the learners still not know despite using the materials?
conducting a neutral analysis and the carrying out of a belief-driven  What can the learners do which they could not do before starting to
evaluation. use the materials?
 Rubdy (2003), who suggests a dynamic model of evaluation in which  What can the learners still not do despite using the materials?
the categories of psychological validity, pedagogical validity and  To what extent have the materials prepared the learners for their
process and content validity interact. examinations?
 Mukundan (2006), who describes the use of a composite framework  To what extent have the materials prepared the learners for their post
combining checklists, reflective journals and computer software to course use of the target language?
evaluate ELT textbooks in Malaysia.  What effect have the materials had on the confidence of the learners?
 McDonough et al. (2013), who focus on developing criteria  What effect have the materials had on the motivation of the learners?
evaluating the suitability of materials in relation to usability,  To what extent have the materials helped the learners to become
generalizability, adaptability and flexibility. independent learners?
 Did the teachers find the materials easy to use? l Did the materials
help the teachers to cover the syllabus?
 Did the administrators find the materials helped them to standardize knowledge to the class; hence the learner’s role in this process is still rather
the teaching in their institution? limited and not truly learner-centered.

In other words, it can measure the actual outcomes of the use of the LIST OF KEY FEATURES IN MATERIALS ADAPTATION
materials and thus provide the data on which reliable decisions about the The following is a list of basic key points to take into account when
use, adaptation or replacement of the materials can be made. Ways of evaluating and adapting courses. However, these can be used simply as a
measuring the post-use effects of materials include: proposal to be developed further and adapted to different classroom
 tests of what has been ‘taught’ by the materials; situations.
 tests of what the students can do; l examinations; 1. Learner-centeredness and critical awareness development
 interviews; There is a large amount of literature on learner-centered
 questionnaires; approaches and principles (Nunan, 1988). However, there are very
 criterion-referenced evaluations by the users; few language teaching and learning materials which, in my opinion,
 post-course diaries; are truly learner-centered, in the sense that their aims are the
 post-course ‘shadowing’ of the learners; development of learners’ critical awareness, linguistic empowerment
 post-course reports on the learners by employers, subject tutors, etc. and therefore learner autonomy. The materials should put learners at
the center of the learning process and make them input providers
The main problem, of course, is that it takes time and expertise to (hence part of the materials adaption process), whereas teachers
measure post-use effects reliably (especially as, to be really revealing, there should be facilitators and coordinators and should provide a stimulus,
should be measurement of pre-use attitudes and abilities in order to provide a starting point, for language exposure as well as for different
data for post-use comparison). But publishers and ministries do have the approaches to learning. Materials adaptation, therefore, should be
time and can engage the expertise, and teachers can be helped to design, shared between materials developers, teachers and learners.
administer and analyse post-use instruments of measurement. Then we will 2. Flexibility and choice
have much more useful information, not only about the effects of particular Materials should be flexible, in the sense that they should
courses of materials but about the relative effectiveness of different types of provide learners with the possibility of choosing different activities,
materials. Even then, though, we will need to be cautious, as it will be very tasks, projects and approaches, thus of adapting the materials to their
difficult to separate such variables as teacher effectiveness, parental own learning needs. At the same time, however, given the fact that the
support, language exposure outside the classroom, intrinsic motivation, etc. majority of learners are not used to this type of approach to learning,
For a description of the process of post-use evaluation of piloted they should also be exposed to a variety of different activities and
materials see Donovan (1998), for descriptions of how publishers use focus approaches, so that they themselves become more flexible learners,
groups for post-use evaluation of materials see Amrani (2011) and for having experienced different ways of learning. Materials, then,
suggestions of how teachers could do post-use micro-evaluations of should, on the one hand, provide choice but, on the other hand, also
materials see Ellis (1998, 2011). For reports of projects which conducted enable learners to develop a variety of skills and learning styles by
post-use evaluation of materials in many different countries see Tomlinson encouraging them to experience a wide range of tasks and approaches,
and Masuhara (2010). so that they may also become more independent learners. Materials
can, for example, include a choice of tasks ranging from analytical
TEACHER-CENTERED AND LEARNER-CENTERED APPROACH ones (such as those based on grammatical awareness) to more creative
ones (such as those based on creative writing). Learners can be
Adapting materials encouraged to experience them all at one point and then also make
Despite the fact that it seems a relatively under-researched discipline, in choices at a later stage.
many ways adapting materials is an inevitable process as it is always 3. Open-endedness and aesthetic experience
carried out as part of classroom practice. The simple fact of using a piece of If materials allow only one possible right answer, they do not
teaching/learning materials inevitably means adapting it to the particular leave space for interpretation and adaptation, whereas if they are
needs of a specific teaching and learning scenario. In the practice of open-ended they can become more relevant to learners. In many ways
language teaching, this has been accepted for quite a long time now this is related to the concept of Aesthetic Experience, an idea which
(Madsen and Bowen, 1978). originated from the theory of Aesthetic Response as put forward by
Rosenblatt (1995). Aesthetic Response refers to the process of
Teacher-centred Approach reacting spontaneously when reading literary texts, hence it involves
A teacher-centred approach to adaptation Having proposed the interaction between readers, language and texts (Iser, 1978; Hirvela,
importance of materials adaptation as a relevant and useful link between the 1996). Some of the major elements of such type of experiential
reality of the language classroom and SLA research findings, there is response, such as the voice of the narrator and that of the reader, as
obviously a great need to develop such a process further and put it into well as the role of the receiver and the one of the producer of the
practice in a more systematic manner. However, materials adaptation, in the literary input, become overlapping and interchangeable. Aesthetic
great majority of cases, is still left to the teachers’ hands, and it is largely Experience, therefore, typically represents the immediate response to
based simply on their intuition and experience. On the one hand, research language and literature experienced by the receiver and the producer,
has, for decades, stressed the importance of the learner and their role in the as well as their later interpretations and reactions. Literature and
language classroom; many areas of research, have extensively explored and Aesthetic Experience are inevitably part of a subjective process which
described the advantages of learner involvement in programme design, is created every time the text is read or written. Reading and
methodology, materials selection and adaptation, since the 1970s and 1980s interpretation are always different: we have different reactions every
(Nunan, 1988; Clarke, 1989). On the other hand, particularly as far as time we aesthetically experience a poem, a novel, etc. (Saraceni,
adapting courses is concerned, learners are traditionally left with a rather 2010).
passive role. A parallel point should be drawn here between aesthetic
experience and materials adaptation. Aesthetic Experience
Learner-centered Approach (Rosenblatt, 1994, 1995; Saraceni, 2010) promotes the subjectivity of
A learner-centered approach to adaptation Clarke (1989) texts and their various interpretations. In a similar way, also materials
provides a typical example of a learner-centered approach to adaptation: he for teaching and learning purposes should promote an aesthetic
acknowledges the importance of learner involvement in the adaptation experience, in the sense that they should, not only be based on
process, and he distinguishes what he calls a Negotiated Syllabus, from an right/wrong testing and practice but, rather, they should also focus on
Externally Imposed Syllabus. The former is internally generated, and it is a open-ended tasks and texts. For example, in relation to texts, materials
result of the product of negotiation between teacher and students. The latter should include also those which are open to many different ideas and
is a syllabus imposed by an external body such as the teacher, the institution points of view and encourage a variety of interpretations. Therefore,
or any other administrative authority. There is, however, a fine line between texts and tasks should be included with the main purpose of
the Negotiated Syllabus and the Externally Imposed Syllabus in the sense promoting a subjective response, whether this be in relation to a
that the former turns out, very often, to also be an imposed syllabus for the reading text or to a listening one. If materials present open spaces or
reasons given below. gaps (Eco, 1993, 1995), they can allow learners to form their own
If we write about the learner’s more active role in the adaptation interpretations and ideas and, therefore, to take control of the
process and his/ her negotiation with the teacher, we are assuming that the adaptation process. In this context, the aim of materials moves from
syllabus is the product of cooperation between the teacher and the learners. comprehension testing, which allows only a rather superficial intake
However, generally in this case, the teacher’s input tends to become the of the input, to a deeper understanding and awareness of the language
dominant one, accepted by the learners as the ‘right one’ and the one to exposure, with the emphasis on individual differences.
follow, whereas the learners’ ideas on adapting materials are very often 4. Relevance
perceived to be ‘wrong’ if different from those of the teacher. This still In an attempt to draw a link between the adaptation process and
comes from the traditional, teacher-centered concept of teachers feeding reading, materials left open-ended, as explained above, have the
potential to become relevant to the learners when they fill those gaps
 with their ideas, interpretations and discussions. It is only at this level
that materials acquire significance and become potentially beneficial
for the learners. It is, in fact, by virtue of such contributions that
materials can be adapted and developed further. Adaptation is,
therefore, essential in making materials relevant and potentially more
effective for learning development.
5. Universality
Materials should be based on universally appealing topics,
which are culturally provoking in the sense that they are culturally
specific but, at the same time, they are present in all cultures. A rich
source of this type of topics comes from Literature, which typically
involves themes based on life experiences, feelings, relationships.
These are present in all cultures, but they can be looked at from
different angles and experienced in different ways. Universality of
topics provides a stimulus for discussion, and it enables learners to
focus on and gain a better understanding of cultural differences as
well as cultural commonalities (Jiang, 2000)
6. Authentic and non-authentic input
Materials should be based on authentic texts, those texts which
have been written for any purpose other than language teaching. At
the same time, there should also be a combination of authentic and
non-authentic tasks, based on realistic scenarios, in order to expose
the learners to realistic input. In my view a significant role is played
by the use of non-authentic tasks with authentic texts. For example,
tasks which aim at drawing the learners’ attention to certain linguistic
features of the input with activities based on texts selected from
authentic sources, can be beneficial for language awareness
development.
7. Provocative topics and tasks
Materials should include topics and activities that can
potentially provoke a reaction, hence an aesthetic experience (whether
it be positive or negative) that is personal and subjective. These can
make learning more engaging and perhaps also more humanistic.
From my point of view, topics are not to be considered
intrinsically provocative, but the activities associated with them can
potentially make the materials more or less provocative, thus more or
less engaging. In my experience, however, certain topics related to
Personal Life, Family, Parents, Relationships, Emotions, Inner Self
can achieve this aim more effectively, rather than those topics very
often associated with controversy such as Politics, War, Racism,
Drugs, etc.
However, although students generally feel engaged when
exposed to provocative topics, at first a few may show some
resistance to such personal depths. Students in general are used to
traditional ways of being taught; they are not always ready to be
challenged and to step beyond the usual safer topics. In some cases,
they are so used to teacher-centered teaching, that they find it more
reassuring and credible. This, however, further demonstrates their
need to be gradually exposed to different types of input, to enable
them to express their opinions and to further develop their
interpretations and points of view, hence, to develop their flexibility
as learners.