Types of materials evaluation

There are many different types of materials evaluation. It is possible to apply

the basic principles of materials evaluation to all types of evaluation but it is not
possible to make generalizations about procedures which apply to all types.
Evaluations differ, for example, in purpose, in personnel, in formality and in
timing. You might do an evaluation in order to help a publisher to make
decisions about publication, to help yourself in developing materials for
publication, to select a textbook, to write a review for a journal or as part of a
research project. As an evaluator you might be a learner, a teacher, an editor, a
researcher, a Director of Studies or an Inspector of English. You might be doing
a mental evaluation in a bookshop, filling in a short questionnaire in class or
doing a rigorous, empirical analysis of data elicited from a large sample of users
of the materials. You might be doing your evaluation before the materials are
used while they are being used or after they have been used. In order to conduct
an effective evaluation, you need to apply your principles of evaluation to the
contextual circumstances of your evaluation in order to determine the most
reliable and effective procedures.

Exercise 1. Mention the different types of evaluation and explain their

differences.

There are various kinds of evaluation that are discussed in this text. These
evaluations serve different purposes and are conducted by different individuals
using various methods. They also occur at different times. Let's take a closer
look at each of these evaluations.

Firstly, there is evaluation for publication decision-making. This type of

evaluation is conducted to determine whether a piece of work is suitable for
publication. It involves assessing the quality and relevance of the content, as
well as the overall structure and organization of the material.

Next, we have materials development evaluation. This evaluation focuses on

the creation and improvement of educational materials. It involves assessing the
effectiveness and appropriateness of the materials in meeting the learning
objectives.
Another type of evaluation is textbook selection evaluation. As the name
suggests, this evaluation is carried out to select the most suitable textbooks for
a particular course or curriculum. It involves reviewing and comparing different
textbooks based on their content, language level, and pedagogical approach.

Review writing evaluation is another important type of evaluation. This

evaluation is conducted by individuals who write reviews of educational
materials. It involves assessing the quality and accuracy of the reviews, as well
as the reviewer's ability to provide constructive feedback.

Research project evaluation focuses on evaluating the quality and validity of

research projects conducted in the field of English language teaching. This
evaluation involves analyzing the research methodology, data collection and
analysis, and the overall contribution of the project to the field.

Learner evaluation is a type of evaluation that assesses the progress and

achievement of learners. It involves measuring their language skills and
knowledge through various assessment methods such as tests, assignments, and
observations.

Teacher evaluation, on the other hand, focuses on assessing the performance

and effectiveness of teachers. This evaluation is conducted to provide feedback
and support for professional development.

Editorial evaluation is carried out to assess the quality and suitability of

materials for publication. It involves reviewing and editing the content,
language, and overall structure of the material.

Director of Studies evaluation is conducted to assess the performance and

effectiveness of the Director of Studies in a language school or institution. This
evaluation involves reviewing their management skills, decision-making
abilities, and overall leadership.

Lastly, Inspector of English evaluation is conducted by an external inspector to

assess the quality and standards of an English language teaching institution.
This evaluation involves reviewing the curriculum, teaching methods, and
overall learning environment.
In conclusion, the different types of evaluation mentioned in this text serve
various purposes and are conducted by different individuals using different
methods. They play a crucial role in ensuring the quality and effectiveness of
educational materials, research projects, teaching, and learning in the field of
English language teaching.

Pre-use evaluation

Pre-use evaluation involves making predictions about the potential value of

materials for their users. It can be context-free, as in a review of materials for a
journal, contextinfluenced as in a review of draft materials for a publisher with
target users in mind or context-dependent, as when a teacher selects a
coursebook for use with her particular class. Often pre-use evaluation is
impressionistic and consists of a teacher flicking through a book to gain a quick
impression of its potential value (publishers are well aware of this procedure
and sometimes place attractive illustrations in the top righthand corner of the
right-hand page in order to influence the flicker in a positive way). Even a
review for a publisher or journal, and an evaluation for a ministry of education
is often ‘fundamentally a subjective, rule of thumb activity’ (Sheldon, 1988, p.
245) and often mistakes are made. Making an evaluation criterion-referenced
can reduce (but not remove) subjectivity and can certainly help to make an
evaluation more principled, rigorous, systematic and reliable. This is especially
true if more than two evaluators conduct the evaluation independently and then
average their conclusions. For example, in the review of eight adult EFL courses
conducted by Tomlinson et al. (2001), the four evaluators devised one-hundred-
and-thirty-three criteria together and then used them independently and in
isolation to evaluate the eight courses before pooling their data and averaging
their scores. Even then, though, the reviewers admitted that, ‘the same review,
conducted by a different team of reviewers, would almost certainly have
produced a different set of results’ (p. 82). Making use of a checklist of criteria
has become popular in materials evaluations and certain checklists from the
literature have been frequently made use of in evaluations.
● McGrath (2002), who suggests a procedure involving materials analysis
followed by first glance evaluation, user feedback and evaluation using
context-specific checklists.
● Riazi (2003), who suggests, surveying the teaching/learning situation,
conducting a neutral analysis and the carrying out of a belief-driven
evaluation.
● Rubdy (2003), who suggests a dynamic model of evaluation in which
the categories of psychological validity, pedagogical validity and process
and content validity interact.
● Makundan (2006), who describes the use of a composite framework
combining checklists, reflective journals and computer software to
evaluate ELT textbooks in Malaysia.
● McDonough et al. (2013), who focus on developing criteria evaluating
the suitability of materials in relation to usability, generalizability,
adaptability and flexibility.

Exercise 2. What does a pre-use evaluation involves? How can it be? How
is it?

Before incorporating educational materials into a teaching setting, it is

crucial to conduct a pre-use evaluation. This evaluation serves the purpose
of determining the suitability, effectiveness, and alignment of the materials
with the desired learning outcomes. There are several methods that can be
employed to carry out this evaluation effectively.

One approach is the review and analysis method, which involves thoroughly
examining the content, structure, and organization of the materials. This
allows educators to assess whether the materials cover the necessary topics,
provide clear explanations, and present information in a logical manner.

Pilot testing is another valuable method in the pre-use evaluation process.

This involves trying out the materials with a small group of students to gather
feedback and identify any areas that may require improvement. This
feedback can be used to make necessary adjustments to the materials before
implementing them on a larger scale.
Expert review is also beneficial in evaluating educational materials. Seeking
input from subject matter experts, experienced educators, or professionals in
the field can provide valuable insights and ensure that the materials are
accurate, up-to-date, and pedagogically sound.

Aligning the materials with the intended learning objectives is essential. This
involves assessing whether the content, activities, and assessments in the
materials are aligned with the desired outcomes. It ensures that the materials
support the learning goals and enable students to achieve the desired
competencies.

Usability assessment focuses on evaluating how user-friendly the materials

are. This involves considering factors such as readability, layout, navigation,
and accessibility. Assessing the usability of the materials ensures that they
are easy to use and understand, facilitating effective teaching and learning.

Engagement and interest are crucial aspects of effective educational

materials. Assessing the level of engagement and interest that the materials
generate among students can help determine their effectiveness. Materials
that are engaging and interesting are more likely to capture students'
attention and enhance their motivation to learn.

Lastly, adapting the materials for the specific teaching context is important.
Considering the unique needs, preferences, and characteristics of the
students and the teaching environment can enhance the effectiveness of the
materials. Adapting the materials ensures that they are relevant, relatable,
and meaningful to the students, promoting successful learning outcomes.

In conclusion, a pre-use evaluation of educational materials is essential to

ensure their suitability, effectiveness, and alignment with the desired
learning goals. By employing methods such as review and analysis, pilot
testing, expert review, alignment with objectives, usability assessment,
engagement and interest evaluation, and adaptation for context, educators
can ensure that the materials meet the needs of their students and facilitate
successful teaching and learning experiences.
 Whilst-use evaluation

This involves measuring the value of materials while using them or while
observing them being used. It can be more objective and reliable than pre-
use evaluation as it makes use of measurement rather than prediction.
However, it is limited to measuring what is observable (e.g. ‘Are the
instructions clear to the learners?’) and cannot claim to measure what is
happening in the learners’ brains. It can measure short-term memory through
observing learner performance on exercises but it cannot measure durable
and effective learning because of the delayed effect of instruction. It is
therefore very useful but dangerous too, as teachers and observers can be
misled by whether the activities seem to work or not. Exactly what can be
measured in a whilst-use evaluation is controversial but I would include the
following:

● Clarity of instructions
● Clarity of layout
● Comprehensibility of texts
● Credibility of tasks
● Achievability of tasks
● Achievement of performance objectives
● Potential for localization
● Practicality of the materials
● Teachability of the materials
● Flexibility of the materials
● Appeal of the materials
● Motivating power of the materials
● Impact of the materials
● Effectiveness in facilitating short-term learning
Exercise 3. What does a whilst-use evaluation involves? How can it be?
How can be achieved a greater reliability?

One way to evaluate educational materials is by conducting a while-use

evaluation, which involves assessing them while they are being used in a
teaching context. The purpose of this evaluation is to gather continuous
feedback on how effective, usable, and impactful the materials are during
actual instruction. There are several methods that can be used to conduct this
evaluation.

Observation is one method that can be used during a while-use evaluation.

By observing teachers and students using the educational materials,
evaluators can gain insights into how well the materials are being utilized,
if they are engaging students, and if they are effectively facilitating
learning.
Feedback collection is another important aspect of a while-use evaluation.
By gathering feedback from teachers, students, and other stakeholders,
evaluators can gain valuable insights into the strengths and weaknesses of
the educational materials. This feedback can help identify areas for
improvement and inform future iterations of the materials.
Assessment results can also provide valuable data during a while-use
evaluation. By analyzing the performance of students who have used the
materials, evaluators can determine if the materials are effectively
facilitating learning and if students are achieving the desired outcomes.
Teacher reflection is another method that can be used during a while-use
evaluation. By encouraging teachers to reflect on their experiences using
the materials, evaluators can gain insights into how well the materials
align with their instructional goals, if they are easy to implement, and if
they meet the needs of their students.
Continuous monitoring is an important component of a while-use
evaluation. By regularly checking in on the progress and impact of the
educational materials, evaluators can ensure that any issues or challenges
are addressed in a timely manner. This ongoing monitoring allows for
adjustments and improvements to be made throughout the implementation
process.
In conclusion, a while-use evaluation is a valuable tool for assessing the
effectiveness, usability, and impact of educational materials during actual
instruction. By utilizing methods such as observation, feedback collection,
assessment results, teacher reflection, and continuous monitoring, evaluators
can gather valuable data to inform improvements and ensure that the
materials are meeting the needs of teachers and students.

Post-use evaluation

Post-use evaluation is probably the most valuable (but least administered)

type of evaluation as it can measure the actual effects of the materials on the
users. It can measure the short-term effect as regards motivation, impact,
achievability, instant learning, etc., and it can measure the long-term effect
as regards durable learning and application. It can answer such important
questions as:

● What do the learners know which they did not know before starting to
use the materials?
● What do the learners still not know despite using the materials?
● What can the learners do which they could not do before starting to use
the materials?
● What can the learners still not do despite using the materials?
● To what extent have the materials prepared the learners for their
examinations?
● To what extent have the materials prepared the learners for their
postcourse use of the target language?
● What effect have the materials had on the confidence of the learners?
● What effect have the materials had on the motivation of the learners?
● To what extent have the materials helped the learners to become
independent learners?
● Did the teachers find the materials easy to use?
● Did the materials help the teachers to cover the syllabus?
● Did the administrators find the materials helped them to standardize the
teaching in their institution?

Exercise 4. What can the post-use evaluation measure? Can you mention
almost three important questions that it can answer? Can you mention some
way to measure the post-use effects of the materials?

A post-use evaluation is a valuable tool for assessing the effectiveness of

educational materials once they have been implemented in a teaching
setting. It allows us to delve into the impact and outcomes of these materials,
providing insights that can inform future improvements and optimizations.

One of the key aspects that a post-use evaluation can shed light on is the
learning outcomes achieved by students. By examining whether the intended
learning objectives were met, we can gauge the effectiveness of the materials
in facilitating knowledge acquisition, skill development, and competency
enhancement. This information is crucial in determining the efficacy of the
materials and their alignment with educational goals.

Engagement and satisfaction are also important factors to consider. It is

essential to assess whether students were actively engaged and motivated
while using the materials. Were they captivated by the content? Did they
find the materials interesting and relevant to their learning needs? By
answering these questions, we can gain insights into the level of student
engagement and satisfaction, which can inform future material design and
selection processes.

Furthermore, a post-use evaluation allows us to evaluate the overall impact

of the materials on students' language proficiency, critical thinking abilities,
problem-solving skills, and other relevant competencies. By assessing the
extent to which the materials contributed to the development of these skills,
we can identify areas of strength and areas that may require further attention
or improvement.
In conclusion, a post-use evaluation provides a comprehensive assessment
of educational materials, enabling us to gauge their effectiveness, measure
their impact, and understand their influence on student learning outcomes.
By addressing key questions related to learning outcomes, engagement and
satisfaction, and the materials' impact on relevant skills, we can make
informed decisions to enhance the quality and effectiveness of educational
materials in future teaching contexts.

To measure the effects of the materials, educators can use different methods:

Assessment Results: Compare students' performance on assessments

before and after using the materials to see if there's improvement.
Feedback and Surveys: Gather feedback from students through surveys to
learn about their experiences and satisfaction.
Observation and Interaction: Observe how students interact with the
materials and their peers during activities to assess engagement and
collaboration.