See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/255566475

ICT for education: A conceptual framework for the sustainable adoption of

technology-enhanced learning environments in schools

Article  in  Technology Pedagogy and Education · October 2012

DOI: 10.1080/1475939X.2012.720415

CITATIONS READS

25 14,532

3 authors, including:

Patricio Rodríguez
University of Chile
21 PUBLICATIONS   693 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

The Geography of education: a territorial intelligence platform to support the implementation and management of new public policies in education View project

Technology Enhanced Collaborative Learning Environments View project

All content following this page was uploaded by Patricio Rodríguez on 17 December 2013.

The user has requested enhancement of the downloaded file.

Paper submitted for peer reviewing — Please do not distribute or cite without permission

ICT for Education: a conceptual framework for the sustainable adoption

of Technology-Enhanced Learning environments in schools
Patricio Rodríguez1, Miguel Nussbaum2 and Lioubov Dombrovskaia3
1
Center for Advanced Research in Education
Universidad de Chile
Periodista José Carrasco Tapia Nº 75, Santiago, Chile
[email protected] // Tel: +56-2-978 2762

2
Department of Computer Science, Engineering Faculty,
Pontificia Universidad Católica de Chile
Edificio San Agustín P4, Av. Vicuña Mackenna 4860, Santiago, Chile
[email protected] // Tel: +56-2-354 4440
3
Informatics Department
Universidad Técnica Federico Santa María,
Av. Vicuña Mackenna 3939, Santiago, Chile
[email protected] // Tel: +56-2-4326600

ABSTRACT —

Currently, the use of ICT in education does not conclusively demonstrate significant effects on learning. However, not all
ICT usage models are designed to affect student outcomes. Therefore, to accurately study the impact of ICT, the concept of
an educational programme supported by ICT must first be defined.

We propose the ICT for Education (ICT4E) programme, an evidenced-based framework to determine a model’s ability to
produce improvements before having to evaluate its results. The framework has four components: implementation,
intervention, transference and total cost.

Based on an explicit definition of the outcomes pursued by a given programme, this framework supports the design of its
activities to promote sustainable changes in pedagogical practices in schools and calculate the programme’s total cost. We
illustrate this with a programme implemented in 6 countries.

In Chile, the ICT4E programme forms part of the policy for educational technology, evaluating the effectiveness and
scalability of such initiatives.
Keywords: Technology-Enhanced Learning, pedagogic model, ICT effects, scalability, sustainability, ICT integration.

1 Introduction
Attempts to improve education with modern machines —such as the phonograph, lantern slides and television— date from
the early 20th century (Castro, 2004; Cuban, 1986; Earle, 2002; Reiser, 2001; Snider, 1992). For the most part such inventions
come and go, yet what happens in the classroom looks pretty much the same (Callister, 1992). Information and
communication technologies (ICT) were also introduced in schools to transform teaching and learning processes and to
improve strategies for better educational attainment (Culp, Honey, & Mandinach, 2003; Kozma, 2003; Sunkel, 2006).

Nonetheless, such desired effects are far from becoming a reality (Moonen, 2008; Reynolds, Treharne, & Tripp, 2003;
Robertson, 2002; Tondeur, Braak, & Valcke, 2007; Trucano, 2005). A wide range of ICT applications and uses were
analyzed in several impact assessments: programme evaluations, specific educational software in particular subjects at large
scale, research and case reviews, and meta-analysis research (see further details in section 2). Some of these showed a
positive impact, while others a neutral or negative impact, leading many authors to argue that there is still no conclusive
answer to the question about the impact and effectiveness of ICT in education (Burns & Ungerleider, 2003; Cartwright &
Hammond, 2007; Cox & Marshall, 2007; Kikis, Scheuermann, & Villalba, 2009; Reeves, 2008; Trucano, 2005). Other

1
Paper submitted for peer reviewing — Please do not distribute or cite without permission

authors even argue that the introduction of ICT in schools served to amplify existing forms of inequality (Toyama, 2010,
2011; Warschauer, Knobel, & Stone, 2004).

However, ICT was not devised with education in mind, it did not appear naturally in educational systems, nor was it
demanded by the teaching community (Sunkel, 2006). It responds to a model — with its concepts, discourse and practices —
which originated outside of educational systems (Kikis et al., 2009; Sunkel, 2006) and was then imposed upon them
(Hennessy, Ruthven, & Brindley, 2005), making the integration a highly difficult process. This worsened with the later
emphasis on technology in pedagogy (Trucano, 2005; Watson, 2001) which ignored the extensive existing research on
teaching-learning (Sutherland et al., 2004).

1.1 Problem definition

When ICT was first being introduced in schools, they were considered so ‘new’ that some policy makers and practitioners
ignored theoretical foundations central to all learning with or without ICT (Sutherland et al., 2004). In addition to learning
theories, other design aspects of curriculum integration were not considered, such as the type of ICT resources and how they
are specifically used in a particular subject (Cox & Marshall, 2007; McFarlane, 2001) and the design and curriculum
implementation of ICT use in the classroom (Cox & Abbot, 2004; Cox & Marshall, 2007; McFarlane, 2001).
By studying ICT integration in schools, it was determined that the way in which this takes place depends on the educational
context. As part of this context, the following factors were also identified: pedagogical approaches and beliefs (Coburn, 2003;
Cox & Webb, 2004; Kirkman, 2000; Trucano, 2005), teacher confidence, attitudes and skills relating to ICT; school ICT
infrastructure, supervision and technical support, involvement and leadership of school principals and time spent by teachers
on meetings, training, exercises and lesson planning (Baskin & Williams, 2006; Conlon & Simpson, 2003; Hayes, 2007;
Pelgrum, 2001; Reynolds et al., 2003; Tondeur et al., 2007).

Thus, after 30 years of research, the main lesson to have been learned is that integrating ICT in education is not a technical
matter (Earle, 2002; Trucano, 2005; Watson, 2001). ICT should be the tool with which we deliver content and implement
educational practices in better ways, based on curriculum and learning principles (Lai, 2008; Marshall & Cox, 2008; Reeves,
2008; Roblyer, 2005; Trucano, 2005). Integration is determined not by the quantity or type of ICT used, but by how and why
it is used (Cox & Abbot, 2004; Cox & Webb, 2004; Earle, 2002; Trucano, 2005). Hereafter, when ICT is used to support
teaching and learning activities, we refer to it as technology-enhanced instructional design or Technology-Enhanced
Learning (TEL) (Dillenbourg, 2008; Kozma, 1991; Roblyer, 2005).

The available evidence suggests that the use of TEL with specific educational objectives, together with an activity that is
planned, guided and structured by a mediator, is what makes the difference in terms of learning results (Cox & Abbot, 2004;
Pilkington, 2008; Trucano, 2005). However, even with effective interventions based on TEL, one problem yet to be resolved
is their scalability in the education system. This issue is not exclusive to ICT interventions, and the few existing models of
scalability identify 4 dimensions to consider: depth, sustainability, spread and shift in reform ownership (Coburn, 2003;
Dede, 2006; McDonald et al., 2006). Depth refers to the changes needed in classroom practice, while sustainability refers to
how to maintain these changes over time and under what conditions (Batchelor & Norrish, 2005). Spread (or scalability)
refers to the diffusion of the innovation to large numbers of classrooms and schools, i.e., the conditions needed to replicate or
scale up the project, and the impact of doing so (Batchelor & Norrish, 2005). The shift in reform ownership refers to the
school’s adoption of the programme.

Since ICT is just one of many available options for supporting teaching and learning processes (Earle, 2002), it should have
to compete for funding with other alternatives, in terms of cost-effectiveness or cost-benefit, regardless of whether these
others are based on ICT (Harris, 2009; Margolis, Nussbaum, Rodríguez, & Rosas, 2006). To calculate the total cost, an
exhaustive inventory must be made of the personnel, software, hardware, materials and resources which are needed to
implement a programme. However, such cost parameters are not set out clearly, as is shown by several studies analyzing the
cost-benefit of TEL environments (Bates, 2005; Laurillard, 2007; Perraton et al., 2002; Rumble, 2001; Steinberg, 2004;
Twigg, 2002). Consequently, the cost parameters to be included and quantified should be specified in operational terms, as
well creating a common language for the educational community (Moyle, 2008).

Finally, in terms of effectiveness, the study of the impact of ICT includes a wide range of technologies and models whose use
is not mainly intended to affect student attainment. Therefore, before subsequently evaluating their effects, we should first
identify which models produce effective instructional practices. Furthermore, any such definition must incorporate the topics
of ICT integration in schools, scalability and the total cost of ownership, so that we may establish methods for the systematic
design, implementation and evaluation of said programmes.

2
Paper submitted for peer reviewing — Please do not distribute or cite without permission

1.2 Purpose and structure of this article

In this article, we use the term ‘educational programme’ as a specific set of replicable procedures, materials and professional
development, whose implementation improves student attainment and that are based on particular resources and/or
instructional processes (Slavin, 2008). Using this concept, we propose a framework — an ICT for education (ICT4E)
programme — to outline what we understand by an educational programme based on TEL. The objectives of this framework
are:
1. To define the elements, concepts, definitions and critical variables involved in developing a deep understanding of how
and why TEL works in real educational settings.
2. To establish the relationships between the expected learning results, the teaching and learning practices supported by
TEL, the skills that these require, and the activities necessary to develop such skills. This will facilitate the specification
of input, process and outcome indicators for the monitoring and evaluation scheme which must be an integral part of the
programme (Batchelor & Norrish, 2005; Rodríguez at al., 2010; Wagner et al., 2005).
3. To define which elements must be present in an ICT4E programme in terms of the processes and resources needed to
achieve the desired learning results.

4. To calculate the real total cost of an ICT4E programme, allowing it to be compared with other educational programmes
in terms of cost-effectiveness.

Taking the ICT4E programme framework as a base, it is possible to then elaborate on the design, implementation and
evaluation of its impact.

The rest of this article is structured as follows: in section 2, we present a framework for an ICT4E programme based on the
evidence available in the literature, indicating its main components, their interrelation and a brief discussion about the
definition provided. In section 3, we exemplify the framework, using a specific programme based on computer supported
collaborative learning. In section 4, we discuss the applications of the ICT4E programme that were implemented in Chile.
The article finishes with the conclusions and future work in section 5.

2 Defining an ICT for Education Programme

To outline our framework, we undertook a wide review of the literature on ICT in education in three categories: curriculum
integration, frameworks for the integration of ICT in schools and impact assessment.

In the first category —curriculum integration of ICT— we found different frameworks that conceptualize the integration of
TEL in teaching/learning models (e.g. contents, activities, roles of the teacher and students). For example, curriculum
components (Van den Akker, 2003), domains of educational innovation (Mioduser et al., 2003), learner-centric ecology of
resources (Luckin, 2008), and social infrastructure (Bielaczyc, 2006).

The second category relates to frameworks for the integration of ICT in schools. Here there is an analysis of the factors which
affect the implementation of ICT in educational settings in countries such as Australia (Baskin & Williams, 2006; Hayes,
2007), Belgium (Tondeur, Valcke, & Van Braak, 2008), Netherlands (Mooij & Smeets, 2001), Israel (Nachmias et al., 2004;
Tubin et al., 2003), Malaysia (Cloke & Sharif, 2001), Great Britain (Kirkman, 2000; Tearle, 2003, 2004) and Singapore (Lim,
2002; Lim & Hang, 2003). There are also international studies such as SITESM2 (Kozma, 2003; Pelgrum, 2001) and
SITES2006 (Plomp, Pelgrum, & Law, 2007).

The third category corresponds to impact assessments of ICT on education, be this in terms of a state of the art summary of
the effect of ICT (Burns & Ungerleider, 2003; Cox, 2008; Cox & Abbot, 2004; Cox & Marshall, 2007; Cox & Webb, 2004;
Lai, 2008; Trucano, 2005), considerations for undertaking monitoring and impact assessments (Agodini et al., 2003; Light,
2008; Penuel, 2005; Wagner et al., 2005), reviews of programmes and cases (Bakia et al., 2009; Balanskat, Blamire, &
Kefala, 2006; Culp et al., 2003), meta-analysis research (Kulik, 2003; Liao & Hao, 2008; Waxman, Lin, & Michko, 2003),
studies based on large scale research design (Barrera-Osorio & Linden, 2009; Campuzano, Dynarski, Agodini, & Rall, 2009;
Dynarski et al., 2007) and research methodologies about this topic (Batchelor & Norrish, 2005; Cox, 2008; Marshall & Cox,
2008; Pilkington, 2008; Reeves, 2008).

Through the second category, we identified three main settings which can be classified by the aspects of integration of ICT in
the school system: classroom, school and external. By ‘classroom’, we mean the setting where teaching and/or learning
occurs in the school, even though it might not take place specifically within a classroom (e.g., in a computer lab). ‘School’

3
Paper submitted for peer reviewing — Please do not distribute or cite without permission

defines the specific educational context in which a TEL is integrated, while the ‘external’ setting considers those processes
which are outside of school (e.g. monitoring and evaluation, public policies, costing parameters).

The classification of the aspects according to the previously defined settings is shown in Table 1. In this table we include the
aspects described in the literature that, in our opinion, are the most relevant, some of which are transversal across the
different settings (A5 to A10).
Table 1: Classification of the aspects involved in ICT in education based on the literature, according to the setting in which they occur.

Setting Aspects involved in ICT integration in schools

A1. Concrete improvement goals (Agodini et al., 2003; Cox & A5. Change in existing practices
Marshall, 2007; Light, 2008; Penuel, 2005; Trucano, 2005) both in (Bielaczyc, 2006; Coburn, 2003;
the immediate short term and in the longer term for the Hayes, 2007; Luckin, 2008; W.
stakeholders and beneficiaries (Batchelor & Norrish, 2005) Penuel, 2005)
Classroom

A2. Support in learning theories and models for teaching (Cox & A6. Role of teachers, students and other
Marshall, 2007; Lai, 2008; Marshall & Cox, 2008; McFarlane, actors inside and outside school
2001; Penuel, 2005; Reeves, 2008; Roblyer, 2005) (Bielaczyc, 2006; Lim, 2002; Lim &
A3. Materials and resources (Luckin, 2008; Nachmias et al., 2004; Hang, 2003; Mioduser et al., 2003;
Plomp et al., 2007; Slavin, 2008; Van den Akker, 2003) Nachmias et al., 2004; Van den Akker,
Inside school

A4. Configuration in which students learn (e.g. individual/group, face- 2003)

to-face/online/blended) (Bielaczyc, 2006; Mioduser et al., 2003; A7. Costing parameters (Bates, 2005;
Plomp et al., 2007; Van den Akker, 2003) Laurillard, 2007; Perraton et al., 2002;
A11. Teacher professional development (Nachmias et al., 2004; Rumble, 2001; Steinberg, 2004;
Pelgrum, 2001; Tearle, 2004; Trucano, 2005), Twigg, 2002)
A12. Educational context where factors helped or hindered its impact A8. Processes that led to the outcomes
(Batchelor & Norrish, 2005; Bielaczyc, 2006; Cloke & Sharif, (Batchelor & Norrish, 2005; Slavin,
2001; Coburn, 2003; Marshall & Cox, 2008; McDonald et al., 2008)
School

2006; Nachmias et al., 2004; W. Penuel, 2005; Reeves, 2008; A9. Monitoring and evaluation scheme to
Tearle, 2004) . measure the differences between
A13. Sustainability (Batchelor & Norrish, 2005; Coburn, 2003; Dede, design and implementation (Batchelor
2006; McDonald et al., 2006) & Norrish, 2005; Reeves, 2008;
A14. Shift in reform ownership (Coburn, 2003; Dede, 2006; McDonald Rodríguez et al., 2010; Trucano, 2005;
et al., 2006; Rodríguez et al., 2010) Wagner et al., 2005)
A15. Scalability (Batchelor & Norrish, 2005; Coburn, 2003; Dede, A10. Total Cost of Ownership (TCO)
External 2006; McDonald et al., 2006) (Moyle, 2008; Scrimshaw, 2002)

Thus, we will define an ICT4E programme as an educational programme which integrates the use of a specific TEL
environment into teaching and learning practices. Such TEL environments should modify teaching and learning process in
ways that could not be achieved without ICT support. We will call the model for the use and integration of the TEL in the
teaching/learning process the pedagogic model.

We specify an ICT4E programme by identifying the processes and the relationship between them in the integration of the
pedagogic model in a school. We assign four attributes to each process: setting, time, aim and outcomes. Setting refers to
where the process operates (i.e., classroom, school or external), while time refers to its duration. The third attribute is the
main aim of the process, which can be classified as one of the following:

1. Implementation: the use of the pedagogic model by the school team.

2. Intervention: the development of skills and practices in school staff. This is normally done by an external team which
carries out training, ensuring the gradual adoption of the pedagogic model and accuracy of its implementation, using a
monitoring and evaluation scheme.
3. Transference: the professional development of the external team so that the intervention can be accurately carried out on
a massive scale.
The fourth attribute is the outcomes, which are the expected skills and/or practices to be developed in the educational actors
(i.e., students, teachers, ICT and school staff). Through the four attributes of the process (setting, time, aim and outcomes), a
causal relationship between what happens in the different settings must be defined. Therefore, in each process — starting
with a set of pre-existing skills and practices amongst the educational actors — new skills or practices are developed as
outcomes. For example, in the implementation, the outcomes are the learning results which the ICT4E programme promotes
amongst the students. In the intervention, the outcomes are the skills and practices which the teachers need in order to

4
Paper submitted for peer reviewing — Please do not distribute or cite without permission

perform implementation. Finally, in the transference, the outcomes are expressed in terms of the skills and practices that the
external team requires to carry out an effective intervention.

In sections 2.1 to 2.4, we will describe the implementation, intervention, transference and calculation of total costs, and how
these are grounded in the aspects of the literature identified in Table 1, to which we will refer with the respective numeration.

2.1 Implementation
This is a set of methodological strategies supported by a TEL environment, whose adoption within a school implies a change
or broadening of the existing practices (A5) and the development of new skills amongst the educational actors, with specific
roles defined by these strategies (A6). The strategies must consider explicit improvement goals (A1), defined within an
educational context where they operate (A12), based on teaching and learning theories and models (A2) and using specific
resources (A3) in a determined configuration (A4). The pedagogic model supported by TEL is defined by 5 elements:

• Purpose (A1): particular subjects, levels and specific curriculum contents.

• Target schools (A12): schools on which the programme is focused, expressed in terms of geographic surroundings
(urban or rural), social vulnerability of its students and their level of attainment (e.g. measured by local or national
standardized tests). Other defining factors could be the type and number of students and school organization (e.g.
multigrade classrooms).

• Implementation resources (A3): all the resources and supporting materials used by the educational actors (e.g. software,
hardware, physical infrastructure, digital and/or non-digital materials, class scripts for teachers and study guides for
students).

• Pedagogic model (A2 to A6): describes how the TEL environment is integrated and supports teaching and learning
processes. Its main elements are:

o How they learn: how teaching-learning process are modified (e.g. preparation of a class and its development); how
the relationships between actors (e.g. teacher-student, student-student) inside and outside the classroom are affected;
with which configuration (e.g. individual, group) they learn; and how implementation resources are integrated into
existing ones.
o Assessment of learning results: how expected effects are evaluated.
o Who participates within or outside the school (e.g. teachers, classmates, parents and family), their roles, and the
skills and practices which they need to develop.
o Specific didactic context(s): where (e.g., classroom, computer laboratory or home) and with whom (i.e., classmates,
parents, online tutors) they are used.
These elements can be described using the terminology of existing integration frameworks, such as curriculum
components (Van den Akker, 2003), domains of educational innovation (Mioduser et al., 2003), learner-centric ecology
of resources (Luckin, 2008), and social infrastructure (Bielaczyc, 2006).
• Implementation outcomes (A1): are the expected effects expressed in terms such as learning results (e.g., scores in
standardized tests, learning of specific curricular contents, acquisition of 21st century skills), personalized learning
environments, or data collection for monitoring student progress.

2.2 Intervention
So that teachers and students develop the necessary autonomy for sustainable use of the pedagogic model (A14) and
according to the educational context where they take place (A12), the external team will carry out professional development
activities (e.g. teacher training, practical experiences, classroom observation) according to a work plan. The outcomes of such
activities are the skills and practices needed by the different educational actors (A8, A11), which are specific to the proposed
pedagogic model, to achieve the learning results set out by the ICT4E (A1). The intervention is defined by 5 elements:

• Work plan (A8): activities and personnel involved (educational actors and/or external team) with the number of person-
hours and professional profile required according to the design of the intervention.

5
Paper submitted for peer reviewing — Please do not distribute or cite without permission

• Monitoring and evaluation scheme (A9, A13, A14): its objectives are: 1) to measure the fidelity of the implementation;
2) to evaluate the outcomes; and 3) to deliver information for decision making during the intervention, especially so as to
take remedial action. The fidelity of the implementation defines metrics of when and how the new teaching and learning
practices should be manifested in the classroom. This implies a diagnostic of the necessary pre-existing skills and
practices amongst the actors (which take on the programme in each setting), as well as following the evolution of the
programme’s adoption over time (A13). This will allow for study of the relationship between the variability of the
implementation, the achieved learning results and the adoption by teachers (A14).

• Intervention resources (A3): all the resources and supporting materials (e.g. documentation, software, hardware,
observation guidelines and software manuals) used by the external team.

• External team (A15): those who carry out the intervention with their respective role (e.g. a distance tutor or field
professional) and professional profile (e.g. studies, expected or desirable work experience, prior skills).

• Intervention outcomes (A8, A11): skills and practices of the school’s educational actors to implement the pedagogic
model.

2.3 Transference
The transference is the professional development and training of the external team so that the intervention may be applied
autonomously, adapting it according to the context of the specific school, following effectiveness standards defined by the
designers of the ICT4E programmes (transference outcomes). For each of the external team roles, the previous background
must be specified, as well as the new skills and practices required to perform the role (e.g. how to apply the pedagogic model,
carry out intervention activities and apply the assessment instruments from the monitoring and evaluation scheme). The
transference also includes the evaluation of the fidelity of the intervention carried out by the external team (A15).

2.4 Total cost

The total cost of the ICT4E programme (A7, A10) refers to the cost of implementation, intervention and transference for the
duration of the programme.

• Implementation cost: is the sum of the marginal costs which the school incurs for participating in the programme: the
person-hours of the educational actors, resources (including software), equipment (including hardware), infrastructure,
materials and others (e.g. administration costs).
• Intervention cost: comprises the fixed and variable costs of the external team. The variable costs depend on the number
of person-hours which the external team spends on the intervention (including associated materials, expenses and
resources), while the fixed costs refer to the general and administrative costs (e.g. management personnel, physical
infrastructure) which must be divided pro-rata according to the number of schools to be serviced and the duration of the
programme (Margolis et al., 2006).
• Transference cost: is the set-up cost of the external team, which is divided pro-rata amongst the participating schools, as
are the fixed costs of the external team.
With all of the above, the total cost of the programme is calculated. It can be expressed in different units such as average cost
per school, per classroom and per participating student, depending on the comparison to be performed.
2.5 Discussion of the proposed framework
In the proposed framework, the processes and settings defined in sections 2.1 to 2.4, cover aspects A1 to A15 identified in the
literature as relevant (Table 1). The ICT4E programme model is outlined in Figure 1.

6
Paper submitted for peer reviewing — Please do not distribute or cite without permission

• Purpose
• Target schools
• Implementation resources
• Pedagogic model
Outcomes:
Learning results

Outcomes: Implementation
Skills & practices
S t udents

E xternal
t e am

Settings • Professional
development
& training
T ransference

Intervention
• Work plan
• Monitoring &
evaluation scheme
• Intervention resources
• External team
T e achers,
ICT & S chool staff O utcomes:
Skills & practices

Figure 1: Process involved in the ICT4E programme framework.

In Figure 1, the block arrows represent the three processes from the ICT4E programme in its corresponding setting,
producing outcomes once each one has been carried out: learning results in students and the respective skills and practices in
the ICT and school staff and the external team. Figure 1 also outlines the relations between the main roles of the ICT4E
programme’s three processes: firstly, the teachers perform the implementation. Secondly, members of the external team and
the ICT and school staff support each other mutually as they carry out the intervention, while the external team finally
monitors and evaluates the implementation. Next, we will discuss how our framework meets the objectives set out in section
1.2.

While the design of the TEL supported pedagogic model can be carried out together with the final users through design
research methodologies (Bielaczyc, 2006; Rodríguez, 2010; Van den Akker et al., 2006), it is very unlikely that the schools
will design, implement and scale-up educational technologies, because is not their role. Thus, it is more probable that an
ICT4E programme take the form of an externally developed school reform, which typically involves a short-term influx of
resources, making it especially vulnerable to sustainability problems, because assistance to facilitate implementation
dissipates over time (Coburn, 2003). To address this problem, the ICT4E programme has at its centre the definition of skills
and practices needed by the actors in each process, as well as the monitoring and evaluation of its adoption: from the teachers
and the school taking control of the programme, to the external team responsible for the intervention.

Our framework establishes an order of precedence for the processes, which starts with the selection and training of the
external team and which concludes with the measurement of student learning results. However, all of these professional
development processes require a prior and explicit definition of the activities in the implementation and intervention. For this
reason, the design of an ICT4E programme must be carried out in reverse order, starting with the expected learning results
and the pedagogic model and finishing with the external team.

With respect to the scalability of the ICT4E programme, the issue of depth is core to the pedagogic model. The sustainability
and shift in reform ownership are the main aims of the monitoring and evaluation scheme from the intervention component,
while the spread is the main concern of the transference component. The ICT4E programme assigns the responsibilities of
these last two dimensions to the external team, which involves designing strategies to achieve these goals right from the
design of the ICT4E programme.

Regarding the costing parameters, as well as being consistent with those previously identified in studies of the costs of TEL
(Bates, 2005; Laurillard, 2007; Perraton et al., 2002; Rumble, 2001; Steinberg, 2004; Twigg, 2002), our framework
incorporates another two which have not previously been identified: the cost associated with monitoring and evaluation, and
the cost of the transference. Our framework does not consider the programme’s design costs because these are considered as

7
Paper submitted for peer reviewing — Please do not distribute or cite without permission

sunk costs. We also assume that the evaluation of the design and effectiveness have already been resolved. This topic is
discussed extensively by Rodriguez et al. (2011) through the Evolutionary Development model for ICT4E programmes.

3 Describing a specific ICT4E programme: the Eduinnova case

In this section we will illustrate how the proposed framework (Figure 1) allows us to specify the intervention and transference
by identifying the skills and practices required by the actors involved for the implementation of the pedagogic model, as
established by the ICT4E programme. This will be exemplified with Eduinnova, an ICT4E programme which uses mobile
technology inside the classroom to support collaborative learning with mobile devices such as laptops, netbooks or Personal
Digital Assistants (PDAs). These devices are used with a 1:1 computing model (Penuel, 2006), but shared by various
classrooms throughout the day.

3.1 Implementation
The purpose of Eduinnova is to support student attainment through Computer Supported Collaborative Learning (CSCL)
(Dillenbourg, 1999), constructivist (Zurita & Nussbaum, 2004b) and collaborative learning approaches (Zurita & Nussbaum,
2004a). Currently, the programme covers the contents of the curriculum for Mathematics and Spanish Language (third to
eighth grade), Science and Social Studies (third and fourth grade) in elementary education; and for Mathematics, Spanish
Language, Physics, Chemistry, Biology, History and Geography in secondary education.

The target schools are elementary and/or secondary urban state-subsidized schools, from lower-middle socioeconomic strata.
In terms of infrastructure, to be run adequately, Eduinnova requires classrooms with at least one power outlet and teacher
access to a computer with a broadband connection.

The implementation resources are: the Eduinnova TEL environment (software for laptop or PDA), a laptop (or PDA)
equipped with wireless communication (Wi-Fi) for each student and teacher; a wireless router which can support at least 45
connections to be used in the classroom; and a cabinet for storing, transporting and charging the devices’ batteries. In
addition, an online library is provided, that: 1) contains CSCL activities previously located by experts within a curriculum
framework, but customizable by teachers in schools; 2) registers the teachers’ use of the technology; and 3) stores the results
of the activities undertaken in class. All of these resources are used to complement the traditional classroom materials (e.g.
books and concrete material).

Next, we will describe the elements of the pedagogic model of Eduinnova for netbooks/PDAs, to be implemented in a school.
Firstly, the way in which the students learn is outlined in Figure 2.

Online
library
CSCL content
generation

e b CSCL activity wireless delivery

Data backing-up for & groups arrangement
analysis

d c

Real-time supervision
of teacher Collaborative work
in groups

Figure 2: Using Eduinnova TEL environment in school

8
Paper submitted for peer reviewing — Please do not distribute or cite without permission

The teacher accesses an online library in which they can search, create and download CSCL activities for their class (Figure
2, step a). The PDAs/netbooks are taken to the classroom in the storage cabinet. In the classroom, using the Eduinnova TEL
environment, the teacher chooses and sends the CSCL activity to their students (Figure 3, step b) and the class is randomly
divided into groups of three (Figure 2, step c). In each group, the students must solve a problem together, with the aid of
traditional materials (e.g. books, notes), while their interaction is mediated by the TEL environment (Figure 3a). In addition,
the software also gives them feedback on the answer and does not allow the students to progress until they arrive at the
solution to the given problem (Cortez et al., 2005; Nussbaum et al., 2009; Rodríguez et al., 2010)
2010).. At the same time, the TEL
environment delivers reports to the teacher in real time about the performance of the students, giving them the opportunity to
mediate the work done by each group (Figure 2, step d and Figure 3b). This information is also backed up on the web for
later use (Figure 2, step e).

(a) General panorama of the classroom (b) Students work together with the help of the teacher
Figure 3: Snapshots of a class which is supported by the collaborative teaching method Eduinnova

Secondly, in the pedagogic model, the assessment of the learning results considers the formative and summative stages. In the
formative stage, the teacher gives feedback to specific groups or to the whole class during the collaborative work. In the
summative assessment stage, the students answer online individual tests in real-time, supported by the TEL environment. The
teacher can also evaluate student learning by administering their own tests.

Thirdly, the roles, skills and practices and educational actors that participate in the implementation are shown in Table 2.
Table 2: Roles, skills & practices of the educational actors in the Eduinnova pedagogic model
Who Roles Skills & practices
Collaborative learning, based on 4 factors (Cortez et al., • Individual responsibility (S1).
2005):
• Mutual support (S2).
• Individual responsibility: each student is in charge of
• Positive interdependence (S3).
their own work and effort to learn.
• Social face-to-face interaction (S4).
• Mutual support: teaching the other members of the
Content learning in specific subject (S5).
Students

group •
• Positive interdependence: collaboration is only
successful when every student has interacted and
accomplished their individual goals to solve the group
problem.
• Social face-to-face interaction: the decision– making
process relies on the ability to efficiently exchange
opinions, negotiate and construct an answer together.

9
Paper submitted for peer reviewing — Please do not distribute or cite without permission

• From ‘expert’ to facilitator, mediator and guide (Condie • Technological handling of the Eduinnova TEL (S6).
& Munro, 2007), supporting, monitoring and providing • Mediate students’ group work inside the classroom and
feedback when students work collaboratively. monitor their outcomes (S7).
• Integrate mobile CSCL activities into a lecture in Give continuous feedback based on the results obtained
Teacher s

•
accordance with the subject, objectives, timing and (S8).
resources (digital or non-digital) involved. • Integrate CSCL methodology into their own teaching
• Create new activities, select or customize existing styles and other activities and resources from their
ones from the online library. subject (S9).
• Shares materials, knowledge and experience of • Adapt digital resources from the programme to suit their
collaborative teaching practices with other teachers. own needs (S10).
• ICT staff: transport, recharging devices and solving • Technical support for the teacher (S11).
ICT & School staff:

technical issues inside the classroom. • Manage the devices in order to guarantee and maximize
• School staff: planning and monitoring the curriculum their access and use (S12).
and progress of the students. • Carry out monitoring control and pedagogical support to
teacher when using the programme (S13).
• Plan the use of technology, consistent with institutional
interests and the pedagogical needs which are detected
(S14).

To finish the description of the pedagogic model, the classroom is the specific didactic context where this TEL environment
is used, with the students from that class and during the school timetable, for the specific subject in which it is applied.

The implementation outcome, which is expected to be achieved following the adoption of the programme, is an effect size
above 0.25 in the subjects in which it is applied. This is according to the evidence available on the impact of ICT on learning
(Agodini et al., 2003; Dynarski et al., 2007; Liao & Hao, 2008).

In sections 3.2 to 3.4, we will describe how skills and practices are developed during the intervention and transference, as
well as how its adoption is monitored and evaluated.

3.2 Intervention
The objectives of the intervention in Eduinnova are: 1) to develop the skills and practices S1 to S14, 2) to measure its
adoption through the systematic application of assessment instruments which form part of the monitoring and evaluation
scheme, and 3) to take remedial actions in case of excessively large gaps in relation to the expected evolution.

Eduinnova shows that, although the ICT4E programme framework sets an order of precedence between implementation and
intervention, these do not have to be strictly sequential. The intervention in Eduinnova consists of an implementation with
coaching inside the classroom which looks to develop a hands-on experience amongst the teachers, with a formative and
summative evaluation of their activities, in order to develop skills and practices S6 to S10. Thus, the implementation and
intervention overlap to support the teachers in the integration of TEL in their classes.

The work plan of the intervention takes place over 2 years and is divided into 4 main phases and 12 activities, as detailed in
Table 3, which also indicates the skills and practices to be developed during each phase.
Table 3: Work plan for Eduinnova programme (Rodríguez et al., 2010)
Duration
Year Phase Main activities and skills and practices to be acquired
(weeks)
1. Introduction of intervention to school staff
2. Supporting adaptation of the pedagogic model to school context and objectives for
1. Introduction ICT integration: subjects and grades: S13 and S14 3
3. Assessment of students’ attainment in achieving school objectives (pre-test): S1 to
S5, S14
pedagogic model :
On e

support of school
remote technical

8. Monitoring and

4. Training teachers and ICT staff: S6 to S8,S10, S11

ICT staff : S6
7. On-site and

5. Coaching teachers inside the classroom, providing them with

adoption of

2. Training and
feedback about the pedagogic model and ICT integration: S6 14
S6 to S9

coaching
to S10, S13
9. Remedial actions phase 2: S6 to S10

10
Paper submitted for peer reviewing — Please do not distribute or cite without permission
Duration
Year Phase Main activities and skills and practices to be acquired
(weeks)

10. Remedial actions phase 3: S6 to S14

3. Remote
11. Assessment of students’ attainment in achieving school 16
supervision
objectives (post-test): S1 to S5, S14

4. Autonomous 12. Assessment of students’ attainment in achieving school

Two

36
implementation objectives (pre & post tests): S1 to S5, S14

The key phase of the plan is Phase 2 (“Training and coaching”), where the teachers learn to use the TEL environment, both
pedagogical and technological, to carry out the practices of the programme (S6 to S10). It is expected that in this phase the
pedagogic model will be adopted, verified by a participant observation of the teachers in their first class using the TEL
environment (activity 5). In this observation, any doubts are cleared up and the work done during the face-to-face training is
reinforced (activity 4). According to the results of this observation, additional support is given to those teachers that have
adopted the pedagogic model to a lesser extent. Subsequently, the level of adoption is only measured in the day-to-day
activity of the teachers with non-participant observation, giving them individual feedback in writing and detailing their main
strengths as well as giving specific suggestions (activity 6). When necessary, remedial actions are taken (activity 9). The
other activities from Phase 2 are: giving technical support to schools (activity 7) and monitoring and evaluating the
programme (activity 8). Skills and practices S11 to S14 are acquired by working with other educational actors within the
school.
The monitoring and evaluation scheme has four adoption indicators: teachers’ technical skills (S6), pedagogical skills in
collaborative learning (S7 and S8), ICT integration within the curriculum (S9 and S10) and ICT management inside the
classroom (S6 plus efficiency of use of classroom time). The assessment instruments are observation protocols based on
previously validated rubrics. Studies indicate that if these indicators do not reach a sufficient final threshold, statistically
significant differences are produced in the learning amongst students whose teachers have different levels of adoption
(Rodríguez et al., 2010). For further details and discussion of this monitoring and evaluation scheme, please refer to
Rodriguez et al. (2010).
The most important intervention resources are the assessment instruments of the adoption of the practices, for example, the
previously mentioned observation protocols (activity 5).

The external team performs, mediates and supports the intervention. The roles are: project-coordinator professionals
who coach teachers inside the classroom (pedagogical supporters) and technicians who support school ICT staff
(technical supporters). The professional profiles of each are the following:
• Project coordinator: educational professional (e.g. teacher or educational psychologist) with more than 3 years of
experience and ICT skills at complex tools level, as established in UNESCO (2008) ICT competency framework for
teachers.
• Pedagogical supporter: teacher with more than 2 years of experience and knowledge of the internal organization of
the schools according to the country where the programme is implemented, and ICT skills at complex tools level.
• Technical supporter: programmer or systems analyst.
In the case of in-classroom support and technical support, each service a certain number of schools depending on the phase of
intervention in which each school can be found, thus, the organizational structure of the external team depends on the number
of participating schools.

The intervention outcomes are the skills and practices S1 to S14, defined in section 3.1 (Table 2).

3.3 Transference
The transference is the professional development and training of the external team in three areas: implementation,
intervention and use of technology applied in the programme at advance level. They are also trained in the application of
assessment instruments from the monitoring and evaluation scheme, so as to guarantee the quality and reproducibility of the
programme’s implementation.

11
Paper submitted for peer reviewing — Please do not distribute or cite without permission

3.4 Total cost

The calculation of the total cost is exemplified by calculating the intervention cost. For this, we will take activity 5
(“Coaching teachers…”) mentioned in section 3.2 (Table 3). Activity 5 is divided into two activities. In the first (“5.1
Observation of each teacher’s first class”) the teacher’s work is observed and supported, reinforcing previously acquired
skills and following an observation guideline after the class. In the second activity (“5.2 Teacher observation…”) the teachers
that show most signs of difficulty are further supported. If each observation lasts approximately 1.5 hours, and assuming that
a maximum of 15 teachers are used in each school with an average rate of 20% of teachers showing a low level of adoption
(of previous experiences), 22.5 and 4.5 person-hours are used for activities 5.1 and 5.2, respectively. Considering in Chile a
cost of US$9/person-hour (pedagogical supporter profile), material costs (US$7) and transport (US$10), we arrive at a total
cost of US$260 per school for this activity, given that it is only carried out during the first year (see Table 3).

Similarly, the variable costs for the remaining activities are calculated. The fixed costs of the intervention and transference
are calculated according the scale of the schools where it is to be implemented.

4 Current applications of ICT4E programme

The ICT4E programme framework is used as a specification for the ICT curriculum integration strategies supported by
Enlaces (ww.enlaces.cl) the educational technology agency from the Chilean Ministry of Education (Rodríguez, 2008). All
programmes that apply for funding from said agency must adjust themselves to this framework. To determine if a programme
fits the specification, an evaluation standard is applied (Román, 2010). This standard uses the Context Input Processes
Products (CIPP) methodology of analysis (Stufflebeam & Shinkfield, 2007) to define the indicators for the quality of the
design, processes and impact of an ICT4E programme.

When a programme passes the evaluation, it is included in the catalogue of digital educational resources promoted by Enlaces
for all schools in Chile (www.catalogored.cl). In addition to Eduinnova, there are other ICT4E programmes currently
included in said catalogue, such as those for 1:1 learning (Lagos et al., 2009; López et al., 2009), project-based learning
(Sánchez & Alarcón, 2003), Mathematics (Díaz, 2010), Spanish Language (Basáez, et al. 2010), Science (Miranda & Feris,
2009; Moenne, 2008, 2010) and Civic Education (Contreras, 2008).

Additionally, the ICT4E programme framework allows for a more accurate estimation of the total cost of integrating ICT,
because it helps to determine all of the materials and resources which the programme needs. Such estimations enable an
analysis of the economic feasibility of the programme, determining whether or not it can be implemented given the financial
restrictions on school and Ministry of Education spending (Margolis et al., 2006). Another possible application is to compare
programmes using a cost-effectiveness analysis (Creemers & van der Werf, 2000; Feinstein & Picciotto, 2001; Harris, 2009;
Margolis et al., 2006; Tan et al., 2001). The ICT4E programme model offers a cost comparison framework, standardizing the
process components and resource prices of the programmes under comparison. For example, the definition of the profiles for
the external team facilitates the comparison between programmes as professionals with similar training and experience
should have similar costs within the same country.

The ICT4E programme supported the design of the Eduinnova programme which, in the last decade, has been tested in
different grade levels and with different subjects: preschool education (Infante et al., 2009), elementary schools (Zurita &
Nussbaum, 2004), secondary schools (Rodríguez et al., 2010) and university level (Bustos & Nussbaum, 2009; Valdivia &
Nussbaum, 2007, 2009). The transference component allowed Eduinnova to also be implemented in schools in Argentina,
Brazil, Guatemala, Great Britain (Nussbaum et al., 2009) and the United States (Roschelle et al., 2010). The effect sizes
found in quasi-experimental studies fluctuate between 0.32 and 0.41 (Rodríguez et al., 2011, 2010), consistent with the
expected learning results for an educational technology (section 3.1).

Finally, another nonetheless important application was the development of a model for the design, implementation and
evaluation of ICT4E programmes. Said model — called Evolutionary Development — builds each component of the ICT4E
programme in stages and validates it in real educational settings through formative and summative evaluations based on
design research methodologies and using experimental or quasi-experimental studies (Rodríguez et al., 2011). Thus, the
efficacy of the implementation, the effectiveness of intervention and the efficiency of the programme as a whole are tested
empirically.

12
Paper submitted for peer reviewing — Please do not distribute or cite without permission

5 Conclusions and future work

In this article, we proposed the ICT4E programme framework to respond to the question about what an educational
programme supported by ICT should look like (section 2). We centred the framework on producing a sustainable change in
time through the effective adoption of new teaching and learning practices supported by TEL environments. The key
components of an ICT4E programme are: the implementation of the TEL supported pedagogic model, intervention,
transference and total costs. The transference component comprises the formation strategies of an external team to guarantee
the replicability and quality of the intervention.

The proposed framework (objective 1 of the article) makes explicit the relationships between each of the activities and the
expected outcomes (at an intermediate and final level) in each process of the ICT4E programme (objective 2). Secondly, it
allows educators to identify which elements (e.g. skills and practices, actors involved, activities, resources) can be absent
(objective 3). Finally, it enables the calculation of the real total cost of the programme (objective 4). These features were
illustrated by the Eduinnova programme in section 3.

Currently, in Chile, the ICT4E programme model is applied to specify ICT projects in education subject to public funding
and evaluate them according to a standard built using the framework as a base (Román, 2010). Other applications are the
analysis of economic feasibility and cost-effectiveness (Margolis et al., 2006). This last case presents practical challenges as
the effectiveness must be measured under criteria from rigorous evidence (Slavin, 2008). Furthermore, only little is known
about how the effectiveness and costs of an ICT4E programme vary with time, especially with respect to long term
substitution of obsolete technology.

Thus, a future challenge for policy making is to determine how the cost-effectiveness varies in the medium and long term,
and how such changes influence the current comparison between programmes. To resolve this problem, some authors
propose the calculation of a present value both of the effect size as well as of the costs, which would imply making
suppositions with respect to the discount rate to be applied and the time horizon for the analysis (Harris, 2009). One empirical
solution would be to carry out longitudinal studies of the population effected by ICT4E programmes to determine if the
effects are sustainable in time, and the rate at which they eventually decline in the long term. Transversal studies every
certain amount of time would verify whether or not an ICT4E programme systematically produces the expected outcomes. In
case it should not, the ICT4E programme model will allow an audit as to which aspects are loss-making and to what extent.

Another form of comparing educational programmes is through cost-benefit studies, which are harder to carry out because
the benefits must be monetarily quantified (Margolis et al., 2006; Tan et al., 2001). However, Laurillard (2007) proposed a
prospective method which allows the identification of specific benefits of a TEL environment. Our ICT4E programme
supports the implementation of said method, as it facilitates the calculation of the costs and also specifies the intermediary
and final outcomes which should be achieved. Thus, after valuing the identified costs and benefits, it would be possible to
calculate the net present value and return of investment of implementing an ICT4E programme in a school or district. With
these indexes policy makers have more quantitative information with which to compare other educational alternatives (with
or without the technology) being available to them at the time of evaluating.

The specification of processes and calculation of the costs that the ICT4E programme enables, also allows the application of
process improvement and cost reduction models — such as Capability Maturity Model Integration (Forrester, Buteau, &
Shrum, 2011) — in order to improve the cost-effectiveness of the programme.

Another aspect to be determined is whether or not there exist significant differences in the impact of ICT4E programmes
which adjust more to the framework, in comparison to those that do it to a lesser degree, a topic which is the subject of
research currently being conducted.

Finally, there are at least two issues for further exploration. Firstly, the proposed framework could be extended to include
other aspects of ICT in education by substituting the pedagogic model with one designed to support such processes (e.g.,
school administration and learning management models). Secondly, our ICT4E programme framework has the potential to
specify less teacher centred models, like lifelong learning or distance education, by redefining the concept of ‘settings’ within
the framework. For example, the ‘classroom’ setting could be extended to other places outside the school such as, the home
or office.

Acknowledgements― This research was funded by the Chilean Ministry of Education through the Enlaces Center for
Educational Technology, and CONICYT under grants FONDECYT 11110424, CIE-05 CIAE and CIE-01 CEPPE. Special
thanks go to Claudia Morchio and Goryet Pandorfa from Enlaces, and Eugenio Severin from Inter-American Development

13
Paper submitted for peer reviewing — Please do not distribute or cite without permission

Bank. This paper is dedicated to the memory of Dr. Gerardo Moenne, from the Instituto de Informática Educativa,
Universidad de la Frontera.

References
Agodini, R., Dynarski, M., Honey, M., & Levin, D. (2003). The Effectiveness of Educational Technology: Issues and
Recommendations for the National Study. Mathematica Policy Research, Inc. Recuperado a partir de
http://www.mathematica-mpr.com/PDFs/edtechrec.pdf
Bakia, M., Means, B., Gallagher, L., Chen, E., & Jones, K. (2009). Evaluation of the Enhancing Education Through
Technology Program: Final Report. U.S. Department of Education, Office of Planning, Evaluation and Policy Development,
Policy and Program Studies Service. Washington, D.C. Recuperado a partir de
http://www2.ed.gov/rschstat/eval/tech/netts/finalreport.pdf
Balanskat, A., Blamire, R., & Kefala, S. (2006). The ICT Impact Report: A review of studies of ICT impact on schools in
Europe. Recuperado a partir de http://ec.europa.eu/education/pdf/doc254_en.pdf
Barrera-Osorio, F., & Linden, L. (2009). The Use and Misuse of Computers in Education: Evidence from a Randomized
Controlled Trial of a Language Arts Program. Recuperado a partir de http://www.columbia.edu/~ll2240/Barrera-
Linden%20Computadores_2009-03-25.pdf
Basáez, C., Bastías, V., Villalobos, R., & Cafena, D. (2010). Piloto @escribo, usando las TIC para potenciar la escritura:
Informe final. Recuperado Abril 22, 2011, a partir de
http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=3709&org=11413
Baskin, C., & Williams, M. (2006). ICT integration in schools: Where are we now and what comes next? Australasian
Journal of Educational Technology, 22(4), 455-473.
Batchelor, S., & Norrish, P. (2005). Framework for the Assessment of ICT Pilot Projects. InfoDev. Recuperado a partir de
http://www.infodev.org/en/Publication.4.html
Bates, A. (2005). Technology, e-learning and distance education (2o ed.). London: Routledge.
Bielaczyc, K. (2006). Designing Social Infrastructure: Critical Issues in Creating Learning Environments with Technology.
Journal of the Learning Sciences, 15(3), 301-329.
Burns, T. C., & Ungerleider, C. S. (2003). Information and Communication Technologies in Elementary and Secondary
Education. International Journal Educational Policy, Research and Practice, 3(4), 27-54.
Bustos, H., & Nussbaum, M. (2009). An experimental study of the inclusion of technology in higher education. Computer
Applications in Engineering Education, 17(1), 100-107.
Callister, T. A. (1992). The computer as doorstep: Technology as disempowerment. Phi Delta Kappan, 74(4), 324–329.
Campuzano, L., Dynarski, M., Agodini, R., & Rall, K. (2009). Effectiveness of Reading and Mathematics Software Products:
Findings from Two Student Cohorts. Recuperado a partir de http://ies.ed.gov/ncee/pubs/20094041/
Cartwright, V., & Hammond, M. (2007). «Fitting it in»: A study exploring ICT use in a UK primary school. Australasian
Journal of Educational Technology, 23(3), 390-407.
Castro, C. (2004). Are new technologies better technologies? For whom? In D.W. Chapman & L.O Mählck (eds.), Adapting
technology for school improvement: a global perspective (págs. 39-54). International Institute for Educational Planning -
Unesco.
Cloke, C., & Sharif, S. (2001). Why use information and communications technology? Some theoretical and practical issues.
Technology, Pedagogy and Education, 10(1), 7-18.
Coburn, C. E. (2003). Rethinking Scale: Moving Beyond Numbers to Deep and Lasting Change. Educational Researcher,
32(6), 3 -12.
Condie, R., & Munro, B. (2007). The Impact of ICT in schools a landscape review. BECTA Research.
Conlon, T., & Simpson, M. (2003). Silicon Valley versus Silicon Glen: the impact of computers upon teaching and learning:
a comparative study. British Journal of Educational Technology, 34(2), 137-150.
Contreras, David. (2008). [email protected], alfabetización digital para ciudadanos en red: Informe final. Recuperado Febrero 3,
2011, a partir de http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=2296&org=11413

14
Paper submitted for peer reviewing — Please do not distribute or cite without permission

Cortez, C., Nussbaum, M., López, X., Rodríguez, P., Santelices, R., Rosas, R., & Marianov, V. (2005). Teachers’ support
with ad-hoc collaborative networks. Journal of Computer Assisted Learning, 21(3), 171-180.
Cox, M. (2008). Research IT in education. In J. Voogt. G. Knezek (Eds.), International Handbook of Information Technology
in Primary and Secondary Education (págs. 965-981).
Cox, M., & Abbot, C. (2004). ICT and attainment: A review of the research literature. ICT in Schools Research and
Evaluation Series – No.17, DfES-Becta. Recuperado a partir de
http://publications.teachernet.gov.uk/eOrderingDownload/ICT and attainment.pdf
Cox, M., & Marshall, G. (2007). Effects of ICT: Do we know what we should know? Education and Information
Technologies, 12(2), 59-70.
Cox, M., & Webb, M. (2004). ICT and pedagogy: A review of the research literature. ICT in Schools Research and
Evaluation Series – No.18, DfES- Becta. Recuperado a partir de http://publications.becta.org.uk/display.cfm?resID=25813
Creemers, B., & van der Werf, G. (2000). Economic Viewpoints in Educational Effectiveness: Cost-effectiveness Analysis of
an Educational Improvement Project. School Effectiveness and School Improvement, 11(3), 361-384.
Cuban, L. (1986). Teachers and Machines: The Classroom Use of Technology Since 1920. New York: Teachers College
Press, Columbia University.
Culp, K. M., Honey, M., & Mandinach, E. (2003, Octubre). A Retrospective on Twenty Years of Education Technology
Policy. U.S. Department of Education, Office of Educational Technology. Recuperado a partir de
http://www.ed.gov/rschstat/eval/tech/20years.pdf
Dede, C. (2006). Scaling Up: Evolving Innovations beyond Ideal Settings to Challenging Contexts of Practice. In R.K.
Sawyer (Ed.), Cambridge Handbook of the Learning Sciences. Cambridge Handbook of the Learning Sciences.
Dillenbourg, P. (2008). Integrating technologies into educational ecosystems. Distance Education, 29(2), 127.
Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., Means, B., et al. (2007). Effectiveness of
Reading and Mathematics Software Products: Findings from the First Student Cohort. U.S. Department of Education,
Institute of Education Sciences. Recuperado a partir de http://ies.ed.gov/ncee/pubs/20074005/
Earle, R. S. (2002). The Integration of Instructional Technology into Public Education: Promises and Challenges.
Educational Technology, 42(1), 5-13.
Feinstein, O. N., & Picciotto, R. (2001). Evaluation and Poverty Reduction. World Bank Series on Evaluation and
Development (Vol. 3). Washington, D.C: Transaction Publishers, New Brunswick, New Jersey.
Forrester, E., Buteau, B., & Shrum, S. (2011). CMMI for Services: Guidelines for Superior Service (2o ed.). Boston:
Addison-Wesley.
Harris, D. N. (2009). Toward Policy-Relevant Benchmarks for Interpreting Effect Sizes: Combining Effects With Costs.
Educational Evaluation and Policy Analysis, 31(1), 3-29.
Hayes, D. N. A. (2007). ICT and learning: Lessons from Australian classrooms. Computers & Education, 49(2), 385-395.
Hennessy, S., Ruthven, K., & Brindley, S. (2005). Teacher Perspectives on Integrating ICT into Subject Teaching:
Commitment, Constraints, Caution, and Change. Journal of Curriculum Studies, 37(2), 155-192.
Infante, C., Weitz, J., Reyes, T., Nussbaum, M., Gómez, F., & Radovic, D. (2009). Co-located collaborative learning video
game with single display groupware. Interactive Learning Environments.
Kikis, K., Scheuermann, F., & Villalba, E. (2009). A framework for understanding andevaluating the impact of information
and communication technologies in education. In F. Scheuermann & F. Pedró (eds.) Assessing the Effects of ICT in
Education: Indicators, Criteria and Benchmarks for International Comparisons (págs. 69-82). Luxembourg: Publications
Office of the European Union.
Kirkman, C. (2000). A model for the effective management of information and communications technology development in
schools derived from six contrasting case studies. Technology, Pedagogy and Education, 9(1), 37-52.
Kozma, R. B. (1991). Learning with Media. Review of Educational Research, 61(2), 179-211.
Kozma, R. B. (2003). Technology and Classroom Practices: An International Study. Journal of Research on Technology in
Education, 36(1), 1-14.

15
Paper submitted for peer reviewing — Please do not distribute or cite without permission

Kulik, J. (2003). Effects of Using Instructional Technology in Elementaryand Secondary Schools: What Controlled
EvaluationStudies Say. Technical Report, SRI International.
Lagos, M. E., Vil, G., Miranda, H., Contreras, D., Rodríguez, J., & Sandoval, K. (2009). Aprendiendo Matemática y
Lenguaje con Tecnología Portátil 1:1: Informe final. Recuperado a partir de
http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=3626&org=11413
Lai, K.-W. (2008). ICT Supporting the Learning Process: The Premise, Reality, and Promise. In J. Voogt. G. Knezek (Eds.),
International Handbook of Information Technology in Primary and Secondary Education (págs. 215-230).
Laurillard, D. (2007). Modelling benefits-oriented costs for technology enhanced learning. Higher Education, 54(1), 21-39.
Liao, Y. C., & Hao, Y. (2008). Large -scale studies and quantitative methods. In J. Voogt. G. Knezek (Eds.), International
Handbook of Information Technology in Primary and Secondary Education (págs. 1019-1035).
Light, D. (2008). Evaluating Educational Technology Interventions: How do we know its working? Center for Children and
Technology, Education Development Center, Inc. Recuperado a partir de http://cct.edc.org/report.asp?id=254
Lim, C.P., & Hang, D. (2003). An activity theory approach to research of ICT integration in Singapore schools. Computers &
Education, 41(1), 49-63.
Lim, Cher Ping. (2002). A theoretical framework for the study of ICT in schools: a proposal. British Journal of Educational
Technology, 33(4), 411-421.
López, M., Parada, D., & Gómez, F. (2009). MIC (1:1) Modelo de Integración Curricular con estrategia 1 a 1. Resultados de
la Implementación: Informe final. Recuperado Febrero 3, 2011, a partir de
http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=3677&org=11413
Luckin, R. (2008). The learner centric ecology of resources: A framework for using technology to scaffold learning.
Computers & Education, 50(2), 449-462.
Margolis, J., Nussbaum, M., Rodríguez, P., & Rosas, R. (2006). Methodology for evaluating a novel education technology: a
case study of handheld video games in Chile. Computers & Education, 46(2), 174-191.
Marshall, G., & Cox, M. (2008). Research Methods: Their design, applicability and reliability. In J. Voogt. G. Knezek (Eds.),
International Handbook of Information Technology in Primary and Secondary Education (págs. 983-1002).
McDonald, S.-K., Keesler, V. A., Kauffman, N. J., & Schneider, B. (2006). Scaling-Up Exemplary Interventions.
Educational Researcher, 35(3), 15 -24.
McFarlane, A. (2001). Perspectives on the relationships between ICT and assessment. Journal of Computer Assisted
Learning, 17(3), 227-234.
Mioduser, D., Nachmias, R., Tubin, D., & Forkosh-Baruch, A. (2003). Analysis Schema for the Study of Domains and
Levels of Pedagogical Innovation in Schools Using ICT. Education and Information Technologies, 8(1), 23-36.
Miranda, S., & Feris, C. (2009). Modelo de Informática Educativa Aprendizaje Interactivo de las Ciencias. Enseñanza Media:
Informe Final. Recuperado a partir de http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=3740&org=11413
Moenne, G. (2008). Dibujos con Vida. Informe final. Recuperado a partir de
http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=2186
Moenne, G. (2010). Atenea: Desarrollo del pensamiento científico a través de las TIC: Informe final. Recuperado Marzo 7,
2011, a partir de http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=3839&org=11413
Mooij, T., & Smeets, E. (2001). Modelling and supporting ICT implementation in secondary schools. Computers &
Education, 36(3), 265-281.
Moonen, J. (2008). Policy from a global perspective. In J. Voogt. G. Knezek (Eds.), International Handbook of Information
Technology in Primary and Secondary Education (pág. 1171—1178).
Moyle, K. (2008). Total Cost Of Ownership And Total Value Of Ownership. In J. Voogt. G. Knezek (Eds.), International
Handbook of Information Technology in Primary and Secondary Education (págs. 615-631).
Nachmias, R., Mioduser, D., Cohen, A., Tubin, D., & Forkosh-Baruch, A. (2004). Factors Involved in the Implementation of
Pedagogical Innovations Using Technology. Education and Information Technologies, 9(3), 291-308.
Nussbaum, Miguel, Alvarez, C., McFarlane, A., Gomez, F., Claro, S., & Radovic, D. (2009). Technology as small group

16
Paper submitted for peer reviewing — Please do not distribute or cite without permission

face-to-face Collaborative Scaffolding. Computers & Education, 52(1), 147-153.

Pelgrum, W. (2001). Obstacles to the integration of ICT in education: results from a worldwide educational assessment.
Computers & Education, 37(2), 163-178.
Penuel, W. (2005). Recommendations for Evaluation Research within Educational Innovations. Center for Technology in
learning, SRI International. Recuperado a partir de
http://ctl.sri.com/publications/downloads/Rec_Eval_Research_within_Edu_Innovations.pdf
Penuel, W. R. (2006). Implementation and Effects of One-to-One Computing Initiatives: A Research Synthesis. Journal of
Research on Technology in Education, 38(3), 329-348.
Perraton, H., Creed, C., & Robinson, B. (2002). Teacher Education Guidelines: Using Open and Distance Learning.
Technology, Curriculum, Cost, Evaluation. UNESCO, Higher Education Division, Teacher Education Section. Recuperado a
partir de http://www.eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED468705
Pilkington, R. M. (2008). Measuring the impact of Information Technology on Student’s Learning. In J. Voogt. G. Knezek
(Eds.), International Handbook of Information Technology in Primary and Secondary Education (págs. 1003-1018).
Plomp, T., Pelgrum, W., & Law, N. (2007). SITES2006–International comparative survey of pedagogical practices and ICT
in education. Education and Information Technologies, 12(2), 83-92.
Reeves, T. (2008). Evaluation of the Design and Development of IT Tools in Education. In J. Voogt. G. Knezek (Eds.),
International Handbook of Information Technology in Primary and Secondary Education (págs. 1037-1051).
Reiser, R. A. (2001). A history of instructional design and technology: Part I: A history of instructional media. Educational
Technology Research and Development, 49(1), 53-64.
Reynolds, D., Treharne, D., & Tripp, H. (2003). ICT - the hopes and the reality. British Journal of Educational Technology,
34(2), 151-167.
Robertson, J. (2002). The ambiguous embrace: twenty years of IT (ICT) in UK primary schools. British Journal of
Educational Technology, 33(4), 403-409.
Roblyer, M. D. (2005). Educational Technology Research That Makes a Difference: Series Introduction. Contemporary
Issues in Technology and Teacher Education, 5(2), 192-201.
Rodríguez, P. (2008). Standard for an ICT for education programme. Enlaces, Center for Technological Education. Ministry
of Education. Recuperado a partir de
http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=2549&param=publico
Rodríguez, P. (2010, Mayo 31). A methodology for replication, scaling-up and transference of ICT supported pedagogical
models (PhD in Engineering Thesis). Department of Computer Science, College of Engineering, Pontificia Universidad
Católica de Chile.
Rodríguez, P., Nussbaum, M., & Dombrovskaia, L. (2011). Evolutionary development: a model for the design,
implementation and evaluation of ICT for education programmes. To appear in Journal of Computer Assisted Learning.
Rodríguez, P., Nussbaum, M., López, X., & Sepúlveda, M. (2010). A Monitoring and Evaluation Scheme for an ICT-
Supported Education Program in Schools. Journal of Educational Technology & Society, 13(2), 166–179.
Roschelle, J., Rafanan, K., Bhanot, R., Estrella, G., Penuel, W., Nussbaum, M., & Claro, S. (2010). Scaffolding group
explanation and feedback with handheld technology: impact on students’ mathematics learning. Educational Technology
Research and Development, 58, 399–419.
Rumble, G. (2001). Analysing Costs/Benefits for Distance Education Programmes. The Commonwealth of Learning.
Recuperado a partir de http://www.col.org/PublicationDocuments/pub_KS2001-02_costs.pdf
Sánchez, J., & Alarcón, P. (2003). Multimedia Project en Chile: Diseño y resultados. Presented at the Taller Internacional de
Software Educativo, Santiago, Chile. Recuperado a partir de
http://www.tise.cl/archivos/tise2003/papers/multimedia_project_diseno_resultados.pdf
Scrimshaw, P. (2002). Total cost of ownership: A review of the literature. DfES: London, UK. Recuperado a partir de
http://partners.becta.org.uk/index.php?section=rh&rid=13611
Slavin, R. E. (2008). Perspectives on Evidence-Based Research in Education--What Works? Issues in Synthesizing
Educational Program Evaluations. Educational Researcher, 37(1), 5-14.

17
Paper submitted for peer reviewing — Please do not distribute or cite without permission

Snider, R. C. (1992). The machine in the classroom. Phi Delta Kappan, 74(4), 316–323.
Steinberg, D. C. (2004). The business model of e-learning in UK higher education. Industry and Higher Education, 18, 187-
198.
Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation Theory, Models, and Applications. John Wiley and Sons.
Sunkel, G. (2006). Las Tecnologías de La Información y la Comunicación (Tic) en la educación en América Latina. Una
exploración de indicadores. CEPAL - Politicas sociales (Division de Desarrollo Social, CEPAL, Naciones Unidas.).
Sutherland, R., Armstrong, V., Barnes, S., Brawn, R., Breeze, N., Gall, M., Matthewman, S., et al. (2004). Transforming
teaching and learning: embedding ICT into everyday classroom practices. Journal of Computer Assisted Learning, 20(6), 413
- 424.
Tan, J.-P., Anderson, J. R., Belli, P., Barnum, H. N., & Dixon, J. A. (2001). Economic Analysis of Investment Operations:
Analytical Tools and Practical Applications. World Bank Publications.
Tearle, P. (2003). ICT implementation: what makes the difference? British Journal of Educational Technology, 34(5), 567-
583.
Tearle, P. (2004). A theoretical and instrumental framework for implementing change in ICT in education. Cambridge
Journal of Education,, 34(3), 331-351.
Tondeur, Jo, Braak, J. van, & Valcke, M. (2007). Curricula and the use of ICT in education: Two worlds apart? British
Journal of Educational Technology, 38(6), 962-976.
Tondeur, J., Valcke, M., & Van Braak, J. (2008). A multidimensional approach to determinants of computer use in primary
education: teacher and school characteristics. Journal of Computer Assisted Learning, 24(6), 494-506.
Toyama, K. (2010). Can Technology End Poverty? Boston Review, (November/December). Recuperado a partir de
http://bostonreview.net/BR35.6/ndf_technology.php
Toyama, K. (2011, Enero). There Are No Technology Shortcuts to Good Education. Educational Technology Debate,
Infodev-UNESCO. Recuperado a partir de http://edutechdebate.org/ict-in-schools/there-are-no-technology-shortcuts-to-good-
education/
Trucano, M. (2005). Knowledge Maps: ICTs in Education. ICT and Education Series. Washington, DC: infoDev / World
Bank. Recuperado a partir de http://www.infodev.org/en/Document.8.aspx
Tubin, D., Mioduser, D., Nachmias, R., & Forkosh-Baruch, A. (2003). Domains and Levels of Pedagogical Innovation in
Schools Using ICT: Ten Innovative Schools in Israel. Education and Information Technologies, 8, 127–145.
Twigg, C. (2002). Improving quality and reducing costs: Designs for effective learning using ICT. Observatory on Borderless
Higher Education.
UNESCO. (2008). ICT Competency Standards for Teachers - Competency Standards Modules. Unesco Paris. Recuperado a
partir de http://cst.unesco-ci.org/sites/projects/cst/The%20Standards/ICT-CST-Competency%20Standards%20Modules.pdf
Valdivia, R., & Nussbaum, M. (2007). Face-to-Face Collaborative Learning in Computer Science Classes. International
Journal of Engineering Education, 23, 434-440.
Valdivia, R., & Nussbaum, M. (2009). Using Multiple Choice Questions as a Pedagogic Model for Face-to-Face CSCL.
Computer Applications in Engineering Education, 17(1), 89-99.
Van den Akker, J. (2003). Curriculum perspectives: An introduction. In J. van den Akker, U. Hameyer, & W. Kuiper (Eds.)
Curriculum landscapes and trends (págs. 1-10). Dordrecht: Kluwer. Recuperado a partir de
http://www.neccs.nl/curriculum/Curriculum_landscapes.doc
Van den Akker, J., Gravemeijer, K., McKenney, S., & Nieveen, N. (2006). Educational Design Research. London:
Routledge.
Wagner, D. A., Day, B., James, T., Kozma, R. B., Miller, J., & Unwin, T. (2005). Monitoring and Evaluation of ICT in
Education Projects: A Handbook for Developing Countries. ICT and Education Series. Washington, DC: infoDev / World
Bank. Recuperado a partir de http://www.infodev.org/en/Publication.9.html
Warschauer, M., Knobel, M., & Stone, L. (2004). Technology and Equity in Schooling: Deconstructing the Digital Divide.
Educational Policy, 18(4), 562 -588. doi:10.1177/0895904804266469

18
 Paper submitted for peer reviewing — Please do not distribute or cite without permission

Watson, D. M. (2001). Pedagogy before Technology: Re-thinking the Relationship between ICT and Teaching. Education
and Information Technologies, 6(4), 251-266.
Waxman, H., Lin, M., & Michko, G. M. (2003). Meta-Analysis: Effects of Educational Technology on Student Outcomes.
Learning Point Associates. Recuperado a partir de http://www.ncrel.org/tech/effects2/
Zurita, G., & Nussbaum, M. (2004b). A constructivist mobile learning environment supported by a wireless handheld
network. Journal of Computer Assisted Learning, 20(4), 235-243.
Zurita, G., & Nussbaum, M. (2004a). Computer supported collaborative learning using wirelessly interconnected handheld
computers. Computers & Education, 42(3), 289-314.

19

View publication stats