Quality, the assurance of improvement

Guide to the evaluation of R&D institutes

© Agència per a la Qualitat del Sistema Universitari de Catalunya
Via Laietana, 28, 5ª planta
08003 Barcelona

© Authors: Esther Huertas Hidalgo and Miquel Vidal Espinar

First edited: November 2007

Legal number: B- 55.290-2007
Index

Foreword ....................................................................................................................................... 5

Introduction.................................................................................................................................... 6

Institutional evaluation................................................................................................................... 9

Stages in the evaluation process ................................................................................................ 11

Self-evaluation ......................................................................................................................... 12

External review ........................................................................................................................ 15

Final report............................................................................................................................... 16

Meta-evaluation ....................................................................................................................... 17

Evaluation protocol...................................................................................................................... 19

1. Mission ............................................................................................................................. 20

2. Vision: policy and strategic plan....................................................................................... 22

3. Organisation and leadership in the institute..................................................................... 24

4. Human resources............................................................................................................. 26

5. Infrastructure .................................................................................................................... 28

6. External relations ............................................................................................................. 30

7. Interaction with society, communication and image ........................................................ 32

8. Results ............................................................................................................................. 34

9. Generation of financial resources. Projects and agreements .......................................... 36

10. Teaching activity ............................................................................................................ 38

11. Evaluation process......................................................................................................... 40

Bibliography................................................................................................................................. 42

Annex I. Institute fact sheet ......................................................................................................... 43

Annex II. Tables with administrative data, statistics and indicators ............................................ 44

Guide to the evaluation of R&D institutes | 3

Foreword

The Agència per a la Qualitat del Sistema Universitari a Catalunya was legally established as a
consortium between the Autonomous Government of Catalonia and the Catalan public
universities on 26 September 1996. Law 1/2003 of February on the Universities in Catalonia
(the Catalan Universities Act) converted the consortium into an organization with its own legal
structure, which changed its name to AQU Catalunya (Agència per a la Qualitat del Sistema
Universitari de Catalunya). While upholding the legacy of the former Agency, AQU Catalunya
has since gone on to extend membership of consortium to include, among others,
representatives of the private universities, and to develop new competencies in the fields of
quality assessment and teaching staff and research assessment.

The ends pursued by the AQU are quality assessment, process certification and accreditation of
the learning process in the higher education system in Catalonia. These serve as a constant
guide for the Agency to adapt to the demands of society, the quality requirements for university
education and the continuous enhancement of its processes within the framework of the
European Higher Education Area.

The Agency's Activities Plan includes various lines of action aimed at the level of the
universities and higher education institutions in Catalonia. As a result of prior experience gained
in the evaluation of the Research Institutes Programme in the 3rd Research Plan for Catalonia,
promoted by the Interministerial Council for Research and Technological Innovation (CIRIT), the
Agency's current Activities Plan also envisages the possibility of the evaluation of different types
of institute and unit as a way to contribute to the improvement of their planning, administration
and policy-making towards excellence.

The Guide has been designed mainly for the evaluation of research and development institutes
and R+D units (both basic and applied research). It can however be applied to the evaluation of
other institutions and institutes, such as those dealing with services, knowledge transfer and
technology innovation of both a public (either belonging to the university framework or
government-funded) and private nature. Within the context of this Guide, the term institute is
used for all institutes, institutes and units of this type, irrespectively.

The Guide has been conceived for the evaluation of institutes that have been running for a
certain period of time or their completion of a specific period of activity. This means that it is not
applicable to ex ante assessment and the setting up of institutes or the evaluation of institutes
that have recently been set up.

The authors would lastly like to express their gratitude to: Roel Bennink (Research Coordinator
with the Quality Assurance Netherlands Universties), Rosa Menéndez (Director of the Spanish
Coal Institute (CSIC) Oviedo), Ramon Pascual (Director of the Synchrotron Laboratory,
Barcelona), and Karen Siune (Director of the Danish Institute for Studies in Research and
Research Policy, Denmark) for their collaboration in the review of this Guide.

Guide to the evaluation of R&D institutes | 5

Introduction

The concept of evaluation, within the framework of this Guide, consists of making a judgment
and assessment of an object or intervention, in this case of an institute. The main purpose of
evaluating an institute is to contribute to the review of the design and definition of its mission;
provide data in order for it to establish its vision and the priorities for its strategic policy; assist in
the more efficient distribution of its resources; enhance its quality and inform on its performance
and the fulfilment of its goals as defined in a contract programme or pre-established strategic
plan.

As in other evaluation processes, the evaluation of institutes can be classified according to

different criteria (see table 1), such as the moment when it is carried out (ex ante; follow-up, and
ex post), the parties involved (internal and external) and its purpose (formative, summative).

The dimensions of structure, activity and results need to be included in the institutional
evaluation methodology (Juárez et al., 2005):

Structure: This involves determining the institute's prospective capability or potential, with
consideration being given to its context and the opportunities available to the institute so it can
develop. Examples of this type of evaluation include the analysis of developments achieved in a
particular field or discipline, the analysis of the staff's research qualifications (capabilities,
experience, adaptability, etc.), technological capability (access to instrumentation, pilot
installations, etc.), economic and administrative organisation (adaptability, level of bureaucracy,
etc.), as well as the analysis of structural factors that facilitate or hinder the institute's activities.

Activity: This normally focuses on examining the institute's mission and vision as these
determine its lines of action. It includes the analysis of aspects such as the purpose of research
that is carried out and its diffusion and relevance in relation to the existing culture. Other
elements of scientific activity are analysed together with these aspects, including staff training,
the publicising of scientific developments, technology transfer, etc.

Results: This evaluation includes an assessment of the degree to which the goals are fulfilled,
an examination of scientific production and the impact of the activities carried out, expressed in
the form of targets achieved. Examples are the number of publications, the number of theses
defended, the number of patents obtained, the institute's academic reputation and recognition at
the international level, and the institute's influence on other institutions, amongst others.

Guide to the evaluation of R&D institutes | 6

Table 1. Classification of the types of institutional evaluation.

Criterion 1. Moment of the evaluation

Ex ante Carried out prior to the institute being set up (prospective evaluation).

Goals:

ƒ To analyse the adequacy and possibilities of the institute's success.

ƒ Analyse the technical project and the institute's organisational chart.

ƒ To contribute to the institute's design and the defining of its mission.

ƒ To contribute to the coherency of activities planned with the strategic goals.

ƒ To make for greater transparency and impartiality.

ƒ To provide instruments for information collection and analysis for the

monitoring and ex post assessment of the results of the institute's activities.

Interim or This is made when the institute has been running for a period of time. The
follow-up moment for the follow-up evaluation must be very clearly defined from the start
of a specific period of activities. Fulfilment of the institute's activities and
envisaged goals is checked and any necessary modifications introduced. This,
together with the ex post assessment, constitutes the retrospective evaluation.

Goals:

ƒ Facilitate the ex post evaluation through the gathering and systematisation

of relevant information in intermediary stages.

ƒ To permit the detection and correction of deviations between activities that

have been programmed and those actually carried out.

ƒ To expedite the ex post assessment through the collection and

systemisation of relevant information in the intermediate stages.

ƒ Enhance the visibility and increase awareness of the effects and impact of
the institute's activities.

Ex post This determines the rate of achievement of pre-established goals. This

assessment is made when an institute's activity comes to an end or a period of
specific activities is completed.

Goals:

ƒ To know the direct results achieved by the institute and the impact of its
activities and the results obtained.

ƒ To provide information in order for the institution to improve the design and
performance of its activity.

Guide to the evaluation of R&D institutes | 7

 ƒ To determine the effectiveness and efficiency of resource management.

Criterion 2. Parties involved

Internal Those responsible for the institute mainly carry out the evaluation, although
external members may also participate. It enables the organisation to
understand what is going on internally.

External Persons from outside the institute make the evaluation, often after a self-
evaluation, and they analyse the results obtained in relation to other institutes in
the same field. The goal of the external review consists basically of the
validation of the institute's self-evaluation process. This process enhances the
credibility of the evaluation through reinforcement of the independence,
objectivity and transparency of the process, as well as helping to improve the
diagnosis made by the evaluated unit and make use of the different expertise of
the members from outside the institute.

Criterion 3. Purpose

Formative Ways to improve and enhance the efficiency and administration of institutes are
examined. This type of evaluation normally focuses on the institute's managers,
the aim being to improve work at the managerial level.

Summative This focuses on the institute's accountability, according to its effectiveness and
efficiency, which may lead to assessment of the prospective allocation of
funding (financial resources).

The methodology set out in this Guide has been designed mainly for the evaluation of research
and development institutes and R+D units (both basic and applied research). It can however be
applied to the evaluation of other institutions and institutes, such as those dealing with services,
knowledge transfer and technology innovation of both a public (either belonging to the university
framework or government-funded) and private nature.

It is important to point out that there is a great diversity of institutes, according to their size and
management structure. In certain case, the institute's physical structure is very well defined,
with a governing body in charge of the management and decision-making, under which come
boards dealing with scientific and commercial matters, and there is often a board of trustees
that watches over compliance with the contract programme. In other cases, the institute or R+D
unit may come under a university (or faculty in the university), with an independent structure
(with independent management, facilities, support staff, etc.) and with a more or less collegial
organisation. The institute or unit may also be run online. This wide diversity of types of institute
may lead to different variations in the evaluation process. Nevertheless, the purpose of the
evaluation is to assist decision-making by the institute's governing body or director on the

Guide to the evaluation of R&D institutes | 8

pathway to excellence, by way of the proposal with means for improvement to enhance the
institute's structure, activities and results.

The proposed institutional evaluation process is, in principle, retrospective and focuses on
broadening the understanding of the institute's characteristics for the purpose of continuous
improvement and, where necessary, accountability to the entity that funds the running of the
institute. In addition, the methodology has been conceived for the evaluation of institutes that
have been running for a certain period of time or that are completing a specific period of activity,
i.e. that are considering an interim or ex post evaluation. In specific terms, for institutes that
have been running for a certain period of time, an evaluation is proposed of the previous five
years of activity is proposed; shorter periods are proposed in the case of institutes set up in the
previous five years.

With regards to those involved in the evaluation, self-evaluation is proposed, followed by an

external evaluation, which lead to a final report that includes the improvement plan as the
fundamental diagnostic tool. The improvement plan contains a series of proposals for action,
resulting from the preliminary diagnosis process, which sets out and formulates the goals for
improvement and the actions aimed at enhancing the strong points and resolving the weak
ones, according to priority and a given schedule (EC, 2004).

Institutional evaluation

Institutional evaluation needs to include the assessment of activities over which an institute has
responsibility and will need to principally deal with aspects associated with research, teaching
(where applicable), services and administration.

Institutional evaluation has the following basic goals (modified from VSNU et al., 2003):

ƒ To define, confirm and/or redefine the institute's mission and vision.

ƒ To enhance the quality of its activities through an internal and external diagnosis (including
the running of support services).

ƒ Improved management and flexibility of administration.

ƒ Analyse the adequacy of the institute's governing structure.

ƒ Enhance the institute's competitiveness and visibility.

ƒ To provide accountability to the organisations that an institute depends on and ultimately to

society in general.

The results of the evaluation need to assist the institute's organisation, management and
researchers in making decisions associated with the institute's future, from the management

Guide to the evaluation of R&D institutes | 9

 and research point of view. The main criteria according to which the evaluation is structured are
(VSNU et al., 2003):

ƒ Quality, which is a measure of the institute's excellence through its recognition at both
national and international levels and the institute's innovative potential, in addition to its
facilities and equipment.

ƒ Productivity, which mainly refers to scientific productivity (for example, publications, theses,
patents, etc.), how the results of the institute's activities are publicised, and the institute's
cultural (for example, services, artistic works, exhibitions, etc.) and socio-economic (for
example, technology transfer) activities.

ƒ Relevance, which covers the scientific, technical, socio-economic and artistic impact of
the work.

ƒ Vitality and feasibility, which refers to the external and internal dynamics of the institute, the
success rate of projects and activities, and its adaptability to initiate and close research
lines.

Figure 1 summarises the institutional evaluation process.

AIMS & CRITERIA DIMENSIONS STAGES IN THE

PROCESS
Aims
STRUCTURE
- Defining of the mission and vision
- Enhancement of quality and activities SELF-ASSESSMENT
- Improved managed and flexibility of
EVALUATION administration
- Enhance the competitiveness and ACTIVITY
visibility
EXTERNAL REVIEW
Criteria

- Quality
RESULTS
- Productivity
- Relevance IMPROVEMENT PLAN
- Vitality and feasibility

IMPLEMENTATION OF THE IMPROVEMENT PLAN

Figure 1. Diagram of the institutional evaluation process

Guide to the evaluation of R&D institutes | 10

Stages in the evaluation process

The evaluation starts with an analysis of the institute's current situation by its self-evaluation
committee, which draws up the self-evaluation document. This report sets out the committee's
appraisal of the questions formulated in the evaluation protocol, together with an analysis of the
situation for each element and aspect evaluated. The analysis must be supported by evidence
(documentation, statistics, administrative data and/or indicators) available to the institute. Once
this document has been drawn up, it is then reviewed by the external review panel. The external
review panel's function is, amongst others, to validate the adequateness of the self-evaluation
process by analysing the self-evaluation document. The external review panel then drafts a
report with its impressions regarding its analysis of the self-evaluation document, the evidence
submitted by the self-evaluation committee, the site visit to the institute and its own experience.
This document is then sent to the self-evaluation committee so it can make any adequate
amendments. The revised and consensual document is subsequently returned to the external
review panel, which draws up the final evaluation report.

Figure 2 shows the stages in the evaluation process.

0. Current
situation

1. Self-evaluation
Data document (self-
assessment
Indicators
committee)
Appraisals

4. Application of
improvement plan
Observations
Appraisals 2. External
Visit review panel's
report

3. Final report and

improvement plan (self-
assessment committee and
external review panel)

Figure 2. Evaluation process and application of the improvement plan

The AQU committee dealing with the quality assessment of university institutions and activities
(ACAU) shall validate the evaluation methodology, give its approval to the final external review

Guide to the evaluation of R&D institutes | 11

report and the executive summary of the same. Responsibility for appointing the members of
the external review panel lies with the AQU Quality Assessment Committee.

Self-evaluation

The self-evaluation is a diagnostic process to detect and enhance as objectively and thoroughly
as possible the institute's areas and activities of excellence, and also to detect areas capable of
being improved.

The process starts with the setting up of the self-evaluation committee, which drafts the self-
evaluation document on the basis of the guidelines given in this Guide. The self-evaluation
committee is responsible for establishing the mechanisms for collecting information on the
institute in the form of documents (for example, by referring to the institute's mission, vision and
organisation) and statistics, administrative data and indicators suggested in the Guide on the
structure, inputs, processes and results of the institute's activities. It must be pointed out that the
self-evaluation committee is free to increase the amount of evidence with the aim of broadening
the scope of information provided and thereby improve the evaluation. Analysing the evidence
will help the self-evaluation committee to better appraise the questions given in the evaluation
protocol, which, together with the analysis of the institute's prospects and expectations, as well
as new opinions and appraisals arising throughout the process, constitute the self-evaluation
document.

The self-evaluation committee

The most important job of the self-evaluation committee is to draw up the self-evaluation
document on the basis of the guidelines given in the Guide.

The self-evaluation committee should not be too large in number although it does need to
ensure the credibility of the self-evaluation process, which involves the participation of all the
institute's significant stakeholders. Its composition may vary according to the institute's
organisational model. In the case of an institute with a governing body, a self-evaluation
committee model is proposed which, in addition to the director, includes members of the
governing board, scientific advisory board, business affairs committee, and the heads of
research and administration. In cases where decisions are made on the basis of collegial
responsibility, a possible model would be the institute's director, between two to four
researchers (who should represent all of the institute's areas and groups, and include both
junior and senior members), one pre-doctoral research fellow, one member of the research
support staff and one member from administration.

Guide to the evaluation of R&D institutes | 12

The procedures that are adopted for gathering information that will form part of the self-
evaluation report need to allow (where applicable) for the opinion of the different stakeholders
and levels in the organisation to be included. Mention is made of the fact that, wherever the
institute's organisational model makes it possible, the self-evaluation report is to be disclosed to
and checked by members of the institute's different stakeholders.

General structure of the self-evaluation document

The self-evaluation document represents the key part of the evaluation model and the main
evidence for the external review process. The drawing up of the self-evaluation document by
the self-evaluation committee must enable both the community at the institute and the external
review panel to know about and understand the institute's standing. In order to lay the
foundations for a good improvement plan, the self-evaluation document must fulfil, amongst
other things, the following requirements:

ƒ Complete and rigorous. It needs to analyse and assess the key elements in the situation
that is to be assessed and improved.

ƒ Based on evidence, in order for it to be more sound, objective and contestable, and less
questionable.

ƒ Systematic and detailed with regard to the analysis of the causes and therefore whatever is
necessary for dealing with improvements.

ƒ Balanced, in terms of both the positive aspects as well as those that need improving.

ƒ Shared by the stakeholders or the communities affected, in order to ensure they are
represented in the analysis, where the institute's organisational model allows for this.

The structure and length of the document are described in the Guide. The self-evaluation
document starts with an introductory chapter and a description of the origins and context in
which the institute was set up, its present context, the defining moments in its development, the
institute's mission and vision, the research and administration policy of the institution that it
depends on, previous experience with evaluation, etc.

For each of the aspects below to be evaluated, the self-evaluation committee will need to
appraise the questions formulated in the evaluation protocol in relation to the aspects to be
evaluated, for which a list of elements is given. The protocol must permit an evaluation to be
based on the criteria of quality, productivity, relevance, vitality and feasibility. The appraisals
made in this part are judged according to a four-point scale. The protocol also gives indications
of standards to orientate the committee's value judgment and a list of evidence to support and
justify the opinions and appraisals made.

Guide to the evaluation of R&D institutes | 13

In the case of this Guide, the evaluation protocol, which has been drafted following the
recommendations of different international agencies (VSNU et al., 2003; EFQM, 2003), is
structured according to the following aspects:

ƒ Mission

ƒ Vision: policy and strategic plan

ƒ Organisation and leadership

ƒ Human resources

ƒ Infrastructure

ƒ External relations

ƒ Interaction with society, communication and image

ƒ Results

ƒ Generating financial resources (funding). Projects and agreements

ƒ Teaching activity (where applicable)

ƒ Evaluation process

For each aspect, the self-evaluation committee must prepare a comprehensive vision or
analysis of the situation of up to no more than 1,500 words on each aspect. On the one hand,
an evaluation needs to be made of the quality (sufficiency, relevance and suitability) of the
evidence provided. Details of any significant changes in the institute's course of development in
recent years need to be given. Recent progress achieved by the institute may constitute an
important piece of data to be included in the institute's current situation, within the framework of
the strategic policy of the institutions that it depends on. This part must also include the
reasoning and justification for the judgments issued in relation to the questions formulated in the
evaluation protocol. The analysis of the situation of each aspect is completed with a SWOT
analysis (strengths, weaknesses, opportunities, threats). Strengths and weaknesses are the
strong and weak points, respectively, which emerge from the analysis of the documentation.
The analysis of opportunities and threats is based on an examination of developments in the
scientific and social frameworks that may affect the institute's activities either positively or
negatively, respectively. The self-evaluation committee may, as a result of the SWOT analysis,
propose changes in goals and strategic policy, modify the institute's mission, establish a new
group of goals, or adapt the strategy to replace the current one.

The self-evaluation document must point out the strategies that seek to mitigate the weak points
detected and enhance the strong ones, which are aspects that will serve as the basis for
establishing the institute's improvement plan. Proposals for improvement need to be shown
clearly and in an orderly way, and their relationship established with each strong and weak point
detected during the self-evaluation process. The proposals for improvement must specify the
level of priority (low, medium, maximum), the time period for execution and impact (for example,

Guide to the evaluation of R&D institutes | 14

short term, 3-6 months; medium term, 6-12 months; long term, 12-24 months), the stakeholders
who are involved and the designation of responsibilities, and their justification and feasibility.

It is recommended that the self-evaluation committee replies to the key questions in the
evaluation protocol after answering the specific questions and analysing the situation of each
aspect.

Forms are provided in the document annexes for the self-evaluation committee to complete the
evidence (data and additional indicators; composition of the self-evaluation committee; minutes
of meetings and work schedule of the self-evaluation committee) for the evaluation process.

The self-evaluation committee has three months to prepare the self-evaluation document from
the start of the evaluation process. The self-evaluation document must be drawn up in English
in order for the members of the external review panel to read and understand it, as it is
envisaged that its members will be international experts.

A training session for the members of the self-evaluation committee will be organised by the
AQU in which the special features of the evaluation methodology will be explained and any
doubts resolved concerning the drafting of the self-evaluation document and the evaluation
process in general.

External review

The aim of the external review is to assist the institute in making the analysis, i.e. to validate the
diagnosis made by the self-evaluation committee and enable it to collaborate in the preparation
of an improvement plan. The credibility and validity of the self-evaluation document need to be
confirmed by the external review panel.

The external review panel analyses the self-evaluation document and visits the institute being
evaluated. On the basis of its observations and information, evidence, and opinions and
appraisals made during the review of the self-evaluation document and the visit, it issues an
external review report. This report is sent to the self-evaluation committee for its consideration
and to make any pleas or adequate remarks.

The external review panel

The members of the external review panel shall be appointed by the Quality Assessment
Committee. The institute shall be informed of the external review panel's composition to prevent
any conflict of interests. The external review panel will be made up of at least three experts in
the institute's field of knowledge and one methodologist who ensures that the methodology is
applied correctly. It is emphasised that, wherever possible, the external review panel shall be
made up mostly of international QA experts in the institute's same area of knowledge, and
external to the framework of the institute.

Guide to the evaluation of R&D institutes | 15

One of the reviewers shall chair the external review panel. A training session for the members
of the external review panel will be organised by the AQU in which the special features of the
evaluation methodology will be explained and any doubts resolved concerning the evaluation
process in general.

The site visit to the institute

Following an interval of at least three weeks after the self-evaluation document has been
received, the visit is made to the institute. The external review panel prepares the contents of
each interview with the groups at the institute that will take place during the two-day visit.

Once the interviews with these groups have been completed, the external review panel's visit to
the institute finishes with a meeting with the self-evaluation committee, where the external
review panel orally presents a draft version of its report.

General structure of the external review report

The external review report is drawn up from an appraisal of the self-evaluation document, the
replies to the key questions in the evaluation protocol on each aspect, and also additional
information collected during the interviews. The report shall include the following sections:

ƒ Introduction: composition of the external review panel, aims, work schedule, incidents.

ƒ Assessment of the self-evaluation process and the self-evaluation document.

ƒ Assessment of just the key questions of each aspect covered in the evaluation Protocol,
including justification for each value judgment.

ƒ General assessment: strong and weak points, and the improvement plan.

Final report

The institute being evaluated has the possibility, through its self-evaluation committee, to qualify
the external review report drafted by the external review panel using a formal mechanism that
ensures that the contentions of the self-evaluation committee are put on record. Once the self-
evaluation committee's amendments have been examined and accepted or not according to the
criteria of the members of the external review panel, the external review panel then considers
the external report to be final, which then constitutes the final evaluation report. The evaluation
process is considered to have ended with the ACAU's approval of the final report. To ensure the
transparency of the process and the visibility of the evaluation, the executive summary of the
external review report shall be made public. The institute shall have the opportunity to submit

Guide to the evaluation of R&D institutes | 16

amendments to the content of the executive summary. The executive summary must be
validated by the ACAU prior to being published.

The institute may, if it considers this to be adequate, draw up its own report for the institute
community and distribute the final report in order to safeguard its commitment to the community
to carry out the actions for improvement proposed in the report.

Meta-evaluation

The aim of this stage, which is conducted by the AQU, is to improve the evaluation process
through the participation of the different stakeholders involved in both the internal and external
stages of evaluation.

Guide to the evaluation of R&D institutes | 17

Evaluation protocol

Index:

1. Mission

2. Vision: policy and strategic plan

3. Organisation and leadership

4. Human resources

5. Infrastructure

6. External relations

7. Interaction with society, communication and image

8. Results

9. Generating financial resources (funding). Projects and agreements.

10. Teaching activity (where applicable)

11. Evaluation process

Scale of assessment:

a) Highly positive / Highly adequate / Highly favourable

b) Positive / Adequate / Favourable

c) Satisfactory / Not very adequate / Not very favourable

d) Unsatisfactory / Inadequate / Unfavourable

Guide to the evaluation of R&D institutes | 19

1. Mission
The institute's mission expresses the institute's raison d' être and is communicated by way of a
statement that defines the fundamental aim of its existence and activity, thereby distinguishing it
from other institutes.

Assessment of the mission helps to determine whether the institute is capable of achieving what
is actually stated in the mission and whether it is well defined, explained and known by both
members of the institute and society in general.

Standards

ƒ The institute's mission must be defined and documented.

ƒ The mission is relevant in relation to the general demands of the social context.

ƒ The mission is duly made public, the members of the institute know about it, and it forms
part of the culture of the institute.

ƒ The mission is implemented in basic elements and specific aims that are coherent and
relevant.

ƒ The mission is coherent with the aims of the institution that the institute depends on.

Evidence

ƒ Explicit documentation on the mission and its implementation in basic elements and specific
aims.

ƒ Validation reports on the relevance of the mission as regards the demands of the context
and the institute's characteristics.

ƒ Evidence (for example, surveys) of the degree to which groups in the institute are familiar
with the mission.

Guide to the evaluation of R&D institutes | 20

1. Mission
Key question:
Is the institute's mission adequately specified?

Highly Adequate not very adequate Inadequate

A B C D
adequate

Elements: a b c d

1.1. Degree to which the institute's mission is defined

1.2. The institute's independence to define its mission

1.3. Relevance of the mission in relation to the general

scientific framework

1.4. Degree to which the members of the institute know

about the mission

1.5. Publicity of the mission in the adequate format and

means

1.6. Visibility and knowledge of the mission outside of the

institute

1.7. The mission is coherent with the aims established by

the institute

Analysis of the institute's mission (maximum 1,500 words):

Guide to the evaluation of R&D institutes | 21

2. Vision: policy and strategic plan
The vision has to do with the expectations generated by the institute. The vision needs to be an
idealised appraisal of what the institute seeks to be in the future and of what the members want
the institute to become in the medium and long terms. The vision may be the result of a
benchmarking study carried out by the institute itself.

The vision is communicated by way of a vision statement that presents the institute's values and
principles, together with its commitments. The institute's policy, understood as being the criteria
for making decisions and designing actions to be undertaken, will guide the defining of the
processes that make up a strategic plan designed to reach a predefined state for the institute,
which will help the institute to plan how it needs to project itself in terms of the future. It is
important to point out that the institute's activities must be flexible and dynamic enough to not
just adapt themselves to and follow the institute's mission, but to be coherent with the vision and
the short and long term strategic plan for its activities.

Standards

ƒ The institute's vision is well defined and documented.

ƒ The institute's vision is viable.
ƒ The institute's policy and strategic activities plan are well defined and documented.
ƒ The institute's policy and strategic activities plan are duly made public and are known by the
members of the institute and those who have relations with it.
ƒ The strategic plan gives a clear definition of the institution's aims, the main fields of action
and the actions to be undertaken.
ƒ The strategic plan provides support for decision-making and the effective allocation of
resources.
ƒ Decision-making mechanisms exist to broaden, reorientate and close research lines.
ƒ The institute needs to consolidate a quality culture based on a policy and goals for quality
that are both known about and publicly accessible.

Evidence

ƒ Explicit documentation on the institute's vision.

ƒ Explicit documentation on the institute's policy and strategic plan.
ƒ Explicit documentation on the institute's policy and goals for quality.
ƒ Number and description of the institute's research lines.
ƒ Evidence (for example, surveys) of the degree to which groups in the institute are familiar
with the vision.
ƒ Evidence (for example, surveys) of the degree to which members of the institute are
satisfied with the institute's policy.

Guide to the evaluation of R&D institutes | 22

2. Vision: policy and strategic plan
Key question:
Are the institute's policy and strategic plan adequate in relation to its mission and vision?

Highly Adequate Not very adequate Inadequate

A B C D
adequate

Elements: a b c d

2.1. Degree to which to the institute's vision is defined

2.2. Level of feasibility of the institute's vision

2.3. The institute's independence to define its vision

2.4. External participation in constructing the institute's

vision

2.5. Degree to which the institute community knows about

the vision

2.6. Adequacy of the institute's policy in relation to its

mission and vision

2.7. Adequacy of the institute's policy in relation to its

context (university / authority that it depends on)

2.8. Adequacy of the strategic plan in relation to the

institute's mission, vision and making decisions
2.9. Assessment and satisfaction of the institute's staff in
relation to the way in which policies and strategies defined
to develop the institute are applied

2.10. Capacity of existing mechanisms to adapt, renew,

adjust and close research lines

2.11. Coherence and innovativeness of the institute's

research programmes and activities

2.12. Degree to which the institute's quality policy is

known by the members of the institute

2.13. Adequacy of the actions to define, approve, review

and enhance the policy and goals for quality
Analysis of the institute's vision (policy and strategic plan) (maximum 1,500
words):

Guide to the evaluation of R&D institutes | 23

3. Organisation and leadership in the institute
Part of the evaluation of the institute involves the examination of the institute's management
organisation and procedures, which affect the running of the institute and decision-making. One
key aspect in the assessment of this aspect is the understanding of the institute's organisational
and management structure and its adequacy for accomplishing the institute's mission and
vision.

Standards

ƒ The institute's organisation and leadership - defined and documented - is relevant in relation
to the institute's mission, vision, decision-making and activities.

ƒ The institute has a quality assurance system linked to follow-up and enhancement actions.

ƒ The institute has identified the persons responsible and the processes for internal quality
management.

Evidence

ƒ Institute's hierarchy.

ƒ List of the institute's consolidated research groups.

ƒ Description of the areas of research.

ƒ Names of the director and head of the institute's consolidated research groups and/or
areas.

ƒ List of the research programmes and activities, and of the respective persons in charge
(heads).

ƒ Description of the team in charge of internal quality management.

ƒ Existence of a process map and/or standardised processes.

Guide to the evaluation of R&D institutes | 24

3. The institute's organisation and leadership
Key question:
Are the institute's organisation and leadership adequate in relation to its mission and vision?

Highly Adequate Not very adequate Inadequate

A B C D
adequate

Elements: a b c d

3.1. Adequacy of the institute's hierarchy and

organisational structure
3.2. Suitability of the mechanisms for the institute's
management and decision-making to fulfil the mission and
vision

3.3. Mechanisms to coordinate the institute's consolidated

groups

3.4. Synergy resulting from the grouping of consolidated

groups in the institute

3.5. Degree of affinity and development of the areas of

knowledge associated with the institute

3.6. Degree of vitality and feasibility of the institute's

research lines

3.7. Suitability of the mechanisms for arranging the work

of research, teaching, administration and services.

3.8. Adequacy of the procedures for organising relations

between the institute and other closely linked entities
3.9. Suitability of the mechanisms for motivating and
recognising staff and the internal communication of
decisions that have been made

3.10. Adequacy of the processes within the framework of

the internal quality assurance system
3.11. Adequacy of the composition of the body
responsible for the follow-up and quality assurance
process

Analysis of the institute's organisation and leadership (maximum 1,500 words):

Guide to the evaluation of R&D institutes | 25

4. Human resources
The adequacy and structure of human resources is a key element in the institute's strategic
plan. Thought needs to be given to the degree to which human resources are currently
adequate in relation to the institute's mission and vision, and necessary actions for improvement
identified in order to correct any possible shortcomings.

Standards

ƒ The institute has an established staff profile.

ƒ Recruitment mechanisms are public.

ƒ Publicity and staff recruitment material clearly and accurately represent the institute's
activity.

ƒ The institute has an established system for staff recruitment, hiring and professional
development, including models for training, assessment and promotion.

ƒ The institute provides the technical research staff with support to develop research.

ƒ The institute provides researchers with auxiliary administration and services staff to carry
out administration work.

Evidence

ƒ Existence of documents on specific and transparent human resources policies (recruitment,

selection, promotion, training and mobility).

ƒ Promotion plan for researchers.

ƒ Tables 1 - 6 (Annex II) give data and indicators on human resources and staff training at the
institute during the period to be evaluated.

ƒ Evidence (for example, surveys) of the staff's degree of satisfaction regarding the institute's
human resources policy.

ƒ Evidence (for example, surveys) of the degree of satisfaction in relation to the suitability of
auxiliary research staff and administration and services staff.

Guide to the evaluation of R&D institutes | 26

4. Human resources
Key question:
Is the staff make-up (qualifications, skills, diversity, experience) adequate in relation to the
institute's mission and vision?

Highly Adequate Not very adequate Inadequate

A B C D
adequate

Elements: a b c d

4.1. Adequacy of the human resources policy (staff

recruitment, selection and promotion)

4.2. Adequacy of staff training plans

4.3. Adequacy of staff mobility and exchange plans

4.4. Beneficial effect on human resources recruitment as a

result of the institute's existence

4.5. Suitability of auxiliary technical staff for research,

administration and services
4.6. The institute's staff's satisfaction with the human
resources policy (recruitment, selection, training,
promotion and mobility)

Analysis of the human resources (maximum 1,500 words):

Guide to the evaluation of R&D institutes | 27

5. Infrastructure
The institute's mission and vision and its activities, both short and long term, call for the
existence of facilities and equipment (offices, administrative premises, laboratories, instruments,
computer resources, etc.), backed up by an infrastructure plan that is adapted to the institute's
requirements.

Standards

ƒ The institute has analysed its requirements concerning infrastructure so as to ensure that
these requirements are sufficiently met.

ƒ The institute has an infrastructure plan.

ƒ The institute guarantees the infrastructure's level of functionality (such as availability and
accessibility) and promotes its correct use.

Evidence

ƒ Documentation connected with the institute's infrastructure plan.

ƒ Documentation on the analysis of the institute's requirements.

ƒ Evidence (for example, records) on the functionality and good use made of the
infrastructure, as well as the time that it is in use.

ƒ Evidence (for example, surveys) of the degree of staff satisfaction with the infrastructure.

ƒ Tables 7 and 8 (Annex II) give data and indicators on the institute's infrastructure (facilities
and equipment).

Guide to the evaluation of R&D institutes | 28

5. Infrastructure
Key question:
Is the infrastructure adequate to the institute's mission and vision?

Highly adequate Adequate Not very adequate Inadequate

A B C D

Elements: a b c d

5.1. Degree to which the facilities and equipment are

suitable for the development of the institute's activities
5.2. Comparison of the institute's facilities and equipment
with other well-known institutes in the same area of
knowledge.

5.3. Level at which the infrastructure development plan

has been realised

5.4. Level of use of the infrastructure

5.5. Degree of access to research data bases

5.6. Degree of accessibility to equipment and facilities in

other institutions

5.7. Staff satisfaction with the available infrastructure

5.8. Degree to which available infrastructures are

profitable

5.9. Adequacy of the processes for controlling, periodically

reviewing and improving infrastructures

Analysis of the infrastructure (maximum 1,500 words):

Guide to the evaluation of R&D institutes | 29

6. External relations
The institute's external relations with its surrounding context need to be examined at both the
national and international levels, with special emphasis put on its relations with the private
sector.

Standards

ƒ The institute has an established model for its external relations that indicates its basic
strategic priorities.

ƒ The institute ensures the necessary support to promote its relations with other institutes.

ƒ The means used to publicise scientific developments represent the institute's activity in a
clear and accurate way.

Evidence

ƒ Existence of a model for its external relations, with the definition of priorities and strategies
for developing this.

ƒ Documentation connected with the need for external relations (at the national, international
and private sector levels).

ƒ Agreements signed with other national and international/foreign institutions.

ƒ Documentation relative to its policies for the diffusion and publicising of scientific
developments.

ƒ Table 9 (Annex II) gives data on the institute's external relations and the organisation of its
scientific-technical and artistic activities.

Guide to the evaluation of R&D institutes | 30

6. External relations
Key question:
Are the institute's external relations adequate for the development of its activities?

Highly adequate Adequate Not very adequate Inadequate

A B C D

Elements: a b c d

6.1. Adequacy of the mechanisms established to manage

external relations

6.2. Projection and adequacy of the institute's relations at

the national level

6.3. Projection and adequacy of the institute relations at

the international level

6.4. Mobility of the institute's staff

6.5. Degree to which stable alliances have been

developed

6.6. Relations with the private sector

6.7. Degree to which the institute participates in projects

that form part of programmes promoted by Spanish
institutions (e.g. Ministry, etc.)

6.8. Degree to which the institute participates in projects

that form part of programmes promoted by international
institutions (e.g. European Union, etc.)

6.9. Adequacy of mechanisms for the diffusion of the

institute's activities in the scientific community

Analysis of the external relations (maximum 1,500 words):

Guide to the evaluation of R&D institutes | 31

7. Interaction with society, communication and image
Communication with society constitutes a central issue for institutes. An institute's prestige is a
function of its capability to offer quality products and services, and of its skill in placing them in
the market.

The assessment of the institute's service to society is based on its high input potential in cultural
and economic sectors (technology, production, industry, etc.). In relation to its input in the
cultural sector, one of its great strengths is the fact that many lines of research are of great
social interest, while its contribution to the economic sector, aside from creating products, is
also based on the training of more highly qualified technical experts to develop projects
involving applied research and technological development.

Standards

ƒ The institute publicises its activities in society, using media that clearly and accurately
describe the institute's activity.

ƒ The institute promotes its social contribution.

Evidence

ƒ Documentary evidence of social publicising policies, marketing and communication.

ƒ Existence of marketing studies.

ƒ Internet presence of the institute's website.

ƒ Evidence (for example, surveys) of the degree to which society's opinion of the institute is
known.

ƒ Table 9 (Annex II) gives data on the institute's communication activities.

Guide to the evaluation of R&D institutes | 32

7. Interaction with society, communication and image
Key question:
Are the institute's interaction with society, communication and image adequate to the activity
that it develops?

Highly adequate Adequate Not very adequate Inadequate

A B C D

Elements: a b c d

7.1. Capacity to provide added value to the cultural and

production sector

7.2. Relevance and social interest of the activities carried

out by the institute

7.3. Mechanisms to establish the level of social demand

and the target market for the institute's activities

7.4. Mechanisms to project the institute's potential in

scientific and financial circles

7.5. Mechanisms for publicising the knowledge produced

to society as a whole

7.6. Existence and updating of the institute's website

7.7. Degree to which there is knowledge of opinions held

by society

7.8. Mechanisms to establish the level of satisfaction of

the institute's funding bodies and users

Analysis of the interaction with society, communication and image (maximum

1,500 words):

Guide to the evaluation of R&D institutes | 33

8. Results
The two basic mainstays for evaluating the institute's results are production and the impact of its
activities. According to the institute's characteristics and the areas covered by its activities, it
may be possible to easily quantify the results, although in certain cases a more qualitative
assessment of the results may be necessary (for example, services, artistic works, the
organisation of exhibitions, etc.).

Standards

ƒ The institute has a policy for publishing and publicising its activities.

ƒ The institute has mechanisms to collect evidence of the results of its activity (publications,
services provided, artistic works, etc.).

ƒ The institute improves its academic reputation through the impact of its production.

ƒ The institute has procedures that allow it to ensure that results are measured and analysed
for decision-making and enhancement of the institute's quality.

Evidence

ƒ Tables 10 and 11 (Annex II) give data and indicators on the results of the institute's
activities.

ƒ Adequacy of the technological results (where applicable).

ƒ Link between the institute's research groups and the corporate world.

ƒ Existence of documentation on the information collection system.

ƒ Description of the quality management processes relative to decision-making and

improvement of the results.

Guide to the evaluation of R&D institutes | 34

8. Results
Key question:
How adequate are the results achieved by the institute?

Highly adequate Adequate Not very adequate Inadequate

A B C D

Elements: a b c d

8.1. Adequacy of the number of scientific publications with

review and their development over time
8.2. Adequacy of the number of scientific publications without
review and their development over time

8.3. Quality of the publications and their impact on their field of

knowledge or area

8.4. Adequacy of the number of theses in progress and

defended

8.5. Adequacy of the number of papers in congresses

8.6. Existence and adequacy of a publications policy

8.7. Degree to which results of R+D are concentrated

8.8. Adequacy of other results not associated with publications

(software, services; artistic works; exhibitions; etc)

8.9. Adequacy of the number of patents and utility models

8.10. Adequacy of the institute's support to industrial incubators

and the setting up of spin-off companies

8.11. The institute's capacity to establish technology 1transfer

agreements with private enterprise

8.12. The institute's academic reputation and international

recognition by other institutes in the same field

8.13. Degree to which the institute has an influence over other

institutions
8.14. Degree to which the members of the institute participate in
science management committees and national and international
programme evaluation
8.15. Adequacy of the systems for gathering information that
provide data on the institute's results

8.16. Adequacy of the strategies and systematics for enhancing

the results
8.17. Adequacy of the processes available to the institute for
regulating and assuring decision-making processes relative to
the results
Analysis of the results (maximum 1,500 words):

Guide to the evaluation of R&D institutes | 35

9. Generation of financial resources. Projects and agreements
The financial resources that the institute generates in the form of projects and agreements,
especially those obtained in competitive schemes, are another important aspect to be
evaluated, particularly in comparison with those from direct allocations by government
authorities and the institutions that it depends on. It is important to reflect on whether the setting
up and existence of the institute has improved funding, in comparison with what the scenario
would be as defined by just the constituent groups. It should be pointed out that some of the
elements under study are analysed according to the source of origin, i.e. of Catalan, Spanish or
international origin, and from public or private sources.

Standards

ƒ The institute has a fund-raising policy for financial resources.

ƒ The institute ensures its staff receives the necessary technical and administrative support in
order to obtain financial resources.

Evidence

ƒ Analysis of the adequacy of financial resources obtained.

ƒ Tables 12 -15 (Annex II) give data and indicators on the generation of financial resources
and also operating expenses.

ƒ Evidence (for example, surveys) of the degree of satisfaction of researchers concerning the
support received to obtain economic resources.

Guide to the evaluation of R&D institutes | 36

 Generation of financial resources. Projects and
9.
agreements
Key question:
Is the generation of financial resources adequate for the activity developed by the institute?

Highly adequate Adequate Not very adequate Inadequate

A B C D

Elements: a b c d

9.1. Trend over time of the generation of resources

9.2. Adequacy of financial resources from competitive

schemes

9.3. Adequacy of the resources originating from sources

other than competitive schemes

9.4. Adequacy of resources of Catalan origin

9.5. Adequacy of resources of Spanish origin

9.6. Adequacy of resources of foreign origin

9.7. Adequacy of resources of public origin

9.8. Adequacy of resources of private origin

9.9. Contribution to resources being generated as a result

of the setting up and existence of the institute

9.10. Satisfaction of the institute's staff in relation to the

support received for generating resources

9.11. Degree to which the institute is dependent on public

funding from the institution or authority that it depends on

9.12. Degree of coherence of funding received with the

institute's lines of research

Analysis of the generation of financial resources (projects and agreements)

(maximum 1,500 words):

Guide to the evaluation of R&D institutes | 37

10. Teaching activity
In certain institutes, as in the case of university institutes, teaching is one of the activities carried
out by the staff. Consideration needs to be given to teaching in both the university context (in
one or more undergraduate and postgraduate degree programmes) and in non-university
courses.

The purpose of analysing this aspect is not to evaluate university degree programmes or
teaching staff, but to reflect on the integration of teaching with the rest of the institute's activities
in which the staff are involved.

Standards

ƒ The institute considers teaching to be one of the institute staff's activities.

Evidence

ƒ Documentation concerning the institute staff's teaching activity.

ƒ Table 16 (Annex II) describes university teaching activity (undergraduate and

postgraduate/PhD degree programmes) in which the institute's staff participates.

ƒ Documentation on non-university teaching activity in which the institute's staff participates.

Guide to the evaluation of R&D institutes | 38

10. Teaching activity
Key question:
Is teaching adequate in relation to the institute's mission and vision?

Highly adequate Adequate Not very adequate Inadequate

A B C D

Elements: a b c d

10.1. Teaching of undergraduate degree programmes

10.2. Teaching of postgraduate and PhD degree

programmes

10.3. Level of the interdisciplinary nature of teaching

undergraduate and postgraduate/PhD programmes

10.4. Adequacy of the university teaching load of the

institute's staff

10.5. Adequacy of the number of teaching staff in relation

to the number of students

10.6. Teaching of non-university courses

10.7. The institute staff's motivation to teach

10.8. Assessment of the institute staff's level of

satisfaction with teaching

10.9 Adequacy and integration of teaching with the

institute staff's other activities

Analysis of teaching (maximum 1,500 words):

Guide to the evaluation of R&D institutes | 39

11. Evaluation process
One final aspect of the evaluation is to assess how much the evaluation process itself ensures
the quality of the self-evaluation document in particular, and of the evaluation in general. It is
important to reflect on the relationship established between the institute's QA committee (if it
existed beforehand) and the self-evaluation committee.

The structure and content of the proposed evaluations should make it easy for the self-
evaluation committee to clarify or expand on any issue or point where it considers this to be
adequate. One should bear in mind that the process of planning the actions for improvement
must assure the quality of the starting point, namely, the self-evaluation.

In order to enhance the usefulness of the external review process, it is suggested that the self-
evaluation committee explains the preliminary remarks that it wishes to make to the external
review panel in order to contextualise both the reading and analysis of the self-evaluation
document itself, and with regard to the external review process. This point should be the self-
evaluation committee's final reflexion regarding the self-evaluation process and in particular the
self-evaluation document, and serves to contextualise and validate the document's content and
to adequately prepare the external review process.

Standards

ƒ The institute has a quality assurance committee.

ƒ The self-evaluation committee ensures the diffusion of information referring to the

evaluation process to the institute community.

Evidence

ƒ List of the members of the quality assurance committee.

ƒ List of the members of the self-evaluation committee.

ƒ Calendar and minutes of the meeting of the self-evaluation committee.

ƒ Documentary archive of the communications, requests for information, suggestions

received, notices, etc., made by the self-evaluation committee.

ƒ Reports by the self-evaluation committee prior to the self-evaluation document.

ƒ Individual assessments (anonymous) by each member of the self-evaluation committee of

each element in the protocol.

Guide to the evaluation of R&D institutes | 40

11. Evaluation process
Key question:
Is the evaluation process positive and adequate?

Highly Adequate Not very adequate Inadequate

A B C D
adequate

Elements: a b c d

11.1. Adequacy of the composition of the self-evaluation

committee

11.2. Attitude and response of the institute community to

the evaluation process

11.3. Support and collaboration of the institute's QA

committee

11.4. Adequacy of the process to draw up the self-

evaluation document
11.5. Actions for the diffusion of the self-evaluation
document and to encourage participation in the evaluation
process

11.6. Overall assessment of the self-evaluation document

Analysis of the assessment process (maximum 1,500 words):

Guide to the evaluation of R&D institutes | 41

Bibliography

European Commission. 2004. Evaluating EU activities. A practical guide for the Commission
services. Published by EU publications office, Luxembourg.

European Foundation for Quality Management - EFQM. 2003. Introduction to excellence.

http://www.efqm.org

Juárez, M.; de Andrés, R.; Martínez, J.; Solís, J. 2005. Un modelo genérico de protocolo para
the evaluación de centros de I+D. Published by FECYT, Madrid.

Vereniging van Universiteiten / Association of Universities in the Netheralands (VSNU),

Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWO), Koninklijke Nederlandse
Akademie van Wetenschappen (KNAW). 2003. Standard evaluation protocol 2003-2009 for
public research organisations. <http://www.nwo.nl>

Guide to the evaluation of R&D institutes | 42

Annex I. Institute fact sheet

Institute fact sheet

NAME OF THE INSTITUTE ACRONYM

Director of the institute

Name:
Contact person
e.mail:
Postal address

e.mail

Website

Telephone

Fax

Year established

Participating entities

Legal representative
(name, address, telephone,
fax, e.mail)

Period of evaluation

FIELDS OF SCIENCE AND

TYPE OF INSTITUTE LEGAL PERSONALITY
TECHNOLOGY

Natural Sciences University

Consortium
Engineering and Interuniversity
Foundation
technology Joint
Public corporation
Medical sciences
No legal personality
Agricultural sciences
Other (specify)
Social sciences
Humanities

Guide to the evaluation of R&D institutes | 43

Annex II. Tables with administrative data,
statistics and indicators

The tables to be used for evidence during the evaluation process are provided below. With
regard to the indicator tables, take note that the indicators given are only for guidance and serve
as examples. The self-evaluation committee is free to extend and modify the tables wherever
necessary in order to provide all of the necessary supplementary information to improve the
institute's evaluation.

Table 1. Overall data on the structure and human resources

Table 2. Human resources indicators

Table 3. Staff training data

Table 4. Staff training indicators

Table 5. Data on grants applied for and awarded per year of application

Table 6. Research grant indicators

Table 7. Data on infrastructure: facilities and equipment

Table 8. Indicators for infrastructure: facilities and equipment

Table 9. Data on the institute's external relations, organisation of scientific-technical and artistic
activities, and communications

Table 10. Data on the results of the institute's activity

Table 11. Indicators of the results of the institute's activity

Table 12. Data on the generation of financial resources: projects and agreements

Table 13. Data on the generation of financial resources (in thousands of euros; %)

Table 14. Indicators on the generation of financial resources

Table 15. Data on the institute's budget and operating expenses (in thousands of euros; %)

Table 16. University teaching by the institute's staff (credits taught)

Guide to the evaluation of R&D institutes | 44

 Table 1. Overall data on the structure and human resources

Current Year - 1 Year - 2 Year - 3 Year - 4

year

No. of areas/lines of action

No. of research groups

No. of consolidated research groups

No. of head researchers (area/group)

No. of PhD researchers on the institute's staff

No. of PhD researchers attached to the institute

No. of visiting PhD researchers

Total PhD

No. of researchers in training

No. of visiting researchers in training

Total no. PhD

Total researchers (PhD + non-PhD)

No. of auxiliary (technical support) research staff

No. of administration and services staff

Total staff

NB: The information given in this table may be extended to include information on the gender, age and teaching posts held
by the institute's staff.

Guide to the evaluation of R&D institutes | 45

 Table 2. Human resource indicators

Current Year -1 Year - 2 Year - 3 Year - 4

year

No. of PhD researchers / total no. of researchers

No. of auxiliary (technical support) research staff / total no. of

researchers

Total no. of researchers / no. of research groups

Other (specify)

Guide to the evaluation of R&D institutes | 46

 Table 3. Data on training staff

Current Year - 1 Year - 2 Year - 3 Year - 4

year

Grants and contracts

No. of post-doctoral grants and contracts

No. of pre-doctoral grants and contracts

Congresses

No. of attendances at national congresses

No. of attendances at international congresses

No. of times presided over the organisation of

Spanish congresses

No. of times presided over the organisation of

international congresses

No. of invitations to Spanish conferences

No. of invitations to international conferences

No. of times presided over Spanish sessions

No. of times presided over international sessions

Stays abroad

No. of stays abroad

Courses

No. of participations in courses for researchers

No. of participations in courses for auxiliary

(technical support) research staff

No. of courses organised

No. of participations in courses for administration

and services staff

Guide to the evaluation of R&D institutes | 47

Table 4. Staff training indicators

Current Year - 1 Year - 2 Year - 3 Year - 4

year

No. of pre-doctoral grants and contracts / no.

of researchers

No. of post-doctoral grants and contracts / no.

of researchers

No. of participations in national congresses /

no. of researchers

No. of participations in international

congresses / no. of researchers

No. of researchers abroad / no. of researchers

No. of participations in courses aimed at

research staff / no. of researchers

No. of participations in courses aimed at

auxiliary technical staff / no. of auxiliary
technical staff

No. of participations in courses aimed at

administration and services staff / no. of
administration and services staff

Other (specify)

Guide to the evaluation of R&D institutes | 48

 Table 5. Data on grants applied for and awarded per year of application

Current year Year -1 Year - 2 Year - 3 Year - 4

Entity
Applied Awarded Applied Awarded Applied Awarded Applied Awarded Applied Awarded
for for for for for

Local

Regional (Autonomous Community)

Ministry of Education and Science

Other Spanish ministries and public bodies

European Union

Private enterprise

Total

No. of predoctoral grants

No. of postdoctoral grants

Total grants (pre and post-doctoral)

Guide to the evaluation of R&D institutes | 49

 Table 6. Research grant indicators

Current year Year -1 Year - 2 Year - 3 Year - 4

No. of predoctoral grants awarded / No. of predoctoral grants

applied for

No. of postdoctoral grants awarded / No. of postdoctoral grants

applied for

Total no. of grants awarded / Total no. of grants applied for

Other (specify)

Guide to the evaluation of R&D institutes | 50

Table 7. Data on infrastructure: facilities and equipment

Current Year -1 Year - 2 Year - 3 Year - 4

year

Work areas

Total surface area (m2)

Surface area for staff (m2)

Surface area for laboratories (m2)

Surface area for materials storage (m2)

Computer and audiovisual resources

Computers for the exclusive use of staff

On-line connections

Audiovisual equipment for staff use

Specialised equipment

Scientific and technical equipment1

Supercomputer calculation equipment1

1
List the most significant equipment

Table 8. Infrastructure indicators: facilities and equipment

Current Year - 1 Year - 2 Year - 3 Year - 4

year

Surface area for staff / no. of persons

Surface area for laboratories / no. of

researchers

No. of computers / no. of researchers

Other (specify)

Guide to the evaluation of R&D institutes | 51

Table 9. Data on the institute's external relations, organisation of scientific-technical and
artistic activities, and communications

Current Year -1 Year - 2 Year - 3 Year - 4

year

No. of coordinated projects

No. of stays by staff from other institutes

No. of invitations received from national

institutions

No. of invitations received from international

institutions

No. of congresses, seminars, workshops,

etc. organised at the national level

No. of congresses, seminars, workshops,

etc. organised at the international level

No. of courses and seminars given

No. of visits / hits on the institute's website

No. of times the institute appears in the

media

No. of times the institute has participated in

projects promoted by Spanish institutions

No. of times the institute has participated in

projects promoted by international
institutions

Other (specify)

Guide to the evaluation of R&D institutes | 52

 Table 10. Data on the results of the institute's activity

Current Year -1 Year - 2 Year - 3 Year - 4

year

Production and scientific publications

No. of articles in reviewed indexed journals

No. of book chapters

No. of books and monographs

No. of theses defended

No. of theses under way

No. of citations of the scientific production

No. of articles in non-reviewed and general interest journals

No. of papers / presentations at national congresses

No. of papers / presentations at international congresses

No. of professional and scientific-technical reports

Technology, services and artistic results

No. of patents and utility models

No. of associated incubator companies

No. of spin-off initiatives

No. of services provided

No. of knowledge transfers to the production sector

No. of journals and books published

No. of artistic works produced

No. of exhibitions organised

Other (specify)

Other results of the institute's activity

Software produced, CD-ROM, etc.

Guide to the evaluation of R&D institutes | 53

No. of awards and distinctions

No. of times the members of the institute have participated

in Spanish science management committees and
evaluation programmes

No. of times the members of the institute have participated

in international science management committees and
evaluation programmes

Table 11. Indicators of the results of the institute's activity

Current Year -1 Year - 2 Year - 3 Year - 4

year

No. of articles in the year / total no. of

articles

No. of book chapters in the year / total no.

of book chapters

No. of theses defended in the year / total

no. of theses defended

No. of articles / no. of PhD researchers

No. of theses defended / no. of PhD

researchers

No. of theses under way / no. of PhD

researchers

No. of citations / no. of articles in reviewed

indexed journals

No. of patents in the year / total no. of

patents

Other (specify)

Guide to the evaluation of R&D institutes | 54

 Table 12. Data on the generation of financial resources: projects and agreements

Current Year -1 Year - 2 Year - 3 Year - 4

year

No. of competitive projects applied to in

Catalonia

No. of competitive projects awarded in

Catalonia

No. of competitive projects applied to at

the national level (Spain)

No. of competitive projects awarded at

the national level (Spain)

No. of competitive projects applied to at

the international level

No. of competitive projects awarded at

the international level

No. of non-competitive projects in

Catalonia

No. of non-competitive projects in Spain

No. of non-competitive projects at the

international level

No. of agreements with companies

No. of participations in national networks

No. of participations in international

networks

No. of participations in networks of

excellence

Guide to the evaluation of R&D institutes | 55

 Table 13. Data on the generation of financial resources (in thousands of euros; %)

Current Year -1 Year - 2 Year - 3 Year - 4

year

Allocations from the university/-ies

Allocations from government sources

Competitive projects in Catalonia

Competitive projects at the national level (Spain)

Competitive projects at the international level

Non-competitive projects in Catalonia

Non-competitive projects at the national level (Spain)

Non-competitive projects at the international level

Agreements with companies

Patents and utility models

Participation in national networks

Participation in international networks

Provision of services, trials and analyses

Organisation of scientific-technical, services and artistic

activities

Total

Guide to the evaluation of R&D institutes | 56

 Table 14. Indicators on the generation of financial resources

Current Year -1 Year - 2 Year - 3 Year - 4

year

No. of competitive projects awarded / no. of

competitive projects applied to (Catalonia)

No. of competitive projects awarded / no. of

competitive projects applied to (national level
- Spain)

No. of competitive projects awarded / no. of

competitive projects applied to (international
level)

No. of competitive projects awarded / no. of

researchers (Catalonia)

No. of competitive projects awarded / no. of

researchers (national level - Spain)

No. of competitive projects awarded / no. of

researchers (international level)

No. of agreements with companies / no. of

researchers

No. of participations in national networks /

no. of researchers

No. of participations in international networks

/ no. of researchers

No. of participations in networks of

excellence / no. of researchers

Funding awarded / no. of PhD researchers

Other (specify)

Guide to the evaluation of R&D institutes | 57

 Table 15. Data on the institute's budget and operating expenses (in thousands of euros;
%)

Current Year -1 Year - 2 Year - 3 Year - 4

year

Research staff

Auxiliary (technical support) research staff

Administration and services staff

Direct costs of the institute's activity

(consumable material; trips; expenses;
procurement of durable equipment)

Ordinary operation

Other (specify)

Total

Guide to the evaluation of R&D institutes | 58

 Table 16. University teaching by the institute's staff (credits taught).
Current year Year -1 Year - 2 Year - 3 Year - 4

Credits Credits Credits Credits Credits

No. of staff

No. of staff

No. of staff

No. of staff

No. of staff
graduate /

graduate /

graduate /

graduate /

graduate /
graduate

graduate

graduate

graduate

graduate
Degree programme:

Under-

Under-

Under-

Under-

Under-
Post-

Post-

Post-

Post-

Post-
PhD

PhD

PhD

PhD

PhD
Full Professor

Associate Professor

Professor at a university polytechnic

Associate professor at a university

polytechnic
ICREA Senior

ICREA Junior

Tenure-track

Teaching assistant

Part-time instructor

Research fellow

Other (visiting professor, non-tenured

lecturer, etc.)

NB: Prepare a table for each degree programme in which the institute's staff participates

Guide to the evaluation of R&D institutes | 59