National Pharmacy Service, Pharmaceuticals Supply Chain and

Medical Equipment Management Monitoring and Evaluation Framework

National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Acknowledgments

The Federal Ministry of Health would like to express its gratitude and appreciation to
all participants and their respective health institutions who were involved in this doc-
ument preparation for their relentless efforts and commitment. The shared technical
knowledge and experiences have produced a comprehensive framework for realiza-
tion of simple, coordinated and effective results-based national M&E system for data
management, dissemination and utilization of strategic information for pharmaceutical
supply chain management, pharmacy service and medical device.

The support and commitment of the members of the technical team at the Ministry of
Health and partner organizations are also very well acknowledged. Their active partici-
pation and engagement in the development process have been the driving force to the
completion of this document successfully.

Sincere appreciation is extended to the following members of the technical team

whose support was central to the development, coordination, and finalization of the
entire M&E framework document

Finally, FMOH would like to express its gratitude to United Nation Population Fund
(UNFPA), Clinton’s Health Access Initiative (CHAI), JSI/AIDSFree, On behalf of the Fed-
eral Ministry for Economic Cooperation and Development (BMZ), the Deutsche Ge-
sellschaft für Internationale Zusammenarbeit (GIZ) and the United States Agency for
International Development (USAID) Global Health Supply Chain Program - Procure-
ment and Supply Management (GHSC-PSM) project in Ethiopia for the dedicated and
full support for successful completion and printing of the M&E framework document.

i
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Acronyms

DIS Drug Information Service

DTC Drug and Therapeutic Committee
DTP Drug Therapy Problem
EPSA Ethiopian Pharmaceuticals Supply Agency
FMOH Federal Ministry of Health
GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit
HFs Health Facilities
IPLS Integrated Pharmaceuticals Logistics System
PMED Pharmaceuticals and Medical Equipment Directorate
PSCM Pharmaceutical Supply Chain Management
RHB Regional Health Bureau
ScHO Sub city Health Office
UDS Unit Dose Dispensing
WoHO Woreda Health Office
ZHD Zonal Health Department

ii
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Table of Contents
Acknowledgments i
Acronyms ii
Foreword v
Executive Summary vi
Chapter one: Introduction 1
1.1. Background 1
1.2 Monitoring and Evaluation Situation Analysis 3
1.3 Rationale for Developing the M&E framework 4
ChapterTwo: Overview of the Monitoring & Evaluation Framework 5
2.1. Objectives of M&E Framework 5
2.2. Guiding Principles 5
2.3. Development Processes of the M&E framework 5
2.4. Summary of the Monitoring and Evaluation Framework 6
2.5 Categorization of Indicators 6
2.6. Intended Users of the Document 7
Chapter Three:The National Pharmaceutical SCM, Pharmacy Service and Medical
Device Monitoring and Evaluation Indicators 9
3.1. Pharmacy Service Indicators 9
PS1. Drug and therapeutics committee (DTC) Functionality 9
PS2. Availability of health facility specific medicine list 10
PS3. Availability of standard treatment guidelines 10
PS4. Percentage of medicines prescribed from the facility’s medicines list 10
PS5. Average number of medicines per encounter 11
PS6. Percentage of encounters with an antibiotic prescribed 11
PS7. Average dispensing counselling time 12
PS8. Percentage of medicines adequately labelled 12
PS9. Patients’ knowledge on correct dosage 13
PS10. Percentage of clients with 100% prescribed drugs filled 13
PS11. Clinical pharmacy service functionality 14
PS12. Hospital with functional unit dose dispensing system (UDS) 14
PS13. Percentage of hospitals with functional DIS 15
PS14. Percentage of hospitals with functional compounding services 15
PS15. APTS functionality 16
PS16. Client satisfaction with dispensing services 16
3.2. Supply Chain Indicators 17
SC1. Forecast accuracy 17
SC2. Supplier fill rate 17
SC3. Average lead time 18
SC4. Stock out duration 18
SC5. Wastage rate 19
SC6. Percentage of facilities that maintain acceptable storage conditions 19
SC7. Inventory accuracy rate 20
SC8. RRF reporting rate 20
SC9. Essential drugs availability 21
SC10. Disposal of unfit-for-use medicines 21

iii
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

3.3. Medical Equipment Indicators 22

MD1. Availability of functional medical equipment 22
MD2. Percentage of health facilities with updated medical equipment inventory 22
MD3.Percentage of health facilities with functional medical equipment management committee
(MEMC) 23
MD4. Percentage of health facilities with scheduled preventive maintenance practice 23
MD5. Percentage of medical equipment installation 24
MD6. Biomedical professional positions filled at health facilities 24
MD7. Availability of medical equipment as per the national standard 24
3.4. Pharmacy Services and Pharmaceutical Supply Chain Management Cross Cutting
Indicators 25
SC-PS1. Pharmacy review meetings conducted 25
SC-PS2. Supportive supervision of health facility pharmacies 25
SC-PS3. Percentage of pharmacy workforce positions filled at health facilities 26
Chapter Four: Data Collection,Analyses and Utilization 27
4.1 Flow of reports 27
4.2 Reporting schedule 28
4.3 Collection and analysis of data 28
4.3.1 Collection and analysis of data at facility level 28
4.3.2 Collection and analysis of data at administrative levels 29
4.4 Utilization of data 29
4.4.1. Utilization of data at health facility level 29
4.4.2. Utilization of reports by WoHOs/RHBs/FMOH 29
4.5 Data quality assurance 30
4.6. Roles and responsibilities 30
Annex 31
Annex 1. Registration formats for pharmacy service indicators 31
Annex 2. Registration formats for supply chain indicators 39
Annex 3. Registration and reporting formats for medical equipment indicators 43
Annex 4: Registration formats for supply chain and pharmacy service crosscutting indicators 47
Annex 5: Reporting format 48
Annex 5.1: Reporting format for Health Center 48
Annex 5.2: Reporting format for Hospital 49
Annex 5.3: Reporting format for Woreda 51
Annex 5.4: Reporting format for Zone 54
Annex 5.4: Reporting format for Region 57
Annex 6: List of workshop participants 59

iv
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Foreword

The Federal Ministry of Health (FMOH) has been coordinating sector wide reforms
that aim to improve equity and quality of health services. As part of these efforts,
the ministry is also exerting concerted efforts to improve accessibility and quality of
pharmaceutical products and services. It is widely known that, the sector is growing in
line the overall growth and transformation plan of the country and the sector is being
guided by the health sector transformation plan (HSTP).

Pharmaceutical supply chain management and pharmacy service activities are an inte-
gral part and a cross cutting activity of the health care system. Managing pharmaceutical
supply chain, pharmacy service and medical device is a key for fulfilling basic customer
satisfaction with regards to obtaining the right product with right quantity and right
condition, at the required time. Therefore, the purpose of this M&E plan is to strength-
en the pharmaceuticals supply chain management, pharmacy service and medical device
management of the country to ensure uninterrupted supply of pharmaceuticals for the
ultimate customers. Also the M&E framework will help FMOH to build the capacity
of professionals working at different levels of the system so as to properly manage
pharmaceuticals SCM, pharmacy service and medical devices. The M&E framework is
developed by the national ME TWG established by plan, ME Directorate.

As the development of this framework is a significant achievement, it would be mean-

ingful only if the M&E framework of all stakeholders engaged in pharmaceutical SCM,
pharmacy service and medical device is built on this common framework. Realization
of this framework requires effective leadership by the government and commitment,
dedication, and concerted action of all stakeholders.

Regasa Bayisa (B.Phram, MSC)

Director, Pharmaceutical and Medical Equipment Directorate
Federal Ministry of Health

v
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Executive Summary

Ethiopia Health Sector Transformational Plan (HSTP 2016-2020) lay emphasis on the
need to have strong health commodity supply chain management and pharmacy ser-
vices to fulfill customer satisfaction with regards to obtaining the right pharmaceutical
with right quantity and right condition, at the required time, for the right client. It
demonstrates the performance of the pharmacy service and logistics system, highlights
successes, and informs the design of appropriate interventions for areas that need
improvement.

Therefore, it’s essential to have implementable monitoring and evaluation (M&E) frame-
work that help continually improve pharmaceutical supply management (PSM), pharma-
cy service (PS) and medical equipment management (MEM) performance.

In Ethiopia, however; the M&E system for PSM, PS and MEM lacked standardization and
was implemented in a fragmented manner. Recognizing this, the FMOH, through Phar-
maceuticals and Medical Equipment Directorate (PMED) in collaboration with partners
developed this national M&E framework.

The aim of M&E framework is to assist the FMOH, EPSA, RHBs, ZHDs, and Woreda
health offices, health facilities, donor agencies and development partners in evaluating
performance and identifying the factors which contribute to its service delivery out-
comes.

The main purposes of the M&E framework are:

(i) To provide guidance for gathering of timely, accurate and complete informa-
tion for monitoring and evaluating PSM, PS and ME.
(ii) To standardize data collection and reporting tools and procedures
(iii) To promote information sharing among stakeholders
(iv) To promote informed decision making
(v) To promote continuous improvement in the pharmaceutical sector through
timely identification and addressing of implementation challenges

This M&E framework comprises a range of indicators at various levels to measure,

monitor and evaluate both implementation and impact of pharmaceutical related in-
tervention. There are a total of 36 indicators grouped into four areas namely: PS, PSM,
MEM, and Supply chain and pharmacy service crosscutting.

Some indicators will be collected and used by each level without reporting to the next
level while selected indicators will be reported to the next level. The data elements for
the four KPIs that can be captured by the routine HMIS system will follow the HMIS
reporting system.

vi
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Chapter one: Introduction

1.1. Background
Health Service in Ethiopia
Ethiopia is located in Sub Saharan Africa. The country has a decentralized administra-
tive system that consists of nine Regional States and two City Administrations. At the
national level, the Federal Ministry of Health (FMOH) is responsible for health policy
making, strategic planning, coordination and harmonization of all health actors and
stakeholders. It coordinates and harmonizes action plans of all actors in the health sec-
tor including national and international organizations; provides technical support and
guidance to Agencies under it, Regional and city administration Health Bureaus, as well
as monitor the execution of the action plans and evaluate performances periodically.

Health system service provision is organized in a three-tier system. These include Pri-
mary Healthcare Units (PHCU) composed of primary hospitals, health centers and
health posts that serve 60,000 – 100,000 people. The second tier consist general hos-
pitals that serve 1-1.5 million people. The third tier consist specialized hospitals that
render tertiary level healthcare for 3.5 - 5 million people.

Currently, the sector is guided by a 20-year strategic plan document, envisioning Ethio-
pia’s path towards universal health coverage through strengthening primary health care.
A 5-years strategic plan, Health Sector Transformation Plan, 2015-2020 (HSTP), that is
derived from the envisioning document and which is also part of the second Growth
and Transformation Plan (GTP II) of the government, is developed detailing the sector’s
priority until 2020. The HSTP aims to transform the health sector so as to further
improve equity, coverage and utilization of essential health services, improve quality of
healthcare, and enhance the implementation capacity of the health sector at all levels
of the system.

In the HSTP, addressing triple burden of diseases namely communicable diseases,

non-communicable diseases and accidents/injuries is given much focus. The HSTP has
identified three key features: quality and equity; universal health coverage and transfor-
mation. It also sets out four pillars of excellence which are believed to help the sector
to achieve its mission and vision. These are:
1. Excellence in health service delivery
2. Excellence in quality improvement and assurance
3. Excellence in leadership and governance
4. Excellence in health system capacity

The strategic initiatives that are prioritized in the HSTP include improving supply chain
and logistics management, pharmacy service, use of technology and innovation, devel-
opment and management of human resource for health.

Major performance measures for improving medicines and medical equipment manage-
ment stated in the HSTP include:
• Increase availability of essential drugs for primary, secondary and tertiary
healthcare to 100%,
• Reduce wastage rate to less than 2%,
• Increase proportion of essential drugs procured from local manufactur-
1
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

ers from 25% to 60%

• Reduce procurement lead-time from 240 days to 120 days.
• 80% of facilities equipped with medical equipment as per the essential
medical equipment list

The government intends to achieve these targets through pharmaceutical supply chain
management, pharmacy service and medical equipment management strategic ini-
tiatives which include scale-up of auditable pharmaceutical transaction and services,
scale-up community pharmacies, enhancing efficiency in selection, quantification and
procurement of essential medicines, developing an essential medical equipment list,
strengthening and scaling-up of the training of biomedical engineers and technicians,
and establishing a medical equipment maintenance center etc.

The Ethiopian Government has endorsed the Ethiopian Hospital Service Transforma-
tion Guideline (EHSTG) and Ethiopian Health Centres Reform Implementation Guide-
lines (EHCRIG) which have chapters on operational standards for pharmacy services
and medical equipment management. These guidelines have indicated performance
standards and guidance to ensure efficient and quality service delivery in hospitals
and health centres. Although all those strategic documents and guidelines have given
considerable focus to the pharmacy service, supply chain and medical equipment man-
agement, there are many challenges in their implementation.

Pharmaceutical supply chain management, medical equipments and pharmacy service

activities are integral parts of the healthcare system. They are key for fulfilling basic
customer satisfaction with regards to obtaining the right pharmaceutical with right
quantity and right condition, at the required time, for the right client.

Cognizant of the pivotal role of pharmaceuticals, the Government of Ethiopia estab-

lished Pharmaceutical Fund and Supply Agency (PFSA), currently renamed as Ethiopian
Pharmaceuticals Supply Agency (EPSA), which is responsible to ensure uninterrupted
supply of quality assured pharmaceuticals to the public at affordable price through
strengthening Integrated Pharmaceutical Logistics System (IPLS), efficient procurement,
improved warehousing and inventory management, and efficient distribution of phar-
maceuticals to health facilities. Routine monitoring reports show that IPLS is improv-
ing information recording and reporting, storage and distribution systems, as well as
the availability of essential commodities at service delivery points (SDPs). The recent
national survey conducted on IPLS to measure system performance at public health
facilities (hospitals, health centers and health posts) indicated that the system has sig-
nificantly improved the availability of essential pharmaceuticals at health facilities.

To further improve the availability and rational utilization of medicines and medical
equipment, a directorate, Pharmaceuticals and Medical Equipment Directorate (PMED),
was established within FMOH. This directorate is composed of three case-teams: Phar-
maceuticals Supply management case-team, Pharmacy service case-team, and medical
equipment case-team. The directorate was first established as Pharmaceuticals Logis-
tics Management Unit (PLMU) in 2012 to oversee and facilitate the smooth implemen-
tation of supply chain management of pharmaceuticals. The unit was established based
on the findings of the mid-term review of the Health Sector Development Plan IV
(HSDP IV).The mid-term review found out that supply chain management and pharma-
cy service activities had become uncoordinated and weak primarily due to unavailability
of strong departments which is responsible to coordinate and oversee SCM activities
at FMOH and lower levels.
2
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

In Ethiopia, Medical equipment (ME) almost exclusively acquired through donation or

through purchase. According to anecdotal data from EPSA, the past few years’ pro-
curement of ME has significantly increased due to increase in construction of primary
hospitals, HCs and health posts. On average, EPSA has procured 2.9, 2.7 and 2.8 billion
birr worth of MEs in 2006 EC, 2007 EC and 2008 EC respectively. In addition to EPSA,
some partners also procured MEs to the Ministry.

Some reports showed that inferior quality medical equipment are procured and distrib-
uted to health facilities. As a result, they are damaged without providing the required
services and eventually risk service disruptions or have a prolonged downtime. In ad-
dition, multiple brands are being supplied which caused significant burden to avail the
required different types of consumables and to address training requirements. Although
standards are set to define which medical equipment to be availed at the different levels
of the health system, the current availability of medical equipment at different levels is
haphazard and not guided strategically.
As there is no standardized monitoring and evaluation (M&E) system, it is difficult to
identify and analyze problems and provide directions in the management of medicines
and medical equipment.

1.2 Monitoring and Evaluation Situation Analysis

The review of the existing national and regional health supply chain and pharmacy
service M&E system indicated that all RHBs and City Administration Health Bureaus
(CAHBs) introduced Health Supply Chain Management and pharmacy service M&E
system since 2014 with the objective of improving health supply chain and pharmacy
services. EPSA has also drafted M&E framework in 2017 to assess and improve SCM
functions which are managed by the agency.

Major findings of the review are summarized below:

• Most of the RHBs and CAHBs have M&E system with defined Key per-
formance Indicators (KPIs).
• Key indicators that are routinely monitored by RHBs and CAHBs include
order fill rate, line fill rate for program and budget items, proportion of
availability of essential tracer drugs, and proportion of stock wasted due
to expiry and damage.
• Health supply chain M&E trainings were provided for RHBs/CAHBs,
ZHDs, WoHOs, and selected hospitals.
• Supportive supervision has been conducted at selected health facilities.
• RHBs and CAHBs have been collecting, analyzing and providing feedback
in the implementation of the health supply chain management M&E re-
ports of WoHOs and health facilities.

The major weaknesses/gaps identified were:

• FMOH did not develop national M&E framework that measure perfor-
mances at all levels of the health system and can serve as reference for
lower level M & E systems design.
• There is lack of harmonization and alignment between the regional
PSCM M&E KPIs and M&E plans implemented by the RHBs and CAHBs
and little or no involvement of regional M&E and planning units
• All regions have different M&E plans, different number of KPIs, reporting
systems and reporting tools. As the involvement of the FMOH was min-
imal, there was no way to standardize the framework.
• Skill gaps were observed in collecting quality data, analysis and reporting
3
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework
• Lack of well-developed data recording and management guides/manuals
• Poor data documentation, utilization and feedback system
• Lack of electronic system for data collection and reporting

1.3 Rationale for Developing the M&E framework

Routine monitoring and evaluation (M&E) of the pharmaceutical supply chain, pharma-
cy service and medical equipment management enhances efficiency and effectiveness.
Having an M&E system helps to ensure that the right product is delivered in the right
quantity, right condition, and at the right time. It demonstrates the performance of the
SCM, medical equipments, and related services; highlights successes, and informs areas
that need improvement.
However, in Ethiopia, the M&E system for pharmaceutical supply chain, pharmacy ser-
vice and medical equipment lacked standardization and was implemented in a fragment-
ed manner.

Recognizing this, the FMOH, through Pharmaceuticals and Medical Equipment Direc-
torate (PMED) and in collaboration with developmental partners, developed this na-
tional M&E framework.
The M&E Framework will provide stakeholders with a tool for well-coordinated, har-
monized and functional M&E systems that enhances evidence-based decision making in
pharmaceutical SCM, Pharmacy service and Medical equipment.

In order to develop appropriate M&E system, it is necessary to define the benefits that
the designed M&E system will bring into the healthcare system. Some of the benefits
that this M&E framework will bring into the Ethiopian health system include, but not
limited to:
• Standardization: Common definitions of indicators, data collection in-
struments, and data management procedures form the foundation for
effective M&E system. Without these, performance cannot be systemat-
ically measured and improved across different geographical locations or
over time.
• Coordination: One national M&E framework, shared by all actors and
stakeholders, is critical for effective M&E system. This principle helps to
avoid duplication of efforts among stakeholders and ensures to generate
complete data that show the full picture of programs.
• Integration: Collecting and reporting of pharmaceutical SCM, pharmacy
service and medical equipment related data in an integrated way brings
efficiency into the system. Each intervention should align with the stan-
dard indicators and reporting format that will guide tracking the pro-
gresses made.
• Decentralization: Analysis and storage of data takes place at the level
where it’s collected and used for evidence-based decision making.
• Simplification: Collecting, analyzing, and interpreting only the information
that is immediately relevant to performance improvement and makes
best use of scarce resources.
• Transparency and Accountability: M&E framework of pharmaceutical
SCM, pharmacy service and medical equipment has to be open and par-
ticipatory for stakeholders at all levels.Those in charge of data collection,
analysis, timely reporting, and policy decisions must take ownership of
and accountable for their actions and be able to professionally defend
their reports and/or decisions. All stakeholders and participants have to
agree on and abide by this key principle.
4
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Chapter Two: Overview of the Monitoring & Evaluation Frame-

work
2.1. Objectives of M&E Framework
General Objectives
• To provide a comprehensive framework for realization of simple, coordinated
and effective results-based national M&E system for data management, dissem-
ination and utilization of strategic information for pharmaceutical supply chain
management, pharmacy service and medical equipment.
• To enhance multi-sector partnerships, networking, collaboration and account-
ability with all stakeholders through strengthening existing platforms for M&E of
pharmaceutical supply chain management, pharmacy service and medical equip-
ment at all levels.

Specific objectives
• To provide guidance for gathering of timely, accurate and complete informa-
tion for monitoring and evaluating pharmaceutical supply chain management,
pharmacy service and medical equipment.
• To standardize data collection and reporting tools and procedures across all
levels.
• To promote information sharing among stakeholders.
• To promote informed decision making.
• To promote continuous improvement in the pharmaceutical sector.

2.2. Guiding Principles

In order to develop appropriate M&E framework, it is necessary to set guiding prin-
ciples, which the system and the measurement items (metrics) and processes can be
screened. Such principles include the following:
• Consistent with pharmaceutical supply chain management, pharmacy service
and medical equipment strategic objectives in HSTP.
• Consistent with both national and international standards
• Feasibility (in terms of cost, time, data collection and capturing burden)
• Relevance of the indicator
• Basic principles of Health Information System (Simplification, Integration, Stan-
dardization and Institutionalization)

2.3. Development Processes of the M&E framework

To develop the M&E framework, a technical team, comprising of different expertise
from ministry of health and development partners and led by pharmaceuticals and
medical equipment directorate has been established. The technical expertise included
pharmacists, supply chain management specialists, biomedical engineers and M&E ex-
perts.

After many consultations at technical team level, two consultative workshops were
conducted to develop the draft M&E framework. Furthermore, the draft M&E frame-
work was presented and shared on the annual PMED review meeting in 2018, to get
inputs from potential stakeholders. After repeated meetings of the technical team to
incorporate comments, a second version of the M&E framework was drafted, which
became ready for larger audience comments and for a validation workshop.
5
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Finally, the validation workshop was conducted from February 13 – 15, 2019 in Bishoftu.
A total of 37 participants from Regional Health Bureaus, selected Zone health depart-
ment and Hospitals, Pharmaceutical Supply Agency (EPSA), Policy Plan Monitoring and
Evaluation Directorate (PPMED), Pharmaceuticals and Medical Equipment Directorate
(PMED), Public Relation (PR), UNFPA, GHSC-PSM, CHAI and JSI/AIDSFree represen-
tatives attended the workshop (list of participants annexed). At the end comments
from the validation workshop participant were reviewed and incorporated to the final
document by the technical team.

2.4. Summary of the Monitoring and Evaluation Framework

The M&E framework provides a foundation for performance monitoring and evalu-
ation of the pharmaceutical supply chain, pharmacy service and medical equipment
management of the country. The framework helps to monitor how program activities
contributes to the achievement of effectiveness and efficiency of pharmaceutical supply
chain management system, availability and quality of pharmacy services and improved
medical equipment availability, utilization and management practices. It is outlined in
Figure 1, showing how inputs are translated into outputs, outcomes and impact. System
inputs, processes and outputs reflect systems capacity, whereas outcomes and impact
reflect systems performance.

Multiple data sources will be used in the implementation of the M&E framework. Data
sources will include routine administrative sources (such as HMIS), surveys and sup-
portive supervision findings.Various input, output, and outcome indicators are included
in the M&E framework. Input indicators will help ensure that resources are properly
mobilized, equitably distributed and efficiently utilized. Output indicators will be used to
measure utilization and coverage. Outcome and impact indicators have the advantage
of being “integrative” (i.e. many different factors are “integrated” into the outcome/im-
pact), reflecting the result of interventions within and outside the sector. A total of 36
indicators are selected to monitor and evaluate the sector. In addition, regions can have
specific indicators related to their operational and program monitoring and evaluation.

The PYRAMID shape information flow will be systematically strengthened by identi-

fying more indicators to be utilized at lower level such as districts and health facilities.
Data analysis will be conducted starting from facility level to national level to be used
for evidence-based decision making. M&E findings will be disseminated to stakeholders
using different channels. Quarterly and annual reports will be produced and shared
to stake holders. The data will be used in performance review meetings to review
strengths and challenges and to agree on future interventions. FMOH will conduct
inspections to verify activities are undertaken at grass roots level. In addition, the in-
volvement of all stakeholders is highly required in the implementation of M&E process
up to use of information.

2.5 Categorization of Indicators

For the ease of implementation and use, this M&E framework document systematically
categorize the list of indicators in to four categories: pharmacy service indictors, supply
chain indicators, medical equipment indicators and supply chain and pharmacy service
crosscutting indicators.

Indicator Reference Sheet (IRS) that defines each performance indicator in each cat-
egory, when and how performance data are collected, analysed and reported is devel-
oped for each indicator and can be found in chapter three.
6
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

For some of the indicators like DTC functionality, all hospitals and health centers
should evaluate their performance using annexed weight based criteria. Hospitals and
health centers should report their summarized performance result to the next level
(WoHO, ZHD and RHB) using the indicators indicated in Chapter 3.

2.6. Intended Users of the Document

The intended users of this M&E framework are Ministry of Health, Food and Drug
Authority (FDA), Ethiopian Health Insurance Agency (EHIA), Pharmaceuticals Supply
Agency (EPSA), Regional Health Bureaus (RHBs), Zonal Health Departments (ZHDs),
Sub city Health Office (ScHO),Woreda health offices (WoHO), health facilities, donors,
UN agencies, and development partners that work in the pharmaceutical sector. The
framework can also be useful to M&E professionals, universities, professional associa-
tions, research institutes, civil society organizations, and experts in the field of policy
analysis and advocacy.

7
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Figure1: Monitoring and Evaluation Framework for Pharmaceutical Supply Chain, Phar-
macy Service and Medical equipment Management
Program: Pharmacy Services, Supply Chain Management and Medical Equipments Management
Program Objectives
• Improve effectiveness and efficiency of pharmaceutical supply chain management system
• Improve availability and quality of pharmacy services
• Improve Medical Equipment availability, utilization and management
Inputs Process Outputs Outcome Impact
Pharmacy - Quantification, - Improved Essential - Improve - Improved
Workforce Procurement and Drug availability patient Health
distribution of - Reduced stock out satisfaction Status
Leadership and drugs of drugs in pharmacy
management - Establishing DTC - Availability of services - Reduced
Indicator - Developing facility national and facility - Improved drug
Domains Coordination specific drug list specific drug list Rational use resistance
- Perform activities - Reduced drug of drugs
Strategies, to implement wastage - Improved - Improved
guidelines APTS - Improved storage knowledge efficiency
- Implement clinical of medicines on rationale and
Finance Pharmacy - Improved disposal use of drugs effectiven
- Perform phar. of unfit for use - Reduced ess in
Information compounding drugs Drug therapy pharmacy
- Capacity Building - Availability of problems services
Logistics activities Quality - Improved and
- Conduct HTA pharmaceutical equitable managem
Technology - Establish Medical products and access to ent
Equipment effective services quality health
management - Availability of DTC, services
committee MEMC - Effective and
(MEMC) - APTS implemented safe
- Perform scheduled - Capacitated utilization of
preventive workforce on medical
maintenance pharmacy services equipment
- Implement IPLS & supply - Improved
- Develop electronic management diagnostics
systems for - Improved capacity of
reporting and use availability of MEs HFs
of data - Improved
- Conduct procurement,
supervision, distribution,
mentorship installation,
maintenance &
disposal of MEs
Data Routine Pharmacy reporting formats. Admin Reports, regular facility Facility Surveys, Population
Collectio surveys surveys
n and HMIS, EHCRIG and EHTG Reports, Supportive supervision reports
Reportin Submission and aggregation of reports with the existing hierarchy of
g health administration
Analysis Data Quality assurance at all levels; Assessment of progress of performance versus plan, use performance
and indicators to discuss during regular performance monitoring meetings
interpret
ation
Dissemin Dissemination of data through different platforms such as regular reporting, quarterly and annual review
ation meetings, publication of bulletins
and use

8
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Chapter Three: The National Pharmaceutical SCM, Pharmacy

Service and Medical Equipment Monitoring and Evaluation In-
dicators
An indicator is a variable that measures one aspect of a program/project and is related
to the program’s goal and objectives. Indicators provide M&E information crucial for
decision -making at every stage of program implementation. FMOH, in collaboration
with its stakeholders, has selected a set of core indicators to inform management of
pharmaceutical SCM, pharmacy service and Medical device program.The breakdown of
these core indicators consists of routine indicators and non-routine indicators.
3.1. Pharmacy Service Indicators

PS1. Drug And Therapeutics Committee (Dtc) Functionality

Indicator Drug and therapeutic Committee (DTC) functionality
Definition Percentage of health facilities that have functional DTC
Formula Number of health facilities that have functional DTC
X 100
Total number of hospitals that established DTC
Interpretation This indicator measures the functionality of health facility DTC. Functional health facility DTC
develops and implements interventions promoting the rational and cost-effective use of medicines.
DTC functionality serves as a proxy indicator of the ability of a health facility to avail
pharmaceuticals and ensure their rational use.
The criteria for functionality of DTC are Assigned DTC members by official letter, Has approved
TOR, Meets regularly at least every months with documented minute, Has developed action plan,
Has updated health facility specific medicine and medical devices list, Has medicine use policy and
procedures (at least two policies, Conduct supply and medicine use problem studies, Take actions
based on the supply and medicine use study findings, Report its performance activities to the
management. If the facility meets 75% of the requirements the facility has functional DTC. Health
facilities measure their DTC functionality using weight-based criteria (Annex 1.1).
Disaggregation By health center and hospital
Sources Documents from DTC secretary (DTC minute, official assignment letters, approved TOR, action
plan, facility specific medicine list, policy & procedures, DTC performance reports, medicine use
study/evaluation reports)
Method of data Survey/Supportive supervision with structured checklist, Routine report
collection
Frequency of HC Hospital WoHO ZHD/ ScHO RHB FMOH
collection/ Quarterly Quarterly Quarterly Quarterly Quarterly Quarterly
Reporting

PS2. Availability Of Health Facility Specific Medicine List

Indicator Availability of Health facility specific medicine list
Definition Percentage of health facilities that have specific health facility medicine list updated every year.
Formula Number of health facilities with facility specific medicine list
X100
Total number of health facilities
Interpretation This indicator measures the extent to which comprehensive facility specific list of medicines,
reagents and supplies, medical device is available at health facilities. The list should be prepared
by the DTC and updated at least every year.
The list is prepared based on relevance to treat prevalent diseases of the catchment area, and
should be categorized by VEN.
Disaggregation By health center and hospital
Sources A copy of facility specific medicine list
Method of data Survey, supportive supervision
collection
Frequency of HP HC Hospital WoHO ZHD/ ScHO RHB FMOH
Reporting Annually Annually Annually Annually Annually
9
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

PS3. Availability Of Standard Treatment Guidelines

Indicator Availability of Standard Treatment Guidelines (STG)
Definition The percentage of health facilities that have recent edition of STG.
Formula Number of health facilities with recent STG
X100
Total number of health facilities
Interpretation This indicator measures the availability and utilization of copies of nationally developed STG.
The availability of STG in a health facility can be used as proxy indicator for rational medicine
use. The STG assessed should be those that are developed for the level of health facility.
At least one copy of the recent STG should be available at adult, pediatric OPDs and at OPD
pharmacy.
Disaggregation By health center and level of hospitals
Sources Copy of STG
Method of data Survey, supportive supervision
collection
Frequency of HP HC Hospital WoHO ZHD RHB FMOH
collection/Reporting Quarterly Quarterly Quarterly Quarterly Annually Annually

PS4. Percentage Of Medicines Prescribed From The Facility’s Medicines List

Indicator Percentage of medicines prescribed from the facility’s medicines list
Definition Percentage of medicines prescribed from those listed on the medicines list of the health facility
(developed by the DTC).
Formula Total number of medicines prescribed from HF medicine list
X100
Total number of medicine prescribed
Interpretation This indicator measures the level of prescribers’ adherence to the health facility specific
medicines list. High level of adherence to the medicine list indicates better rational prescribing
practices. For health facilities the assessment tool and method is indicated on Annex 1.2.
Disaggregation None
Sources Dispensing register, Prescription paper
Method of data Survey
collection
Frequency of HP HC Hospital WoHO ZHD RHB FMOH
collection/Reporting Quarterly Quarterly

PS5. Average Number Of Medicines Per Encounter

Indicator Average number of medicines per encounter
Definition The average number of medicines prescribed per encounter/prescription at OPD
Formula Total number of medicines prescribed
Total number of encounters
Interpretation This indicator measures the degree of polypharmacy. Polypharmacy is prescribing many
medicines for a single encounter. In this analysis, the known combination drugs are counted as
one. This analysis should be done only in outpatient pharmacy. If a patient comes with two
prescriptions in one encounter, the two prescriptions will be considered as one.
The expected level of average number of medicines per encounter is less than two. If the
number of medicines per encounter is more than two, it indicates probability of polypharmacy
and is subjected to further drug use evaluation. For health facilities the assessment tool and
method is indicated on Annex 1.2.
Disaggregation By health center and hospital
Sources Dispensing registration book, prescription paper, routine report
Method of data Survey, supportive supervision with structured checklist
collection
Frequency of HP HC Hospital WoHO ZHD/ RHB FMOH
collection/Reporti ScHO
ng Quarterly Quarterly Quarterly Quarterly Annually Annually
10
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

PS6. Percentage Of Encounters With An Antibiotic Prescribed

Indicator Percentage of encounters with an antibiotic prescribed
Definition The percentage of encounters with one or more antibiotics prescribed at OPD
Formula Total number of encounters with one or more antibiotic
X100
Total number of encounters
Interpretation This indicator measures the overall level of antibiotics use. Imprudent use of antibiotics leads to
antimicrobial resistance. For health facilities the assessment tool and method is indicated on
Annex 1.2. The target is 20-30 %.

Disaggregation Health center, Hospital

Sources Prescription papers, prescription registration book
Method of data Survey
collection
Frequency of HP HC Hospital WorHO ZHD/ ScHO RHB FMOH
collection/Reporting Quarterly Quarterly Quarterly Quarterly Quarterly Quarterly

PS7. Average Dispensing Counselling Time

Indicator Average dispensing counselling time
Definition Average time (in seconds) spent for counseling on the proper use of dispensed
medicines by dispenser; calculated for a sample of series of counseling encounters.
Formula Total time for counseling on medicines dispensed for series of encounters
Number of encounters observed
Interpretation This indicator measures the average time dispensers spend on counseling patients
about their medicines.
It is calculated by observing and recording the time (in seconds) the dispenser takes to
counsel a series of encounter. For health facilities the assessment tool and method is
indicated on Annex 1.3.
Disaggregation Health center, hospital
Sources Observation of counseling encounters
Method of data Survey
collection
Frequency of HP HC Hospital WoHO ZHD RHB FMOH
collection Annually Annually Annually Annually Annually Annually
/Reporting

11
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

PS8. Percentage Of Medicines Adequately Labelled

Indicator Percentage of medicines adequately labeled
Definition Percentage of medicine packages that are labeled with adequate information to enable
the rational use of medicines by patients.
Formula Number of medicines with adequate label
X100
Total number of medicines dispensed
Interpretation This indicator measures the degree to which dispensers record essential information on
dispensed medicine packages. It is very important that medicines are labelled with the
necessary information that enables their rational use by patients. A medicine is
adequately labelled, at least when it is labelled with patient name, name of the medicine,
dose, frequency, duration of use/quantity dispensed, and route of administration.
Medicine information written directly on blisters and strips by the manufacturer or
dispenser cannot be considered as labeling information. For health facilities the
assessment tool and method is indicated on Annex 1.4.

Disaggregation Health center, Hospital

Sources Observation of dispensed medicine by exit interview
Method of data Survey
collection
Frequency of HP HC Hospital WoHO ZHD RHB FMOH
collection/Repo Quarterly Quarterly Quarterly Quarterly Annually Annually
rting

PS9. Patients’ Knowledge On Correct Dosage

Indicator Patients knowledge on correct dosage
Definition Percentage of patients who understood the correct dosage of their dispensed
medicines.
Formula Number of patients with adequate knowledge on correct dosage
X100
Total number of patients interviewed
This indicator measures the effectiveness of the information given to patients on
the dosage of medicines dispensed to them. Correct dosage includes dose,
Interpretation frequency, route, and duration. For health facilities the assessment tool and
method is indicated on Annex 1.4.

Disaggregation Health center, Hospital

Sources Client, label of medicine dispensed
Method of data Survey
collection
Frequency of HP HC Hospital WoHO ZHD/ RHB FMOH
collection/Reporting Quarterly Quarterly Quarterly Quarterly Quarterly Quarterly

12
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

PS10. Percentage Of Clients With 100% Prescribed Drugs Filled

Indicator Percentage of clients with 100% prescribed drugs filled
Definition Percentage of clients who get all the prescribed medicines (100%) from dispensary among all the
clients who received prescriptions in a given time period.
Formula Number of clients who received all prescribed drugs
X 100
Total number of clients who received prescriptions
Interpretation This indicator measures proportion of clients who get all the prescribed drugs. It is one of the
indicators that tell about continuous availability of medicines. Getting prescribed drugs within the
facility pharmacy improves patient satisfaction and overall trust and confidence in the health
sector. Percentages of clients who get all the prescribed drugs (100%) from dispensary is
expected to be 100 percent. The registration book is indicated in annex 1.5.
Disaggregation Health center, Hospital
Sources Dispensing registration book, Prescription paper
Method of data Routine through DHIS2
collection
Frequency of HC Hospital WoHO ZHD/ ScHO RHB FMOH
Reporting Monthly Monthly Monthly Monthly Monthly Monthly

PS11. Clinical Pharmacy Service Functionality

Indicator Clinical pharmacy service functionality
Definition Percentage of hospitals with functional clinical pharmacy service (CPS)
Formula Number of hospitals with functional CPS
X 100
Total number of hospitals
Interpretation This indicator measures the extent of the provision of pharmaceutical care in inpatient wards by
pharmacists to maximize therapeutic benefits and minimize risk of medicines. A functional clinical
pharmacy service requires the provision of pharmaceutical care from admission to discharge, at
all times, and in all inpatient wards. A hospital is considered as functional when it fulfills 75 % of
the criteria indicated in Annex 1.6.
Disaggregation None
Sources Clinical pharmacy records and report
Method of data Survey, routine report
collection
Frequency of HC Hospital WorHO ZHD/ ScHO RHB FMOH
collection/ Semi-annually Semi-annually Semi-annually Annually
Reporting

13
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

PS12. Hospital With Functional Unit Dose Dispensing System (UDS)

Indicator Percentage of hospitals with functional unit dose system (UDS).
Definition Percentage of hospitals with functional unit dose dispensing system (UDS) in their inpatient ward
pharmacies in the reporting period.
Formula Number of hospitals with functional UDS
Total number of hospitals X100

Interpretation This indicator shows the presence and functionality of UDS in the inpatient ward pharmacy of
the hospital. The presence of functional UDS is expressed in terms of availability of dedicated
ward pharmacy(s), dedicated pharmacist, medicines are dispensed in a single dose package, in a
ready to administer form and only for 24 hours with a pharmacy specific documentation. A
health facility UDS is considered functional when a minimum of 75% score is achieved using a
checklist. Health facilities measure their UDS functionality using criteria indicated in Annex 1.7.
Disaggregation None
Sources Direct observation
Method of data Survey
collection
Frequency of HC Hospital WorHO ZHD/ ScHO RHB FMOH
Reporting
Annually Annually Annually Annually Annually

PS13. Percentage Of Hospitals With Functional Drug Information Service

Indicator Percentage of hospitals with functional Drug information service (DIS)

Definition Percentage of hospitals with functional drug information services in the reporting period.

Formula Number of hospitals with functional DIS

X100
Total number of hospitals that established DIS
Interpretation This indicator measures the provision of DIS to health professionals, patients and the public. A hospital
DIS is considered functional when 75% of the following criteria are fulfilled: availability of dedicated
room, dedicated pharmacy professional, adequate reference materials and equipment, standard
operating procedures, completed query response forms, medicine education program and report,
sample alerts/newsletters prepared, action plan and performance reports. Health facilities measure their
DIS functionality using criteria indicated in Annex 1.9.
Disaggregation None
Sources Observation Completed DIS recording and reporting form
Method of data Routine aggregation of health facility DIS record and report, survey
collection
Frequency of HC Hospital WorHO ZHD/ ScHO RHB FMOH
collection/
Reporting Semi-annually Semi-annually Semi-annually Annually Annually

14
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

PS14. Percentage Of Hospitals With Functional Compounding Services

Indicator Percentage of hospitals with functional compounding services.
Definition Percentage of hospitals fulfilling the criteria for functional compounding services
Formula Number of Hospitals with functional compounding services
X 100
Total number of Hospitals
Interpretation This indicator measures the presence of compounding capability of a hospital pharmacy to prepare
non-sterile preparations. The preparations may include dermatological preparations (ointments,
creams) and bulk preparations (e.g. hand rubs, hydrogen peroxide, gentian violet,). A hospital
compounding service is considered functional when 75% of the following criteria are fulfilled: separate
room/area dedicated for compounding, dedicated pharmacist, equipment, chemicals, SOP, completed
compounding registration form. Health facilities measure their compounding functionality using criteria
indicated in Annex 1.8.
Disaggregation None
Sources Observation, Compounding registration form
Method of data Survey and supportive supervision through Observation
collection
Frequency of HC Hospital WoHO ZHD/ ScHO RHB FMOH
Reporting Annually Annually Annually Annually Annually

PS15. APTS Functionality

Indicator APTS functionality
Definition Percentage of health facilities with functional APTS
Formula The number of health facilities with functional APTS
X 100
Total Number of Health facilities implementing APTS
Interpretation This indicator measures the number of health facilities that fulfilled the requirements and implemented
APTS. APTS is considered functional when 75% of the following criteria are fulfilled: Designed workflow,
started APTs in all dispensaries and stores, produce daily summary and monthly report, bin ownership,
conduct audit, fulfill pharmacy workforce, availability of skilled personnel, APTS vouchers, sales tickets and
registers, conduct physical inventory as per the standard, ABC/VEN and stock status analyses. Health
facilities measure their APTS functionality using criteria indicated in Annex 1.10.
Disaggregation By level of health facility
Sources Observation, APTS records and report
Methods of data Routine report, survey
collection
Frequency of HC Hospital WoHO ZHD/ ScHO RHB FMOH
collection/ Quarterly Quarterly Quarterly Quarterly Quarterly Quarterly
Reporting

15
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

PS16. Client Satisfaction With Dispensing Services

Indicator The percentage of clients satisfied with dispensing services
Definition The proportion of patients satisfied with dispensing services among all interviewed patients
from all dispensing outlets.
Formula Number of clients satisfied with dispensing services
Total number of clients interviewed X 100

This indicator measures the overall outcome of all reform activities to improve pharmacy
services in general and dispensing activities in particular. It indicates the degree to which
Interpretation dispensing service meets clients’ expectations. It can be measured in terms of availability of
medicines, information provision, premises and personnel. A minimum of 80% client satisfaction
with dispensing service is considered acceptable. (Annex 1.11)
Disaggregation By health center, hospital
Sources Client
Methods of data Survey
collection
Frequency of HP HC Hospital WoHO ZHD RHB FMOH
collection/Reporting Quarterly Quarterly Quarterly Quarterly Annually Annually

16
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

3.2. Supply Chain Indicators

SC1. Forecast Accuracy

Indicator Forecast accuracy
Definition Forecast accuracy is the percentage difference between forecasts previously made for specified
period of time and the actual consumption or issues data for that period.

Formula

Interpretation This indicator measures the degree of accuracy of a forecast or quantification exercises in
facilities that perform forecasting of their own medicine requirement. Higher calculated value
indicates there is a correspondence between the forecasted quantities and the actual
consumption and this tells the forecasting accuracy is high. Hundred percent accuracy is difficult
to achieve in any facility but values greater than or equal to 75% are considered high.
The health facility should calculate this indicator for tracer items for which a forecast is made
by the health facility not more than 10 products. Health facilities should use the formats
indicated in Annex 2.1.
Disaggregation By program, by tracer product
Sources Facility forecast data/document, facility consumption data from bin card/DAGU, actual
dispensary records (dispensing registration book)
Frequency of HP HC Hospital WoHO ZHD/ RHB FMOH
Reporting/ ScHO
Collection Annually Annually Annually

SC2. Supplier Fill Rate

Indicator Supplier fill rate
Definition The percentage of all items ordered by health facility from a distribution source (EPSA, or other
private supplier) over a period that are filled correctly at least 80% in terms of quantities
requested of those items
Formula Number of line items delivered at least 80%
X 100
Total no. of line items requested
Interpretation This indicator measures supplier’s ability to fill orders completely in terms of items and quantity
during a definite period of time. This indicator measures the percentage of items ordered that
are received to determine whether an order is filled in the correct quantities with the correct
products at least 80%. For health facilities, it may be necessary to identify which items are
causing the most problems and find another mechanism for obtaining those items. Health
facilities can measure supplier fill rate using the format indicated in Annex 2.2.
Disaggregation By supplier (EPSA, others) and by Programs
Sources RRF report, Receiving voucher of HF, approved procurement
Method of data DHIS2
collection
Frequency of HP HC Hospital WoHO ZHD/ScHO RHB FMOH
Reporting/ Quarterly Quarterly Quarterly Quarterly Quarterly Quarterly
collection

17
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

SC3. Average Lead Time

Indicator Average Lead Time
Definition The average amount of time between facilities place order to supplier and when the products are
delivered to a facility.
Formula Summation of the number of days it takes by supplier to deliver products once orders are submitted
to the supplier
Number of orders submitted to the supplier
Interpretation This indicator measures the average amount of time it takes by supplier to deliver products once
the facilities placed order to supplier. Facilities submit order up to the 10th day of the month
following the end of the reporting period. EPSA will use the data from RRF to resupply health
facilities within 20 days. Therefore, this indicator helps to measure on time delivery. A lead time
average exceeding 20 days is considered as delay. Health facilities can measure the average lead time
by using the format indicated in Annex 2.3.
Disaggregation By supplier (EPSA, others), program/RDF
Sources RRF report, approved procurement request and Model 19
Method of data Document review & routine report
collection
Frequency of H HC Hospital WoHO ZHD/ ScHO RHB FMO
Reporting P H
Bi-annually Bi-annually Bi-annually Bi-annually Bi-annually

SC4. Stock Out Duration

Indicator Stock out duration for tracer items
Definition The number of days in which the tracer drug was not available in a specified period of time
Formula Sum of stock out days of specific tracer drug in specific review period
Interpretation The availability of tracer items is a measure of service availability. Tracer items should always be
available at the health facility. If there is any stock out of tracer items, the facility should act to
identify and address the cause.
This indicator provides a proxy measure of the ability of a program to meet clients’ needs with a
full range of items. Health facilities can measure stock out duration of tracer items by using the
tracer drug availability and stock out duration tally sheet and registration format indicated in
Annex 2.4.
Disaggregation By specific tracer product, By level of health facility
Sources Bin card, DAGU and tracer drug availability tally sheet
Method of data Survey and SS (Review of documents and observation)
collection
Frequency of HP HC Hospital WoHO ZHD/ ScHO RHB FMOH
Reporting/ Monthly Monthly Quarterly Quarterly Annually Annually
collection

18
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

SC5. Wastage Rate

Indicator Wastage rate of health products
Definition The percentage of the stock of products, in value, that are unusable because of expiration or damage
during a period to the total value of the products received during the same period plus the quantity
of the products found during the beginning of the period.
Formula Unusable stock of products during a period in monetary value
X 100
Beginning stock + received stock during the same period in monetary value
Interpretation This indicator can be calculated for any facility that manages pharmaceutical of interest. It can be
measured over any period but it is preferable to be calculated for unusable stock with in a quarter. It
is usually calculated after a physical inventory is taken.
Unusable stock that has been accumulated for long period and were not disposed previously
(expired and damaged items that were transferred from previous quarter) should not be included
during calculation of this indicator. In addition, Items that were unusable during the quarter reviewed
but were disposed with in the quarter should be taken in to consideration during calculation.
This indicator is one of the performance indicators to have efficiency gain and one of the HSTP
indicators. The target in HSTP is to reduce wastage of pharmaceuticals to less than 2%.
This indicator is calculated for medicines, reagents, chemicals and supplies by using the registration
format indicated in Annex 2.5.
Disaggregation By program, RDF
Sources Bin cards, Model 19, inventory sheet, disposal reports, electronic records
Method of DHIS2
data collection
Frequency of HP HC Hospital WoHO ZHD/ ScHO RHB FMOH
Reporting/ Quarterly Quarterly Quarterly Quarterly Quarterly Quarterly
collection

SC6. Percentage Of Facilities That Maintain Acceptable Storage Conditions

Indicator Percentage of facilities that maintain acceptable storage conditions
Definition This indicator measures the percentage of facilities that meet acceptable storage conditions.
Formula Number of facilities that meet acceptable storage condition
X 100
Total number of facilities
Interpretation This indicator measures the conditions of pharmaceutical store against a list of storage
conditions required to protect the integrity of products. Evaluators can apply the indicator
at pharmaceutical stores identify facilities that need improvement. The good storage
guideline standards are a set of standards that a well-functioning pharmacy store should
maintain and have in place. There is a total of 13 standards (Annex 2.6). Storage facilities
are expected to meet at least 80% of the requirements according to standard checklist.
Disaggregation Hospital and Health Center
Sources Checklist for standard storage condition
Method of data Survey, supportive supervision
collection
Frequency of HP HC Hospital WoHO ZHD/ RHB FMOH
Reporting/Collection ScHO
Annually Annually Annually Annually Annually Annually

19
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

SC7. Inventory Accuracy Rate

Indicator Inventory accuracy rate
Definition This indicator measures the accuracy of stock balances recorded in stock keeping records (bin
card, electronic) versus physical count over a range of items as a percentage of stock balances
reviewed for accuracy.
Formula Number of items where stock record balance equals physical stock count
X100
Total number of items counted
Interpretation This indicator measures the accuracy of logistics data as the percentage of discrepancy between
physical count and stock record. The calculation is performed for randomly selected 10 tracer
products. High accuracy rate (80% and above) indicates good inventory practice. For
administration levels, this indicator measures the percentage of health facilities that had 80%
and above inventory accuracy rate when bin cards were compared to a physical inventory
count out of the total number of facilities under review during a defined period. Health facilities
can measure their inventory accuracy rate by using the format indicated in Annex 2.7.
Disaggregation By Health Center and Hospital
Sources Bin cards, Electronic records, physical count
Method of data Survey, supportive supervision
collection
Frequency of HP HC Hospital WoHO ZHD RHB FMOH
Reporting Quarterly Quarterly Quarterly Quarterly Annually Annually

SC8. Rrf Reporting Rate

Indicator RRF reporting rate
Definition The proportion of Report and Requisition Forms (RRFs) submitted on time
Formula Total number of RRF submitted on time
X100
Total number of expected RRF
Interpretation This indicator provides an overall measure of whether timely reports and requests are sent to
EPSA. All health facilities are expected to send RRF report every two months until the 10th day
of the following month. Health facilities can measure their RRF reporting rate by using the
format indicated in Annex 2.8.
Disaggregation By Hospital and Health Center
Source RRF, Electronic report, RRF submission monitoring log book, RRF tracking dashboard
Method of data Document review for WoHO and ZHD, EPSA RRF tracking dashboard for FMOH
collection
Frequency of HP HC Hospital WoHO ZHD RHB FMOH
Reporting Quarterly Quarterly Quarterly Quarterly Quarterly Quarterly

20
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

SC9. Essential Drugs Availability

Indicator Essential drugs availability
Definition The percentage of tracer drugs available throughout the month averaged over all tracer drugs
under the review in the month

Formula Number of tracer drugs available

X 100
Number of tracer drugs under review
Interpretation Essential drugs should always be available. Essential drug availability is the proportion of tracer
drugs under review which are available throughout the month. The type of essential drug that
needs to be available differs by type of health facility.
This indicator measures product availability (or absence) over a period and serves as a proxy
indicator of the ability of a program to meet clients’ needs with a full range of products and
services. If a product is not available (stocked out) for one day in the month, then it’s considered
as not available for the whole month. Health facilities can measure stock out duration of tracer
items by using the tracer drug availability and stock out duration tally sheet and registration
format indicated in Annex 2.4.
Disaggregation By each product, program products
Sources Bin card, Electronic records and tracer drug availability sheet
Method of data DHIS 2
collection
Frequency of HP HC Hospital WoHO ZHD/ RHB FMOH
Reporting ScHO
Monthly Monthly Monthly Monthly Monthly Monthly Monthly

SC10. Disposal Of Unfit-For-Use Medicines

Indicator Disposal of unfit-for-use medicines
Definition Percentage of health facilities that have disposed unfit-for-use medicines at least once in the past
12 months.
Formula Number of health facilities that have disposed of their unfit-for-use medicines
X 100
Total number of health facilities having unfit-for-use medicines
Interpretation This indicator measures the performance with which health facilities dispose unfit-for-use
pharmaceuticals as per the national disposal directive. Unfit-for-use pharmaceuticals include
expired or damaged, pharmaceuticals with quality problems. Health facilities should be able to
dispose of these products in a timely fashion to avoid their inadvertent use by patients due to
dispensing errors and enables efficient utilization of storage space. The indicator assumes that
pharmaceuticals are disposed of at least once in 12 months. Health facilities can measure their
disposal of unfit-for-use medicines by using the format indicated in Annex 2.9.
Disaggregation Health center and hospital
Sources Disposal certificate, bin card, Electronic record, physical count
Method of data Survey, supportive supervision, document review,
collection
Frequency of HC Hospital WoHO ZHD RHB FMOH
Reporting Annually Annually Annually Annually Annually Annually

21
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

3.3. Medical Equipment Indicators

ME1. Availability Of Functional Medical Equipment

Indicator Availability of Functional Medical Equipment
Definition Percentage of functional medical equipment from the health facility’s updated medical equipment
inventory list.
Formula Number of functional medical equipment in the health facility
Total number of medical equipment in the health facility from updated X 100
medical equipment inventory list
Interpretation This indicator measures percentage of functional medical equipment in the health facility during
the review/data collection. Functional medical equipment are instruments which are giving the
expected services. To monitor and evaluate this indicator, the health facility should establish
computer based or manual medical equipment inventory system and also should update the
inventory whenever additions or omissions of medical equipment occur to the health facility.
Health facilities should use the Medical Equipment Inventory Form (Annex 3.1) to register
medical equipment that is available in the health facility. Health facilities can measure this
indicator by using the format indicated in Annex 3.1.
Disaggregation By type of health facility
Sources Updated medical equipment Inventory
Method of data Document review and observation
collection
Frequency of HC Hospital WoHO ZHD RHB FMOH
Reporting Annually Annually Annually Annually Annually Annually

ME2. Percentage Of Health Facilities With Updated Medical Equipment In-

ventory
Indicator Availability of updated Medical Equipment inventory list
Definition Percentage of health facilities that have updated their medical equipment inventory annually.
Formula Number of health facilities with updated medical equipment inventory
X 100
Total number of health facilities
Interpretation This indicator measures the proportion of health facilities that have updated medical equipment
inventory list annually. Medical equipment inventory is a list of technology on hand including
details of the type and quantity of equipment and the current operating status. This indicator
enables the health facility and administrative bodies to take action on procurement, distribution,
installation, maintenance and disposal of medical equipment. Health facilities should use the
Medical Equipment Inventory Form (Annexed) to register medical equipment that are available
in the health facility. Health facilities can measure this indicator by using the format indicated in
Annex 3.2.
Disaggregation By type of health facility
Sources Inventory record
Method of data Document review, survey
collection
Frequency of HC Hospital WoHO ZHD RHB FMOH
Reporting Annually Annually Annually Annually Annually Annually

22
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

ME3. Percentage Of Health Facilities With Functional Medical Equipment

Management Committee (MEMC)
Indicator Health facilities with functional MEMC
Definition Percentage of health facilities that have functional Medical Equipment Management committee
Formula Numbers of health facility with functional MEMC X100
Total numbers of health facility
Interpretation This indicator measures functional MEMC at each health facility that advices the management of
medical equipment in the facility. Medical Equipment Management committee (MEMC) is a
committee that is established at health facilities to play an advisory role on management of
medical equipment in the facility. MEMC is considered functional, if it meets 80% of the below
criteria: having defined TOR, assigned members officially, annual action plan, regular meeting
with minutes, develop medical equipment list, and conducts regular inventory (Annex 3.3).
Disaggregation By health facilities
Sources MEMC documents
Method of data Document review, Survey
collection
Frequency of HC Hospital WoHO ZHD RHB FMOH
Reporting Quarterly Quarterly Quarterly Quarterly Annually Annually

ME4. Percentage Of Health Facilities With Scheduled Preventive Mainte-

nance Practice
Indicator Health facilities with scheduled preventive maintenance practice
Definition The proportion of medical equipment that has undergone scheduled preventive maintenance
as per the manufacturer’s recommendation.
Formula Number of scheduled preventive maintenance performed
X 100
Total numbers of expected preventive maintenance
Interpretation This indicator measures scheduled preventive maintenance performed to maintain the
functionality of medical equipment. Preventive maintenance refers to regular, routine
maintenance to help keep equipment up and running, preventing any unplanned downtime and
expensive costs from unanticipated equipment failure. It requires careful planning and
scheduling of maintenance on equipment before actual problem happens. Preventive
maintenance schedule includes regular inspection, testing, safety and calibration for each
medical equipment as per the manufacturer's service manual.
If the manufacturer’s manual is not available, inspection, testing and preventive maintenance
should be conducted at a minimum every six months. A facility is considered as having a
scheduled preventive maintenance practice, if it meets 80% of the criteria: care and cleaning
schedule, safety procedures in place, functional and performance, calibration testing,
preventive maintenance checks for at least 80% of medical equipment. Health facilities
measure their preventive maintenance practice using criteria indicated under Annex 3.4.
Disaggregation By health facilities
Sources Medical equipment history file, log sheet, scheduled preventive plan and report
Method of data Review of documents, Survey
collection
Frequency of HP HC Hospital ZHD RHB FMOH
Reporting
Annually Annually Annually Annually

23
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

ME5. Percentage Of Medical Equipment Installation

Indicator Medical equipment Installed within six months
Definition Percentage of Medical equipment delivered to the health facility within the past six months and
installed
Formula Number of installed medical equipment within the past six months
Total number of medical equipment delivered to the health facility in the past X 100
six months that needs installation
Interpretation This indicator indicates that all delivered medical equipment are installed and commissioned in
accordance with the manufacturer’s specifications and undergoes acceptance testing within the
contract agreement. The supplier should provide staff in-service training on the correct and safe
use of equipment and basic troubleshooting and preventive maintenance. See annex 3.5 for
reporting formats.
Disaggregation By time taken for installation (< 3 months, > 3 months, by health facility
Sources Medical equipment history file, distribution list, Inventory data, survey
Frequency of HC Hospital WoHO ZHD RHB FMOH
Reporting Quarterly Quarterly Quarterly Quarterly Quarterly Quarterly

ME6. Biomedical Professional Positions Filled At Health Facilities

Indicator Biomedical professional positions filled at health facilities
Definition The percentage of Biomedical workforce positions filled at health facilities
Formula Number of Biomedical professionals at health facilities
X 100%
Number of Biomedical workforce positions
Interpretation This indicator measures the number of Biomedical staffs deployed as per the approved
workforce position for the health facilities. The measurement of this indicator shows the gap on
the number of Biomedical professionals against the approved structure and help to fulfill
through recruitment or other means. The percentage of biomedical professionals’ positions
filled can be measured using the format indicated in annex 3.6.
Disaggregation By level of Health Facility
Sources HR records
Method of data Review of documents, Survey
collection
Frequency of HC Hospital WoHO ZHD RHB FMOH
Reporting Annually Annually Annually Annually Annually Annually

ME7. Availability Of Medical Equipment As Per The National Standard

Indicator Availability of medical equipment as per the national standard
Definition This indicator measures the percentage of health facilities that have medical equipment as per
the national standard.
Formula Number of health facilities that have medical equipment as per the national
standard X 100
Total number of health facilities
Interpretation All health facilities are required to be equipped as per the national standards. Those health
facilities that are equipped as per the national standard are expected to deliver quality health
services and satisfy the needs of the health professionals and the population. This indicator
measures medical equipment availability (or absence) over a period and serves as a proxy
indicator of the ability of a program to meet clients’ needs with a full range of products and
services. Health facilities that have 80% of the medical equipment according to the national
standard for the level are considered as acceptable. Evaluators may assess reasons for
unavailability to help program managers address the underlying causes.
Disaggregation By level of health facility (except tertiary hospital)
Sources BIN card, Inventory record, List of standard medical equipment (EFDA)
Method of data Review of documents
collection
Frequency of HC Hospital WoHO ZHD RHB FMOH
Reporting Annually Annually Annually Annually Annually Annually
24
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

3.4. Pharmacy Services and Pharmaceutical Supply Chain Management

Cross Cutting Indicators

SC-PS1. Pharmacy Review Meetings Conducted

Indicator Number of review meetings conducted
Definition The number of pharmaceutical supply chain and pharmacy service related review meetings
conducted within a year per administrative level
Formula Number of review meetings
Interpretation This indicator measures the presence of coordination, leadership, and commitment.
Pharmaceutical good governance is critical to realize sustainable commodity security and quality
of pharmacy services. Resilient pharmaceutical systems require the involvement of stakeholders
that are involved in all aspects of the system strengthening efforts. It is to be noted that these
review meetings should also participate development partners to align plans and monitor
progress in a timely fashion. Hence, review meetings at respective administrative bodies can
play an important technical and political role by coordinating the different actors working in the
pharmaceutical sectors. Review meeting are expected to be conducted at least annually. This
indicator can be measured using the format indicated in annex 4.1.
Disaggregation By RHB
Sources Reports of review meetings
Frequency of HP HC Hospital WoHO ZHD RHB FMOH
Reporting/ Annually Annually
collection

SC-PS2. Supportive Supervision Of Health Facility Pharmacies

Indicator Supportive Supervision of Health Facility Pharmacies
Definition The percentage of health facility pharmacies that received supportive supervision on their
pharmacy activity by immediate administrative units using standard checklist within the specified
time-period.
Formula Number of health facility pharmacies supervised
X100
Total number of health facilities under immediate administrative level
Interpretation This indicator measures the percentage of health facilities that received technical and
administrative support on their pharmaceutical supply chain and pharmacy service activities. The
supervision should be conducted using standard checklist which is approved by RHB/FMOH.
The feedback and action points obtained from the supportive supervision should be
documented at both the supervised health facility and the supervisor’s office.
Higher rate of supportive supervision will help facilities to improve supply chain efficiency and
pharmacy services and will help to solve gaps at health facility levels. This indicator can be
measured using the format indicated in annex 4.2.
Disaggregation By ZHD & RHB
Sources Completed checklist, copy of written feedback provided
Method of data Survey and supportive supervision (Document review)
collection
Frequency of HP HC Hospital WoHO ZHD RHB FMOH
Reporting/ Quarterly Quarterly Bi-annually Annually
collection

25
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

SC-PS3. Percentage Of Pharmacy Workforce Positions Filled At Health Fa-

cilities
Indicator Percentage of pharmacy workforce positions filled
Definition The percentage of pharmacy workforce positions filled by at health facilities
Formula Number of pharmacy workforce at health facilities
X100%
Number of pharmacy workforce positions
Interpretation This indicator measures the number of pharmacy staff deployed at health facilities as per the
structure/determined by workload analysis. The measurement of this indicator shows the
pharmacy staff gap and help to fulfill the pharmacy department through recruitment. The
percentage of pharmacy workforce positions filled can be measured using the format indicated in
annex 4.3.
Disaggregation By type of health facility, type of professionals (pharmacy professional and other pharmacy
workforce)
Sources HR records
Method of data collection Document review and routine report
Frequency of Reporting HP HC Hospital WoHO ZHD RHB FMOH
Annually Annually Annually Annually Annually Annually

26
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Chapter Four: Data Collection, Analyses and Utilization

The M&E Framework consists of a total of 36 indicators. Some indicators will be col-
lected and used by each level without reporting to the next level while selected in-
dicators will be reported to the next level. Some indicators will be tracked by higher
administrative levels using different data collection methods such as surveys and sup-
portive supervisions. The data elements for the four KPIs that can be captured by the
routine HMIS system that will follow the HMIS reporting system. There will not be
parallel reporting system for these indicators.

4.1 Flow of reports

Data elements that are selected for reporting from one level to the next will follow the
existing hierarchy of report flow in the health system. Report flow from the lowest to
the highest levels of the health system and the flow of feedback in the reverse direction
is depicted in figure 2 below. Accordingly,
• Health centers report to woreda health offices
• Woreda health offices aggregate and report to Zonal Health Departments
• Zonal Health Departments aggregate and send to Regional Health Bureaus
• Regional Health Bureaus aggregate and report to the FMOH

In addition to the above-mentioned reporting hierarchy, some regions may have a dif-
ferent administrative organization for whom their unique administrative hierarchy will
be used for reporting flow. Example: Some regions that have no zonal administrative
structure, Woreda Health Offices will directly report to the RHBs.

Data will be collected, analyzed, reviewed and reported via an electronic reporting tool
that will be developed for this purpose.

Figure 2: Pharmaceutical Supply Chain, Pharmacy Service and Medical Equipment M&E
Indicators Reporting Flows

27
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

4.2 Reporting schedule

Reportable data elements will be reported to the next administrative level using a stan-
dardized reporting format for a specified reporting period. A reporting timeline, which
is in line with the HMIS reporting schedule, is set for each level, Accordingly, a monthly
report of a health facility is compiled from the 21st of the previous month up to the
20th of the reporting month and submitted to the next level the latest by the 26th of
the reporting month. Example: For Tikimt 2011 EC monthly report, the data should be
collected from Meskerem 21 up to Tikimt 20, 2011. The reporting channel and period
of public health facilities and administrative health units will follow the following sched-
ule, as depicted in the table below.

Table 1: Reporting hierarchy, frequency and schedule of public health facilities and ad-
ministrative health units
Unit Reports Timeline Latest date report should be Type of reporting form
to submitted*
Health WoHO Quarterly 2 6t day of the last month of the Reporting form f or h ealth
Centre h quarter centres
Hospital Z HD/RHB Quarterly 26t day of the last month of the Reporting form for hospitals
quarter
h

WoHO Z HD Quarterly 2n d ay o f the 1st m onth o f the Reporting form for Woreda
next quarter Health Offices
ZHD RHB Quarterly 7t d ay o f the 1st m onth o f the Reporting form f or Z onal
d next quarter Health Departments
RHB F MOH Quarterly 15t d ay o f the 1st m onth o f the Reporting form for RHBs
next quarter

Quarterly reports consist ofh data for three months according to the Ethiopian fiscal
year. It should follow the following periods:
• Quarter 1: Sene 21-Meskerem 20
• Quarter 2: Meskeremh 21- Tahsas 20
• Quarter 3: Tahsas 21- Megabit 20
• Quarter 4: Megabit 21- Sene 20

Also annual reports contain data for a one-year period from Sene 21 of the previous
fiscal year to Sene 20 of the current fiscal year.
Example:
For the 1st quarter of the Ethiopian Calendar, health facilities should submit their
quarterly reports of the first quarter the latest by Meskerem 26; WoHO will aggregate
the reports and submit to ZHD until Tikimt 2; ZHD will submit their report to RHBs
until Tikimt 7; and RHBs should submit their quarter report to the FMOH by the 15th
of Tikimt.

4.3 Collection and analysis of data

4.3.1 Collection and analysis of data at facility level
Health facilities will use the standard forms to collect routine and survey data.They will
review and assure the quality of data before analysis and use. During analysis, the indi-
cator reference sheet should be used as a reference for the description and interpre-
tation of the indicator. Health facilities should use the data to assess their performance
and take actions accordingly.
28
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

The pharmacy department of the health facility is the data owner for PSCM and PS data
and is responsible to generate the report and submit to the health facility head. More-
over, the Biomedical Engineering department head generates data related to medical
equipment, compiles and submits the report to the pharmacy head. The health facility
head will submit to the next administrative level as per the reporting schedule. The
pharmacy head of the facility or his delegate should be a member of the facility’s per-
formance monitoring team (PMT) and should present performance reports to the PMT.

If the facility is using non-electronic reporting system, the report will be prepared in
two copies from which the 1st copy will be summited to Woreda/Zone; and the second
copy to be archived at the health facility. Ideally, the focal person should submit the
report using electronic platforms such as emails. If this is not possible, the pharmacy
department focal person should submit the hard copy of the data elements to the next
level.

4.3.2 Collection and analysis of data at administrative levels

All administration levels should assign a focal person to receive and compile reports.
The focal person is responsible to follow the timeliness and completeness of reports.
The administrative levels will use the aggregation formats to aggregate reports submit-
ted from the lower units. In order to simplify aggregation and analysis, an electronic
data base (Excel based or other) will be developed and used. The electronic system
will help simplify aggregation of reports and to generate and display the results in the
form of tables, graphs in a dashboard. Every quarter, the results will be analyzed, and
feedbacks will be given to lower levels.The results will also be shared to the next higher
administrative level as per the agreed timeline.

4.4 Utilization of data

4.4.1. Utilization of data at health facility level
The pharmacy department is responsible not just for reporting of data, but primarily
to use the data for performance and service improvement through evidence-based
decision making.
Useful questions to consider when reviewing the data include:
• How does this result compare to the last reporting period? How and why has
the change in performance happened?
• How does the data compare to the target for the reporting period? Has the
target been reached? If the target has not been reached, why not?
• Is there a need for further improvement on the indicators?
• Is further support required from health facility management, administrative
levels or other partners to support the facility to make improvements?
The pharmacy department, together with PMT should analyze the performance and
develop action plan to improve performance.The PMT will oversee performance mon-
itoring and improvement across the health facility.

4.4.2. Utilization of reports by WoHOs/RHBs/FMOH

The RHBs, ZHDs and WoHOs should aggregate and analyze reports received from all
health facilities and provide feedback. When reviewing reports, the RHB/ZHD/WoHO
should consider the same questions as outlined above. In addition, performance of
health facilities should be compared:
• Which facilities are showing the best performance overall? Which are showing
poor performance?
29
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

• Which facilities are improving? Which facilities show slow or no improve-

ment?
• What are the strengths in the region/Zone/Woreda? What are the weakness-
es?
The RHB should give feedback to each ZHDs/hospitals on the reports, asking for
clarification or further information whenever required. The RHB should also use the
reports to identify areas for action. The reports can be used as an input for subsequent
supportive supervision visits.

4.5 Data quality assurance

All health facilities and administrative health units should provide a due attention to
the quality of data generated and used at each level of the health system. In order to
ensure the quality of data, each unit should avail adequate inputs and make sure that
data quality assurance processes are in place. They will review the following elements
to ensure data quality:
• Availability of standardize data collection, aggregation, and reporting tools
• Written standard operating procedures (SOPs) are in place for data collection
• Data quality assessments will review whether these procedures are in place,
implemented consistently, and reviewed periodically for effectiveness and effi-
ciency)
• Initial training and ongoing refresher training provided for all relevant staff
• Implement SOPs that have a system to check for and remove duplicate data
• Safeguards are in place to prevent unauthorized access to and changing of data
• Original source documents are maintained and readily available.
• Carry out system assessment to identify underlying causes for poor data qual-
ity

The national Health Data Quality Guideline of the HMIS provides guidance to compre-
hensively measure the level of data quality, to assess the underline data management
system, and to build an internal data quality assurance mechanism for health facilities
and administrative levels. To conduct data quality checks at health facilities, LQAS (Lot
Quality Assurance Sampling) methodology can be used. (Refer to the national Health
Data Quality Guide and training manual to understand the details of LQAS method-
ology).

Use of information for decision making will have a positive reinforcing effect to improve
data quality. FMOH and RHBs will technically support cross referencing and linking
of logistic system performance to program or service delivery performance. FMOH/
RHBs will make technical data reviews to improve the data quality and provide inter-
pretation on the use of the reports. The data review might also suggest adjustment of
performance for decision making such as supply planning, redistribution of commodi-
ties, and allocation of resource.

4.6. Roles and responsibilities

Each health institution at all levels of the health system has specific roles and responsi-
bilities in implementing and monitoring the implementation of the M&E plan for phar-
maceutical supply chain, pharmacy services, and medical device. Table x below outlines
the major roles and responsibilities of each health institution/stakeholder. FMOH, to-
gether with the RHBs, will review the M&E plan/framework every two to three years to
determine if adjustment is needed on the indicators, and data collection tools.
30
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Table 2: Roles and Responsibilities

Institution Role and Responsibilities
Design the M&E system
Periodically review and update the M&E plan
FMOH/RHBs/EPSA Develop standardized reporting forms and electronic database
Follow the implementation of the M&E plan
Collect performance data from lower levels
Analyze data and use for performance improvement
Provide feedback to health facilities or administrative levels
Assign focal persons for data management
Conduct supportive supervision visits
Conduct research and evaluations
Provide capacity building to staff at all levels of the health system
Conduct data quality assessments
Organize and conduct national performance review meetings
Follow the implementation of the M&E plan
Collect performance data from lower levels
Analyze data and use for informed decision making
Provide feedback to health facilities
ZHDs/WoHOs Assign focal persons for data management
Conduct supportive supervision visits
Provide trainings and other capacity building activities
Conduct data quality assessments
Present the data in review meetings and other platforms
Assign focal person for data management
Health Facilities Maintain the primary data source(s) for KPI information
(Hospitals/Health Centers) Compile data regularly, perform data quality checks
Compute indicators and conduct self-assessment
Receive feedback and take actions
Provide data for monthly progress review meetings
Submit quarterly reports to the next level

31
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex
Annex 1. Registration formats for pharmacy service indicators

Annex 1.1: DTC functionality criteria

DTC functionality criteria
S.N Criteria Weight Score
1 Assigned DTC members by official letter 10
2 Has approved TOR 10
3 Meets regularly at least every months with documented minute 10
4 Has developed action plan 10
5 Has updated health facility specific medicine and medical 15
devices list
6 Has medicine use policy and procedures (at least two policies) 10
7 Conduct supply and medicine use problem studies 10
8 Take actions based on the supply and medicine use study 15
findings
9 Report its performance activities to the management 10
DTC functionality (%) Sum of total score
Functionality of DTC if >75%, Yes, If < 75%, No

32
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 1.2: Data collection form for indicators obtained from prescriptions/ prescrip-
tion registration book

1. Data Collection Form for Indicators Obtained from Prescriptions

Health Facility: ______________________

Investigator: ________________________
Reporting period: from ___________ to __________
SN # # Generics Injection Antibiotics # on Diagnosis
Drugs (0/1) (0/1) FSML*
1
2
3
4
5
6
7
8
9
10
--
--
--
100
Total X XXX YYY X
Average X X X X X X
Percentage X % of total drugs % of cases % of total cases % of total X
drugs

*FSML: Facility Specific Medicines List

For this M&E framework, Antibiotics (XXX) and # on FSML*(YYY) are reported to the next administrative level.

Take a sample of 100 prescriptions using systematic random sampling from the pre-
scription register/prescription paper during the fiscal year

33
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 1.3: Counseling time registering form

Counseling time in seconds
Patie
nt #

T1 T2 T2-T1
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.

Method
Observe a series of at least 100 patients and record the time spent for each encounter.
Time is recorded when a patient receives the medicine during which instruction on the
use of medicine is provided.

34
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 1.4: Data collection form for patient knowledge and labeling interview

Labelling and knowledge data is obtained by observing a sample of at least 100 clients
during exit interview.To analyze knowledge, the label of medicine dispensed to patients
can be checked.

NB. When regional/national assessments are conducted, take 30 encounters from each
of 20 health facilities.

35
 Annex 1.5: Health facility dispensing registration book
Region ________ Woreda __________ Name of Health Facility _______________

Medicines Level of Importance by VEN

Prescribed
Diagnosis Therapeutic Vital Essential Non Overall*
Dispensed (Y/N)

SN Patient Name Age Sex (NCoD) Category (1,0) Remark

(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14)

Count total patient Count Total 1

Overall*: Enter '1' only if all the prescribed medicines are dispensed and enter '0' if one or more medicines no t
dispensed. FMOH V1 2009
Medical Equipment Management Monitoring and Evaluation Framework
National Pharmacy Service, Pharmaceuticals Supply Chain and

36
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 1.6: Functionality of clinical pharmacy

Annex 1.7: Criteria to measure UDS functionality

Annex 1.8: Criteria to measure compounding functionality

37
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 1.9: DIS functionality

Annex 1.10: Functionality of APTS

Annex 1.11: Client satisfaction with dispensing services

38
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 2. Registration formats for supply chain indicators

Annex 2.1 Forecast accuracy
Forecast accuracy for tracer products
Quantity
Forecasted Consumed Forecast error Forecast
S.N. Tracer products quantity (Issued) Quantity (P3) Accuracy
(P1) (P2) (P1-P2)/P2 (1-P3x100)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Summary Forecast
accuracy

Annex 2.2. Supplier fill rate

39
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 2.3 Average lead time

40
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 2.4 Tracer drug availability and stock out duration tally sheet and registration
format

41
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 2.5 Wastage rate

Annex 2.6 Percentage of facilities that maintain acceptable storage conditions

check list to evaluate good storage condition
Assess the storage conditions of main storage area only. Place a check (tick) mark in
the appropriate column based on visual inspection of the storage area. To qualify for a
“Yes” response, all products must meet the criteria for each item.

Good storage condition criteria

S.N. Criteria Met
Yes (1) No (0)
1 Products are arranged on shelves with arrows pointing up, and with identification
labels, expiry dates, and manufacturing dates clearly visible.
2 Drugs are stored and organized to FEFO procedures and are accessible for counting
and general stock management.
3 Outer cartons are in good condition (not crushed, perforated, stained, or otherwise
visibly damaged).
4 Damaged and expired products are separated from usable products in the storeroom,
and procedures exist for removing them from inventory.
5 Drugs are stored in a dry, well-lit, well-ventilated storeroom. (Visually inspect roof,
walls, and floor of storeroom.)
6 Cartons and products are protected from direct sunlight.
7 There is no evidence of rodents or insects in the storage area. (Visually inspect the
storage area for evidence of rodents [droppings] or insects that can damage or contaminate
the products.)
8 Storage area is secured with a lock and key but is accessible during normal working
hours; access is limited to authorized personnel.
9 Products are stored at the appropriate temperature according to product
temperature specifications (8°–30°C) and including cold chain storage (2°–8°C), as
required for certain products.
10 Roof is maintained in good condition to avoid sunlight and water penetration.
11 Storeroom is clean, with all trash removed, no evidence of food and drinks, products
stored on sturdy shelves/bins, and boxes organized neatly.
12 Current storage space is sufficient for existing products and planned program
expansion.
13 Drugs are stored separately from insecticides, flammable products, and chemicals.
Total number of Yes
Storage condition score (%)=Total Yes *100
13
If storage condition score is > 80%, say acceptable

42
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 2.7 Inventory accuracy rate

S.No. List of Tracer Drugs Bin Physical Count Bin Card Balance
Card/Electronic equals with
Record Balance physical count
(if yes put 1, if no
put 0)
1
2
3
4
5
6
7
8
9
10
Number of items where bin card (Manual or Electronic) balance equals physical stock count
Sum=Total number of “1” Checks
Inventory accuracy rate =Total yes *100
10

Annex 2.8 RRF reporting rate

Date entry: (enter 1 if the facility reported using the RRF, 0 if the facility does not use
RRF report in the reporting period. Please fill for each period).

Annex 2.9. Disposal of unfit-for-use medicines

43
 Operational condition
N Loca Nam Inv M Ser Ma C Yea Supp fu Non- wh W Ho Tra Train Spa Do Do
u tion/ e of ent o ial nuf ou r of lier/ nc Func Non y hen w ine ed re you you
m Depa Equi ory d No act nt ma Loca tio tiona - do do ofte d Tech par have have
b rtme pme No el . ur ry nuf l nal l but Fun wn wn n is Op nicia ts user serv
e nt nt . er of act agen repa ctio it era n/En ava Man ice
r O urin t rable nal Use tor ginee ilab uals? Man
rig g and d (ye r ility (Yes/ uals
in not s/N (yes/ (Ye No) (Yes
repa o) No) s/N /No
rabl o) )
e
Medical Equipment Inventory Form

Date of conducting survey……………………

Annex 3.1 Availability of functional medical equipment

Name of health facility ………………………………………….…

Annex 3. Registration and reporting formats for medical equipment indicators
Medical Equipment Management Monitoring and Evaluation Framework
National Pharmacy Service, Pharmaceuticals Supply Chain and

44
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 3.2. Percentage of health facilities with updated medical equipment inventory

Annex 3.3. Criteria to functionality of medical equipment committee

S.N Functional
Criteria
o Yes No
1 Assigned medical equipment committee members by official letter
2 Has approved TOR
3 Meets regularly at least every two months with documented minute
4 Has annual action plan and monitor performance
5 Has updated model medical equipment list
6 Conduct annual medical equipment inventory
7 Has medical equipment policy and procedures
8 Maintain equipment history profile for all model medical equipment
9 Follow disposal of non-functional medical equipment
Follow the reporting and implementation of medical equipment
10 indicator findings
Review and follow medical equipment procurement and installation
11 request
Total number of "yes"
Total Criteria 11
Percentage functionality of MEMC = Total number of "yes"
Total criteria
Functionality of MDC (>80%) If functional write “1” If not functional
write ”0”
Note: A health facility is considered as having functional MEMC if
80% of the above requirements are met.

45
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 3.4. Criteria for the scheduled preventive maintenance practice

Criteria
No Care and Safety Functional Calibratio Preventive
cleaning procedure and n testing maintenanc
schedule s in place performance (Calib.) e checks
Medical equipment (C & C) (Safety P.) (F & P) (PMC)
Yes No Yes No Yes No Yes No Yes No

ME4.1=Total no. of Yes

ME4.2=Expected PPM
Percent PPM performed =
ME4.1/ ME4.2
Overall average PPM % C&C + % Safety P+ % F & P+% Calib.+ %PMC
performed 5 = ____________

NB: it is “yes” if it meets 80% of the preventive maintenance criteria for 80% of medical
equipment.
Annex 3.5. ME5. Percentage of medical equipment installation

Note: ME5.2 and ME5.4 (total medical equipment delivered within the past six months)
are the same number
Annex 3.6. ME6. Biomedical professionals’ positions filled at health facilities

46
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 3.7. ME7. Availability of medical equipment as per the national standard

Annex 4: Supply chain and pharmacy service crosscutting indicators

Annex 4.1. Pharmacy review meetings conducted

Annex 4.2. Supportive supervision of health facility pharmacies

Annex 4.3. Percentage of pharmacy workforce positions filled at health facilities Annex

47
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 5: Reporting Formats

Annex 5.1: Health Center

48
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 5.2: Hospital

49
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

50
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 5.3: Woreda

51
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

52
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

53
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 5.4: Zone

54
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

55
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

56
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 5.5: Region

57
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

58
National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Annex 6: List of workshop participants

S.No. Full Name Organization
1. Andualem Ababu FMOH/PMED
2. Anteneh Tsige GHSC-PSM
3. Asnake Mebrat Gambella RHB
4. Azeb Fisseha JSI/AIDSFree
5. Beshir Abdi Somalia RHB
6. Bethlem Hailu FMOH/PMED
7. Buzuayehu W/Hitsan Black Lion Specialized Hospital
8. Dagim Damtew FMOH/DPCD
9. Deresse Abera Oromia RHB
10 Edmealem Ejigu GHSC-PSM
11. Elias Germew GHSC-PSM
12. Fasika Berhanu Dire Dawa Administration HB
13. Fikreslassie Alemu GHSC-PSM
14. G/Egziabeher W/Giorgis Tigray RHB
15 Hassen Seid EPSA-main office
16 Kaleb Terefe SNNPR HB
17 Lemlem Degifu FMOH/PMED
18 Lucha Geneti Oromia RHB
19 Marye Yehuala Amhara RHB
20 Melkamu Kumsa Melka Oda General Hospital
21 Meseret Zerihun JSI/AIDSFree
22 Mesret Adugna FMOH/PMED
23 Miraf Tesfaye FMOH/MCH
24 Mohammed-Aman Jemal FMOH/PMED
25 Mustafa Mohammed Benishangul-Gumz RHB
26 Seid Ali CHAI
27 Seid Mohammed Afar RHB
28 Seife Demisse Addis Ababa City Administration HB
29 Selam Kifle FMOH/PRD
30 Selamawit Meressa JSI/AIDSFree
31 Shegaw Mulu FMOH/PPD
32. Solomon Abdella FMOH/PMED
33 Solomon Nigussie Adama EPSA Branch
34 Sufyan Abdulber FMOH/PMED
35 Tadele Gedif Dangila Hospital
36 Wondowesen Shewarege FMOH/PMED
37 Yidenkachew Degifa FMOH/PMED

59
 National Pharmacy Service, Pharmaceuticals Supply Chain and
Medical Equipment Management Monitoring and Evaluation Framework

Monitoring and Evaluation Framework for Pharmaceutical Supply Chain, Pharmacy Ser-
vice and Medical Equipment Management