Information Products to Drive

Decision Making: How to Promote

the Use of Routine Data Throughout
a Health System

MEASURE Evaluation
Eric Geers, Peter Nghui, Akaco Ekirapa, Viola Rop,
Selemani Mbuyita, Jackie Patrick, and Sono Kusekwa
Kenya Ministry of Health, Division of Health Information Management
David Soti and Martha Muthami
Tanzania Ministry of Health, Community, Development, Gender,
Elderly and Children
Claud Kumalija
Abbreviations
ANC antenatal care
ART antiretroviral treatment
CCHP comprehensive council health plan
CHF community health funds
CHMT county health management team(s)
CHP county health profile
CTC care and treatment clinic
DMO decision-making officer
EID early infant diagnosis
DVDMT district vaccine and data management tool
HCMIS human capital management information system
HCMP health commodities management platform
HMIS health management information system(s)
HRHIS human resources for health information system(s)
iHRIS integrated human resource information system(s)
IDSRS integrated disease surveillance response system(s)
ILS integrated logistics system(s)
IT information technology
KEMSA Kenya Medical Supplies Authority
LMIS logistics management information system(s)
MCH maternal and child health
M&E monitoring and evaluation
mRDT Malaria Rapid Diagnostic Test
NGO nongovernmental organization
NSMS nutritional status monitoring system(s)
NTLP National Tuberculosis and Leprosy Program
PMTCT prevention of mother-to-child transmission
RCH reproductive and child health
RMNCH reproductive, maternal, newborn, and child health
USAID United States Agency for International Development

2
Data are fundamental in health
communications
An information product contains According to the Health Metrics Network,
sound analysis of good-quality
data routinely collected at health without data-informed decision making,
facilities. Products present the a health ministry limits the return on
analysis in a compelling format
that changes the audience’s
investment from a routine data collection
understanding, influences system, because it becomes simply a reporting
decision makers, and leads them tool, not a driver of action and persistent
to improve health services or
policies. improvement in the delivery of health services.
Source: Health Metrics Network, World Health Organization, 2008
Health behavior-change
communication with the public
takes time, effort, target audience
pretesting, and tracking of Conclusion: To have data and not use them is
message impact. Creating health counterproductive for a health system.
information products to promote
data use for decision making in
the health system has these same
requirements.

3
What kind of data?
The design of information products for data use involves the analysis
of raw data routinely collected by health facilities to answer important
questions about health services, such as:
• Are we meeting targets?
• Are we reaching clients who need services?
• Are resources adequate to provide the package of services and
maintain standards of care?

4
Who can or should produce information
products?
In many countries, it’s the health
workers at the national and Conclusion: Greater responsibility for analyzing data for health
regional levels who have the information products to improve health services may logically belong
technical ability to transform at the regional and national levels, where there are more resources
raw health data into information and time for the task.
products that influence decision
making. They also often receive
the most external support
from implementing partners,
universities, and other experts.
National & Regional District & Facility
Health workers at the district and
Technical ability &
health facility levels prioritize
delivering health services, followed
external support Technical ability &
by routine data collection. They external support
frequently have little technical
ability, technology, or external Data collection &
support to analyze the data they reporting burden Data collection &
collect. They frequently cite reporting burden
an insufficient work force or
technical capacity to fulfill their Service provision
triple burden: services, data
collection, and data use. Service provision

5
What does this tell us, so far?
A. Analysis of available data packaged in useful health Many working in the Kenya and Tanzania health systems
information products is necessary in order to make good agree that the DHIS 2 platform to manage routine health
management decisions regarding health. This is data use. data has promoted data use because it:

B. Health systems could achieve more data use in several • Includes up-to-date, key health indicators from the point of
ways. service delivery
• Has multiple ways to manipulate and visualize data
• Features potential linkages to other data sources, such
as health commodities,1 human resources,2 and disease
surveillance3
1
Health commodities databases—Kenya: DHIS 2, HCMP, KEMSA, and LMIS; Tanzania: ILS.
2
Human resource databases—Kenya: iHRIS; Tanzania: HCMIS, HRHIS, and iHRIS.
3
Disease surveillance databases—Tanzania: IDSRS.

6
“DHIS 2 is the mother database, so To promote data use, the facility and district levels need technical
assistance to use the tools in DHIS 2, such as custom dashboards.
reports generated from DHIS 2 Regional and national program experts should assist in analyzing data,
because they have more time and technical ability than do those close to
are the most important ones in service delivery.
informing routine health service
delivery practices.”
– District Conclusion: Information products that reduce the
burden of analysis and data visualization for health
workers, whose responsibilities are to their clients,
will increase the likelihood that data will be used to
make decisions on managing health programs.

$$
$
7
What types of information products
are most effective to drive data use?
MEASURE Evaluation on regional, district,6 and health
conducted qualitative facility levels to:
studies in Kenya4 and 1. Explore how routine
Tanzania5 to find the data are disseminated in
answer. The research aimed information products, to
to learn what products whom, for what purpose
from routine health data Here’s
were available, if they 2. Describe organizational
Did you an idea.
could be improved, and support or obstacles
how service providers to using routine data know?

atadatdataa
contained in these
data
could best use them.
Qualitative interviews
with key informants in the
ministries of health focused
products
3. Understand how target
audiences understood
d
datadata
and interacted with
4
Kenya: Ministry of Health (MOH).
available information
5
Tanzania: Ministry of Health, products
Community, Development, Gender,
Elderly and Children (MOHCDGEC). 4. Identify other supports
6
In Tanzania, the health system or barriers to using
consists of regions and districts. In
Kenya, it is divided into counties and
information products in
subcounties. For consistency, this decision making.
report will describe different levels
of the health system as regions and
districts.

8
Group discussions were held with
57 staff managing services from
the national to the facility level.
The staff represented units providing services in HIV and AIDS; maternal and child
health (MCH); laboratory, pharmacy, logistics, monitoring and evaluation (M&E)
units; and regional and district health management teams in Kenya and Tanzania.

Discussions at the health facility level were chiefly with nurses, health center in-
charges, and data managers.

In Kenya, discussions took place from September 2015 to April 2016 in Nairobi,
Machakos, and Migori counties (n=23).

In Tanzania, discussions were held from March to July 2016 in four districts in
four regions: Central (Bahi); Dar es Salaam (Temeke); Mbeye (Rungwe); and Pwani
(Mkuranga) (n=34).

9
Interview process
Group interviews permitted observations of the working
relationships between those who managed health programs and
those who managed health data.

After the interviews, informants reviewed samples of actual

information products to interpret and to develop recommended
actions based on data.

10
Analysis
The research included an analysis of The analysis of themes suggested
interview reports, and employed a common patterns, based on
topical codebook created from the how one theme aligned with or
interview questions devised for data contradicted another, and provided
collection. answers to the research questions.
All themes were supported by
Researchers used the topically coded informant responses.
sections to inform the development
of interpretative themes; all coded
text was analyzed manually in
Microsoft Excel and Word.

11
 ??
Sample questions
The interviews included questions about data sources:
• How did respondents access data to assess program performance?
• What specific data systems did they use, such as for services, human
resources, or commodities?
• How did they triangulate data from multiple data sources?

?
We asked informants about their experience with specific information products
developed from these routine data sources:
• Dashboards or reports
• Relevance of these products to their work
• Preferences for receiving information
• Organizational procedures around data reviews
• Challenges and motivations for using data for decision making

12
 This report provides findings and
recommendations, divided into
four chapters.

Chapter 1. Data use promotion................................................................ 14

Chapter 2. Capacity-building needs........................................................ 36
Chapter 3. Health systems strengthening............................................ 46
Chapter 4. Respondents’ observations on the utility of .................. 55
information products

13
 Chapter 1
Promotion of data use
This section summarizes findings on data use shared by informants. Given these
findings and experiences of the MEASURE Evaluation project in strengthening
health information systems, we recommend the activities below.
Summary of findings and recommendations on promotion of data use:

Findings
Information product design:
• Data are essential for regional/district, less so at health facility (see page 16)
• Capacity to analyze data declines with proximity to service delivery level (see page 18)
• Few standardized information products are tailored to local information needs,
regularly disseminated, and used (see page 19)
• Dissemination of information products has greatest impact prior to key planning
events (see page 21)

Recommendations
Information product design: (see page 30)
• Identify information needs at district and health facility level
• Design visual presentations of data that highlight key messages in the data
• Pilot-test with health managment information system (HMIS) focal points and program coordinators
• Orient HMIS focal points and local decision makers to information product
• Disseminate prior to key planning events
14
Chapter 1 Promotion of data use

Findings
Feedback:
Informants value feedback on performance and advice on how to improve services
(see page 24)
Recommendations
Feedback:
Provide performance feedback within the information product (see page 33)

Findings
Information sharing:
• Analysis and interpretation of data are not shared across the health system
(see page 26)
• When decision makers and HMIS focal points work together, data is likely to be
used to improve programs. (see page 28)

Recommendations
Information sharing:
• Provide opportunities for performance review meetings (see page 35)
• Facilitate access to program experts

15
 Chapter 1 Promotion of data use

Data initiate actions at the

regional and district levels...
District uses for data Regional uses for data
    building
Assess facility Staff Planning and Capacity
reporting rates performance performance bonus and supervision
and performance evaluations payments* (in
Tanzania)

Plan supervision, *Based on data completeness, reporting
rates, service indicators, disease
trainings, and incidence, commodities management, etc.
health promotion

“… We CHMT [the county health management team]

are motivated [to use data] because our work involves
a lot of reporting and measuring performance of our
health providers... In doing so, we need data and we
are happy to have them… Health providers are not
equally motivated and find the data collection tasks and
reporting just a burden.” —District

16
Chapter 1 Promotion of data use

...but less so at the facility level

However, often data are not used to support any primary roles or They did not have access to or receive any processed data from
tasks at the service-delivery level: the reports they submitted. They were familiar only with the
HMIS through their own use of paper-based data collection tools.
“A barrier to information use is when facilities do Their primary interactions with the district level were to clarify
not really feel like the data they report gets used to reports or receive new guidance on data collection.
support them. They believe that these reports only
benefit the national-level program.” —District

Conclusion: There is insufficient effort to engage

In the health information system, health providers collect data
service delivery staff to understand the data that they
every day in service registers and every month in summary
collect.
reports for different data systems: service delivery, human
resources, commodities management, disease-specific, and
others. “We need to change the whole culture and practices
of information sharing to motivate health providers.
Informants at health facilities were not aware of what happens They need to feel and see the value of the data they
to the data they collect at other levels of the health system. collect by sharing with them frequently the outcome of
the work they do on data.” —National

“We don’t know how the data

are being processed…” —Facility

17
Chapter 1 Promotion of data use

Less capacity to analyze data at

district level
At the district level, program coordinators often are assigned to specific health areas. They produce standard reports that contain basic
frequency tables of key indicators and a narrative. Informants gave four chief reasons why available data often are analyzed in topic-
specific ways and without more complex analysis: workload, capacity, insufficient data to meet demand, and job responsibility.

   
Workload Lack of skills Data ≠ demand Not in job
There is limited time to synthesize data into Informants said limited “County-level managers description
useful information. The HMIS unit spends about knowledge of basic would also do poorly on “Our staff do not
one to two days a week resolving data-related computer skills is a the interrogation and produce their own
issues. Program coordinators spend about a barrier. use of these reports analysis or displays.
quarter of their time on data issues, report for their own decision Their job description
“CHMT staff do not have
writing, and presenting reports at meetings. making, saying that does not require
the capacity to produce
the data they need them to do so.”
“The policy requires a specific person [to manage their own displays in
is not what is being
data]. But in practice, one person cannot manage DHIS 2.”
collected…”
everything. Our unit has five people, of whom
three are data clerks.”
“We also have data clerks who conduct data entry. Conclusion: In order to promote additional analysis of routine health
They are just volunteers.” data, district-level staff need to see and understand more sophisticated
“Vertical programs that come with vertical analysis approaches, and the review of information products should be
tools for data collection tend to increase service integrated in routine operations.
providers’ workload.”

18
Chapter 1 Promotion of data use

Few standard information products

are tailored to local information
needs, and they are not regularly
disseminated and used.
One of the more popular information products
was the RMNCH* scorecard (see respondents’ “The RMNCH Scorecard is also linked
observations on information products on page
66). Informants also mentioned that they transfer to the PMTCT** indicators and is one
raw data from DHIS 2 to Excel to form tables and
charts for annual reports. None of the informants of the very comprehensive summaries
mentioned doing these types of analysis with other
information systems, such as human resource or of reproductive and child health
commodities systems.
indicators as well as regional, district,
and facility performance.” —District
*reproductive, maternal, newborn, and child health
**prevention of mother-to-child transmission (of HIV)

19
Chapter 1 Promotion of data use

Respondents were asked why information

products are not used in decision making.
   
Staff lack capacity to Data lack credibility Products tend to be Insufficient
interpret data: or do not match health-sector specific: dissemination:
“[The] RMCNH scorecards are information needs: One informant said that some Some informants had not
difficult to interpret, but are “It is quite rare that we use products—for instance, the seen the RMNCH or PMTCT
the most emphasized by the DHIS to make a decision. We distribution list for HIV test scorecards and others said
national level…It is difficult to tend to use the surveys more, kits—are only used for USAID- they were not consistently
define the baseline versus the because they are felt to be more funded HIV programs. available.
current figures.” — District accurate, and reporting rates in Another said that commodities
DHIS are so-so. Some months were listed in the monthly
it is good and other months it request and report prepared
“The problem with the RMNCH is wanting… and even the data by the pharmacist and district
Scorecard is that no one apart quality is still in question for health secretary.
from the national level knows DHIS.” —National
how to compute it.” —Region
Another informant thought the
RMNCH Scorecard emphasized
more long-term impact and so Conclusion: In order for an information product to be
successfully used for decision making, (1) its design
did not reflect more manageable
should conform to the users’ capacity to interpret the
short-term outcomes.
content; (2) dissemination should be consistent and
shared across health sectors.

20
Chapter 1 Promotion of data use

Dissemination of information products

is most effective prior to key planning
events
The best time to receive new information and analysis is in
preparation for key decision-making moments, when there is a need “The CCHP assessment
for evidence to justify future activities and budgets. Opportunities
occur during a specific health promotion campaign or at monthly, report is more useful to
quarterly, or annual performance reviews.
For example, annual health plans, such as the county health profile
the national and regional
(CHP) in Kenya or comprehensive council health plans (CCHPs) in levels [than to the district
Tanzania, establish performance targets. These targets may create an
“accountability effect” that motivates teams to monitor their targets and facility levels] to
and also a “competitive effect” as teams compare their performance
with that of other health catchment areas. provide support in effective
However, some informants are skeptical, because they believe these
annual health plans are more useful for management at the national
planning.” —District
or regional level and less important at the district level.

21
Chapter 1 Promotion of data use

The main channels of communication for

feedback on routinely reported data vary at
different levels of the health system.

National Regional District “The [Regional] HMIS coordinator has

Phone X X established a ‘WhatsApp’ group with all
Email X X X district HMIS coordinators and hence
In-person discussion X X can contact them and receive a pictorial
IT/DHIS 2 application X
spreadsheet of data from them.”
Formal meeting X
—Regional
“We communicate by phone and
emails, although, to be honest, the
emails are never responded [to].”
—District

22
Chapter 1 Promotion of data use

Whatever communication “We provide feedback on issues

channel might be used, many related to data and information
informants considered the products. But…we are not sure if
frequency of feedback to be our views are taken.” —District

insufficient.
“The ministry [of health] has
They also did not always see their views or expert advice from
others in the ministry reflected in the feedback they received. established an email group, but
Communication was mostly about clarifying a report or ensuring
accuracy of the health register.
it is not effective. It is more of
sharing concerns than receiving
feedback.” —District
“The problem is not sending the
feedback, but [for] the national
level to act and address the Conclusion: More consistent feedback on the interpretation
of data and the implications for service delivery is needed.
concern.” —Regional

23
Chapter 1 Promotion of data use

Informants value feedback on

performance and advice on how to
improve services.
“If available, an information product
would be useful to provide feedback on
performance. It would help [us] to set
new targets reasonably. It would help [us]
measure our own performance. It would
help [us] to correct ourselves where we go
wrong. It would guide us where to put
more efforts.”
—District

24
Chapter 1 Promotion of data use

“They usually give us feedback on our

performance, how far or near we are towards
nationally set targets, quality issues… in fact,
they’re the first people to come and ask you, ‘Are
you sure you have these percentages of couples
tested?’... and, ‘What is this positivity rate we are
seeing [among] key populations here?’ ”
—District

25
Chapter 1 Promotion of data use

Analysis and interpretation of data

are not shared across the health
system.
In general, if staff across all health sectors and levels of service can In contrast, a health facility worker said: “There is
have more opportunities to network and share ideas on what data
are saying about service performance, they will be more likely to inadequate time to do data review because [of]
use data in decision making. other duties … Only the indicators that are performing dismally are
therefore highlighted.”
Based on informant responses, most of these opportunities occur
in the districts, followed by regions, then national, and, last, in However, some district-level informants said they
health facilities.
do not engage in data sharing and reviews outside
For example, a district official said: “That’s why we meet weekly of preparing annual plans. “We rarely get the opportunity to review
and monthly to discuss about data. Even today, in the morning, data on a monthly basis. These meetings are conducted on an ad hoc
we met to discuss about strengthening our emphasis to improve basis.”
the quality of data during supportive supervision.”

26
Opportunities to
network and share data

Districts
More
• Several informants said they discuss data during regular daily,
weekly, or monthly county health management team meetings
• One added that they also review information products

Regions
• Data review once a month or quarterly
• Not often on the agenda of regular regional health
management team meetings
National
• Annual planning session meeting
• Annual sector-wide review meetings

Health Facility
• Data sources are HMIS registers and summary reports
• Little experience in DHIS 2 and data visualization
• Lack skills in data analysis
Less More Likelihood of data use


in decision making

Conclusions: To promote data use, service providers at the health

facility level need more opportunities to talk with others about the
data they collect.

27
Chapter 1 Promotion of data use

When decision makers and HMIS focal

persons work together, data is likely to
be used to improve programs.
The job of an HMIS focal person7 is (1) to communicate HMIS “Heads of other [health program] units
policy and information products, (2) to manage data, and (3) to are expected to make use of the data by
generate tables and charts for standard reports.
themselves, but many of them lack analytical
“The HMIS focal person is the link between skills.” —District
the national and the district and the lower National informants also spoke of the increasing need to
levels.” —District staff managing health
involve
Daily interaction with data gives the HMIS focal person more commodities among those helping to use data to
opportunities to become familiar with the data and how best to improve programs.
display them.

health program
In contrast, the job of
“… the people we are targeting to work with
coordinators is to manage and implement services. are the county pharmacists [and]…the county
Daily interaction with programs gives them more opportunities to
understand the challenges of implementation, but they may have lab coordinators and county nutritionists,
trouble describing program performance in a report. because those are the major areas where we
have commodities being procured and being
distributed.” —National
7
Often a staff member would be designated as a “focal person” and be assigned to manage and
report routine health information in addition to their existing service delivery duties.

28
Chapter 1 Promotion of data use HMIS focal person
(identifies what is
happening based on routine
data):
• Communicator
• Data manager
• Report generator

Program coordinator
(can explain the “why” Pharmacy/lab technician/
of performance based on Team approach nutritionist (can explain the
experience): to data use for “why” of performance based on
decision making experience):
• Manages service delivery
• Manages commodities

“We never make a presentation, or even give

anybody any data, before consulting each other, Conclusion: A close collaboration among all
because there are some things you will understand these staff is essential in order to use data
to describe health system performance and
as a program manager for HIV, and there are to plan activities.
some things you need to understand in data
management.”
—District

29
Chapter 1 Promotion of data use

Recommendations on promotion
of data use
Based on these findings, this assessment recommends activities that would effectively
promote the use of information products in program decision making.

A) design, B) test, and C) promote use

30
Chapter 1 Promotion of data use

A) Recommendations on information
product design
Identify information needs by region, district, and facility levels.
• Systematically develop a series of standard information products
tailored to the information needs of these audiences.
• Ensure products identify select indicators relevant to each health-sector level.
If a product is intended for those closer to the service-delivery level, the need increases
for design that effectively communicates key messages contained in routine data.
• Competing priorities at this level mean there is less capacity to analyze and
review data, so products must be more intuitive.

31
Chapter 1 Promotion of data use

A) Recommendations on
information product design
Design visuals to highlight key messages from the data:

Experiment with different data

visualization methods that
communicate the key messages
within the data.
“People don’t like reading. People want stuff that • Tables: Informants say these are the easiest
to understand and interpret and they enable
jumps out at them… PowerPoint is an excellent comparison of individual values of the data.
way, instead of sending 15 pages of prose and once
• Bar charts: These explain trends and patterns,
in a while a table.”—District identify exceptions in the data, and represent
individual values of data.
“I can just click a button and just get a flash of
• Narratives: Informants say these help explain key
what is happening in the subcounties… So that messages in tables and charts.
when you go for the quarterly meeting, you • Dashboards: Informants liked online dashboards
already know at least where you are.”—District of key indicators for specific health sectors.

32
Chapter 1 Promotion of data use

A) Recommendations on
information product design
Provide brief written feedback within the information product, such as:

• Performance comparisons among health catchments areas

• Recommendations on how to improve performance
• Suggestions on where to focus resources or where to find resources
• Sharing of best practices
• Predictions of problems before they occur
• Recognition of good performance
• Guidance on how to access other relevant data sources
• Guidance on whom to involve in the review of an information product in
terms of program expertise and decision-making authority

33
Chapter 1 Promotion of data use

B) Recommendations on
information product testing
Pilot-test the proposed information product for accessibility,
understanding, and utility.
N
• Testing ensures you communicate key messages and that the
target audience finds them useful.
• HMIS focal points and program coordinators work most with

W
routine health or HMIS data and should be included in the
testing.
• Test to ensure that information products meet the needs of
those who have authority to use data in program management
decisions:
o Regional and district medical officers or health

E
directors
o Regional and district health secretaries
o Pharmacists
o Lab coordinators

•
o Health facility in-charges
Before key planning events, orient decision makers on how to
S
use the information product.

34
Chapter 1 Promotion of data use

C) Recommendations for sharing and

promoting the use of information
products
mance
Discuss perfor
• Provide opportunities for performance review meetings
• Facilitate access to program experts

Outside of key planning events, data reviews for program

performance are not a regular agenda item for meetings of
regional and district health management teams. More Build data skills
opportunities for health staff—especially at the district level—to
discuss specific performance would help increase familiarity with
available data sources and give staff a chance to build their skills in
data analysis and interpretation.
erts
Informants also said they need access to health experts, either in
Confer with exp
person or remotely. Health teams, particularly at the subnational
level, need help developing effective program strategies to address
performance or to interpret discrepancies between different but
similar indicators.

35
 Chapter 2
Capacity building needs
Findings
• Limited skills in data analysis and information product design such as DHIS 2 dashboards (see page 37)
• Limited use of data triangulation from multiple sources (see page 39)

Recommendations
• Regional/district training of trainers (see page 43):
o Information product development
o Use of multiple databases
• Supportive supervision/coaching to prepare for performance reviews
• Working sessions to develop and share DHIS 2 custom dashboards or other information products (see page 44)
• Sessions on engagement with a variety of databases (see page 45)

36
Chapter 2 Capacity building needs

Limited skills in data analysis and

creating information products,
such as DHIS 2 dashboards
Many informants said they need better skills in data
management, analysis, visualization, and report development,
“You can see everyone is
particularly at the district level. Others mentioned needing
training on how to access and generate information products
struggling to come up with
using DHIS 2. the tables and then come up with
the graphs.” —District
“The data from the DHIS is quite “CHMT staff do not have the capacity
relevant at subcounty level… to produce their own displays in
But it requires that the officers DHIS 2. While they have access to
be familiar with the system and the DHIS 2, they have limited
set up their own information capacity to conduct simple
products (i.e., dashboards or pivot analysis and produce needed graphical
table summaries).” —District displays.” —District

37
Chapter 2 Capacity building needs

Informants said that platforms exist—separate from HMIS or DHIS 2—that are useful for assessing performance,
setting priorities, budgeting, and general decision making. They also said, however, that a staff member assigned
to manage a particular data source is typically the only one to use it.

Data sources by frequency of mention (1 being mentioned most often)

1. Health facility registers: often cited as initial data source or to verify reported data
2. Logistics management information system (LMIS) to manage, reallocate, and send stockout
alerts for commodities
3. Human resources information system (HRHIS) to manage staff allocation and training
4. HIV sector-specific databases:
a) Care and treatment clinic (CTC)7
b) Early infant diagnosis
c) Viral load
5. Malaria Rapid Diagnostic Test (mRDT)
6. National Tuberculosis and Leprosy Program (NTLP)
7. Nutritional status monitoring system (NSMS)
8. Integrated disease surveillance response system (IDSRS)
9. District vaccine data management tool (DVDMT)
10. Community programs activity report

7
The CTC was managed by a nongovernmental organization (NGO) in Tanzania working in the MCH and HIV and AIDS health sectors. The CTC
database was created by the U.S. Centers for Disease Control and Prevention. It includes a pharmacy or commodities module and tracks home-
based care.

38
Chapter 2 Capacity building needs

Limited triangulation of data from

multiple sources
Because several health-related but separate databases exist, the researchers investigated whether
informants compared data from multiple sources to produce useful information for program decision
making and management.

Informants were aware of possible comparisons of separate databases, such as among:

1. DHIS 2 health service indicators and commodities

2. Specific program databases, such as for malaria and commodities
3. Indicators from different health services, such as reproductive and child health data compared with
HIV services data
4. Reports from vaccinations campaigns and DHIS 2
5. DHIS 2 reports on healthcare provider training/performance and human resource databases

39
 Chapter 2 Capacity building needs
Did data triangulation occur? Respondents described three scenarios:

YES; YES; NO
led by decision led by
maker program
(R/DMO) coordinator

Scenario 1: R/DMO requests reports and Scenario 2: Program coordinators “This culture of triangulation of
presentations from program coordinators collated data from sources relevant to their information, data, and reports is missing
for their health sector and leads efforts health sector and discussed conclusions here.”
to merge this information into an annual as a team.
plan. “I can’t remember when as a team
“As one person, such practice [of using we brought in triangulation data or
more than one data source] is rare. But as information products from different
“The RCH* coordinator makes use of the a team, we tend to triangulate information sources to arrive at a decision.”
DHIS 2 data more than other sources, extracted from various sources.”
while the… pharmacist makes use of the “We have never been able to compare
ILS** database… Each of them would “…CHMT as a team conducts data commodity versus service data. But it is
prepare their reports based on those triangulation all the time. This is because something that we are hoping we can
data sources, and the DMO would be when planning, all data sources are do.”
expected to lead the CHMT in merging consulted. The DHS*** will present
these reports into one health plan…” HRHIS+, the district pharmacist presents
ILS, and the tuberculosis and leprosy
“The CCHP is prepared by merging presents the NTLP++.”
plans submitted by different heads of
units within the CHMT… Since funding is “The other time that you might find
always limited, some aspects of the plans yourself looking at DHIS and… maybe
from each individual unit plan would need what [vaccines] the facilities have
to be removed and in such a discussion ordered, is when you want to make
there is usually… a lot of disagreement a decision of what you want to order. * reproductive and child health

among CHMT members.” Because maybe the figure you get with ** integrated logistics system
the DHIS compared with the commodities *** district health secretary
+
human resources for health information system
that you had in hand and what you ++
National Tuberculosis and Leprosy Program
distributed is not making sense.”
40
Chapter 2 Capacity building needs

Barriers to data triangulation occurred when:

1. Program coordinators only worked with one type of database Barriers to
2. There were technical challenges in reconciling data from
separate systems with different management structures multiple
3. Informants were unaware of or lacked access to other data source
databases
triangulation
Specialization: Conflicting data
1. “… [A] challenge is the specialization of key reconciliation:
actors [such as program coordinators, DHS, “Conducting
and DMO] who become ignorant of how triangulation is
other data sources work.” technically challenging
and time consuming.
Lack of awareness/access: Different periods: Not many can do this.”
1. “[A] challenge is navigating through more
than one system. Thus, an integrated system 1. “I have only interacted with “Using more than one routine
will be more useful and easy to use.” DHIS. I have not had a data source is challenging
chance to access the iHRIS* because they may not be
and KEMSA**. I believe the available at the same time
KEMSA system is used by the or may have been produced
pharmacist.” with reference to different time
2. “… When I need data for periods.”
malaria, I have to go through
another person. [This] often
[happens] when doing
* integrated Human Resource Information System
performance review, thus
creating delays.” ** Kenya Medical Supplies Authority

41
Chapter 2 Capacity building needs

Recommendations on capacity
building
A. Regional/district training of trainers
• Information product development
• Access to multiple databases
B. Supportive supervision/coaching to prepare for performance reviews
C. Working sessions to develop and share DHIS 2 custom dashboards or
other information products
D. Promote engagement with a variety of databases

42
Chapter 2 Capacity building needs

Recommendations on capacity
building
A. Regional/district training of trainers B. Supportive supervision/coaching to prepare for
Formal training is most useful when introducing a new concept, such performance review
as systematic methods to develop new information products or how to Ways to reinforce learning are to provide regular supportive
access multiple databases. supervision and coaching to prepare teams for performance
However, formal training can be expensive, especially with high staff reviews and program planning.
turnover. A way to spread training costs is to train facilitators at the
regional or district level so they can provide the same training to
other districts and health facilities. Another cost-effective strategy is to “For future motivation,
provide a guidance booklet with instructions that staff can reference as
needed on how to access and use DHIS 2, perform common indicator [we] should target health
analyses, and facilitate data review meetings to develop plans to address
performance.
providers by providing them
Also useful are tutorials or online courses. However, this approach with seminars and on-the-job
would be limited to those with a consistent Internet connection.
orientations on data issues,
“There is a need for subcounty including conducting simple
officers to be trained on analyses.”—District
visualization of the data in order
to know how they are performing
on specific indicators.”—District
43
Chapter 2 Capacity building needs

Recommendations on capacity
building
C. Working sessions to develop and share DHIS 2 custom dashboards or
other information products

Teams should work together to develop their own tailored dashboards and
other data visualizations and share this work with others.

“Building their skills so they can also

access DHIS 2 and be able to develop
their own graphic displays would further
motivate them in [the] future.”
—District
DHIS 2

44
Chapter 2 Capacity building needs

Recommendations on capacity
building needs
D. Promote engagement with a variety of databases
Several databases track healthcare delivery, each of them often managed by specific
health staff. An effort to engage other health staff unfamiliar with these potential
resources within different databases would encourage more triangulation of data
for decision making.

Data use for decisions can be encouraged, by making sure that information
products are available and all participating actors are familiar with them.

45
Chapter 3
Health systems strengthening
Findings
• Data collection burden (see page 47)
• Need for regular infrastructure maintenance (see page 48)
• Confidence in the quality of routine data (see page 50)
• Request for further systems integration and access (see page 51)

Recommendations
• Reduce data collection burden at the service delivery level (see page 53)
• Implement annual maintenance reviews of the HIS (see page 53)
• Work with subnational teams to identify initial data sources/indicators essential for system integration (see page 54)
• Provide multiple links within DHIS 2 to other data sources (see page 54)

46
Chapter 3 Health systems strengthening

Burden of data collection

All routine databases, health service indicators, commodities,
human resources, disease surveillance, etc., draw from “We are overburdened with
facilities that submit monthly aggregate reports. lots to do. We usually have
Many facilities have no online access to DHIS 2. Facility
staff use paper copies of registers to record data and then
long queues when attending our
summarize them in reports for a district HMIS focal person patients and we cannot fill in all
to enter in an online system.
the details required. So we opt to
Informants spoke about the burden of data collection on
health providers, who spend from a quarter to half fill in some of the gaps later and
of their time on data collection: completing registers,
compiling reports, and resolving data issues. sometimes some of us tend to
forget, hence having gaps in the
register.” —Health facility

47
 Chapter 3 Health systems strengthening

Need for regular infrastructure

maintenance
In terms of barriers to data use, the concerns informants
mentioned most often were infrastructure issues: “Where there is poor network,
1. Reliable Internet access like at [the subcounty hospital],
2. Sufficient number of computers we stayed for almost two weeks
3. Replacement data collection registers before I got the data I needed,
4. Dependable power sources
because they were telling me the
Internet connectivity was hampered by technical issues and lack of
funds to pay for the service.
network is low.” —District

48
Chapter 3 Health systems strengthening

Some informants said they needed more laptops distributed Although a dependable power source was mentioned by some
among both HMIS focal persons and program coordinators. informants, there have been local efforts to deal with this issue.
Others would simply wait until power resumed.
“If I had a laptop of my own,
I would be able to assist my “We are in the process of
colleagues where I would be.” installing our own standby
—District generator to solve the power
Another issue identified by district-level informants was delays
cuts problem.” —District
in replacing the data collection registers at facilities. Informants
said that responsibility for the distribution of new registers had
recently shifted to the districts, and some were not prepared to
take on this task.

“This has created a huge problem

due to procurement policies,
and thus we are into a fourth
month now with scarcity of
HMIS registers in our facilities.”
—District

49
 Chapter 3 Health systems strengthening

Confidence in the quality of

routine data
Most informants expressed confidence in the quality of routine data that are entered in DHIS 2 from
summary reports of data contained in health-facility registers. They attributed this confidence to:
1. Data quality checks built into the DHIS 2 system
2. Phone reminders sent to health facilities
3. Frequent training on registers and summary reports
4. Collective efforts of the HMIS focal person and program coordinators to review
data for quality

“Quality of data in the various However, some informants did question the accuracy
of the health registers and attributed this to inadequate
data sources is good and has supervision or limited follow-up on outliers in the data.

been improving over time.”

—District “Some of the data that is
entered, if you compare
with what is on the ground,
sometimes it does not reflect.”
—District
50
Chapter 3 Health systems strengthening

Request for more complete system

integration and access
Although some links between DHIS 2 and other data Some informants said access to other databases was limited,
systems were in place, informants said there should be more because those authorized to access databases were not
links among databases to overcome barriers to data use. available. Others said that access was not a challenge. Still
others said that access might be denied, because passwords
expired when staff did not access the system.
“All data sources should be
linked so that one can have “Some other CHMT members
access to any data source. have reported [access
This will reduce workload failure] when they do not
and production of different have passwords and those
reports (e.g., the CTC authorized are absent.” —District
database could be linked with
PMTCT indicators).” —District

51
 Chapter 3 Health systems strengthening

Recommendations on health
systems strengthening
A. Reduce the burden of data collection at the service-delivery level
B. Implement annual system-maintenance reviews
C. Work with subnational teams to identify initial data sources/indicators
essential for system integration
D. Provide multiple links within DHIS 2 to other data sources

52
Chapter 3 Health systems strengthening

Recommendations on health
systems strengthening
A. Reduce the burden of data collection at the service- B. Implement annual health information system-
delivery level maintenance reviews
Reducing this burden would encourage health facilities to use data. Computer and Internet-based information systems require constant
Ways to reduce the burden are to reduce or harmonize data and systematic infrastructure maintenance to ensure data are
collection procedures. available. We recommend that subnational teams conduct an annual
maintenance review of the HMIS.

“Massive data that are being This review would provide staff with recommendations to ensure
availability of: Internet connectivity, access to functioning computers
collected through various by all essential staff, IT support, availability of data collection tools,
registers could be ‘compressed’ and power sources.
Where consistent Internet connection is a problem, downloading
or reduced, hence reducing the specific data sets or an information product may be useful so users
workload of health providers.” can review the information they need.

—District

53
 Chapter 3 Health systems strengthening

Recommendations on health
systems strengthening
C. Work with subnational teams to identify initial data sources or indicators
essential for system integration
Separate information systems create yet another barrier for the triangulation and use of
data for decision making.
Informants wanted to see more integration of multiple databases in order to understand
all aspects of service delivery. This integration should be incremental, especially as other
health areas can gain confidence in the data. One way to initiate integration is to work
with subnational teams to identify the information they need most frequently from
other data sources and then work with the managers of those data to find technical
solutions for the integration of selected indicators.

D. Within DHIS 2, provide multiple links to other data sources

This should be done among databases where data comparisons would be useful.

54
Chapter 4
Product utility
Respondents’ observations on the
utility of information products
After the interviews, informants looked at samples of information We asked informants to:
products, interpreted them, and recommended actions based on 1. Interpret information
data.
2. Indicate questions about the HIV and AIDS programs in their area
The following page analyzes data from Kenya on HIV-positive
pregnant women who were not documented to have received 3. Identify actions for HIV and AIDS services
maternal antiretroviral therapy (ART) prophylaxis for PMTCT. The 4. Suggest recommendations after reviewing the information product
data are from January to March 2015.

55
Chapter 4 Product utility

Analysis of HIV-positive pregnant

women not on ART

56
Chapter 4 Product utility

Respondents’ observations on the

utility of information products
With the PMTCT information product, informants were able to identify the message:
Many counties were doing a poor job of distributing ART to mothers.
Informants’ immediate reaction was to question the quality of these data. The action
they suggested was to look at the source registers to verify the validity of the data in
this information product.

57
Chapter 4 Product utility

Observations on the PMTCT

information product
Questions Observation
What does this information tell me? “… We are looking at performance for
PMTCT… We look at the gap… We need to
figure out why those mothers who are positive
are not getting the prophylaxis.”

Any questions about the program? “… Quality issues with the data… You can’t
know which health facility, or even which
county.”

Next steps? “…Interrogate the data… A lot of our mothers,

if you go back to the registers, actually get
prophylaxis. So we would probably do a DQA
[data quality audit].”

Recommend? “I would recommend that the team conduct

a facility-level data analysis to identify which
facility is reporting this data.”

58
Chapter 4 Product utility

Respondents’ observations on the

utility of information products
After this exercise, interviewers asked informants to look at graphic displays of similar
hypothetical data that were visually enhanced (pages 60, 61, and 62) for better
communication of the message in the data.

Some of the visual enhancements were:

• Presenting a cascade of services, from antenatal care (ANC) clients tested for
HIV, to the number of PMTCT clients receiving test results, to HIV-positive
maternal clients receiving prophylaxis

• Displaying dual bar graphs with the analysis or calculation in percentages or

differences on the left and the actual numbers on the right

• Changing the order of subcounties to highlight locations with the most health
service issues

• Changing the color of the numbers displayed for areas with the most health-
service issues

59
Chapter 4 Product utility

PMTCT data charts

* All data is hypothetical

60
 8

Chapter 4 Product utility

* All data is hypothetical

61
 Chapter 4 Product utility

* All data is hypothetical

62
Chapter 4 Product utility

Based on the concept dashboard design on the previous slides, we asked

informants to comment on whether these data visuals supported or
inhibited their ability to understand, interpret, and make decisions

???
based on the data.

At this point, instead of discussing data quality, informants discussed

interventions to improve service performance. They also wanted to visit
the facilities to find out exactly which ones were having issues.

63
 Chapter 4 Product utility

Observations on the PMTCT

information product
Questions Observation
What does this information tell me? “… you can… kind of find out… how many of them did
get their prophylaxis, the total number of maternal ANC
patients, and the total number of those who are positive…
So you can even now try to judge: are they doing very well
or is there an area that they need to do an intervention.”

Any questions about the program? “… there are areas with more [HIV-positive pregnant
women]… Is it because of population… or way of life…”

Next steps? “This can tell you a lot… give you a lot to plan for
interventions.”

Recommend? “… It will require a field visit after digging deeper…”

64
Chapter 4 Product utility

Respondents’ observations on the

utility of information products
For the MCH sector, informants were asked to interpret the RMNCH Scorecard (on the
following slide).
Some of the visualization techniques used were:
• Linking MCH and HIV- and AIDS-related indicators
• Using color to highlight indicators that needed immediate attention
• Displaying trends with an arrow

65
 Chapter 4 Product utility

RMNCH observation exercise (Kenya)

66
Chapter 4 Product utility

RMNCH observation exercise (Tanzania)

67
 Chapter 4 Product utility

Respondents’ observations on the

utility of information products
Reactions to the scorecard were dramatically different in These informants were able to identify some of the broad areas
Kenya and Tanzania. The reason may have been that Kenya of service in the scorecard, such as newborn and child health,
respondents had been oriented to the tool more recently. labor and delivery, and pregnancy. However, many were unable
to describe the information depicted.
In Kenya, informants immediately began describing
performance relative to targets. Informants questioned the An informant requested that the data be in a table. Another
accuracy in reporting for some indicators, but felt they informant wanted to know why these indicators were selected.
could still use the data. In addition to addressing data Some informant recommendations for action were:
quality, informants discussed where they needed to focus their • Staff training
efforts.
• Finding additional resources for services
In Tanzania, several informants found it a challenge to
interpret every indicator or differentiate the color patterns. • More supportive supervision
Often they would not refer to the legend but go directly to the • Promotion of delivery services (attended birth)
data. One informant said there were too many indicators.
Other informants wanted more orientation on how to use the
scorecard.

Conclusion: These informant responses suggest

that pilot testing and orientation to information
products may increase the likelihood that these
products will be used to manage programs.

68
Chapter 4 Product utility

Kenya Tanzania
Questions
MCH Sector Observations MCH Sector Observations
“As a county we are able to tell where we are. We “Why is half of the cell green and another half is yellow
What does this or red? What does it mean?”
can identify the best county and even decide to do
information tell
benchmarking there … You don’t have to struggle so “We lack the knowledge on how to interpret the
me?
much, you see how the colors ... are …” scorecard …”
“Why is CHF [community health funds] coverage low in
“… with time the accuracy will improve. For now almost every district?”
Questions about we can still use the data … When partners come to
the program the county, we are able to tell them, ‘these are the “We find it difficult to tell the meaning of the arrows
indicators we are not doing well on.’” … The color key is helpful, but why do some cells have
triangles …?”

“Like this one here [points to family planning “The district needs to think of interventions to promote
Next steps
commodity coverage]. It means we need to do a lot of facility delivery.”
advocacy, especially at the community level.”

• “We should have a subcounty one …”

“We recommend orientation of the scorecard to all
Recommendations • “Is it possible to have this on the dashboard ...?”
CHMT members.”
• “… not everyone would access the computer …”

69
Summary of key findings
and recommendations
Provide information products Target HMIS focal points Informants value feedback Relate routine data
at the point of service and decision makers. HMIS on performance and to service delivery.
delivery. Information products focal points and program advice on how to improve. Performance review
that communicate key messages coordinators are good sources to This includes performance meetings and other team
from data are most effective for pilot-test information products. comparisons, implementation working sessions to develop
staff working close to the point of However, products should target strategy, forecasting tailored dashboards or other
service delivery. decision makers such as regional problems, recognition of good information products and
and district medical officers performance, and guidance on to share this work would be
or health directors, regional the use of additional data or opportunities to learn how
and district health secretaries, engaging others to use data. to use data. Reinforcing the
pharmacists, lab coordinators, links between the HMIS focal
and health facility in-charges. person, program coordinators,
and decision makers is
essential.

70
Target information product Reduce the burden of data Conduct annual maintenance Integrate information
development in capacity collection at health facilities. reviews on the HMIS to assess system databases with
building efforts, interpretation Looking at ways to reduce the Internet access, computer DHIS 2. Although some
of analyses, and access to burden of data collection, by functionality and access, linkages exist, further
multiple databases. reducing or harmonizing data- information technology integration would improve
Provide access to health collection procedures. support, availability of data- access and overcome
collection tools, and access to many data-triangulation
experts. This could
power sources. challenges.
help subnational teams
develop effective program
implementation strategies to
address service performance
or to interpret discrepancies
between different—but
similar—indicators.

71
This publication was produced with the support of the United States Agency for International Development (USAID)
under the terms of MEASURE Evaluation cooperative agreement AID-OAA-L-14-00004. MEASURE Evaluation is
implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF
International; John Snow, Inc.; Management Sciences for Health; Palladium; and Tulane University. Views expressed
are not necessarily those of USAID or the United States government. SR-17-145

[Link]
72