Report of the EGH subgroup on measuring ICT skills using

household surveys
August 2024

Summary of proposed recommendations

A. The subgroup recommends the modified wording and organization of ICT skills
indicators shown in Table 1 and detailed in Annex 1 for future data collection.
• Examples detailed in Annex 1 should be adapted to reflect the most popular local
or national services.
• Countries should consider their need for a filter question on Internet use where
individuals can accomplish near-equivalent tasks using widely adopted non-
Internet ICT services such as mobile banking or integrated voice response (IVR).

Table 1.
Proposed ICT skills indicators, by ICT skill area. Bold text indicates proposed changes to wording
from current recommendations.

Information and data literacy

Verifying the truthfulness of information found online

Finding information about goods or services

Accessing news or books in a digital format

Finding health information

Communication and collaboration

Sending content in messages

Making calls (telephoning over the Internet/VoIP)

Participating on social networking platforms

Taking part in consultations via the Internet to define civic or social issues

Digital content creation

Editing text documents, spreadsheets or presentations using digital tools

Duplicating or moving data, information and content in digital environments

Creating content combining different digital media

Using spreadsheet software

Programming or coding in digital environments

Safety

Taking security measures to protect devices and online accounts

 Taking measures to protect privacy on your device, account or app

Problem solving

Connecting new devices

Installing software or apps

Using Internet or mobile banking

Doing an online course or accessing online learning material

Purchasing or ordering goods or services

B. The subgroup agreed that countries which collected sufficient data on the five skill
areas should calculate overall digital skill levels for individuals using the criteria shown in
Table 2.
• Countries collecting data for only three or four of the skill areas are encouraged to
calculate overall skills levels for use at the national or local level.

Table 2.
Definition of overall ICT skill levels for individuals

Category Definition

Above basic skills Above basic skills in all five areas

Basic skills At least basic skills in all five areas – can be basic or above
basic, but not all five at above basic

4 of 5 Basic or above basic in any four areas and no skills in one area
(at least basic in four of five areas).

3 of 5 Basic or above basic in any three areas and no skills in two

areas (at least basic in three of five areas).

2 of 5 Basic or above basic in any two areas and no skills in three

areas (at least basic in two of five areas).

0-1 of 5 No skills in four or five areas (at least basic in one or fewer of
five areas).

C. The subgroup agreed that for the moment, at a global level, it is challenging to identify
a reduced set of indicators that can provide near-equivalent information on ICT skills in a
country compared to the full set.
• Given differences between conditions in countries and changing technologies
identifying a common reduced set across countries is a challenge.
• If interested, countries are encouraged to investigate the possibility of using a
reduced set of indicators by comparing results against the full set while also
considering conceptual issues (e.g. criteria of having all five skills areas).
 1. Background
In 2013, the Expert Group on ICT Household Indicators (EGH) added indicator HH15 to
the Core list of ICT Indicators. This indicator examines the activities individuals carry out
on digital devices as a proxy for digital skills to help link ICT usage and its impact. These
data may be used to inform targeted policies to improve ICT skills, and thus contribute to
an inclusive information society. The UN Sustainable Development Goals (SDGs) also
reference ICT skills through SDG Indicator 4.4.1: Proportion of youth and adults with
information and communications technology (ICT) skills, by type of skill.

At its 2017 meeting, EGH agreed to create a subgroup to improve the measurement of
ICT skills based on ICT household data and make proposals for a conceptual framework
and dimensions of digital skills to be monitored through ICT household data. The
subgroup operated from 2018 to 2020 amending the response categories of HH15,
reducing redundancy and filling data gaps in the skills that are currently measured.

At its 2021 meeting, EGH decided to revive the subgroup on ICT skills to reconsider ways
to aggregate indicators on skills in a meaningful way given the additional skills indicators
that were added. The revived subgroup proposed several key recommendations that
were accepted by EGH at its 2022 and 2023 meetings:

• To discontinue the grouping of indicators by levels (basic/intermediate/advanced).

• To include component indicators from HH9 to complement and rebalance the
data, aligning with the conceptual framework and its skills areas (DigComp).
• Individuals should be assessed on the number of activities within a skill area they
report having done in the last three months using the following progression:

None Basic Above basic

0 activities 1 activity More than 1 activity

o Skill levels should not be assessed in skill areas where fewer than two indicators
are collected.
o Indicators should be weighted equally within each skill area.
o Skill areas with different numbers of components should be treated equally.

EGH prolonged the mandate of the subgroup for 2024 to build on the work of previous
years and investigate how the set of ICT skills indicators could be made more robust and
resilient to technological changes. It also requested the subgroup to examine further
country examples and investigate whether and which indicators within each skill area could
be identified as “mandatory” for calculating aggregates. Lastly, EGH requested that the
subgroup also consider further conceptual and practical issues, including whether or when
data availability might be sufficient to recommend assessments of overall ICT skill levels for
individuals.
 2. Reflections from the sub-group on measuring ICT skills
The subgroup met five times in 2024 through videoconference. The focus was three-fold.
First, to examine the wording and scope of the current recommended ICT skills indicators
and make proposals to improve their robustness, modernity and clarity. Second, to assess
data availability in order to recommend assessments of overall ICT skill levels for
individuals. Last, to consider the practical aspects of developing comparable skill
aggregates given differences between countries in the questions included in their
household surveys.

Members of the subgroup were invited to provide inputs that were discussed during the
monthly calls. Cetic.br (Brazil) and Statistics Canada provided data-driven analyses related
to identifying a common reduced set of skills indicators (more information in Annex 4 and
Annex 5). Members from a consortium led by the University of Cape Town organized a
workshop in London on digital skills in low- and middle-income countries. This workshop
included sessions where attendees provided feedback on the improved ICT skills
indicators.

AnadditionalinputtothesubgroupwasfromthepreliminaryresultsofITU'sdata
collection from its annual short questionnaire (see
Annex 3). As part of its normal data collection cycle, ITU sent a questionnaire to countries
to collect ICT household survey data in March 2024 where, for the first time, data on
aggregates for each skill area were requested. The ITU received and validated data from
this questionnaire through June 2024 with 41 countries providing data on ICT skills
aggregates for at least one skill area (of 61 countries with any data on ICT skills indicators).
Of these, 19 were Eurostat countries providing data calculated according to the DSI 2.0
methodology. Results based on the other 22 countries which followed the 2023 EGH
recommendations helped to inform the subgroup on how countries can provide these
data.

A. Updating indicators
The subgroup examined each of the existing ICT skills indicators to assess whether they
could be improved keeping in mind three overarching goals:

• Maintain comparability over time by avoiding major changes.

• Maintain relevance by modernizing indicator wording and using mobile-friendly
language where warranted.
• Maintain specificity by keeping indicators comprehensible to respondents and
avoiding overly vague wording.

The changes ranged from minor wording changes to improve clarity and modernize
terms to more substantive changes which adjust the conceptual emphasis of indicators.
The changes also focused on examples included in the indicators, the subgroup
recommends that countries adapt examples to include the most popular local or
national services. Some details will be explained below, the complete set of changes can
be found in Annex 1.
Another consideration of the subgroup was the perspective shared from lower-income
countries. In some of these countries, technologies such as integrated voice response
(IVR) are widely used and may allow individuals with low literacy and without access
to the Internet to demonstrate near-equivalent ICT skills. In such cases, adjustments to
surveys may include consideration of whether to include filters on Internet use or include
reference to locally available services that do not require an Internet connection (e.g.
mobile banking services, IVR programs). Such services may be considered for the
following indicators:

• Internet or mobile banking

• Finding health-related information
• Finding information about goods or services
• Sending content in messages

While outside the scope of this year's work, the subgroup also recognized the importance
of devising simple and clear survey questions based on cognitive interviews to ensure
consistent responses from all segments of population [2]. The indicators, as defined in this
report, may in some cases need to be divided into separate questions to ease the
cognitive response burden.

Last, the subgroup agreed that adding new indicators (such as using mapping tools, using
spam blockers, creating files/folders) would be out of scope for this year’s revision. While
new indicators could be considered in future iterations of the subgroup, this year's
subgroup recommends stability for several years before further changes.

Minor changes in wording

For many indicators, some minor changes in wording were required to clarify the meaning
of the indicator or modernize some of the terminology.

In the Information and data literacy skill area, the following minor changes in wording
were recommended:

Current indicator Proposed indicator Comments

Verifying the reliability of Verifying the Truthfulness more to the point

information found online truthfulness of (even if conceptions of truthfulness
information found online can vary).

Getting information about Finding information Excluding unsolicited

goods or services about goods or services advertisements

Seeking health-related Finding health-related Focus on successful searches

information information
In the Communication and collaboration skill area, the following minor changes in
wording were recommended:

Current indicator Proposed indicator Comments

Participating in social Participating on social Emphasizing passive

networks (creating user networking platforms (e.g. participation
profile, posting messages or creating user profile, reading Adding additional examples
other contributions to or posting messages and (though countries can tailor
Facebook, Twitter, Instagram, other contributions to locally relevant examples)
Snapchat, etc.) Facebook, X, Instagram,
Snapchat, TikTok)

Taking part in consultations Taking part in consultations Emphasize civic nature of

or voting via the Internet to via the Internet to define civic consultations vs political.
define civic or political issues or social issues (e.g. urban Voting only one example
(urban planning, signing a planning, signing a petition,
petition etc.) voting)

In the Digital content creation skill area, the following minor changes in wording were
recommended:

Current indicator Proposed indicator Comments

Using copy and paste tools to Duplicating or moving data, Copy and paste not always
duplicate or move data, information and content in the primary means of
information and content in digital environments (e.g. duplicating and moving
digital environments (e.g. within a document, between
within a document, between devices, on the cloud)
devices, on the cloud)

Programming or coding in Programming or coding in Examples not informative

digital environments (e.g. digital environments
computer software, app
development)

Using basic arithmetic Using spreadsheet software Emphasizing use of

formulae in a spreadsheet (e.g. using basic arithmetic spreadsheets rather than
formulae, functions, arithmetic formulae only
macros)

Using software run over the Editing text documents, Adding examples for clarity
Internet for editing text spreadsheets or
documents, spreadsheets or presentations using digital
presentations tools (e.g. Google Docs,
Sharepoint, Apple iCloud)

In the Safety skill area, the subgroup recommended a modernization of examples of both
security and privacy measures that individuals can take. For both indicators the emphasis
was placed on active measures taken. For security measures, the word “effective” was
removed as this is a subjective assessment.

Current indicator Proposed indicator

Setting up effective security measures (e.g. Taking security measures to protect

strong passwords, log-in attempt notification) to devices and online accounts (e.g.
protect devices and online accounts changing passwords, setting up two-
factor authentification, avoiding
unsecure links or downloads)

Changing privacy settings on your device, Taking measures to protect privacy on

account or app to limit the sharing of personal your device, account or app (e.g. to limit
data and information (e.g. name, contact the sharing of personal data and
information, photos) information, restrict access to social
network profiles or geolocation,
prevent targeted marketing)

In the Problem solving skill area, the following minor changes in wording were
recommended:

Current indicator Proposed indicator Comments

Connecting and installing new Connecting new devices Wireless speakers/headphones

devices (e.g. a modem, (e.g. camera, printer, more frequently connected than
camera, printer) through wireless speakers or modems. Wired or wireless
wired or wireless technologies wireless headphones) technology is self-evident.
Installation no longer often
required

Finding, downloading, Installing software or Installing infers that the software

installing and configuring apps and apps have already been
software and apps found, downloaded.
“Configured” is not clear to all
respondents. An individual
should respond that have done
the activity if they have installed
software or apps (not required to
have installed both)

Also in the Problem solving skill area, two adjustments were made to indicators that were
previously excluding certain types of activities.

Current indicator Proposed indicator Comments

Internet banking Using Internet or mobile Removing exclusion of activities

(includes…looking up banking (includes that would nearly always require
account information; electronic transactions online banking (addition of
excludes electronic with a bank for payment, “mobile banking” explained below)
 transactions via the Internet transfers, etc. such as M-
for other types of financial Pesa, or for looking up
services such as share account information)
purchases, financial services
and insurance)

Purchasing or ordering Purchasing or ordering Removing exclusion of

goods or services (purchase goods or services (via cancelled/incomplete orders. This
orders placed via the the Internet whether or still shows the activity or skill of
Internet …; excludes orders not payment was made interest.
that were cancelled or not online; includes
completed; includes purchasing of products
purchasing of products such such as music, travel and
as music…) accommodation via the
Internet)

Substantive changes

In the Communication and collaboration skill area the subgroup recommended the
below change to emphasizing sending content as the important aspect of the indicator.
The revised indicator also expands the concept from attached files only to sending
content more generally. Attaching files is one of many ways to send messages with
additional content.

Current indicator Proposed indicator

Sending messages (e.g. e-mail, messaging Sending content (e.g. document, picture,
service, SMS) with attached files (e.g. video through attached files, embedded
document, picture, video) content, hyperlinks) in messages (e.g. e-
mail, messaging service, MMS)

The Digital content creation skill area was a particular focus of the subgroup as the
current set of activities included several possibly overlapping indicators. The subgroup
addressed this by making two recommendations. The first is an adjustment to the
indicator “Creating electronic presentations with presentation software”. The subgroup
recognized that while digital skills required for office jobs remain important, the indicators
should also recognise digital content created in other aspects of life. The proposed
indicator (shown below) will include activities that can be performed with a smartphone as
well. This also aligns with a similar existing Eurostat indicator.

A second change is recommended to drop the indicator on uploading content from the
set of ICT skills indicators. The subgroup considered sharing online to be more
representative of confidence than digital skills. The indicator will be retained as an
indicator of activities by Internet users (HH9) as it continues to be of interest for other
research outside of digital skills.

Current indicator Proposed indicator

 Creating electronic presentations with Creating content combining different digital
presentation software media

Uploading self/user-created content to a [Removed from digital skills indicators]

website to be shared

The subgroup expanded the definition further for two indicators in the Problem solving
skill area. For Internet banking, the subgroup recognized that many individuals in some
countries can do activities comparable to Internet banking using non-Internet connected
mobile phones. It is therefore important that individuals that have performed similar tasks
are also included when assessing comparable digital skills in a population.

For online learning, the subgroup also considered that the amount of learning material
available online has expanded considerably in recent years. Enrolment in a formal course
is less meaningful than it was even several years ago. While acknowledging that an
additional indicator in HH9 already exists1 the subgroup still felt it was important to
expand the definition of online learning to cover informal learning despite some overlap
between the indicators.

Current indicator Proposed indicator

Internet banking (includes…) Using Internet or mobile banking (includes…)

Doing an online course (in any subject) Doing an online course or accessing online
learning material (e.g. video tutorials,
webinars, learning apps)

Cross-cutting issues

Finally, the subgroup also identified significant overlap between the two indicators below.
To simplify and avoid confusion for respondents, the subgroup recommended dropping
one indicator. This indicator should be removed from surveys as it does not provide
additional information.

Skill area Current indicator Proposed indicator

Digital content Using copy and paste tools to Duplicating or moving data,
creation duplicate or move data, information and content in digital
information and content in environments
digital environments

Problem solving Transferring files or applications [Removed]

between devices

1
Consulting wikis (Wikipedia etc.), online encyclopaedias or other websites for formal or informal
learning purposes
B. Overall skills aggregate

Definition

Another objective of the group was to review the possibility of assessing overall ICT skills
for individuals. Ideally such a measure would inform policy on digital skills by identifying
the breadth of digital skills in a population. Data on overall ICT skills can provide an
indication of the gaps that a population needs to bridge to achieve basic levels of ICT
skills. DSI 2.0 is used in European countries to provide such information [1] and the
subgroup agreed that this definition was suitable as well as an international
recommendation. The subgroup agreed that the overall ICT skills aggregates can only
be calculated for countries reporting aggregates for all five skill areas. The agreed
definitions are provided in Table 2 (see Summary of proposed recommendations).

ITU Short Questionnaire results

The results of the ITU short questionnaire collected in 2024 (Annex 3) show how data
comprehensiveness can make implementing an overall ICT skills aggregate measure
challenging. Of the 21 countries providing data on ICT skills aggregates only 10 reported
data on all five skill areas. According to the definition above, it would not be possible to
calculate overall ICT skill levels for the 11 countries reporting aggregates for fewer than
five skill areas. Of note, the Safety skill area was the least reported skill area with only 12 of
the 21 countries reporting skills aggregates. As noted in previous subgroup reports this is
a skill area with newer indicators than the others and increases in data availability in the
future are possible and strongly recommended.

Other considerations

Based on data availability reported by the ITU, the subgroup recognized that many
countries will not be able to provide data on overall ICT skill levels immediately. Until
comprehensive data are collected for all five skill areas, data would not be internationally
comparable. However, there are many countries which collect data for three or four of the
five skill areas. The subgroup agreed that in these cases, countries could still gain valuable
information about their populations by calculating overall ICT skill levels using similar
criteria, but for the subset of skill areas that they collect. For example, in a country only
collecting data for four skill areas, an individual with basic skills in all four skill areas
collected would be considered to have basic skills. This modified ICT skill level calculation
could be used at the national level to gain valuable insights even if it is not
internationally comparable.

C. Common reduced set of indicators

Motivation

The current proposed set of indicators consists of 20 activities. The subgroup agreed that
the recommended set provides depth and breadth to assessments of digital skills in a
population. Nevertheless, there are good reasons why a smaller set of indicators could be
preferable in certain circumstances – two are elaborated below.

The variation in indicators collected by countries can result in challenges with

comparability. As noted in the 2023 subgroup report, countries often do not include all
indicators in their household surveys although this is recommended. Hence the
requirement for calculating an aggregate measure: at least two indicators within a skill
area. However, as these activities can vary between countries, skills aggregates for
different countries can be based on a different set of indicators – with no overlap in some
cases. Where this occurs, the comparability of this measure will depend on the similarity
of the underlying likelihood that an individual will have done each activity within a skill
area.

Another motivation for a reduced set is where questions on ICT activities are being
included in multi-purpose surveys, there may be limitations on the number of questions
that can be added. In certain countries, where digital skill levels are low, the inclusion of a
long set of questions, where a respondent is likely to say ‘no’, may result in respondent
frustration. This was cited as a challenge in the previous round of UNICEF's Multiple
Indicator Cluster Surveys (MICS)2 where the previously recommended set of 9 questions
on digital skills was asked for the first time.

Identifying candidate indicators

Given this background, the subgroup attempted to mitigate these issues by identifying a
common and reduced set of required indicators for the calculation of skills aggregates –
two indicators per skill area.

One set of inputs on the topic addressed the issue of whether a reduced set of indicators
could reasonably replicate the results of aggregating using the full set of indicators. Brazil
provided an interesting analysis providing evidence that using a data-based approach
such a set of indicators could be identified (see Annex 4). It noted that this type of analysis
should be done for each country, as underlying conditions are likely to vary between
countries. Taking inspiration from the Brazil analysis, Canada also provided their own
data-driven analysis using a different method. They also eventually arrived at a reduced
set, which reasonably replicated the results of the full set. However, the specific indicators
that best replicated the results for Canada differed in most skill areas from those derived
for Brazil (see Annex 5). Annex 5

A similar data-driven approach was presented where a reduced set of 10 items was
selected using Eurostat data from 27 EU countries [1, p.20-21]. The method based on
Item Response Theory (IRT) modeling to select items that maximize the information

2
Sixth round of MICS (MISC6)
content and reliability of the shortened scale. However, due to differences in indicators
and the cultural contexts, the results yield little insights for the use at a global level.

Such data-driven approaches show that a reduced set of indicators is a reasonable

proposition. However, given the lack of capacity in many countries, for the moment the
subgroup does not recommend at a global level using such a data-driven approach to
identifying this reduced set of indicators.

A complementary approach taken by the subgroup was to identify the common required
set based on their perceived importance in measuring digital skills and without a goal of
replicating the results of the full set. This expert-driven approach was used in an exercise
conducted in the London Digital Skills workshop. In this exercise participants (a group of
~20 researchers generally focusing on conditions in low- and middle-income countries)
were asked which two activities in each skill area would be most important to retain in a
reduced set of digital skills indicators. While participants shared very interesting
perspectives during the exercise, no clear choices emerged in most skill areas.

Last, the share of ITU member states providing data on these indicators since 2021 was
considered, however, the subgroup emphasized that the recommendations should be for
the best indicators in any common reduced set of indicators regardless of data
availability. This recognizes that informative indicators, that are not widely collected today,
could be collected in the future. The subgroup's recommendations should encourage
more countries to collect the current ICT skills indicators with their improved
modifications rather than be influenced by current levels of data availability.

Table 3 shows the data availability along with a comparison of the other different
methods. Based on these four sources of information and discussion during its meeting,
the subgroup noted the following:

• In the Communication and collaboration skill there was no consensus – all four
indicators are possible selections.
• In the Digital content creation skill area there was no consensus – all five indicators
are possible selections. However, the indicator Duplicating or moving data,
information and content in digital environments was recommended by three of the
four sources.
 • Finding information about goods/services was recommended in all cases in
the Information and data literacy skill area. While there was no consensus for the
second indicator, Accessing news or books in a digital format was the next most
recommended.
• No consensus was found for the Problem solving skill area.
• For Safety, the existing 2 indicators are recommended.

Table 3.
Proposed reduced sets of indicators, various sources.
Indicator 1 Indicator 2 Indicator 3 Indicator 4 Indicator 5 Indicator 6 Indicator 7
Taking part
Participating
Making calls Sending in Using
on social
Comm. and over the content in consultation instant
networking
collab. internet (E-L) messages (L- or voting via messaging
platforms (B-
[58%] C) [36%] Internet (B) (E)
C) [61%]
[45%]
Creating
Digital content Using Programmin Uploading
Duplicating or Editing…usin Using word
combining spreadsheet g or self/user-
content moving data...
different
g digital tools
software (E) coding... (B) created
processing
creation (L-B-C) [47%] (L-C) [15%] software (E)
digital media [55%] [55%] content (B)
(L) [55%]
Finding
Information information Accessing Finding
Verifying the
news or reliability of
and data about goods
books.. (E-L-
health info (L)
information
literacy or services (E- [58%]
C) [55%] (B) [50%]
B-L-C) [58%]
Purchasing Changing
Installing Internet or
Doing an Connecting or ordering settings of
Problem software or mobile
online course new devices goods or software,
solving apps (E-L) banking (L-C)
… (L) [57%] (B) [18%] services (B- app or
[58%] [61%]
C) [60%] device (E)
Checked that
the website
Taking Taking
Limited where you
measures to security
Safety protect measures
access to provided
profile … (E) personal
privacy [27%] [16%]
data was
secure (E)
E = Eurostat recommended indicators
B = Brazil recommended indicators (data-driven based on model)
C = Canada recommended indicators (data-driven)
L = Most recommended indicators from London Digital Skills Workshop (concept-based)
[xx%] = Data availability (share of ITU members with data since 2021)
Gray = Only two indicators in Safety skill area
Red = Not included in ITU digital skills indicators

ITU Short Questionnaire results

Another point to consider is that of those who provided validated data on skills
aggregates in the ITU Short Questionnaire using the EGH recommendations3, most
collected nearly all indicators in skill areas for which they reported aggregates. This is a
self-selected group, but still may show that there may be fewer concerns about
comparability. Variations on this same pattern were observed across skill areas for
countries providing aggregate ICT skills data.

• Communication and collaboration: 11 of 19 provided data for all activities in the

skill area and 17 provided data for at least three of the four activities in the skill
area.
• Digital content creation:16 of 20 provided data for at least five of the six activities
• Problem solving: 15 of 20 provided data for at least five of the six activities and
• Information and data literacy: all 17 countries providing ICT skills aggregate data
for had provided data for at least three of the four activities in this skill area.

3. Conclusions
It is recommended to close the subgroup as its work has been completed. However, the
subgroup recognizes that the digital skills required to fully benefit from digital technology
will continue to evolve. Consequently, it is likely that this topic should be revisited in the
future. To balance the need for relevance and stability, the subgroup recommends a
pause in EGH’s work on this topic for at least three years to provide stability. Digital skills
should remain open as a discussion topic in the EGH Forum.

To communicate these changes, the ITU will also request the Interagency Working Group
on SDGs (IAEG-SDG) to implement an update to the metadata for SDG 4.4.1 and consider
the most efficient way to update the Manual for measuring ICT access and use by
households and individuals with these new recommendations.

Annex 1: Mapping of current indicators to recommended

indicators

Current indicator Proposed indicator

3
Analysis excludes 19 European countries which reported data using the DSI 2.0 methodology
 Information and data literacy

Verifying the reliability of information found Verifying the truthfulness of information

online found online

Getting information about goods or services Finding information about goods or services

Reading or downloading newspapers, Accessing news or books in a digital format

magazines or electronic books in a digital (e.g. reading online news, watching news
format videos online, reading e-books on an e-
reading device)

Seeking health-related information (on injury, Finding health information

disease, nutrition etc.)

Communication and collaboration

Sending messages (e.g. e-mail, messaging Sending content (e.g. document, picture,
service, SMS) with attached files (e.g. video through attached files, embedded
document, picture, video) content, hyperlinks) in messages (e.g. e-
mail, messaging service, MMS)

Making calls (telephoning over the Making calls (telephoning over the
Internet/VoIP using Skype, WhatsApp, Viber, Internet/VoIP using Skype, WhatsApp, Viber,
iTalk, etc.; includes video calls via webcam) iTalk, etc.; includes video calls via webcam)

Participating in social networks (creating user Participating on social networking

profile, posting messages or other platforms (e.g. creating user profile, reading
contributions to Facebook, Twitter, Instagram, or posting messages and other contributions
Snapchat, etc.) to Facebook, X, Instagram, Snapchat, TikTok)

Taking part in consultations or voting via the Taking part in consultations via the Internet to
Internet to define civic or political issues define civic or social issues (e.g. urban
(urban planning, signing a petition etc.) planning, signing a petition, voting)

Digital content creation

Using software run over the Internet for Editing text documents, spreadsheets or
editing text documents, spreadsheets or presentations using digital tools (e.g.
presentations Google Docs, Sharepoint, Apple iCloud,
etc.)

Using copy and paste tools to duplicate or Duplicating or moving data, information and
move data, information and content in digital content in digital environments (e.g. within a
environments (e.g. within a document, document, between devices, on the cloud)
between devices, on the cloud)

Creating electronic presentations with Creating content combining different

presentation software (including text, images, digital media (including text, images, sound,
sound, video or charts) video or charts)
Using basic arithmetic formulae in a Using spreadsheet software (e.g. using
spreadsheet basic arithmetic formulae, functions,
macros)

Programming or coding in digital Programming or coding in digital

environments (e.g. computer software, app environments
development)

Uploading self/user-created content to a [Removed from digital skills indicators]

website to be shared (text, images, photos,
videos, music, software, etc.)

Safety

Setting up effective security measures (e.g. Taking security measures to protect devices
strong passwords, log-in attempt notification) and online accounts (e.g. changing
to protect devices and online accounts passwords, avoiding unsecure links or
downloads, setting up two-factor
authentication)

Changing privacy settings on your device, Taking measures to protect privacy on your
account or app to limit the sharing of personal device, account or app (e.g. to limit the
data and information (e.g. name, contact sharing of personal data and information,
information, photos) restrict access to social network profiles or
geolocation, prevent targeted marketing)

Problem solving

Connecting and installing new devices (e.g. a Connecting new devices (e.g. camera, printer,
modem, camera, printer) through wired or wireless speakers or wireless headphones)
wireless technologies

Finding, downloading, installing and Installing software or apps

configuring software and apps

Internet banking (includes electronic Using Internet or mobile banking (includes

transactions with a bank for payment, electronic transactions with a bank for
transfers, etc. such as M-Pesa, or for looking payment, transfers, etc. such as M-Pesa, or for
up account information; excludes electronic looking up account information)
transactions via the Internet for other types of
financial services such as share purchases,
financial services and insurance)

Doing an online course (in any subject) Doing an online course or accessing online
learning material (e.g. video tutorials,
webinars, learning apps)

Purchasing or ordering goods or services Purchasing or ordering goods or services (via

(purchase orders placed via the Internet the Internet whether or not payment was
whether or not payment was made online; made online; includes purchasing of products
excludes orders that were cancelled or not such as music, travel and accommodation via
completed; includes purchasing of products the Internet)
 such as music, travel and accommodation via
the Internet)

Transferring files or applications between [Removed]

devices (including via cloud-storage)

Annex 2: Comparison with Digital Skills Indicator (DSI) 2.0

Background
In February 2023, select members of the ITU’s ICT Data and Analytics team were granted
access to Eurostat microdata for several research purposes including for pilot analysis of
EGH recommendations for aggregating data on ICT skills at the individual level. The
current microdata include data for years 2009-2022 for all Eurostat member states as well
as code descriptions. The most recent year where data on digital skills were collected was
20214.

Mapping to ITU indicators

Once downloaded, Eurostat data for 2021 (the latest year with ICT skills data) were
mapped to ITU indicators and aggregated using available individual and household
weights. This can be summarized in the following steps:

• Logical mapping to ITU codes – In some cases, transformation of Eurostat codes

was necessary to create yes/no options required for aggregation under the ITU
definitions5. While most were clear correspondence, partial correspondence was
also used in several cases, most notably for editing where the Eurostat definition is
focused on editing photos/videos/audio and the ITU definition on editing
documents/presentations.
• Summarizing microdata – Data were aggregated by multiplying the mapped
individual responses against individual weights and summing these weighted
responses across selected variables, grouping by country.

Structural comparison
The table below shows the difference in the structure of the two recommendations.
Notably, Information and data literacy is nearly perfectly aligned between the two. Other
skill areas show differences:

4
This analysis is based on data from Eurostat, Information and Communications Technologies (ICT)
usage by Households and Individuals, 2021, released 8 Feb 2024. The responsibility for all
conclusions drawn from the data lies entirely with the author(s).
5
For example, using the response code for “in the last three months” for the Eurostat question
When did you last buy or order goods or services for private use over the internet?
 • Communication and collaboration – EGH recommendation excludes three
indicators that are included in DSI 2.0: Using instant messaging (IUCHAT1),
Expressing civic/political opinions (IUPOL2), Sending/receiving emails (IUEM) while
including a different indicator: Sending content in messages. Likely results in
higher assessed skills using DSI 2.0 – more indicators plus sending content in
messages is a subset of sending messages.
• Digital content creation - EGH recommendation excludes two indicators that are
included in DSI 2.0: Using word processing software (CWRD1), Using advanced
features of spreadsheet software (CXLSADV1). Likely results in higher assessed
skills using DSI 2.0 – more indicators (though advanced spreadsheets is a subset of
using spreadsheet software).
• Problem solving - EGH recommendation excludes three indicators that are
included in DSI 2.0: Changing settings of software, app, or device (CCONF1),
Selling goods/services (IUSELL), Looking for a job (IUJOB). It also excludes
purchasing goods/service between 3 and 12 months ago. The EGH
recommendation include a different indicator: Connecting and installing new
devices. Effect is unclear as selling goods/services and looking for a job are
unlikely to increase the numbers of positive responses.
• Safety - EGH recommendation excludes one indicator included in DSI 2.0:
Checked the website where you provided personal data was secure
(MAPS_CWSC). This is subset of the missing EGH recommended indicator for
Taking security measures to protect devices and online accounts. It is not sufficient
to covering this indicator. All other DSI safety indicators are related to EGH
recommended indicator Taking measures to protect privacy on your device,
account or app. Since only one of two ITU indicators is collected in the skill area,
the aggregate cannot be calculated under the EGH recommendation.
Table 4.
DSI 2.0 structure compared to 2023 EGH recommendations

Skill Common
area indicators DSI 2.0 only EGH only
Info. and 4/4 TICXND: Did not check the
data truthfulness of the information or
literacy content you found on the internet
news sites or social media because
already knew that information,
content or source was not reliable

Comm. 3/4 IUCHAT1: Using instant messaging, Sending content in

and i.e. exchanging messages, for messages
collab. example, via Skype, Messenger,
WhatsApp, Viber, Snapchat
IUPOL2: Expressing opinions on civic
or political issues on websites or in
social media
IUEM: Sending / receiving e-mails
 Digital 5/5 CWRD1: Using word processing
content software
creation CXLSADV1: Using advanced features
of spreadsheet software to organise,
analyse, structure or modify data
Safety 1/2 MAPS_CWSC: Checked that the Taking security measures to
website where you provided personal protect devices and online
(5 DSI data was secure accounts
indicators
related to USLCOOK: Use software
privacy) that limits the ability to track
your activities on the
internet on any of your
devices
Problem 4/5 CCONF1: Changing settings of Connecting and installing
solving software, app or device new devices
IUSELL: Selling goods or services via
a website or app
IUJOB: Looking for a job or sending
a job application
IBUY=2: Bought or ordered goods or
services for private use over the
internet between 3 months and a year
ago

Data comparison
In addition to mapping 2021 Eurostat indicators to ITU codes, the DSI 2.0 indicators were
also imported from the 2021 Eurostat microdata for comparison. This comparison shows
that skill levels using the DSI 2.0 methodology and the EGH recommendations are quite
similar for individuals with at least basic skills in Eurostat countries. As expected, based on
the structural differences, the Information and data literacy skill area yields the most similar
results with no differences more than 2 percentage points (all DSI greater than EGH). Only
the Problem solving skill area showed some additional discrepancy with the share of
individuals with at least basic skills 10 percentage points higher using DSI 2.0 in one
country (Romania). This was an exception with most shares in countries remaining 0-2
percentage points higher under DSI 2.0.
Figure 1
Distribution of percentage point differences between DSI 2.0 and EGH recommendations, at least
basic skills, 2021

Note: Percentage point differences expressed as (DSI – EGH). The horizontal axis shows the
percentage point differences, the black tick marks indicate the differences, the height of the waves
indicate the density of the distribution. The red vertical lines mark the group average values.

However, these differences were greater when comparing shares of individuals with
above basic skill levels – partly due to different criteria for attaining above basic skills.
Again, the Information and data literacy skill area yields the most similar results with no
differences more than 2 percentage points (all DSI greater than EGH). Other skill areas
showed larger discrepancies. To be assessed as having above basic skills in the Problem
solving and Digital content creation skill areas DSI requires that individuals have three
activities as opposed to two in the EGH recommendations. As a result of this and
structural differences the EGH shares of individuals with basic skills averaged over seven
percentage points higher than the same for DSI in Problem solving skill area. Smaller but
still meaningful differences were observed for the Digital content creation and
Communication and collaboration with DSI share of individuals greater for
Communication and collaboration and lower for Digital content creation (where DSI
requires three activities for above basic skills).

Figure 2
Distribution of percentage point differences between DSI 2.0 and EGH recommendations, above
basic skills, 2021
Note: Percentage point differences expressed as (DSI – EGH). The horizontal axis shows the
percentage point differences, the black tick marks indicate the differences, the height of the waves
indicate the density of the distribution. The red vertical lines mark the group average values.

Annex 3: Preliminary results from ITU household short

questionnaire

Background
As part of its normal data collection cycle, ITU sends questionnaires twice a year to
countries to collect data. A short questionnaire requesting data on key ICT indicators only
(including ICT skills) is sent in March and a detailed questionnaire requesting data on all
core indicators as well as socio-economic disaggregations is sent in September (see
Figure 3).
Figure 3.
ITU data collection/dissemination cycle

In March 2024, data on aggregates for each skill area were requested for the first time
through the ITU short questionnaire. The ITU received and validated data from this
questionnaire through June. Of 83 countries submitting questionnaires, 72 provided
updated data and 61 provided data on at least one ICT skills indicator. ITU provided
documentation of the new EGH recommendations in the questionnaire for calculating ICT
skill levels for individuals and provided guidance during the validation process.
Ultimately, 40 provided data on skill levels for at least one skill area. 19 were Eurostat
countries providing data calculated according to the DSI 2.0 methodology (see Annex 2
for information on comparability between these methods). Results based on the other 21
countries which followed the 2023 EGH recommendations helped to inform the subgroup
on how countries can provide these data and are shown below.
Communication and collaboration
Results show that communication and collaboration skills are the most highly developed
among individuals using the Internet (Figure 4). In nearly every country, the share of those
using the Internet and having no communication and collaboration skills is very low.
However, there is much variation between countries. For example, Uruguay reports a high
skill level with nearly all Internet users at the above basic skill level. This is somewhat
higher than countries with similar levels of Internet use. In Viet Nam on the other hand,
Internet use is only slightly lower than the other countries reporting data, but skill level
data show knowledge gaps should be addressed.
Figure 4.
Share of individuals with ICT Communication and collaboration skills, most recent year 2022 or
later

Digital content creation

Digital content creation skill levels are substantially lower and skill levels vary widely
among countries with similar levels of Internet use (Figure 5). Notably, there are high skill
levels reported among Internet users in the Republic of Korea, Oman, and Brunei
Darussalam. In Latvia, low levels of digital content creation skills are reported. This may be
an issue with data comparability.
Figure 5.
Share of individuals with ICT Digital content creation skills, most recent year 2022 or later

Information and data literacy

Information and data literacy skill levels are again lower than communication skill levels
and again skill levels vary widely among countries with similar levels of Internet use. There
are notably high skill levels among Internet users in the Republic of Korea, Canada, and
Uruguay. In many other countries the share of individuals with above basic skills is much
lower. This demonstrates a need to focus digital skills education on information and data
literacy skills in these countries.
Figure 6.
Share of individuals with ICT Information and data literacy skills, most recent year 2022 or later

Problem solving
Data reported on problem solving skill levels show especially important divides between
even countries with similar levels of Internet use (Figure 7). For example, high skill levels
are reported among Internet users in the Republic of Korea, Malaysia, and Brunei
Darussalam. Conversely, there may be a need to focus digital skills education on problem
solving skills in countries like the Dominican Republic, Palestine and Azerbaijan with lower
levels of above basic skills.
Figure 7.
Share of individuals with ICT Problem solving skills, most recent year 2022 or later

Safety
Reported ICT safety skill levels are the lowest of the skill areas on average (Figure 8).
Again, skill levels vary widely among countries with similar levels of Internet use. While
high skill levels among Internet users are reported in Canada and Brunei Darussalam
there is a clear need to focus digital skills education on safety skills in nearly all countries
reporting data.
Figure 8.
Share of individuals with ICT Safety skills, most recent year 2022 or later

Annex 4: Brazil analysis of reduced set of indicators

Introduction
This annex presents an exploration of skills data in Brazil by Cetic.br/NIC.br – collected
through a survey conducted in 2023 – to understand which of the full spectrum of 22 skill
sub-indicators6 are the most necessary to adequately calculate the aggregate skills
indicator by individuals. We do not discuss the conceptual issues related to the list of
indicators but present a methodology that could be applied to different countries and to a
different list of indicators if a new list is to be selected. In short, the process stablishes the
road to collect a small number of essential indicators given a complete agreed-upon list.
The exercise is done considering the existing list as the skills indicators to be collected by
countries.

The subsequent sections outline the methodology employed, present the findings, and
concluding remarks.

Methodology
In 2023 the Brazilian ICT Households Survey collected all 22 indicators listed to calculate
the individuals’ skills indicators. The list of indicators is shown in Figure 9 below.

6
Based on 2023 EGH recommendations
Figure 9.
Skills indicators list

Areas and Indicators Variable Code

Information / data literacy

1. Verifying the reliability of information IDL1

2. Getting information about goods or services IDL2

3. Reading or downloading newspapers, etc IDL3

4. Seeking health-related information IDL4

Communication / collaboration

1. Sending messages (e.g. email, messaging service, SMS) with attached files ICC1

2. Making calls (Telephoning over the Internet ICC2

3. Participating in social networks ICC3

4. Taking part in consultation or voting via Internet ICC4

Digital content creation

1. Using copy and paste tools IDC1

2. Creating electronic presentations IDC2

3. Using basic arithmetic formula in a spreadsheet IDC3

4. Writing a computer program IDC4

5. Editing online text, spreadsheets, presentations IDC5

6. Uploading self/user-created content IDC6

Safety

1. Changing privacy settings ISF1

2. Setting up effective security measures ISF2

Problem solving

1. Finding, downloading, installing and configuring software IPS1

2. Connecting and installing new devices IPS2

3. Transferring files or applications between devices IPS3

4. Electronic financial transactions IPS4

5. Doing an online course IPS5

6. Purchasing or ordering goods or services IPS6

Table 5 and Source: Brazilian ICT Households 2023 survey.

Table 6 shows the results for the aggregate and disaggregate measures:

Table 5.
Proportion of individuals within each skill category (aggregated categories)

Skill Level Percentage Confidence interval

No overall level 11,72 10,55 12,89
Basic level 56,01 53,93 58,08
Above basic level 16,42 14,29 18,55
No Internet use (last 3 months) 15,85 14,65 17,05
Source: Brazilian ICT Households 2023 survey.

Table 6.
Proportion of individuals within each skill category (disaggregated categories)

Skill Level Percentage Confidence interval

No overall level 11,72 10,55 12,89
Limited overall level 8,25 7,49 9,00
Narrow overall level 12,31 10,66 13,95
Low overall level 14,6 13,32 15,88
Basic level 20,85 19,21 22,5
Above basic level 16,42 14,29 18,55
No Internet use (last 3 months) 15,85 14,65 17,05
Source: Brazilian ICT Households 2023 survey.

Based on this classification, we want to identify what is the minimal set of skills sub-
indicators needed to reach results as close as possible to those presented in table 1.

To do that, we integrate two different methods:7

- Multiple correspondence analysis (MCA).

- Random Forest (RF).

a. Step 1 - Multiple correspondence analysis (MCA)

The MCA8 method was used to identify for each skills area the variables (indicators) that
contributed more to the principal axes of the MCA. The directions of the choice of variables
were:

- At least two variables by skill dimension.

- After running the MCA, retain as much axes as to represent more than 85% of the
total variability of the respondents.
- For the axes retained, select the variables (indicators) that contributed most for the
construction of the axes.

7
We also tried the use of ordinal linear regression (R Package survey, function svyolr) but the results
were better for the RF. We choose not to present them here to be more concise.
8
This approach utilizes sampling weights as the only required information when performing MCA
on databases with complex sampling designs for R package FactoMineR (Lê, Josse, & Husson, 2008).
 b. Step 2 – Random Forest (RF)

The RF method was used to estimate the observed overall skill (dependent variable) – based
in the indicators selected on Step 1 – reduced set of select indicators (MCA choices). The
process was carried out in three steps:

1) Run a RF for disaggregated skills categories considering sampling weights for the whole
dataset and evaluate the results.
2) Select 200 samples of the Brazilian ICT Households survey the same way the sample
was selected from the population frame [3].
3) For the samples in (2) run the RF (with weights) and evaluate the results.
4) (3) gives results in terms of identifying the empirical confidence intervals for the
estimates, cross validating the observed results (avoiding overestimation).

Results
Initially we present the results of MCA. As a reminder, we retained for each skills area the
indicators that contributed more to MCA axes that added up to at least 85% of the whole
variability of the data – bounded by a minimum of two indicators by dimension.

a. Multiple correspondence analysis (MCA)

The tables below show the contributions of each indicator to each axe for the MAC analysis
for all the skill dimensions.
Table 7.
MCA results: Information and Data Literacy

Information and Data Literacy Factor 1 Factor 2 Factor 3 Factor 4

IDL1 23,8 44,1 30,2 1,9
IDL2 26,2 20,1 0,9 52,8
IDL3 24,4 10,5 59 6,1
IDL4 25,6 25,2 9,9 39,3
Cumulative Variance 90,5 94,3 97,7 100
Source: Brazilian ICT Households 2023 survey.

Table 8.
MCA results: Communication and Collaboration

Communication and Collaboration Factor 1 Factor 2 Factor 3 Factor 4

ICC1 24,7 0,4 74 0,9
ICC2 32,1 8,3 14,5 45,1
ICC3 34,3 5,9 5,7 54
ICC4 8,8 85,4 5,8 0,0
Cumulative Variance 90,5 94,3 97,7 100
Source: Brazilian ICT Households 2023 survey.
Table 9.
MCA results: Digital content creation

Digital content Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6

creation
IDC1 23,2 0,2 0,0 13,1 5,5 58
IDC2 18,6 18,9 4,0 0,0 34,5 23,9
IDC3 18,7 13,6 9,7 0,0 57,5 0,6
IDC4 6,3 57,2 31,1 1,8 0,1 3,6
IDC5 19,1 4,2 5,4 69,5 0,1 1,6
IDC6 14,1 5,9 49,7 15,5 2,4 12,3
Cumulative 66,8 78,1 86,8 91,8 96,1 100
Variance
Source: Brazilian ICT Households 2023 survey.

Table 10.
MCA results: Safety

Safety Factor 1 Factor 2

ISF1 50,0 50,0
ISF2 50,0 50,0
Cumulative Variance 93,4 100
Source: Brazilian ICT Households 2023 survey.

Table 11.
MCA results: Problem-solving

Problem Solving Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6

IPL1 19,2 2,6 1,1 18,1 58,7 0,2
IPL2 11,9 58 6,2 22,1 1,7 0,0
IPL3 19,1 0,3 0,6 38,8 36,9 4,2
IPL4 18 23 3,9 15,7 0,8 38,6
IPL5 11,7 0,4 85,9 2,0 0,0 0,0
IPL6 20,1 15,6 2,3 3,3 1,9 56,9
Cumulative Variance 82 88 93,1 96,1 98,4 100
Source: Brazilian ICT Households 2023 survey.

Based on the results presented on the tables above, 11 of 22 indicators were considered
fundamental for the skills variability of the Brazilian respondents’ dataset. The indicators
retained were: 'IDL1', 'IDL2', 'ICC3', 'ICC4', 'IDC1', 'IDC4', 'IDC6', 'ISF1', 'ISF2', 'IPL2', and
'IPL6'.

b. Random Forest (RF)

Based on the retained indicators, a RF analysis was done for disaggregated categories
considering full sample and replications for 200 samples of the ICT Households survey. The
results are presented on tables 12 and 13.
Table 12.
Comparing RF prediction with skills indicator for disaggregated categories

Skill Level Prediction Rate Confidence interval

No overall level 95,65% 91,61% 97,69%
Narrow overall level 79,24% 65,88% 84,13%
Low overall level 33,10% 27,95% 43,49%
Limited overall level 64,91% 43,46% 65,04%
Basic level 55,19% 31,47% 56,87%
Above basic level 96,34% 92,21% 98,80%
Source: Brazilian ICT Households 2023 survey.

Table 13.
Comparing RF prediction with skills indicator for aggregated categories

Skill Level Prediction Rate Confidence interval

No overall level 95,65% 91,61% 97,69%
Basic level 86,35% 82,33% 90,06%
Above basic level 96,34% 92,21% 98,80%
Source: Brazilian ICT Households 2023 survey.

While, on one hand, the use of disaggregated categories captures adequately the variation
inside “Basic level” users, considering the original 22 skills sub-indicators, this
disaggregation is not well captured by the RF modelling of the 11 most “important”
indicators. This was expected, since the lower number of indicators considered (almost half
of the total number) is an obstacle for correctly classify the inner categories of “Basic level”.

On the other hand, the methodology was able to achieve a good estimation for the
aggregated classification of skill for the individuals. Table 14 shows the comparison of
results.
Table 14.
Comparison between observed skills classification and estimated skills classification

Estimated skill level

Observed Skill Level
No overall level Basic level Above basic level
No overall level 95,6 4,4 0,0
Basic level 5,3 86,4 8,3
Above basic level 0,0 3,7 96,3
Source: Brazilian ICT Households 2023 survey.

Final remarks
The study showed that, with 11 of 22 indicators, it is possible to reliably estimate the
aggregated skills indicator for Brazil. The estimation allows comparison between countries
for the aggregated skills level, even for the cases that the sub-indicators selected as the
most important in the analysis differ. By following these methods, other countries can
replicate the study for their data sets to identify the most critical sub-indicators.
While there are relevant reasons to reduce questionnaire length by analysing the results
of studies like this, since the methodologies used are data driven, it is advisable to collect
the full set of sub-indicators every three years, at least, for the methodology to be verified
and updated.

Annex 5: Canada analysis of reduced set of indicators

Applying Multiple correspondence analysis (MCA) to Canada

To help assess the wider applicability of the minimal set identified for Brazil, Canada
replicated the MCA procedure using the 2022 Canadian Internet Use Survey (CIUS). The
goals of this analysis were to examine:

• How the minimal set identified for Canada compared with the one identified for
Brazil; and
• How the minimal set performed in predicting skill levels in Canada.

Before proceeding to the results of this analysis, it is important to provide a few caveats:

1. In the problem solving (PS) skill area, Canada does not collect “connecting and
installing new devices,” which was one of the indicators the MCA procedure
selected for Brazil. Therefore, it was impossible for MCA to select an identical
minimal set for PS for Canada.

2. Canada did not use RF modelling to calculate skill levels using the minimal set.
Instead, we used the normal skill area calculation method, but excluded any
indicators that weren’t selected by MCA. This decision was made since the primary
purpose of the minimal set discussion is to determine which indicators are most
critical for participating in the existing skill calculations.

3. For the purposes of comparing the performance of the minimal set to the
expected skill levels based on the full set, Canada first compared predictions to
expectations at the skill area level, resulting in 5 observations per respondent (one
per skill area). All observations were then aggregated (using survey weights) to
determine the overall rate of successful predictions. We did not use the new
formula Brazil created for classifying overall skill levels.

Results
Multiple correspondence analysis (MCA) results

The indicators selected by MCA for Canada were the same as Brazil for information & data
literacy (IDL), communication and collaboration (CC), and safety (S), but differed for digital
content creation (DCC) and PS. For consistency with Brazil, Canada also selected three
indicators for DCC (Table 15).
Table 15.
Indicator Code Same as
Brazil?
Information & data literacy
Verifying the reliability of information IDL1 Yes
Getting information about goods or services IDL2 Yes
 Communication and collaboration
Participating in social networks CC3 Yes
Taking part in consultation or voting via CC4 Yes
Internet
Digital content creation
Writing a computer program DCC4 Yes
Using online software for editing text, DCC5 No
spreadsheets, or presentations9
Uploading self/user-created content DCC6 Yes
Safety
Changing privacy settings S1 Yes
Setting up effective security measures S2 Yes
Problem solving
Electronic financial transactions PS4 No
Doing an online course PS5 No

Performance of the MCA-selected minimal set

Table 16.
Expected
Basic Above basic
None 35.3% 2.6%
Predicted Basic 64.7% 57.5%
Above basic 0% 39.9%

In Table 16, the denominator in each column is the weighted number of cases that scored
as “basic” or “above basic” respectively when using the full set of indicators. Overall, the
MCA-selected indicators performed poorly; predictions were least accurate for “above
basic,” with only 39.9% of cases being correctly predicted.

Exploring other potential minimal sets

With the MCA-selected indicators performing poorly, we wanted to explore if other

minimal sets could perform better for Canada. By using a program to check all possible
combinations (two indicators per skill area), we identified the following minimal set as
performing best for Canada (Table 17).
Table 17.
Indicator Code
Information & data literacy
Getting information about goods or services IDL2
Reading or downloading newspapers, etc. IDL3
Communication and collaboration
Sending messages (e.g. email, messaging CC1
service, SMS) with attached files10
Participating in social network CC3
Digital content creation

9
The CIUS did not specify that this software had to be used over the Internet.
10
The CIUS did not specify that the messages had to have attachments.
 Using copy and paste tools DCC1
Using online software for editing text, DCC5
spreadsheets, or presentations11
Safety
Changing privacy settings S1
Setting up effective security measures S2
Problem solving
Electronic financial transactions PS4
Purchasing or ordering goods or services PS6

Performance of the best-performing minimal set

Table 18.
Expected
Basic Above basic
None 7.0% 0.1%
Predicted Basic 93.0% 8.9%
Above basic 0% 91.0%

The best-performing set correctly predicted over 90% of cases for both skill levels, far
higher than the MCA-selected set (Table 18).

Final remarks
When using the normal method to compute skill levels, indicators selected by MCA for
Canada did not provide accurate predictions. This indicates that in the absence of
introducing a new calculation procedure for determining skill levels (e.g., RF modelling),
MCA is not a viable method for determining a good minimal set for Canada. However, by
verifying the performance of all possible minimal sets, we were able to identify another set
that performed much better, demonstrating that it is possible for a minimal set to perform
well using the normal calculation method.

References
[1] Vuorikari, R., Jerzak, N., Karpinski, Z., Pokropek, A. (2022). Measuring Digital Skills
across the EU: Digital Skills Indicator 2.0. Publications Office of the European Union,
Luxembourg. https://publications.jrc.ec.europa.eu/repository/handle/JRC130341

[2] Scott, K., Ummer, O., & LeFevre, A. E. (2021). The devil is in the detail: reflections on
the value and application of cognitive interviewing to strengthen quantitative surveys in
global health. Health policy and planning, 36(6), 982-995.

[3] Wieczorek, J., Guerin, C., & McMahon, T. (2022). K-fold cross-validation for complex
sample surveys. Stat, 11(1), e454. https://doi.org/10.1002/sta4.454

11
The CIUS did not specify that this software had to be used over the Internet.