ADVANCING THE THEORY OF EFFECTIVE USE

THROUGH OPERATIONALIZATION

Research paper

Eden, Rebekah, School of Information Systems, Queensland University of Technology,

Queensland, Australia, [email protected]
Fielt, Erwin, School of Information Systems, Queensland University of Technology, Queens-
land, Australia, [email protected]
Murphy, Glen, QUT Business School, Queensland University of Technology, Queensland,
Australia, [email protected]

Abstract
Despite rigorous conceptualization, quantitative investigations of effective use are largely lacking.
This, in part, may be due to the complexity in operationalizing and measuring the construct. Moreo-
ver, failure to effectively measure the construct could result in erroneous findings. Recognizing this
challenge, we sought to understand how effective use can be operationalized and measured. In doing
so, we proposed two approaches: a formative multidimensional approach and a first-order approach
with relationships between the dimensions of effective use. We drew upon seminal effective use re-
search, and refined a survey instrument for measuring effective use. Our analysis provides support for
both operationalizations with nomological validity established. We believe this research will help ad-
vance the theory of effective use and will enable researchers and practitioners to assess how effective-
ly systems are being used.
Keywords: Effective Use, Survey, Operationalization, Individual Impact

1 Introduction
With effective use serving as the lynchpin through which benefits are obtained from systems, the ‘the-
ory of effective use’ (e.g., Burton-Jones and Grange, 2013) has the potential to be a pivotal, native
theory with implications for research and practice. Fulfilling repeated calls for research to shift from
use to effective use, the theory explicated the complexity of effective use decomposing it into measur-
able dimensions. This rigorous conceptualization of effective use serves as the starting point of the
journey to understand “what effective use actually involves, [which] is a complex challenge, [where]
the route is long and difficult” (Burton-Jones and Volkoff, 2017, p. 117). Nevertheless, despite rigor-
ous conceptualization, empirical investigations particularly of a quantitative nature are lacking and
when quantitative assessments are performed the measures are often idiosyncratic with content validi-
ty issues. To progress the theory, guidance into how to operationalize and measure effective use is
paramount. Without guidance there is the potential for cumulative research efforts to stall (as per Chin
et al. (1997), Salisbury et al. (2002)), and for inappropriate recommendations to be made to practice if
the operationalisation and measurement is not in keeping with the intent of effective use.
The ‘theory of effective use’ defines effective use as “using a system in a way that helps attain the
goals of using the system” and conceptualizes it to consist of three dimensions: transparent interaction,
representational fidelity, and informed action (Burton-Jones and Grange, 2013, p. 633), which were
grounded on representation theory (Wand and Weber, 1995). When conceptualising effective use,
Burton-Jones and Grange (2013) provided hints into the construct’s operationalization and measure-

1
ment, however they do not quantitatively assess the theory rather quantitative assessment is left to fu-
ture research.
In terms of operationalization, Burton-Jones and Grange (2013, p. 643) state effective use “is an ag-
gregate construct formed by its dimensions”. This indicates that effective use can be operationalized as
a multidimensional formative construct (Law et al., 1998). However, it is also specified that the di-
mensions are “hierarchically related”, indicating an omission of a second-order construct with rela-
tionships between first-order dimensions1 (which we refer to as first-order for brevity) (Burton-Jones
and Grange, 2013, p. 644). Subsequent research building on the theory of effective use, has not deter-
mined the appropriateness of when to use each approach. Early research has favoured the first-order
approach with little justification, whereas the formative approach has seemingly been left unnoticed.
In terms of measurement, some example measurement items are provided in the seminal work
(Burton-Jones and Grange, 2013). However, these are seldom applied. While some have quantitatively
assessed effective use (e.g., Haake et al., 2018, Marchand and Raymond, 2017), their measurement
items do not completely reflect the meaning of the underlying dimensions. The potential content valid-
ity issues surrounding the measurement of effective use is expected, as these papers represent much
needed first attempts at exploring how to measure such a complex and under-researched topic. Yet, it
is imperative that these measurement concerns are resolved in future research. As Burton-Jones and
Grange (2013, p. 653) note “researchers need instruments to test the relationships in [the] model, and
practitioners need instruments to assess how effectively systems are being used”.
Due to the importance of effective use, the multiple ways it can be operationalized, coupled with is-
sues surrounding measurement, this research seeks to provide insights into the following: “How can
effective use be operationalized and measured?” and provide areas for future research. As such, we
conduct a survey examining the effective use of enterprise systems. To support the operationalization
and measurement we examine the nomological validity (see: Straub et al., 1995, Cenfetelli and
Bassellier, 2009) of effective use with individual impact, which is a relationship long proffered but
never empirically investigated. We believe this research will help advance the theory of effective use
by providing empirical support behind its measurement and modes of operationalization. From a prac-
tical perspective, our measurement instrument can be used to better measure effective use, which will
help practitioners to identify where investments need to be made to improve how the system is used
which will ultimately improve the benefits obtained from their information systems investments.
This paper is structured as follows: Next, we provide a brief overview of effective use literature. Sub-
sequently, we describe two operationalizations of effective use and hypothesize their relationships
with individual impact. Then, the method and measurement items are detailed. Following, we present
the results and discuss how our findings enhance effective use literature.

2 Background Literature on Effective Use

This section provides an overview of the current state of the theory of effective use (e.g., Burton-Jones
and Grange, 2013) and subsequent related research drawing upon Dubin’s (1978) components of theo-
ry building. To engender cumulative research, Dubin (1978, pp. 7-8) specifies seven components that
a theory requires: i) Units are the constructs that form the model; ii) Relationships are the laws of in-
teraction among the units; iii) Boundary conditions to which the model holds true; iv) System states
are areas in the boundary conditions that differ to the rest of the model; v) Propositions are the truth
statements of the model; vi) Empirical indicators are the measures for determining the value of a unit;
and vii) Testable hypotheses in which the proposition is broken into a series of testable statements.

1 Theterm “hierarchical” can refer to a multidimensional construct. However, in the theory of effective use, the relationship
was depicted as separate relationships between transparent interaction, representational fidelity, and informed action.

2
The focal unit of the theory is effective use, which is defined as “using a system in a way that helps
attain the goals of using the system” (Burton-Jones and Grange, 2013, p. 633). In conceptualising ef-
fective use, Burton-Jones and Grange (2013) drew upon representation theory (see: Wand and Weber,
1995, Wand and Weber, 1990) which asserts that any information system consists of three structures:
physical, surface, and deep structure. The physical structure, refers to the hardware and infrastructure;
the surface structure refers to the user interface and the format of outputs, and deep structure repre-
sents the phenomenon that the system is designed to model (Eden and Burton-Jones, 2018). Central to
representation theory is that users need to interact with the physical and surface structures to access the
representations in the deep structure, however these representations are not infallible (Eden and
Burton-Jones, 2018). Rather the users need to be able to determine the faithfulness of representations
they leverage before they act on the information in the deep structure. This view on information sys-
tems is central to effective use, and resulted in Burton-Jones and Grange (2013, p. 642) conceptualis-
ing effective use to consist of three dimensions (i.e., units):

• Transparent Interaction: “The extent to which a user is accessing the system’s representations un-
impeded by its surface and physical structures.”
• Representational Fidelity: “The extent to which a user is obtaining representations from the system
that faithfully reflect the domain being represented.”
• Informed Action: “The extent to which a user acts upon the faithful representations he or she ob-
tains from the system to improve his or her state.”
Due to its underpinning of representation theory, the theory of effective use is largely technology ag-
nostic in nature, where the dimensions are purported to apply to any information system. However,
some have argued the need to conceptualise the dimensions based on the context under investigation.
For instance, in the healthcare context Burton-Jones and Volkoff (2017) conceptualised effective use
to consist of the dimensions accuracy, consistency, and reflection in action. In the context of emergen-
cy management Bonaretti and Piccoli (2019) conceptualised effective use to consist of the dimensions
promptness, currency, and responsiveness. In this study, we focus on the dimensions in the original
theory of effective use, with a goal to develop a foundation for cumulative research efforts.
As indicated in the Introduction, in terms of relationships, effective use is an “aggregate construct
formed by its dimensions” that are hierarchically related (Burton-Jones and Grange, 2013, p. 643). As
such, there are two approaches for the relationships: (i) multidimensional formative construct formed
by its dimensions; and (ii) Relationships between the first-order dimensions. Research has tended to
focus on the latter approach (e.g., Trieu, 2013, Marchand and Raymond, 2017, Haake et al., 2018),
seldom is the formative approach examined (Eden et al., 2019). Moreover, scholars typically do not
recognize these multiple approaches nor do they justify their approach. Given how a construct is oper-
ationalized has both theoretical and practical implications (see: Coltman et al., 2008) it is important to
understand and assess the appropriateness of each approach.
In terms of boundary conditions, which examines the applicability of effective use across contexts,
according to Burton-Jones and Grange (2013, p. 650), “a full analysis of context is a major undertak-
ing outside [their] scope”. Initial research provides support in contexts including healthcare (Eden et
al., 2019), marketing (Campbell and Roberts, 2019), finance (Haake et al., 2018), and emergency
management (Bonaretti and Piccoli, 2019). Within boundary conditions, system states exist which de-
lineate specific areas within the boundary that differ to the rest of the model. In the social sciences it is
common for there to be only one system state present in a theoretical model (Dubin, 1978, Sedera et
al., 2013). Currently, no specific system states have been identified in past literature on effective use.
The overarching propositions in the theory of effective use, is that people can take several steps to im-
prove their effective use of systems, and the effective use of systems can influence their performance.
Testable hypotheses were not provided in the seminal paper on the theory of effective use, however
some have hypothesised relationships between the dimensions (Trieu, 2013), between individual di-
mensions with performance (Campbell and Roberts, 2019), and between the overarching effective use
construct with performance (Eden et al., 2019).

3
To test the hypotheses empirical indicators are necessary, and guidelines have been provided by
Burton-Jones and Grange (2013). However, in many instances these guidelines have not been fol-
lowed and some content validity issues have arisen. For instance, transparent interaction is often as-
sessed in terms of perceived ease of use, which does not link to whether content is impeded due to
hardware and user interfaces as Burton-Jones and Grange (2013) recommend. For representational
fidelity, in some cases information quality measures are used without anchoring them in usage behav-
iour as per Burton-Jones and Grange (2013). For informed action the measures largely inappropriately
represent impacts from use. Inappropriate measurement can ultimately impede the cumulativeness of
the theory (Mohr, 1982) and result in inaccurate recommendations (Salisbury et al., 2002).
To summarise, there are many areas for future research required to progress the theory of effective use
in terms of Dubin’s (1978) guidelines. In this research, we focus on providing clarity into both: (i) the
units and their measurement, as well as (ii) the relationships between the units. We believe this will
provide the foundation necessary for cumulative research to extend the boundary conditions.

3 Operationalizing Effective Use and Theorizing its Relation-

ships
When operationalizing a construct, it is imperative to establish nomological validity, which “reflects
the extent to which predictions about constructs and their measures are accurate from the perspective
of reasonably well established theoretical models” (Straub et al., 1995, p. 1331). A prominent native
information systems nomological net developed by Benbasat and Zmud (2003), highlights the necessi-
ty for systems to be used for impacts to be experienced. This relationship is also central to the Infor-
mation Systems Success model (DeLone and McLean, 1992, Delone and McLean, 2003), which pro-
poses a relationship between use and impacts at multiple levels. Nevertheless, many have argued
against use being a necessary and sufficient condition for benefit attainment (Seddon, 1997) rather the
use must be effective (Trieu, 2013, Marcolin et al., 2000, Burton-Jones and Grange, 2013). Drawing
on these bodies of literature, we assess the nomological validity of both operationalizations of effec-
tive use (i.e., multidimensional formative construct, first-order dimensions with relationships between
dimensions) with individual impact, which is defined as: “the extent to which the IS has influenced the
capabilities and effectiveness, on behalf of the organization, of key users” (Gable et al., 2008, p. 389).
In the following sections, we discuss each operationalization in turn.

3.1 Model I: Multidimensional Formative Operationalization of Effective Use

The first operationalization treats effective use as an “aggregate construct formed by its dimensions”
(Burton-Jones and Grange, 2013, p. 643). As such, effective use is operationalized as a multidimen-
sional second order formative construct, comprised of transparent interaction, representational fidelity,
and informed action, whereby the removal of any dimension changes the underlying meaning of the
construct (Figure 1).
Multidimensional, Second Order Formative
Transparent Representational Informed
Interaction Fidelity Action

H1 Individual
Effective Use
Impact

Figure 1. Model I – Multidimensional Operationalization of Effective use with Individual Im-

pact

4
Little empirical research has assessed the relationship between the multidimensional construct of ef-
fective use and individual impact. Hence, we first look at use literature and then integrate conceptual
and qualitative effective use research to support the relationship. As previously specified the nomolog-
ical nets of Benbasat and Zmud (2003) and the Information Systems Success Model (DeLone and
McLean, 1992) support the relationship between use and individual impact. Empirical evidence has
also supported the relationship, for instance: in a meta-analysis, the positive relationship between use
and individual impact is widely supported (Petter and McLean, 2009). Moreover, Sedera et al. (2013),
Lin et al. (2006), and Fan and Fang (2006) all reported a positive, statistically significant relationship
between use and impact. Many argue that use is not sufficient for benefit attainment and rather the
effectiveness of use should be considered as a necessary and sufficient condition (Burton-Jones and
Grange, 2013). Qualitative research has evidenced a relationship between the effective use of systems
and concepts related to individual impact. For example, Burton-Jones and Volkoff (2017) identified
that effective use can improve decision making. In line with the positive relationships between use and
impacts coupled with literature highlighting the utility of effective use over use as well as findings
from qualitative studies, we hypothesize:
H1: Effective use positively influences individual impact

3.2 Model II: First-Order Operationalization of Effective Use

The second approach focuses on the dimensions of effective use being “hierarchically related”
(Burton-Jones and Grange, 2013, p. 644) and decomposes effective use into its dimensions, which in-
fluence each other (Figure 2). Unlike, the multidimensional formative approach in which a higher or-
der construct is formed, in Model II the relationships between the dimensions need to be hypothesized.

First-Order Dimensions

Transparent H2 Representational H3 Informed H4 Individual

Interaction Fidelity Action Impact

Figure 2. Model II – First-Order Operationalization of Effective Use with Individual Impact

When interacting with systems, users should be able to interact with the surface and physical struc-
tures (e.g., user interfaces and supporting hardware) in an unimpeded manner (i.e., transparent interac-
tion). Surface and physical structures offer users with sensory and physical affordances that enable
users to both sense and do things (Burton-Jones and Grange, 2013). For instance, positioning and la-
belling of fields and checkboxes accompanied with asterisks denoting mandatory fields provide users
with guidance on both the data to input and how it should be inputted. Improving transparent interac-
tion should facilitate users entering in more accurate data in the system and as a result the system
should more accurately represent the domain that the data pertains to, improving the fidelity of repre-
sentations users input and extract from the system. Therefore, in line with Trieu (2013) and Haake et
al. (2018), we hypothesize:
H2: Transparent interaction positively influences representational fidelity
In accordance with Burton-Jones and Grange (2013), being able to obtain representations that faithful-
ly reflect the domain enables individuals to effectively act. Trieu (2013) also asserts that improved
fidelity of representations will help decision makers make better informed decisions. This is substanti-
ated by Marchand and Raymond (2017) who highlights representational fidelity enables users to in-
formate. Therefore, we hypothesize:
H3: Representational fidelity positively influences informed action.
When operationalizing effective use, with the first-order approach, scholars need to consider which
specific dimension of effective use accounts for variation in performance outcomes, in this case indi-

5
vidual impact. In accordance with Burton-Jones and Grange (2013) transparent interaction is an ena-
bler of representational fidelity, which is an enabler of informed action. We are conscious that users
must make these informed decisions for productivity improvements and efficiency gains to be real-
ized. For instance ill-informed actions can result in errors, decreasing efficiency, whereas when those
actions are informed, users will be less likely to make errors. Therefore, we hypothesize:
H4: Informed action positively influences individual impact

4 Survey Method
To test the operationalization of effective use and its dimensions the survey method was used. The
survey method is appropriate as we seek to empirically test a series of hypothesized relationships at
the individual level of analysis, which the survey method facilitates (Gable, 1994). In this section, we
first provide details into the measurement and operationalisation of the constructs in the study, we then
provide details of the organization where the survey was distributed.

4.1 Measurement and Operationalization of Constructs

To develop the instrument, where possible existing scales were adapted from literature and contextual-
ized if necessary (Froehle and Roth, 2004). For effective use, we used the measures proposed by
Burton-Jones and Grange (2013) as the starting point as they clearly represent the meaning of each
dimension. We opted for this approach as despite these items not being previously validated, we felt
this would be a useful starting point to maintain content validity. However, we did not use this as the
only way for establishing content validity rather additional steps were performed to ensure “developed
instruments are measuring what they are supposed to be measuring” (Straub, 1989, p. 150), including
pretesting, pilot testing, and preliminary interviews to understand the context.2
As per the guidelines of Rubio et al. (2003) the survey was pre-tested to assess its face validity. Face
validity is a form of content validity, whereby the instrument is assessed by an expert panel consisting
of between three to ten experts (Rubio et al., 2003) who examine the instrument for potential empiri-
cal, theoretical, and practical issues (Hair et al., 2010). We pre-tested the survey, with eight respond-
ents, including experienced academics and PhD students familiar with the domain. The respondents
provided guidance to improve the understandability of the items and the overall format of the survey.
After the pretest, we conducted a pilot study. The pilot was conducted in a small and medium organi-
zation operating in the mining industry in Australia. In total 17 employees, who were operational users
of an enterprise system participated. Due to the low number of respondents, quantitative assessment
was limited. Notwithstanding, in accordance with Hunt et al. (1982) as the sample was between 12 and
30 the pilot was adequate. We identified participants were unsure on the negatively worded items orig-
inally associated with transparent interaction, as such we reworded these items.
In formulating the measurement items, we were conscious of the nature (i.e., formative versus reflec-
tive) and dimensionality (i.e., unidimensional versus multidimensional) of the constructs as this has
implications for measurement (Coltman et al., 2008). Constructs can be reflectively or formatively
measured (Shin and Kim, 2011). In reflective models “covariation among measures is explained by
variation in an underlying latent factor” (MacKenzie et al., 2005, p. 711). Therefore, the direction of
causality is from the construct to the measurement items, as such the items all measure the same facet,
are interchangeable and need to covary (MacKenzie et al., 2005, Petter et al., 2007). Whereas, in
formative models “the measures jointly influence the composite latent construct, and meaning ema-
nates from the measures to the construct” (MacKenzie et al., 2005, p. 712). Thus, the direction of cau-
sality is from the measurement items to the construct, as such the items all measure different facets
and form the constructs definition (MacKenzie et al., 2005, Petter et al., 2007). As previously high-

2 For brevity, the interviews are not reported in this papers. Details of the interviews are provided in Eden et al. (2017)

6
lighted, in Model I transparent interaction, representational fidelity and informed action combine to
form effective use. Therefore, while the dimensions are reflectively measured, the multidimensional
construct of effective use is formative in nature. As such, effective use is considered a first order re-
flective-second order formative construct. Due to the formative nature of effective use a global meas-
urement item for effective use was included in the survey (Hair et al., 2016). Figure 3 depicts the
measurement model of effective use. In accordance with Gable et al. (2008) individual impact is a
first-order formatively measured construct. As such, it also requires a global indicator.

Figure 3. Effective Use Measurement Model

The refined measurement items are present in Appendix 1. All items were measured on a 7-point Lik-
ert Scale ranging from strongly disagree to strongly agree. In addition, to understand the sample, de-
mographic characteristics were also collected, including gender, age, job role, and experience.
Both IBM SPSS and SmartPLS were used to analyze the data. IBM SPSS was used to clean and trans-
form the data, and SmartPLS was used to analyze the formative measurement model and structural
model. SmartPLS is an appropriate data analysis tool to use for formative measurement models, and
84% of publications examining formative constructs use the tool (Polites et al., 2012)

4.2 Field Setting

We distributed the survey at a single case organization, while this limits generalizability, it provides
the opportunity to have an in-depth understanding of the use context, through preliminary interviews,
and provides the opportunity to ensure the survey is worded in a way respondents can understand. The
organization is a large tertiary education provider in Australia with over 12,000 staff employed across
17 divisions. The organization implemented the enterprise system, Oracle Financials, in the 1990s.
Since its implementation it has undergone several updates and is beyond the shakedown phase, which
makes it an ideal case study to understand the use and impacts of the system (Markus and Tanis,
2000). The enterprise system includes both self-service and standard modules, providing functionality
into asset management, accounts payable, accounts receivable, general ledger, and purchasing. To
identify the individuals in the organization who used the enterprise system, user logs were analysed
with managerial staff in the organization assisting in identifying the relevant participants. Specifically,
we surveyed users who used the system in an operational capacity to complete finance related tasks.
This included some management users as they also completed similar tasks in the system.

5 Results
Of 393 potential respondents, 250 surveys were returned (response rate: 63.81%). Of the 250 surveys
returned, 213 surveys were considered usable (response rate: 54.2%). This is greater than the average
response rate observed for physically distributed surveys (Baruch and Holtom, 2008). The sample de-
mographics are summarised in Table 1. In the following sections, we provide validity and reliability
assessments of the measurement models and analysis of our structural models.

7
 Gender Job Role Experience (average years) Age
Male Female Operational Managerial Organizational Experience us- (average)
Experience ing the system
51 (23.9%) 162 (76.1%) 188 (88.3%) 25 (11.7%) 9 7 41.6
3
Table 1. Respondent Demographics

5.1 Measurement Model

Prior to analyzing the structural model, reliability and validity assessments of the constructs were per-
formed following the guidelines of Hair et al. (2016). Reflectively measured constructs (i.e., transpar-
ent interaction, representational fidelity, and informed action) were assessed for: internal consistency
reliability, convergent validity, and discriminant validity. Formatively measured constructs (i.e., effec-
tive use, individual impact) were assessed for convergent validity, multicollinearity, and relevance of
formative indicators/dimensions.
For effective use, we first analyze the validity and reliability of the reflectively measured dimensions,
followed by the second-order formative construct. All effective use dimensions exhibited discriminant
validity, composite reliability, and convergent validity (table 2, appendix 2) (Hair et al., 2016). In ad-
dition, indicator reliability for transparent interaction and representational fidelity was established.
One informed action item (IA3) did not exhibit indicator reliability, however, it was retained as com-
posite reliability and convergent validity are above the required threshold (Hair et al., 2016).

Construct Items Indicator Reliability* Composite Reliability** Convergent Validity***

Transparent TI1 0.883 0.937 0.833
Interaction TI2 0.955
TI3 0.899
Representational RF1 0.883 0.933 0.776
Fidelity RF2 0.925
RF3 0.822
RF4 0.890
Informed IA1 0.914 0.835 0.642
Action IA2 0.918
IA3 0.496
Note: In accordance with Hair et al. (2016):
*Indicator reliability established when outer loadings are greater than 0.708. However outer loadings between 0.40
and 0.70 can be considered acceptable if composite reliability and AVE above threshold.
**Composite reliability of construct established when composite reliability is between 0.70 and 0.95.
***Convergent validity established when average variance extracted is greater than 0.50
Table 2. Reliability and Validity of Reflectively Measured Effective Use Dimensions
In terms of the second-order formative perspective of effective use, as evident in Model I, it needs to
be assessed for convergent validity, multicollinearity, and relevancy of dimensions. For convergent
validity a redundancy analysis was performed where the path weight between the formatively meas-
ured effective use and the reflective global measure of effective use was analyzed. The path weight (β

3 Our prior interviews (Eden et al., 2017) and analysis of the demographic questions in our survey indicated that the opera-
tional and managerial users used the system in similar ways in terms of tasks performed and time spent using the system.

8
= 0.627) was below the recommended threshold, and as such convergent validity could not be estab-
lished. There were no multicollinearity issues with effective use, with Variance Inflation Factor (VIF)
values below 10 (MacKenzie et al., 2011). Bootstrapping was performed to assess the relevancy of the
dimensions. All dimensions were statistically relevant, representational fidelity was the strongest (β =
0.543), followed by transparent interaction (β = 0.432), and informed action (β = 0.223). In sum, there
may be potential issues with the convergent validity of effective use as a multidimensional construct,
however this could be due to the global indicator as no other issues were present.
Due to the formative nature of individual impact, it needs to be assessed for convergent validity, mul-
ticollinearity, and relevancy of indicators. Using a similar approach to effective use, convergent validi-
ty was established, and no multicollinearity issues were present. Two indicators did not have a signifi-
cant outer weight but as per Hair et al. (2016) they were retained as past research supports the
measures (Gable et al., 2008).
To provide further evidence that the constructs possessed convergent and discriminant validity a con-
firmatory factor analysis was performed (Appendix 3). All items loaded onto their respective con-
struct, with a factor loading of greater than 0.50 and no cross loadings above 0.40 (Hair et al., 2010).

5.2 Model I Analysis – Multidimensional Formative Operationalization of Ef-

fective Use
For the multidimensional operationalization, we hypothesized that effective use influences individual
impact (H1) this is statistically significant (β = 0.682, P<0.001), with effective use accounting for
46.5% of the variance in individual impact, with a large effect size (f2 = 0.869) (Figure 3).

Multidimensional, Second Order Formative

Transparent Representational Informed
Interaction Fidelity Action

Individual
Effective Use
Β = 0.682*** Impact
R2 = 46.5%

Figure 3. Structural Equation Model of Model I

5.3 Analysis of Model II – First Order Operationalization of Effective Use

Figure 4 illustrates the results of the first order approach. H2 was supported (β = 0.670, P<0.001) with
a large effect size (f2 = 0.813) with transparent interaction accounting for 44.9% of the variance in rep-
resentational fidelity. H3 was supported (β = 0.375, P<0.001), with a medium effect size (f2 = 0.164),
with representational fidelity explaining 14.1% of the variance in informed action. H4 was again sup-
ported (β = 0.474, P<0.001), with a medium effect size (f2 = 0.290), with informed action explaining
22.5% of the variance in individual impact.

First Order Dimensions

β= 0.670*** β= 0.375*** β= 0.474***
Representational Informed Individual
Transparent
Fidelity Action Impact
Interaction
R2 = 44.9% R2 = 14.1% R2 = 22.5%

Figure 4. Structural Equation Model of Model II

Following Cepeda et al. (2017), we also performed additional exploratory mediation analyses:

9
• An examination of whether representational fidelity mediates the relationship between transparent
interaction and informed action.
• A multiple mediation analysis examining whether representational fidelity and informed action
mediates the relationship between transparent interaction and individual impact.

For the first analysis, we identified representational fidelity completely mediated the relationship be-
tween transparent interaction and informed action (β = 0.234, P <0.05, [95% CI: 0.074, 0.418]).
For the multiple mediation analysis we examined the indirect effects on individual impact. As detailed
in Table 3, we found an indirect relationships may exist between transparent interaction and individual
impact through representational fidelity through informed action. However, examination of the VAF
indicated that no mediation was present (VAF < 20%). Rather, the path was better explained from
transparent interaction through representational fidelity to individual impact (VAF = 25.7%) (Hair et
al., 2017). During this analysis we also identified that direct relationships between transparent interac-
tion and representational fidelity existed with individual impact.

Direct Effects Path Weight 95% Confidence Interval VAF

Lower CI (2.5%) Upper CI (97.5%)
TI -> RF β = 0.667, p<0.01 0.584 0.738 NA
TI -> IA β = 0.170, p>0.05 -0.008 0.333 NA
TI -> II β = 0.390, p<0.01 0.235 0.541 NA
RF -> IA β = 0.265, p<0.01 0.114 0.419 NA
RF -> II β = 0.249, p<0.01 0.091 0.409 NA
IA -> II β = 0.205, p<0.01 0.062 0.344 NA
Indirect Effects on II
TI -> RF -> IA -> II β = 0.037, p<0.05 0.008 0.077 5.9%
TI -> IA -> II β = 0.035, p>0.05 -0.002 0.082 NS
TI -> RF ->II β = 0.162, p<0.01 0.060 0.273 25.7%
Total Indirect Effect β = 0.234, p<0.05 0.138 0.342 31.6%
*TI: Transparent Interaction; RF: Representational Fidelity; IA: Informed Action; II: Individual Impact; CI:
Confidence Interval; VAF: Variance Accounted For; NS: Non significant
Table 3. Multiple Mediation Analysis

6 Discussion
Overall, our research provides support for the effective use dimensions, and established nomological
validity for two operationalisations of effective use: multidimensional and first-order relational. In this
section, we first reflect on the dimensions and their measurement followed by the operationalisations.
In terms of the units (i.e., dimensions), our analysis demonstrates that transparent interaction, repre-
sentational fidelity, and informed action are all statistically significant dimensions of effective use. We
also provide initial evidence validating the measures of effective use based on the measures proposed
by Burton-Jones and Grange (2013). There are some limitations with the measures as detailed in ap-
pendix 1, as such we recommend future research performs additional face validity assessments (see:
Hardesty and Bearden, 2004). Additionally, research should triangulate the self-reported measures of
effective use with independent assessments.
Our analysis identified the appropriateness of the dimensions in a single context, which was the opera-
tional use of an enterprise system to complete finance-related tasks. According to Burton-Jones and
Grange (2013), these dimensions are considered to be applicable across all system types, including
network IT, functional IT, and enterprise IT (see: McAfee (2006)). Therefore, rather than the dimen-

10
sions of effective use being informed in an inductive context-driven manner, elements of the system
use context – user, task, system (Burton-Jones and Straub, 2006) – are embedded in the measurement
items. However, these dimensions are informed by representation theory, where the underlying as-
sumption is that the purpose of any information system is to provide an accurate representation of
some phenomenon (Wand and Weber, 1990, Wand and Weber, 1995). Yet, this may not be the case
for all systems. For instance, understanding representations within social media contexts may go be-
yond simple representational fidelity criteria such as completeness and correctness (Emamjome,
2014). This suggests there is value in performing a grounded approach to understanding effective use
in different contexts. However, when grounding the dimensions, it is important that they still faithfully
reflect the overarching definition of effective use for appropriate conclusions to be drawn. Recognis-
ing this Burton-Jones and Volkoff (2017) provide guidelines into how to contextualise effective use.
We recommend, further research be performed into examining the efficacy of both the context agnos-
tic (Burton-Jones and Straub, 2006) and context specific (Burton-Jones and Volkoff, 2017) perspec-
tives of effective use.
In terms of relationships, based on Burton-Jones and Grange (2013), we proposed two operationaliza-
tions of effective use: multidimensional and first-order relational. Our research supports both ap-
proaches, yet is limited in its ability to explain causality. Our findings extend current literature by
demonstrating the appropriateness of the multidimensional approach to effective use (e.g., Eden et al.,
2019), and supports nascent empirical research examining the first-order approach (e.g., Trieu, 2013,
Marchand and Raymond, 2017, Haake et al., 2018). However, we acknowledge that neither approach
on its own is perfect as traditional measurement models fall short in their ability to demonstrate the
combination of a multidimensional construct with relationships between its first order dimensions.
Nevertheless, given both operationalisations are statistically significant the choice of which operation-
alisation to use should be based on the theoretical considerations informed by the study’s objective.
We provide some initial insights below.
When the objective of the research is to understand the consequences of effective use, it is likely to be
more appropriate to use the multidimensional approach as it enables a holistic representation of com-
plex phenomena in a comparatively simple abstraction (Polites et al., 2012). It also offers key ad-
vantages including retaining parsimony, allowing for insightful explanations, and maintaining theoret-
ical implications (Petter et al., 2007). However, the ability of a multidimensional construct to provide
insights is limited to how rigorously the underlying dimensions have been formulated.
Alternatively, when the objective of the research is to understand how to improve effective use, the
first order relational approach is especially useful. For instance, different mechanisms can be enacted
to improve transparent interaction, representational fidelity, and informed action. The exploratory ex-
amination of the first order relational model also highlighted some nuanced relationships. For instance,
we identified all dimensions directly impacted individual impact. When comparing the direct effects
for each dimension, transparent interaction had the strongest influence on individual impact, and an
indirect effect through representational fidelity – yet no indirect effect through informed action. We
believe this is the result of our user cohort being largely operational users. Traditionally, operational
users complete highly fragmented, routinized tasks, with limited ability to make decisions, whereas
managerial and strategic level users typically make informed decisions on the data present in the sys-
tem (Wickramasinghe and Karunasekara, 2012). This could suggest that different dimensions of effec-
tive use may be more relevant for different user groups. Therefore, future research should conduct a
multi-group examination of effective use.
Another potential way to determine which operationalization should be investigated is based on
whether the objective of the research is to develop a process or variance theory (e.g., Mohr, 1982). A
variance approach to theorizing examines causal relationships between constructs and, as such, is built
on the assumption that the relationship between constructs is necessary and sufficient (Mohr, 1982,
Burton-Jones et al., 2015). Alternatively, a process approach to theorising examines the sequence of
events, where an outcome event is only probabilistically reached if the sequence of events occur in
that specific order (Mohr, 1982, Seddon, 1997, Lyytinen et al., 2008). As such, the assumption of a
process approach is that the relationship between constructs is necessary but insufficient (Burton-Jones

11
et al., 2015). In terms of the operationalization of effective use, the multidimensional formative ap-
proach (Model I) best aligns with a variance perspective when examined in conjunction with its con-
sequences, whereas the first-order approach (Model II) is more in keeping with a process approach4.
To summarise, we provide some initial insights to improve the foundations of the theory of effective
use, to minimise model instability and to help foster a cumulative research tradition. We call for fur-
ther research to be performed into critically understanding the operationalisation of effective use as
well as for future research to expand the nomological network surrounding effective use.

7 Conclusion
In conclusion, this research sought to provide insights and initial guidance into the operationalization
and measurement of effective use. We demonstrated two approaches to operationalizing effective use:
(i) second-order formative, first-order reflective construct, and (ii) decomposed into its dimensions
with relationships between. In addition, we refined the measurement items of effective use. This re-
search is timely as few studies have quantitatively assessed effective use, and those studies that have
quantified effective use have questionable measurement items raising measurement concerns. Future
research on effective use can draw upon our refined items and operationalizations.

8 Appendix

8.1 Appendix 1. Measurement Items

Construct Item* Descriptives
Transparent TI1: When using [System], I have seamless access to the content that I need Mean: 4.44
Interaction to complete my job task. St. Dev: 1.42
TI2: When using [System], I have difficulty obtaining the content I need to Range: 1-7
complete my job task because of [System]’s interface.
TI3: When using [System], I have difficulty obtaining the content I need to
complete my job task because of the physical characteristics of the device I
use to access [System].
Representational RF1: When completing my job task using [System], the information pro- Mean: 4.91
Fidelity vided is complete. St. Dev: 1.22
RF2: When completing my job task using [System], the information pro- Range: 1-7
vided is clear.
RF3: When completing my job task using [System], the information pro-
vided is correct.
RF4: When completing my job task using [System], the information pro-
vided is meaningful.
Informed Ac- IA1: When I obtain information from [System], I look for the relevant as- Mean: 4.92
tion5 pects that I can act upon to improve my task performance. St. Dev: 1.19
IA2: When I obtain information from [System], I seek ways to leverage Range: 1-7

4 Even though in this research we examined the first-order relational approach using a variance perspective (although causal

inferences could not be drawn), it may be more applicable to use a process perspective. This is to an extent also implied by
Burton-Jones and Grange (2013) as each dimension is considered necessary but not sufficient for the higher level dimension.
For instance, transparent interaction enables users to determine representational fidelity yet it does not directly cause repre-
sentational fidelity to occur.
5 Future research could seek to further refine the informed action items, so they reflect the extent to which appropriate, rele-
vant, and accurate information is acted upon. For instance: When I obtain information from [System], I look for the relevant
aspects and act on that information to improve my task performance.

12
 Construct Item* Descriptives
good pieces of information for my job.
IA3: When I obtain information from [System], I avoid acting on infor-
mation that I think is suspect.6
Effective Use GEU: Overall I effectively use [System] to complete my job tasks.7 Mean: 5.42
Global Item St. Dev: 1.19
Range:1-7
Individual Im- II1: I have learnt much through the presence of [System]. Mean: 4.55
pact II2: [System] enhances my awareness and recall of job related information. St. Dev: 1.24
II3: [System] enhances my effectiveness in the job. Range: 1-7
II4: [System] increases my productivity.
GII: Overall, the impact of [System] on me has been positive
*Effective Use and Individual Impact measures from Burton-Jones and Grange (2013) and Gable et al. (2008).

8.2 Appendix 2. Fornell Larker discriminant validity assessment

Construct Informed Action Representational Fidelity Transparent Interaction
Informed Action 0.801
Representational Fidelity 0.377 0.881
Transparent Interaction 0.343 0.6700 0.913

8.3 Appendix 3. Factor Analysis

Item Transparent Interaction Representational Fidelity Informed Action Individual Impact
TI1 0.818 0.194 0.156 0.290
TI2 0.787 0.399 0.064 0.312
TI3 0.782 0.379 0.072 0.191
RFI1 0.350 0.806 0.128 0.121
RFI2 0.348 0.798 0.054 0.304
RFI3 0.151 0.759 0.201 0.243
RFI4 0.180 0.816 0.172 0.266
IA1 0.244 0.116 0.857 0.118
IA2 0.127 0.156 0.842 0.186
IA3 -0.055 0.083 0.577 0.093
II1 0.170 0.291 0.052 0.861
II2 0.113 0.275 0.175 0.863
II3 0.335 0.193 0.208 0.774
II4 0.362 0.153 0.249 0.674

6 While Informed Action had adequate composite reliability and convergent validity, it did not demonstrate indicator reliabil-
ity. In examining the items, we believe this could stem from the use of the word ‘avoid’. For instance, if one avoids acting on
irrelevant information it does not mean they act on correct information. As such, we propose the following item: “when I
obtain information from [System], I use key parts of it to find solutions for problems encountered in my work.”
7 All dimensions of effective use were relevant and no collinearity issues present. However, there was an issue with the re-

dundancy analysis, this is likely due to an issue with the global item used. For instance, respondents may perceive effective
use to be simply the inputting of data and running of reports rather than the use of the output. As such, we propose the fol-
lowing item: Overall, I effectively use [system] and the output that it provides to complete my job tasks.

13
References
Baruch, Y. and B. C. Holtom (2008). "Survey response rate levels and trends in organizational
research". Human relations 61 (8), 1139-1160.
Benbasat, I. and R. W. Zmud (2003). "The identity crisis within the IS discipline: Defining and
communicating the discipline's core properties". MIS Quarterly 27 (2), 183-194.
Bonaretti, D. and G. Piccoli. 2019. Unifying the emergency management research program in IS: A
representation theory perspective for effective use in chaotic environments. Louisiana State
University.
Burton-Jones, A. and C. Grange (2013). "From use to effective use: A representation theory
perspective". Information Systems Research 24 (3), 632-658.
Burton-Jones, A., E. McLean and E. Monod (2015). "Theoretical perspectives in IS research: From
variance and process to conceptual latitude and conceptual fit". European Journal of Information
Systems 24 (6), 664-679.
Burton-Jones, A. and D. Straub (2006). "Reconceptualizing system usage: An approach and empirical
test". Information Systems Research 17 (3), 228-246.
Burton-Jones, A. and O. Volkoff (2017). "How can we develop contextualized theories of effective
use? A demonstration in the context of community-care electronic health records". Information
Systems Research 28 (3), 468-489.
Campbell, D. E. and N. Roberts (2019). "Effective use of analytic DSS and job performance: Looking
beyond technology acceptance". Journal of Organizational Computing and Electronic Commerce
29 (2), 125-138.
Cenfetelli, R. T. and G. Bassellier (2009). "Interpretation of formative measurement in information
systems research". MIS Quarterly, 689-707.
Cepeda, G., C. Nitzl and J. L. Roldán 2017. Mediation analyses in partial least squares structural
equation modeling: Guidelines and empirical examples. Partial Least Squares Path Modeling.
Springer.
Chin, W. W., A. Gopal and W. D. Salisbury (1997). "Advancing the theory of adaptive structuration:
The development of a scale to measure faithfulness of appropriation". Information Systems
Research 8 (4), 342-367.
Coltman, T., T. M. Devinney, D. F. Midgley and S. Venaik (2008). "Formative versus reflective
measurement models: Two applications of formative measurement". Journal of Business Research
61 (12), 1250-1262.
DeLone, W. H. and E. R. McLean (1992). "Information systems success: The quest for the dependent
variable". Information Systems Research 3 (1), 60-95.
Delone, W. H. and E. R. McLean (2003). "The DeLone and McLean model of information systems
success: a ten-year update". Journal of Management Information Systems 19 (4), 9-30.
Dubin, R. (1978). Theory development. New York: Free Press.
Eden, R. and A. Burton-Jones (2018). Beyond effective use: A journey to understand inconsistencies
in use. International Conference of Information Systems. San Francisco, USA.
Eden, R., A. Burton-Jones and R. Donovan (2019). Testing the links from fit to effective use to
impact: A digital hospital case. International Conference of Information Systems. Munich,
Germany.
Eden, R., E. Fielt and G. D. Murphy. Who are the Operational Users of Enterprise Systems: Does One
Size Fit All? Proceedings of the 28th Australasian Conference on Information Systems (ACIS
2017), 2017.
Emamjome, F. A theoretical approach to conceptualize information quality in social media.
Australasian Conference on Information Systems, 2014 Auckland, NZ.
Fan, J. C. and K. Fang (2006). ERP implementation and information systems success: A test of
DeLone and McLean's model. Technology Management for the Global Future. IEEE.
Froehle, C. M. and A. V. Roth (2004). "New measurement scales for evaluating perceptions of the
technology-mediated customer service experience". Journal of Operations Management 22 (1), 1-
21.

14
Gable, G. G. (1994). "Integrating case study and survey research methods: An example in information
systems". European Journal of Information Systems 3 (2), 112-126.
Gable, G. G., D. Sedera and T. Chan (2008). "Re-conceptualizing information system success: The IS-
impact measurement model". Journal of the Association for Information Systems 9 (7), 18.
Haake, P., S. Schacht, B. Mueller, J. Lauterbach and A. Maedche (2018). Toward an
operationalization of effective use. European Conference on Information Systems. Portsmouth, UK.
Hair, J., G. Hult, C. Ringle and M. Sarstedt 2016. A Primer on Partial Least Squares Structural
Equation Modeling (PLS-SEM), Sage Publications.
Hair, J. F., W. C. Black, B. J. Babin and R. E. Anderson 2010. Multivariate data analysis: A global
perspective, Upper Saddle River, N.J. ;, Pearson Education.
Hair, J. F., M. Sarstedt, C. M. Ringle and S. P. Gudergan 2017. Advanced Issues in Partial Least
Squares Structural Equation Modeling, Sage publications.
Hardesty, D. M. and W. O. Bearden (2004). "The use of expert judges in scale development:
Implications for improving face validity of measures of unobservable constructs". Journal of
Business Research 57 (2), 98-107.
Hunt, S. D., R. D. Sparkman Jr and J. B. Wilcox (1982). "The pretest in survey research: Issues and
preliminary findings". Journal of Marketing Research, 269-273.
Law, K. S., C.-S. Wong and W. M. Mobley (1998). "Toward a taxonomy of multidimensional
constructs". Academy of Management Review 23 (4), 741-755.
Lin, H.-Y., P.-Y. Hsu and P.-H. Ting (2006). "ERP systems success: An integration of IS success
model and balanced scorecard". Journal of Research and Practice in Information Technology 38
(3), 215.
Lyytinen, K., T. Keil and V. Fomin (2008). "A framework to build process theories of anticipatory
information and communication technology (ICT) standardizing". International Journal of IT
Standards and Standardization Research 6 (1), 1-38.
MacKenzie, S. B., P. M. Podsakoff and C. B. Jarvis (2005). "The problem of measurement model
misspecification in behavioral and organizational research and some recommended solutions".
Journal of Applied Psychology 90 (4), 710.
MacKenzie, S. B., P. M. Podsakoff and N. P. Podsakoff (2011). "Construct measurement and
validation procedures in MIS and behavioral research: Integrating new and existing techniques".
MIS Quarterly 35 (2), 293-334.
Marchand, M. and L. Raymond (2017). Characterizing, explaining and valuing the effective use of an
IT artefact: A field study of performance management information systems in SMEs. Hawaii
Internatational Conference on System Sciences. Hawaii, US.
Marcolin, B. L., D. R. Compeau, M. C. Munro and S. L. Huff (2000). "Assessing User Competence:
Conceptualization and Measurement". Information Systems Research 11 (1), 37-60.
Markus, M. L. and C. Tanis (2000). "The enterprise systems experience-from adoption to success".
Framing the domains of IT research: Glimpsing the future through the past 173, 207-173.
McAfee, A. (2006). "Mastering the three worlds of information technology". Harvard Business
Review 84 (11), 141.
Mohr, L. B. 1982. Explaining Organizational Behavior, Jossey-Bass.
Petter, S. and E. R. McLean (2009). "A meta-analytic assessment of the DeLone and McLean IS
success model: An examination of IS success at the individual level". Information & Management
46 (3), 159-166.
Petter, S., D. Straub and A. Rai (2007). "Specifying formative constructs in information systems
research". MIS Quarterly 31 (4), 623-656.
Polites, G. L., N. Roberts and J. Thatcher (2012). "Conceptualizing models using multidimensional
constructs: A review and guidelines for their use". European Journal of Information Systems 21,
22-48.
Rubio, D. M., M. Berg-Weger, S. S. Tebb, E. S. Lee and S. Rauch (2003). "Objectifying content
validity: Conducting a content validity study in social work research". Social Work Research 27
(2), 94-104.

15
Salisbury, W. D., W. W. Chin, A. Gopal and P. R. Newsted (2002). "Better theory through
measurement—Developing a scale to capture consensus on appropriation". Information Systems
Research 13 (1), 91-103.
Seddon, P. B. (1997). "A respecification and extension of the DeLone and McLean model of IS
success". Information Systems Research 8 (3), 240-253.
Sedera, D., R. Eden and E. McLean (2013). "Are we there yet? A step closer to theorizing information
systems success".
Shin, B. and G. Kim (2011). "Investigating the reliability of second-order formative measurement in
information systems research". European Journal of Information Systems 20 (5), 608-623.
Straub, D., M. Limayem and E. Karahanna-Evaristo (1995). "Measuring system usage: Implications
for IS theory testing". Management Science 41 (8), 1328-1342.
Straub, D. W. (1989). "Validating instruments in MIS research". MIS Quarterly 13 (2), 147-169.
Trieu, T. V. H. (2013). Extending the theory of effective use: The impact of enterprise architecture
maturity stages on the effective use of business intelligence systems. International Conference on
Information Systems. Milan, Italy.
Wand, Y. and R. Weber (1990). Toward a theory of the deep structure of information systems.
International Conference on Information Systems.
Wand, Y. and R. Weber (1995). "On the deep structure of information systems". Information Systems
Journal 5 (3), 203-223.
Wickramasinghe, V. and M. Karunasekara (2012). "Perceptual differences of enterprise resource
planning systems between management and operational end-users". Behaviour & Information
Technology 31 (9), 873-887.

16