{Journals}acfi/40_3/z126/makeup/z126.

3d

Accounting and Finance 40 (2000) 191± 210

A study of the internal control structure for

electronic data interchange systems using the
analytic hierarchy process

Catherine Hardy a, Robert Reeve b

a
School of Financial Studies, Charles Sturt University, Wagga Wagga, Australia
b
Division of Economic and Financial Studies, Macquarie University, Sydney, Australia

Abstract

Electronic data interchange (EDI) systems involve the direct exchange of

structured business data between trading partner computer systems. A reliable
internal control structure is the primary means of providing assurance of
information integrity in EDI systems. This paper reports the results of a study
that examined information system (IS) managers' and computerised
information system (CIS) auditors' judgements of the relative importance of
elements of the internal control structure for EDI systems, using the analytic
hierarchy process (AHP). It then assessed the degree of consensus in their
judgements. Generally consensus was found to be high. However, the areas
where there was lack of consensus may indicate potential areas of control
weakness in EDI systems.

Key words: Analytic hierarchy process (AHP); Computerised information

system (CIS) auditors; Electronic data interchange (EDI); Information system
(IS) managers; Internal control structure

JEL classification: M41

1. Introduction

Electronic data interchange (EDI) systems involve the exchange of electronic

business data, in a standard and structured format, between trading partner

The authors are grateful to the Coopers & Lybrand Visiting Professor Scheme
which enabled Professor Ron Weber, of The University of Queensland, to visit
Charles Sturt University and discuss ideas and issues in the early stages of this
study. They also acknowledge the constructive comments from delegates at the
1998 AAANZ Annual Conference, and from participants at a Macquarie
University research seminar, on earlier drafts. The authors have received
valuable advice from the editors and referees.

# AAANZ, 2000. Published by Blackwell Publishers, 108 Cowley Road, Oxford

OX4 1JF and 350 Main Street, Malden MA 02148, USA.
{Journals}acfi/40_3/z126/makeup/z126.3d

192 C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210

computer systems via a telecommunication network (Chan et al., 1993;

Jamieson, 1994; Aggarwal and Rezaee, 1996). Despite the well-documented
benefits of EDI, there has been a slower than expected growth of EDI
applications in Australia due to a number of risks and barriers. These include
technological, security, legal, control and organisational issues (ASCPA, 1994;
Cullen, 1995). The trend from value added network (VAN) based EDI to
Internet based EDI is assisting in overcoming some of the technological
barriers (Greenstein and Feinman, 2000). However, the question of risk, both
in the systems, and to the parties trading online, remains a major concern
(Tradegate ECA, 1996; Ratnasingham, 1998).
EDI systems introduce additional complexities in initiating, recording
and executing transactions using telecommunication networks (Hansen and
Hill, 1989). The consequences of unreliable records and information are
more far reaching in EDI systems due to the high speed of data interchange
and low level of human intervention in the conduct of on-line business
(Chan, 1992). Therefore, control over EDI systems is essential to reduce the
risks, ensure the integrity of the information, and achieve maximum benefits
from the technology (Powers and Carver, 1990; Weiner, 1995). Further,
management and auditors are compelled to review computer based controls
more closely due to the heavy reliance placed on these controls with the
elimination of the traditional paper audit trail (Jamieson, 1994; Ryrie,
1994).
The purpose of this study is to examine information system (IS) managers'
and computerised information system (CIS) auditors' judgements of the
importance of elements of the internal control structure in EDI systems, and
then assess the degree of consensus in their judgements. There are two main
motivations for this study.
First, EDI has the potential to significantly impact the accounting function
of organisations through changes to accounting, auditing and control
procedures (ASCPA, 1994; Weiner, 1995). While the literature reveals an
inventory of internal control elements relevant to EDI systems, the research
has been largely descriptive. To date there has been no specific standard issued
by the Australian Auditing Standards Board 1 on the audit implications of
EDI. Much of the authoritative literature on evaluating internal control is
provided in AUS214 Auditing in a CIS Environment (consistent with ISA401)
by reference to AUS402 Risk Assessment and Internal Controls (consistent with
ISA400). However, the diversity of control technology increases with the
emergence of CIS technologies (Weber, 1988) particularly those which `feed
information directly into the financial system ... [as] ... the overall sophistica-
tion of CISs and the complexity of the specific applications ... [affected] ... may

1
This is now the Australian Auditing and Assurance Standards Board

# AAANZ, 2000
{Journals}acfi/40_3/z126/makeup/z126.3d

C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210 193

increase risk...'. (AUS 214 para 17). This study is a unique effort to focus on IS
managers' and CIS auditors' judgements of the relative importance of control
categories and control elements in the internal control structure of EDI
systems. An examination of these judgements will assist in determining which
of the control categories and elements presented in the professional statements
and other prescriptive sources are judged by CIS professionals to be important
constituents of a reliable internal control structure. The findings from this
study should provide a preliminary basis for authoritative bodies such as the
Australian Auditing Standards Board, Tradegate ECA, and auditing educators
to discuss the extent and nature of authoritative guidance necessary to support
managers and auditors in implementing, maintaining and evaluating internal
control structures in EDI systems.
Second, to fulfil their legal and professional responsibilities, both IS
managers and CIS auditors need to ensure that EDI systems are properly
controlled (Ryrie, 1994). Risks of theft, destruction, interception, alteration,
stalling or re-routing of data, as well as forged messages must be minimised
(Greenstein and Feinman, 2000). The reliability of the internal control
structure can be assessed by identifying the strengths and weaknesses in the
structure, weighting the importance of these strengths and weaknesses,
imposing some organisation upon them in order to facilitate decision making,
and performing an overall evaluation (Weber, 1980; Gill and Cosserat, 1993;
Gul et al., 1994). Evaluating internal control in EDI systems is further
complicated by shared interfaces between trading partners. Perceptions of
internal control importance may differ between organisations resulting in EDI
systems considered to be not mutually reliable. Therefore, IS managers and
CIS auditors need to work together, both within their own organisations, and
with their peers from trading partner organisations, to establish EDI systems
with reliable internal control structures.
Furthermore, because IS managers and CIS auditors need to work
together, the level of consensus (agreement) or disagreement between their
judgement of the importance of control areas can help identify control areas
requiring further analysis. For example, finding consensus in internal control
judgements will identify control areas where it is likely that generally accepted
practice exists. On the other hand, differences would indicate areas where
such generally accepted practice may not exist, and could indicate a
potentially weak control area. In these areas IS managers and CIS auditors
could have different perspectives or use different types of knowledge and
information in making judgements, which could be due to their different
educational and professional backgrounds, or association with different
industry and professional bodies. For example, IS management may not
consider a control area important enough to ensure adequate internal
control. CIS auditors should ensure adequate control exists in those control
areas where a significant lack of consensus has been identified. This paper
attempts to shed light on these issues.

# AAANZ, 2000
{Journals}acfi/40_3/z126/makeup/z126.3d

194 C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210

Given the exploratory nature of this research we have not developed formal
hypotheses, but have used the following research objectives to guide our
empirical work:

1. Identify and categorise internal controls relevant to EDI systems.

2. Develop analytic hierarchy process (AHP) judgement models for IS

managers and CIS auditors in order to:

(a) examine IS managers' and CIS auditors' judgements of the relative

importance of these controls; and,
(b) assess the degree of agreement (consensus) or disagreement between the
judgements of IS managers and CIS auditors.

Only the financial statement audit is investigated in this study. However, the
results should be informative for other types of audits as they are interrelated
to some degree (Landry et al., 1989).

2. Background and literature review

2.1. Nature of EDI and internal control

EDI systems replace manual documents with electronic documents and

share supply chain data with key trading partners to reduce procurement lead
times and processing costs by eliminating the preparation of manual
documents and errors in data entry (Greenstein and Feinman, 2000). However,
these benefits are not achieved without risk.
The overall reliability of a system is dependent upon its control activities
(Weber, 1988). However, as CISs evolve into more complex forms, the process
of designing an appropriate configuration of controls becomes more difficult
and more critical (Adolphson and Hansen, 1991). Many traditional controls do
not exist in EDI systems because of their very nature (e.g., paperless
transactions and shared interfaces between trading partners) (Cullen, 1995).
Further, as EDI systems transcend the legal boundaries of organisations,
traditional control assessment procedures need to be extended to include an
examination of trading partner and third party relationships.

2.2. Internal control evaluation and AHP models

A substantial body of literature exists in which the audit task of evaluating

internal controls has been investigated: see, for example, the meta-analysis of
internal control judgements of internal control systems by Trotman and Wood
(1991). Two major issues emerge. First, auditors are usually faced with systems
of interrelated elements of varying complexity when evaluating the reliability of

# AAANZ, 2000
{Journals}acfi/40_3/z126/makeup/z126.3d

C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210 195

internal control structures. Therefore, the greater their understanding of the

function and purpose of each system element, the better is their ability to make
sound decisions regarding the reliability of the system. Second, the human mind
organises and processes information in particular ways. Bonner and Pennington
(1991) posit that if one is able to model these processes for decision tasks then
this should prove beneficial in ensuring judgement consistency. A multi-criteria
decision model, such as the analytic hierarchy process (AHP), can assist in
modelling internal control judgements and in analysing judgement consistency.

3. The empirical study

3.1. Overview of AHP model

The AHP is used in this study to model the internal control judgements of IS
managers and CIS auditors, and to provide measures of the consistency of
those judgements. The AHP judgement model is based on a multiple criteria
method of choice (Saaty, 1987; Apostolou and Hassell, 1993a). It represents
problems in a hierarchical structure, in order to develop priorities for
alternatives based on a decision maker's judgement (Saaty, 1980, 1987, 1988,
1990a, 1990b). While the AHP is conceptually simple to use, it is still powerful
enough to handle complex issues (Saaty, 1990b). The areas in which the AHP
has been applied are diverse and numerous (Zahedi, 1986; Shim, 1989; Vargas,
1990; Apostolou and Hassell, 1993a; Palmer, 1999). The AHP was introduced
into the accounting literature in the early 1980s and has primarily been used by
accounting researchers to model decision processes of individuals, and as an
aid in making choices (Arrington et al., 1984; Harper, 1988; Messier and
Schneider, 1988; Apostolou and Hassell, 1993a; Yau and Davis, 1993; Emby
and Etherington, 1996; Schniederjans and Garvin, 1997).
There are two major phases to the AHP, namely, designing the hierarchy
and evaluating the hierarchy (Vargas, 1990). The hierarchy employed in this
study is based on the framework set out in Tradegate ECA's EDI Control
Guide (1990) and Practical Internal Control Guide for EDI Implementations
(1995). Tradegate ECA is the prime user association for electronic commerce in
Australia. The EDI internal control hierarchy developed for this study is
summarised in Figure 1. This hierarchy is consistent with the five major areas
of concern organisations face when implementing and using EDI identified by
Chan (1992), and salient aspects of internal control for EDI found in the
literature (Hansen and Hill, 1989; Chan et al., 1993; Walden and Braganza,
1993; Jamieson, 1994).
Level 1 of the hierarchy is the overall objective of the judgement, that is
internal control structure reliability. Level 2 identifies the major control
categories set out in the control guide literature from Tradegate ECA, namely,
organisational, application, authentication, security, and third party (network
and mailbox storage security) controls. Level 3 consists of 24 control elements

# AAANZ, 2000
 {Journals}acfi/40_3/z126/makeup/z126.3d
# AAANZ, 2000

196
C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210
Level 1 Internal control structure reliability

Level 2 Organisational Application Controls Authentication Controls Security Controls Third Party Controls
Controls

Level 3 Managerial tone= In=Outbound transaction Authentication of trading Logical security controls Logical &
Control controls partner & transaction operational security
content

Audit committees Document standards Authorisation of Physical security controls Third party
transaction initiation agreements

Internal audit File retention Contingency controls Operational security Audit trails

External influences Application development= Reporting, logging & audit Audit of third party
Program change controls trails provider

Implementation Trading partner profiles Backup and recovery

management procedures

Segregation of duties Encryption techniques

Fig. 1. EDI internal control hierarchy

{Journals}acfi/40_3/z126/makeup/z126.3d

C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210 197

which are based on the Tradegate ECA literature, Auditing Standards (AUS
214 and AUS 402), audit manuals and prior studies.

3.2. Data collection

The data were collected by means of a mail survey. Despite the problems
associated with self-administered questionnaires, especially lower internal
validity, the survey method was chosen because of its effectiveness on the basis
of accessibility, timeliness, cost, and potentially higher external validity. The
survey questionnaire was administered by mail because of the advantages of
ensuring a representative sample at lower costs, which outweighed the
disadvantages of low response rates and having to use closed-ended questions
(de Vaus, 1991).
The target population comprised IS managers and CIS internal auditors
from organisations which were members of Tradegate ECA, and CIS external
auditors from Big 6 accounting firms. Selection of the sample was restricted to
the states of New South Wales and Victoria because 75 per cent of Australian
EDI users had their headquarters located in those states (Coopers and
Lybrand, 1994). The IS managers were then randomly selected from a list of
EDI users obtained from Tradegate ECA, and the CIS internal auditors and
external auditors from the directory of members of the Institute of Chartered
Accountants in Australia.
The survey yielded 54 responses from 159 questionnaires mailed, of which 48
were useable: 23 from IS managers, and 25 from CIS internal and external
auditors. The auditor responses were combined for further statistical analyses.
Statistical tests showed that there were no significant differences in the
judgements of the CIS internal auditors and CIS external auditors for the five
major control categories.
Given the relatively low response rate, questionnaire non-response bias was
considered. The surrogate method for testing non-response bias, which treats
late responders as equivalent to non-responders, was used (Wallace and
Mellor, 1988). Statistical tests showed that there were no significant differences
in the judgements of the early and late responders for the five major control
categories, reducing the likelihood of non-response bias.

3.3. AHP model development

The development of the AHP judgement models required each subject to

make all possible pairwise comparisons of the control categories and elements
at each level of the hierarchy using Saaty's (1986b) measurement scale, which
indicates the degree of importance of the elements relative to each other. The
scale is bounded at one (equally important) and nine (absolute importance)
with varying degree of importance between these bounds. The questionnaire
contained a definition of each of the control categories and elements in the

# AAANZ, 2000
{Journals}acfi/40_3/z126/makeup/z126.3d

198 C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210

hierarchy (Figure 1), an explanation of the measurement scale, and instruc-

tions on how to complete the pairwise comparison task. Subjects were asked to
also consider the cost=benefit trade-off when making their judgements as they
need to not only determine the best set of controls for reducing an array of
risks but also the associated resource implications (Bagranoff, 1989). The
pairwise comparisons were used to construct six matrices which were used to
compute weights which measure: (i) the relative importance of each control
element within each of the five control categories; and, (ii) the relative
importance of each category to the overall goal of internal control reliability.

4. Data analysis and results

4.1. Consistency of responses

The AHP provides a ratio measure of the consistency of the subject's

judgement model. In the AHP, consistency refers to transitivity and
magnitude. A perfectly consistent response would be one where, for example,
if A is preferred to B by a multiple of three, and B is equally preferred to C,
then A is preferred to C by a multiple of three (Apostolou and Hassell, 1993a).
A consistency ratio of zero represents perfect consistency. Saaty (1986a) argues
that a ratio of more than 0.10 is undesirable but both Harper (1988), and
Apostolou and Hassell (1993b), consider the 0.10 threshold to be overly
restrictive.
Seven consistency ratios were calculated for each respondent: one for the
control elements within each of the five major internal control categories; one
for the control categories to the overall goal (referred to as CR); and one
overall consistency ratio for the hierarchy based on the synthesised
(aggregated) weights (referred to as CRH) to provide an indication of
consistency for the full judgement model. In order to avoid loss of data,
responses with a consistency ratio greater than 0.10 were included in the
analysis.
The MANOVA results in Table 1, Panel 1, show that the combined
dependent variables (internal control mean weights) were not significantly
different between highly consistent (CRH Å 0.10), and less consistent
(CRH > 0.10), respondents (Wilks' criterion p ˆ 0.08) (Apostolou and Hassell,
1993b). Note that when testing AHP weights with MANOVA one variable has
to be excluded to overcome the linear dependence which is an artefact of AHP
weights. In this MANOVA the third party control category was excluded
because it had the lowest mean weight. The third party control category was
tested using one-way ANOVA and showed a significant difference ( p ˆ 0.01)
indicating an area which could warrant further investigation. For instance, the
responses may be influenced by circumstances relating to specific third party
arrangements. For the other four categories the mean weights for highly

# AAANZ, 2000
{Journals}acfi/40_3/z126/makeup/z126.3d

C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210 199

Table 1
Panel 1
Significance of differences in internal control category mean weights between respondents with
different consistency ratios for the hierarchy (CRH)

CRH Å 0.10 CRH > 0.10

Control category IC mean weights IC mean weights Significance

Organisational 0.19 0.16

Application 0.23 0.31
Authentication 0.21 0.22
Security 0.21 0.21
Third party 0.17 0.10 0.01 (2)

Wilks' criterion 0.08 (1)

(1) Results from MANOVA excluding the variable with the lowest mean weight (Third party).
F(4,43) ˆ 2.23 (p ˆ 0.08). Exclusion of a variable is necessary to overcome the linear dependence
which is an artefact of AHP weights.
(2) Results from one-way ANOVA. F(1,46).

Panel 2
Significance of differences in consistency ratios (CR and CRH) between IS managers and CIS
auditors

Consistency IS managers CIS auditors

ratios mean mean Significance

CR 0.25 0.17 0.36 (1)

CRH 0.22 0.19 0.63 (1)

(1) Results from one-way ANOVAs. F(1,46).

Panel 3
Significance of differences in consistency ratios for control categories between IS managers and CIS
auditors

IS managers CIS auditors

Control category mean CR mean CR Significance

Organisational 0.22 0.31

Application 0.15 0.24
Authentication 0.12 0.09
Security 0.14 0.19
Third Party 0.36 0.28

Wilks' criterion 0.22 (1)

(1) Results from MANOVA. F(5,42) ˆ 1.15 (p ˆ 0.22).

# AAANZ, 2000
{Journals}acfi/40_3/z126/makeup/z126.3d

200 C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210

consistent respondents (CRH Å 0.10) were not significantly different from those
of less consistent respondents (CRH > 0.10).
The one-way ANOVA results in Panel 2 show no significant difference
between the IS manager and CIS auditor groups for either the consistency ratio
(CR) (Wilks' criterion p ˆ 0.36), or for the consistency ratio for the hierarchy
(CRH) (Wilks' criterion p ˆ 0.63). Finally, a MANOVA was performed to test
for differences, between the IS manager and CIS auditor groups, for the
combined control category consistency ratios (CR). The results in Panel 3 show
that the combined dependent variables were not significantly different (Wilks'
criterion p ˆ 0.22).
The results summarised in Table 1 show that although, on average, neither
IS managers nor CIS auditors had highly consistent responses (CRH Å 0.10),
neither group was more consistent in their judgements. Therefore, as the
internal control mean weights did not differ significantly between highly
consistent respondents (CRH Å 0.10), and less consistent respondents
(CRH > 0.10), the validity of the responses was considered acceptable.

4.2. Internal control importance weights

Table 2 summarises the average results of the individual AHP judgement

models for the 48 respondents in terms of the mean weights of the synthesised
(aggregated) models, and the percentage frequency with which control
categories and elements were judged as highly important (mean weight å 0.20),
moderately important (mean weight <0.20 å 0.10), or less important (mean
weight <0.10) (Harper, 1988).
Inspection of Table 2 shows that the application control category was
judged as the most important in the internal control structure of EDI systems.
Its mean weight of 0.29 was the highest, and a large majority of respondents
(83.3 per cent) weighted it at the highly important level (mean weight å 0.20).
Within the application control category in=outbound transaction control was
judged to be the most important control element and document standards the
next most important.
Authentication controls and security controls were the next most important
of the internal control categories. Authentication controls had a mean weight
of 0.22, and a majority of respondents (54.2 per cent) weighted it at the highly
important level. Within the authentication controls category authentication of
trading partner and transaction content was judged to be the most important
control element and authorisation of transactions initiation the next most
important.
Security controls had a mean weight of 0.21, and a majority of respondents
(62.5 per cent) weighted it at the highly important level. Within the security
controls category backup and recovery procedures was clearly judged to be the
most important control element.

# AAANZ, 2000
{Journals}acfi/40_3/z126/makeup/z126.3d

C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210 201

Table 2
AHP judgement models: summary of results

IC weight IC weight IC weight IC weight

mean value å 0.20 <0.20 å 0.10 <0.10
(%) (%) (%)

Control categories
Organisational controls 0.16 39.6 20.8 39.6
Application controls 0.29 83.3 12.5 4.2
Authentication controls 0.22 54.2 35.4 10.4
Security controls 0.21 62.5 29.2 8.3
Third party controls 0.12 20.8 29.2 50.0

Organisational control elements

Managerial tone=Control 0.26 66.7 27.0 6.3
Audit committee 0.15 33.3 29.2 37.5
Internal audit 0.17 52.1 14.6 33.3
External influences 0.12 18.7 29.2 52.1
Implementation management 0.30 72.9 22.9 4.2
Application control elements
In=Outbound transaction controls 0.25 62.5 35.4 2.1
Document standards 0.20 35.4 39.6 25.0
File retention 0.11 2.1 56.2 41.7
Application development= 0.17 33.3 47.9 18.8
Program change control
Trading partner profiles 0.14 18.7 37.5 43.8
Segregation of duties 0.13 6.2 56.3 37.5
Authentication control elements
Authentication of trading partner= 0.37 85.4 14.6 0.0
Transaction content
Authorisation of transactions 0.32 79.2 16.6 4.2
initiation
Contingency controls 0.30 54.2 16.6 29.2
Security control elements
Logical security controls 0.18 31.2 41.7 27.1
Physical security controls 0.15 14.6 45.8 39.6
Operational security 0.13 12.5 39.6 47.9
Reporting=Logging=Audit trails 0.19 27.1 50.0 22.9
Backup & recovery procedures 0.20 52.1 35.4 12.5
Encryption techniques 0.15 20.8 43.8 35.4
Third party control elements
Logical=Operational security 0.31 81.2 4.2 14.6
Third party agreements 0.22 62.5 12.5 25.0
Audit trails 0.25 68.7 20.9 10.4
Third party audits 0.23 66.7 25.0 8.3

# AAANZ, 2000
{Journals}acfi/40_3/z126/makeup/z126.3d

202 C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210

The importance placed on application control, authentication control, and

security control categories is consistent with the literature. Deterrents to rapid
growth in EDI systems primarily relate to concerns regarding security and
reliability (Greenstein and Feinman, 2000). Reliability concerns include the
loss or delay of data transmitted, response time, and accuracy of data. Security
concerns include the safety of mission-critical data and the unauthorised
interception and=or alteration of transaction data that is displayed and
transmitted over the network. Application, authentication, and security
controls are recognised as a means of reducing such risks. These findings are
closely aligned with observations made in the literature regarding the major
effects of EDI on internal control, namely: increase in the automation of
control; identification and follow up of errors more quickly; possible changes
in accounting processes; evidence in mainly electronic form; and, increasing
importance in security (Wright 1992 cited in Jamieson 1994, 21).
Organisational controls was judged as the fourth most important category.
It had a mean weight of 0.16, and a large minority of respondents (39.6 per
cent) weighted it at the highly important level. Within the organisational
controls category implementation management was judged to be the most
important control element and managerial tone the next most important.
Third party controls were judged as the least important category. It had a
mean weight of 0.12, and only a minority of respondents (20.8 per cent)
weighted it at the highly important level. Within the third party controls
category logical and operational security was judged to be the most important
control element.
These results suggest that controls which may be more directly influenced by
the respondents, such as application, authentication, and security controls, are
seen as more significant than controls which are more dispersed throughout the
organisation, such as organisational and third party controls. The results are
consistent with the findings of Haskins (1987) and Harper (1988). For instance,
Harper (1988) found that, in a local area network environment, the controls
that could be implemented and administered centrally were generally
considered more important than those dispersed throughout the organisation.
These results also contradict the significance placed in the literature on
organisational controls for providing an overall framework for the control
system (Haskins, 1987). Further, the low weights given to the organisational
controls in this study and the Haskins (1987) and Harper (1988) studies
contrast with the findings of Weber (1980) on such matters. The reasons for
this difference appear to have been foreseen by Weber (1980) who stated that
the emphasis placed on management controls in his research may have been `a
carry-over from the audit of manual systems ... [and] ... a tendency still to audit
around rather than through the computer'. These issues await further
investigation.

# AAANZ, 2000
{Journals}acfi/40_3/z126/makeup/z126.3d

C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210 203

4.3. Consensus between the judgements

Table 3 summarises results from the MANOVA and ANOVA tests used to
compare the mean synthesised internal control weights of the IS manager and
CIS auditor groups. Recall that when testing AHP weights with MANOVA,
one variable has to be excluded to overcome the linear dependence of AHP
weights. In these MANOVAs the variable with the lowest mean weight was
excluded and subsequently tested using one-way ANOVAs.
Inspection of the control categories section of Table 3 shows that the mean
weights of the combined control category variables (with the third party
category excluded) were not significantly different between IS managers and
CIS auditors (Wilks' criterion p ˆ 0.18). The excluded third party control
category did show a significant difference in the mean weights of IS managers
and CIS auditors ( p ˆ 0.03). Here internal control judgements may be
influenced by circumstances relating to specific third party arrangements.
However, inspection of the third party control elements section of Table 3
shows that the mean weights of the combined control element variables (with
the third party agreements element excluded) were not significantly different
between IS managers and CIS auditors (Wilks' criterion p ˆ 0.29). Further-
more, the excluded third party agreements element did not show a significant
difference in the mean weights of IS managers and CIS auditors ( p ˆ 0.60).
An examination of the remaining control element categories in Table 3
shows no significant differences between the mean weights for each of the two
expert groups except for the file retention element ( p ˆ 0.02) within the
application control element category and the security control elements
category (Wilks' criterion p ˆ 0.02). The lower weightings and differences in
judgements between the two expert groups for file retention controls may be
partly explained by the area being highly dependent on specific organisational
and regulatory policies.
As the mean weights of the combined security control element variables
(with the operational security element excluded) were significantly different
between IS managers and CIS auditors, one-way ANOVAs were carried out on
all control elements. None showed a significant difference between IS
managers and CIS auditors except for the encryption techniques element
( p ˆ 0.02) and the operational security control element ( p < 0.01). Operational
security and encryption techniques were weighted among the least important in
each expert group, with backup and recovery procedures ranked the highest.
These findings appear to be consistent with observations made in the literature.
Providing a technological environment in which secure messages can be sent
between trading partners is a crucial aspect of EDI systems. Methods such as
encryption techniques provide some means of achieving the goal of a safe,
business-oriented messaging environment. However, because the potential
payoffs to hackers that can circumvent controls are high, existing methods
need to be continuously tested, improved and new security methods explored

# AAANZ, 2000
{Journals}acfi/40_3/z126/makeup/z126.3d

Table 3
Significance of differences in mean internal control weights between IS managers and CIS auditors

Mean IC weight Mean IC weight

IS managers CIS auditors Significance

Control categories
Organisational controls 0.16 0.16
Application controls 0.26 0.32
Authentication controls 0.23 0.21
Security controls 0.21 0.21
Third party controls 0.14 0.09 0.03 (2)

Wilks' criterion 0.18 (1)

Organisational control elements
Managerial tone=Control 0.23 0.28
Audit committee 0.14 0.16
Internal audit 0.20 0.15
External influences 0.12 0.11 0.56 (2)
Implementation management 0.31 0.29

Wilks' criterion 0.47 (1)

Application control elements
In=Outbound transaction controls 0.25 0.25
Document standards 0.20 0.20
File retention 0.13 0.09 0.02 (2)
Application development= 0.16 0.19
Program change control
Trading partner profiles 0.13 0.14
Segregation of duties 0.13 0.12

Wilks' criterion 0.10 (1)

Authentication control elements
Authentication of trading partner= 0.33 0.31
Transaction content
Authorisation of transaction 0.38 0.37
initiation
Contingency Controls 0.29 0.32 0.67 (2)

Wilks' criterion 0.89 (1)

Security control elements
Logical security controls 0.17 0.19 0.59 (2)
Physical security controls 0.14 0.16 0.73 (2)
Operational security 0.17 0.10 0.01 (2)
Reporting=Logging= 0.20 0.17 0.34 (2)
Audit trails
Backup & recovery procedures 0.20 0.20 0.92 (2)
Encryption techniques 0.11 0.18 0.02 (2)

Wilks' criterion 0.02 (1)

Third party control elements
Logical=Operational security 0.33 0.29
Third party agreements 0.21 0.23 0.60 (2)
Audit trails 0.23 0.26
Third party audits 0.24 0.22

Wilks' criterion 0.29 (1)

(a) Results from MANOVAs, excluding the variable with the lowest mean weight. Exclusion of a
variable is necessary to overcome the linear dependence which is an artefact of AHP weights.
(2) Results from one-way ANOVAs.
{Journals}acfi/40_3/z126/makeup/z126.3d

C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210 205

(Greenstein and Feinman, 2000). Further, even the best designed system
cannot control unforeseen man-made (e.g., viruses, hardware failure) or
natural disasters (e.g., fires, floods). Backup and recovery procedures are seen
as being essential to minimise such interruptions to organisational operations.

5. Discussion

5.1. Limitations

While this study of the internal control structure in EDI systems has several
limitations, none are considered to be of such significance as to negate the
contribution to understanding made by the results. The survey method
employed in this study imposed several limitations. First, the design of the
questionnaire represented a tradeoff between realism and subject fatigue
(Landry et al., 1989). To assist in minimising this limitation, procedures were
employed and detailed instructions were provided in the questionnaire to place
the research task as close as possible to actual real world task concerns.
This study did not address specific EDI system installations. Therefore
agreement `in fact' as analysed by Ashton (1974) and others was not possible
(Harper, 1988). As the study was based on a very general situation,
respondents had to determine their own reference in which to evaluate the
internal control elements (Messier and Schneider, 1988; Harper et al., 1992).
However, a tradeoff with this limitation is that the results are generalisable.
It may be argued that the results `are an artefact of a prespecified hierarchy'
(Messier and Schneider, 1988). The selection of the controls included in the
study and the subsequent categorisation may have resulted in controls being
inadvertently omitted. However, the hierarchy was developed based on a
thorough literature review and underwent a review process prior to the
instrument being disseminated so as to overcome this possibility. Therefore,
while it may argued that the hierarchical abstraction may be perceived by
individuals differently it still enables one to `communicate a sense of judgement
which involves common understanding (but not without differences)' (Saaty,
1980).
Another limitation in using the AHP is that the direct assessment of
importance of each of the 24 control elements vis-a-vis each other was not
obtained. Therefore, the weights in the full judgement models are dependent on
the validity of the AHP synthesis (aggregation) process. However, without the
use of the AHP, each respondent would have had to make a total of 295
pairwise comparisons, compared to a total of 59 pairwise comparisons in this
study, with consequent effects on the response rate.
IS managers and CIS auditors have a similar mandate when viewed from the
shareholders' perspective. Therefore, it is expected that their judgements on
internal control will be similar. However, care has been taken in this study to

# AAANZ, 2000
{Journals}acfi/40_3/z126/makeup/z126.3d

206 C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210

separately capture the judgements of these two expert groups responsible for
information integrity and control assurance in EDI systems (Chan et al., 1993).

5.2. Implications and future research

This research study was exploratory and consequently presents opportunities

for a refined theoretical framework and further empirical testing. Nevertheless,
the findings of this study represent a unique effort to focus on IS managers'
and CIS auditors' judgements of the relative importance of control categories
and control elements in the internal control structure of EDI systems.
Accounting professionals must have the appropriate technology skills to
understand EDI-related control mechanisms and be able to adequately
perform an assessment of major risks related to EDI transactions.
Our examination of the consensus between the two professional groups has
yielded useful results in that a substantial degree of consensus (Ashton, 1985;
Murray and Regel, 1992; Trotman, 1998), regarding the importance of
controls, was found between IS managers and CIS auditors. However, care
must be taken to avoid placing too much reliance on such consensus in a
specific EDI setting. Although IS managers and CIS auditors may agree on the
necessity for internal controls, the controls may not have been implemented.
Furthermore, those areas where there is a lack of consensus between IS
managers and CIS auditors (encryption techniques and operational security
controls) should be earmarked for further analysis. For example in areas where
IS managers perceive controls to be less important than do CIS auditors, there
may be a weakness in control because the IS manager did not consider it
worthwhile or cost-effective enough to implement what the CIS auditor
considers to be sufficient control. The reverse may also be true, i.e., that
unnecessary controls have been implemented. If so, discontinuing the
operation of the unnecessary controls may result in cost savings.
EDI impacts the scope and methods of traditional assurance engagements,
such as evaluation of the internal control structure. Because of the loss of the
paper audit trail, auditors need to consider and develop other methods for
collecting and evaluating evidence such as continuous auditing. By using such
an approach, the auditor is able to specify transaction selection criteria and
perform tests as transactions occur. In addition, further extension and
refinement of the AHP model used in this study to evaluate controls in
specific EDI systems, may suggest possible audit tools and techniques for the
CIS auditing profession. These issues await further investigation.
The results may also be useful for researchers who conduct AHP research by
mail survey and are unable to work iteratively with participants to ensure that
consistency ratios of less than 0.10 are obtained (Apostolou and Hassell,
1993b). The results of this study indicate that if the primary purpose of the
research is to obtain a general understanding of the relative importance of
factors using AHP weights, and relatively small samples are used, then

# AAANZ, 2000
{Journals}acfi/40_3/z126/makeup/z126.3d

C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210 207

including AHP results with consistency ratios of greater than 0.10 in the
analysis, may not distort the results. Apostolou and Hassell (1993b) found
similar results when the sample size was relatively large, but Harper's (1988)
study, which used a small sample size, did not. Therefore, to determine whether
the results are generalisable, future research could investigate other contextual
variables, such as the type of task and the number of factors assessed, to
determine when departure from the 0.10 consistency ratio threshold is
appropriate (Apostolou and Hassell, 1993b).
EDI has evolved substantially over the past three decades in both extent of
use and technological sophistication. As efficient supply chain management
becomes an increasingly important goal, organisations are discovering that
fully integrated EDI systems are a necessary enabling mechanism (Greenstein
and Feinman, 2000). Further, the widespread infrastructure of the Internet is
also greatly increasing connectivity and interoperability. However, the success
of EDI and Internet EDI systems ultimately depends not on technology but on
the trust business partners and customers place in these systems. Achieving this
trust requires IS managers and CIS auditors to work together to ensure the
reliability of the internal control structures in EDI systems. This study has
found a substantial body of consensus or `generally accepted practice' between
these two expert groups with regard to the relative importance of the elements
of the internal control structure in EDI systems. Therefore, while the
achievement of reliable internal control structures in EDI systems will not be
easy, it should not be unduly complicated by differences in judgement between
IS managers and CIS auditors.

References

Adolphson, D.L. and J.V. Hansen, 1991, Evaluating software controls for electronic
data interchange, European Journal of Operational Research 51, 42 ± 46.
Aggarwal, R. and Z. Rezaee, 1996, EDI risk assessment, The Internal Auditor 53, 40 ± 44.
Apostolou, B. and J.M. Hassell, 1993a, An overview of the analytic hierarchy process
and its use in accounting research, Journal of Accounting Literature 12, 1 ± 28.
Apostolou, B. and J.M. Hassell, 1993b, An empirical examination of the sensitivity of
the analytic hierarchy process to departures from recommended consistency ratios,
Mathematical and Computer Modelling 17, 163± 170.
Arrington, C.E., W. Hillison, and R.E. Jensen, 1984, An application of analytical
hierarchy process to model expert judgement on analytical review procedures, Journal
of Accounting Research 22, 298± 312.
Ashton, R.H., 1974, An experimental study of internal control judgments, Journal of
Accounting Research 12, 143±157.
Ashton, A.H., 1985, Does consensus imply accuracy in accounting studies of decision
making?, The Accounting Review 60, 173± 185.
Australian Auditing Standard (AUS) 214 Auditing in a CIS environment, in Australian
Society of CPAs and The Institute of Chartered Accountants in Australia, 1997,
Auditing Handbook (Australian Auditing Standards Board, Australian Accounting
Research Foundation, Prentice Hall).
Australian Auditing Standard (AUS) 402 Risk assessment and internal controls, in:

# AAANZ, 2000
{Journals}acfi/40_3/z126/makeup/z126.3d

208 C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210

Australian Society of CPAs and The Institute of Chartered Accountants in Australia.

1997 (Auditing Handbook, Australian Auditing Standards Board, Australian
Accounting Research Foundation, Prentice Hall).
Australian Society of CPAs (ASCPA), 1994, EDI Ð A business perspective (ASCPA
Information Technology Centre of Excellence).
Bagranoff, N.A., 1989, Using the analytic hierarchy approach to design internal control
systems, Journal of Accounting and EDP, Winter, 37 ± 41.
Bonner, S.E. and N. Pennington, 1991, Cognitive processes and knowledge as
determinants of auditor expertise, Journal of Accounting Literature 10, 1± 50.
Chan, S., 1992, Establishing reliability in an EDI environment, The EDP Auditor
Journal, II, 47 ± 51.
Chan, S., M. Govindan, J.Y. Picard, and E. Leschiutta, 1993, EDI for managers and
auditors 2nd ed., (The Canadian Institute of Chartered Accountants, Canada).
Coopers & Lybrand, 1994, EDI surveyÐ 1994 analysis of responses (Coopers &
Lybrand Consultants, Australia).
Cullen, S., 1995, Electronic data interchange: Implementation and control issues,
in Perspectives on contemporary auditing (ASCPA Audit Centre of Excellence)
58 ± 66.
de Vaus, D.A., 1991, Surveys in social research 3rd ed., (Allen and Unwin, London).
Emby, C. and L.D. Etherington, 1996, Performance evaluation of auditors: Role
perceptions of superiors and subordinates, Auditing: A Journal of Practice and Theory
15, 99 ± 109.
Gill, G.S. and G.W. Cosserat, 1993, Modern auditing in Australia 3rd ed.,. (John Wiley
& Sons, Brisbane).
Greenstein, M. and T. Feinman, 2000, Electronic commerce: Security, risk management
and control (McGraw Hill Irwin, Boston).
Gul, F.A., H.Y. Teoh, B.H. Andrew, and P. Schelluch, 1994, Theory and practice of
Australian auditing 3rd ed., (Thomas Nelson, Australia).
Hansen, J.V.,and N.C. Hill, 1989, Control and audit of electronic data interchange,
MIS Quarterly 13, 403± 413.
Harper, R.M., 1988, AHP judgement models of EDP auditors' evaluation of internal
control for local area networks, Journal of Information Systems 3, 67 ± 85.
Harper, R.M., N.G. Apostolou, and B.P. Hartman, 1992, The analytic hierarchy
process: An empirical examination of aggregation and hierarchical structuring,
Behavioral Research in Accounting 4, 97 ± 112.
Haskins, M.E., 1987, Client control environments: an examination of auditors'
perceptions, The Accounting Review 62, 542± 563.
Jamieson, R., 1994, EDI: An audit approach, Monograph Series 7 (The EDP Auditors
Foundation, inc., USA).
Landry, R. Jr., W.C. Letzkus, and T. Cronan, 1989, An examination of consensus
between external and internal auditors, The EDP Auditor Journal IV, 53 ± 63.
Messier, W.F. and A. Schneider, 1988, A hierarchical approach to the external auditor's
evaluation of the internal auditing function, Contemporary Accounting Research 4,
337± 353.
Murray, D. and R.W. Regel, 1992, Accuracy and consensus in accounting studies of
decision making, Behavioral Research in Accounting 4, 126± 139.
Palmer, B., 1999, Click here for decisions, Fortune May 10, 87 ± 89.
Powers, W.J. and T. Carver, 1990, EDI: Control and audit issues, The EDP Auditor
Journal, I, 25 ± 30.
Ratnasingham, P., 1998, Internet-based EDI trust and security, Information Manage-
ment and Computer Security 6, 33 ± 39.
Ryrie, T., 1994, Paperless trading, Charter, February, 28 ± 29.

# AAANZ, 2000
{Journals}acfi/40_3/z126/makeup/z126.3d

C. Hardy, R. Reeve / Accounting and Finance 40 (2000) 191± 210 209

Saaty, T.L., 1980, The analytic hierarchy process (McGraw-Hill, New York).
Saaty, T.L., 1986a, Decision making for leaders (RWS Publications, Pittsburgh).
Saaty, T.L., 1986b, Absolute and relative measurement with the AHP: The most livable
cities in the United States, Socio-Economic Planning Science 20, 327±331.
Saaty, T.L., 1987, Concepts, theory and techniques, rank generation, preservation and
reversal in the analytic hierarchy decision process, Decision Sciences 18, 157± 177.
Saaty, T.L., 1988, The analytic hierarchy process (RWS Publications, Pittsburgh).
Saaty, T.L., 1990a, How to make a decision: The analytic hierarchy process, European
Journal of Operational Research 48, 9 ±26.
Saaty, T.L., 1990b, Multicriteria decision making: The analytic hierarchy process (RWS
Publications, Pittsburgh).
Schniederjans, M.J. and T. Garvin, 1997, Using the analytic hierarchy process and
multi-objective programming for the selection of cost drivers in activity-based
costing, European Journal of Operational Research 100, 72 ± 80.
Shim, J.P., 1989, Bibliographical research on the analytic hierarchy process, Socio-
Economic Planning Sciences 23, 161± 167.
Tradegate ECA (formerly EDI Council of Australia and then ECA Australia) and
Information Systems Audit and Control Association (formerly EDP Auditors
Association), 1990, EDI control guide (Tradegate ECA, Australia).
Tradegate ECA (formerly EDI Council of Australia and then ECA Australia)
Accounting and Finance Working Party, 1995, Practical internal control guide for
EDI implementations (Tradegate ECA, Australia).
Tradegate ECA (formerly EDI Council of Australia and then ECA Australia), 1996,
Electronic Commerce Newsletter. August 5, 1 ± 8.
Trotman, K. and R. Wood, 1991, A meta-analysis of studies on internal control
judgments of internal control systems, Journal of Accounting Research 29, 180± 192.
Trotman, K., 1998, Audit judgement research: Issues addressed, research methods and
future directions, Accounting and Finance 38, 115± 156.
Vargas, L.G., 1990, An overview of the analytic hierarchy process and its applications,
European Journal of Operational Research 48, 2 ± 8.
Walden, I. and A. Braganza, 1993, EDI: audit and control (NCC Blackwell,
Manchester).
Wallace, R.S.O. and C.J. Mellor, 1988, Nonresponse bias in mail accounting surveys: a
pedagogical note, British Accounting Review 20, 131± 139.
Weber, R., 1980, Some characteristics of the free recall of computer controls by EDP
auditors, Journal of Accounting Research 18, 214± 241.
Weber, R., 1988, EDP auditing, conceptual foundations and practice 2nd ed., (McGraw-
Hill, New York).
Weiner, S., 1995, Business risk, internal control and audit implications of EDI, The
CPA Journal 65, 56 ± 61.
Yau, C. and T. Davis, 1993, Using the analytic hierarchy process (AHP) to prioritise
auditing tasks for large scale software systems, Journal of Systems Management 44,
26 ± 31.
Zahedi F., 1986, The analytic hierarchy processÐ a survey of the method and its
applications, Interfaces 16, 96 ± 108.

# AAANZ, 2000