Download the full version of the textbook now at textbookfull.

com

XcalableMP PGAS Programming Language From

Programming Model to Applications Mitsuhisa
Sato

https://textbookfull.com/product/xcalablemp-pgas-
programming-language-from-programming-model-to-
applications-mitsuhisa-sato/

Explore and download more textbook at https://textbookfull.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Programming Language Concepts Peter Sestoft

https://textbookfull.com/product/programming-language-concepts-peter-
sestoft/

textbookfull.com

ADA Programming Everything you need to know about ADA

Programming language 2nd Edition Claudia Alves

https://textbookfull.com/product/ada-programming-everything-you-need-
to-know-about-ada-programming-language-2nd-edition-claudia-alves/

textbookfull.com

Programming 101: The How and Why of Programming Revealed

Using the Processing Programming Language 1st Edition
Jeanine Meyer
https://textbookfull.com/product/programming-101-the-how-and-why-of-
programming-revealed-using-the-processing-programming-language-1st-
edition-jeanine-meyer/
textbookfull.com

Wordsworth and Coleridge: The Radical Years 2nd Edition

Nicholas Roe

https://textbookfull.com/product/wordsworth-and-coleridge-the-radical-
years-2nd-edition-nicholas-roe/

textbookfull.com
From Alienation to Forms of Life The Critical Theory of
Rahel Jaeggi Amy Allen

https://textbookfull.com/product/from-alienation-to-forms-of-life-the-
critical-theory-of-rahel-jaeggi-amy-allen/

textbookfull.com

The 'Mere Irish' and the Colonisation of Ulster, 1570-1641

1st Edition Gerard Farrell (Auth.)

https://textbookfull.com/product/the-mere-irish-and-the-colonisation-
of-ulster-1570-1641-1st-edition-gerard-farrell-auth/

textbookfull.com

Trends in Policing: Interviews with Police Leaders Across

the Globe, Volume Five 1 Edition Edition Baker

https://textbookfull.com/product/trends-in-policing-interviews-with-
police-leaders-across-the-globe-volume-five-1-edition-edition-baker/

textbookfull.com

The Right s First Amendment The Politics of Free Speech

the Return of Conservative Libertarianism Wayne Batchis

https://textbookfull.com/product/the-right-s-first-amendment-the-
politics-of-free-speech-the-return-of-conservative-libertarianism-
wayne-batchis/
textbookfull.com

Introduction To Embedded Systems 2nd Edition K. V Shibu

https://textbookfull.com/product/introduction-to-embedded-systems-2nd-
edition-k-v-shibu/

textbookfull.com
Plant Genome Diversity Volume 2 Physical Structure
Behaviour and Evolution of Plant Genomes 1st Edition Prof.
Pamela S. Soltis
https://textbookfull.com/product/plant-genome-diversity-
volume-2-physical-structure-behaviour-and-evolution-of-plant-
genomes-1st-edition-prof-pamela-s-soltis/
textbookfull.com
Mitsuhisa Sato Editor

XcalableMP PGAS
Programming
Language
From Programming Model
to Applications
XcalableMP PGAS Programming Language
Mitsuhisa Sato
Editor

XcalableMP PGAS
Programming Language
From Programming Model to Applications
Editor
Mitsuhisa Sato
Programming Envt Research Team
RIKEN Center for Computational Science
Kobe, Hyogo, Japan

ISBN 978-981-15-7682-9 ISBN 978-981-15-7683-6 (eBook)

https://doi.org/10.1007/978-981-15-7683-6

This book is an open access publication.

© The Editor(s) (if applicable) and The Author(s) 2021

Open Access This book is licensed under the terms of the Creative Commons Attribution 4.0 Inter-
national License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation,
distribution and reproduction in any medium or format, as long as you give appropriate credit to the
original author(s) and the source, provide a link to the Creative Commons license and indicate if changes
were made.
The images or other third party material in this book are included in the book’s Creative Commons
license, unless indicated otherwise in a credit line to the material. If material is not included in the book’s
Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the
permitted use, you will need to obtain permission directly from the copyright holder.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface

XcalableMP is a directive-based parallel programming language based on Fortran

and C, supporting the PGAS (partitioned global address space) model for distributed
memory parallel systems. PGAS is a programming model related to distributed
memory system with a shared address space that distinguishes between local (cheap)
and remote (expensive) memory access. It provides an easy and intuitive program-
ming model to describe remote data access. PGAS is an emerging programming
model for exascale computing.
The XcalableMP project has been started in 2008 as a part of MEXT e-science
project, led by Prof. Yutaka Ishikawa, University of Tokyo, Japan. The language
specification working group were organized by Mitsuhisa Sato, University of
Tsukuba, with members from academia, government laboratories, and industries,
who are interested in parallel programming languages. In 2011, version 1.0 of
XcalableMP was published. Since 2011, after the e-science project was ended, the
specification working group has been managed under the PC Cluster Consortium,
Japan. At the same time, the Omni XcalableMP compiler has been developed as
a production-level reference implementation of XcalableMP compiler for C and
Fortran 2008 by RIKEN CCS and University of Tsukuba.
The XcalableMP project has taken an evolutional approach with the following
strategies:
• We focus on migration from existing codes: To facilitate the migration, we
took the directive-based approach to enable parallelization by adding direc-
tives/pragma in the global view programming. As a local view programming,
the coarray feature was supported to replace the MPI communication for the
migration from the MPI.
• Learn from the past: In the past, the Japanese HPC language community had a
lot of experience on High-Performance Fortran. This experience provided useful
insights for the design of the global view programming model.
• Specification designed by community: Programming languages should be shared
by many programmers. For the design of the language specification, the specifi-
cation working group has been organized to collect the idea from the community.

v
vi Preface

Currently, the working group is organized under the PC Cluster Consortium,

Japan.
• Design based on the PGAS model and Coarray Fortran: In XcalableMP, the
PGAS model is adopted as the main programming model since it is an emerging
programming model for exascale computing. In addition, we extend it with the
idea taken from HPF for global view programming. Coarray feature is taken as a
local view programming.
• Used as a research vehicle for researches for programming language: As
advanced researches, an extension to accelerator, XcalableACC, and global task-
parallel programming for XcalableMP 2.0 are explored based on the XcalableMP
language.
This book presents XcalableMP language from its programming model and basic
concept to the experience and performance of applications described in XcalableMP
and some extended research projects using XcalableMP.
Chapter “XcalableMP Programming Model and Language” presents the
overview of XcalableMP programming model and language, followed by
implementation and performance evaluation of a reference prototype compiler
and Omni XcalableMP compiler, in Chapter “Implementation and Performance
Evaluation of Omni Compiler”. Chapter “Coarrays in the Context of XcalableMP”
presents how to design and implement the Coarray feature in the XcalableMP
compiler.
XcalableACC is an extended project to integrate the OpenACC with XcalableMP
for the programming of the cluster with accelerators. Chapter “XcalableACC: An
Integration of XcalableMP and OpenACC” describes the XcalableACC with the
implementation and performance evaluation.
Chapter “Mixed-Language Programming with XcalableMP” presents the mixed-
language programming with XcalableMP. It demonstrates how to use XcalableMP
with other programming languages such as Python. The global view programming
of XcalableMP provides very simple and intuitive programming tools to describe a
part of program controlled by Python.
Chapters “Three-Dimensional Fluid Code with XcalableMP”, “Hybrid-View
Programming of Nuclear Fusion Simulation Code in XcalableMP” and “Paralleliza-
tion of Atomic Image Reconstruction from X-ray Fluorescence Holograms with
XcalableMP” present applications experiences, “Three-Dimensional Fluid Code,”
“Nuclear Fusion Simulation Code,” and “Atomic Image Reconstruction from X-ray
Fluorescence Holograms” described in XcalableMP.
Chapter “Multi-SPMD Programming Model with YML and XcalableMP”
presents the international collaboration with French and German partners.
Framework and Programming for Post-Petascale Computing (FP3C) project
conducted during September 2010–March 2013 aimed to exploit efficient
programming and method for future supercomputers. In the FP3C project, the
mSPMD programming had been proposed with the integration of the XcalableMP
and YML workflow programming environment. The priority program “Software for
Exascale Computing” (SPPEXA) by the collaboration with three countries, Japan,
Preface vii

Germany, and France, had been conducted to address fundamental research on the
various aspects of HPC software during 2016–2018 (phase II). The project “MUST
Correctness Checking for YML and XMP Programs (MYX)” had been selected.
The correctness checking program, MUST is applied to the XcalableMP program.
XcalableMP was taken as a parallel programming language project in FLAG-
SHIP 2020 project which was to develop the Japanese flagship supercomputer,
Fugaku, for improving the productivity and performance of parallel programming.
XcalableMP is now available on Fugaku and the performance is enhanced by the
Fugaku interconnect, Tofu-D. Chapter “XcalableMP 2.0 and Future Directions”
presents the current status of XcalableMP on Fugaku and a proposal for the next
version, XcalableMP 2.0. I conclude this book with challenges for future PGAS
models and some retrospectives about XcalableMP.
I hope this book will provide useful information to the XcalableMP program-
mers. Furthermore, for future exascale computing and beyond, an important role
of programming model is to bridge the gap between a new architecture and
programmer’s view as well as providing a comprehensive architectural view of the
system. I believe that the PGAS model and its extensions will be still an important
and suggestive model integrating computation and memory.
Finally, I would like to thank the members of the XcalableMP Specification
Working Group and all the people who contributed to the XcalableMP project.

Kobe, Japan Mitsuhisa Sato

July 2020
Contents

XcalableMP Programming Model and Language .. . . . . . .. . . . . . . . . . . . . . . . . . . . 1

Hitoshi Murai, Masahiro Nakao, and Mitsuhisa Sato
Implementation and Performance Evaluation of Omni Compiler.. . . . . . . . . 73
Masahiro Nakao and Hitoshi Murai
Coarrays in the Context of XcalableMP .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 97
Hidetoshi Iwashita and Masahiro Nakao
XcalableACC: An Integration of XcalableMP and OpenACC .. . . . . . . . . . . . . 123
Akihiro Tabuchi, Hitoshi Murai, Masahiro Nakao, Tetsuya Odajima,
and Taisuke Boku
Mixed-Language Programming with XcalableMP .. . . . . .. . . . . . . . . . . . . . . . . . . . 147
Masahiro Nakao
Three-Dimensional Fluid Code with XcalableMP . . . . . . . .. . . . . . . . . . . . . . . . . . . . 165
Hitoshi Sakagami
Hybrid-View Programming of Nuclear Fusion Simulation Code
in XcalableMP .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 181
Keisuke Tsugane, Taisuke Boku, Hitoshi Murai, Mitsuhisa Sato,
William Tang, and Bei Wang
Parallelization of Atomic Image Reconstruction from X-ray
Fluorescence Holograms with XcalableMP . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 205
Atsushi Kubota, Tomohiro Matsushita, and Naohisa Happo
Multi-SPMD Programming Model with YML and XcalableMP . . . . . . . . . . . 219
Miwako Tsuji, Hitoshi Murai, Taisuke Boku, Mitsuhisa Sato,
Serge G. Petiton, Nahid Emad, Thomas Dufaud, Joachim Protze,
Christian Terboven, and Matthias S. Müller
XcalableMP 2.0 and Future Directions . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 245
Mitsuhisa Sato, Hitoshi Murai, Masahiro Nakao, Keisuke Tsugane,
Tesuya Odajima, and Jinpil Lee
ix
XcalableMP Programming Model
and Language

Hitoshi Murai, Masahiro Nakao, and Mitsuhisa Sato

Abstract XcalableMP (XMP) is a directive-based language extension of Fortran

and C for distributed-memory parallel computers, and can be classified as a parti-
tioned global address space (PGAS) language. One of the remarkable characteristics
of XMP is that it supports both global-view and local-view parallel programming.
This chapter describes the programming model and language specification of XMP.

1 Introduction

Distributed-memory systems are generally used for large-scale simulations. To pro-

gram such systems, Message Passing Interface (MPI) is widely adopted. However,
programming with MPI is difficult because programmers must describe inter-
process communications with consideration of the execution flow of their programs,
which might cause deadlocks or wrong results.
To address this issue, a parallel language named High Performance Fortran (HPF)
was proposed in 1991. With HPF, programmers can execute their serial programs in
parallel by inserting minimal directives into them. If the programmers specify data
distribution with HPF directives, the compilers do all other tasks for parallelization
(e.g. communication generation and work distribution). However, HPF was not
widely accepted eventually because the compilers’ automatic processing prevents
the programmers from performance tuning, and the performance depends heavily
on the environment (e.g. compiler and hardware)

Note For more details, please refer: Ken Kennedy, Charles Koelbel and Hans Zima:
The Rise and Fall of High Performance Fortran: An Historical Object Lesson, Proc.
3rd ACM SIGPLAN History of Programming Languages Conf. (HOPL-III), pp. 7-
1–7-22 (2007).

H. Murai () · M. Nakao · M. Sato

RIKEN Center for Computational Science, Kobe, Hyogo, Japan
e-mail: [email protected]; [email protected]; [email protected]

© The Author(s) 2021 1

M. Sato (ed.), XcalableMP PGAS Programming Language,
https://doi.org/10.1007/978-981-15-7683-6_1
2 H. Murai et al.

In such circumstance, to develop a new parallel programming model that enables

easy parallelization of existing serial programs and design a new language based on
it, “the XMP Specification Working Group” was established in 2008. This group
utilized the lessons from the experience of HPF to define a new parallel language
XcalableMP (XMP). The group was reorganized to one of the working groups of PC
Cluster Consortium in 2011.
It is learned from the lessons of HPF that more automatic processing of compilers
increases the gap between a program and its execution, and, as a result, decreases the
usability of the language. In XMP, the programmers specify explicitly the details of
parallel programs on the basis of compiler directives to make their execution easy to
understand. In particular, they can specify explicitly communication, synchroniza-
tion, data mapping, and work mapping to facilitate performance tuning. In addition,
XMP supports features for one-sided communication on each process, which was
not available in HPF. This feature might enable programmers to implement parallel
algorithms more easily.
In this chapter, an overview of the programming model and language specifica-
tion of XMP is shown. You can find the latest and complete language specification
of XMP in: XcalableMP Specification Working Group, XcalableMP Specification
Version 1.4, http://xcalablemp.org/download/spec/xmp-spec-1.4.pdf (2018).

1.1 Target Hardware

The target of XcalableMP is distributed-memory multicomputers (Fig. 1). Each

compute node, which may contain several cores, has its own local memory (shared
by the cores, if any), and is connected with the others via an interconnection
network. Each node can access its local memory directly and remote memory (the
memory of another node) indirectly (i.e. via inter-node communication). However,

Fig. 1 Target hardware of XMP

Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
XcalableMP Programming Model and Language 3

it is assumed that accessing remote memory may be much slower than accessing
local memory.

1.2 Execution Model

The execution entities in an XMP program are referred to as XMP nodes or, more
simply, nodes, which has its own memory and can communicate with each other.
An XcalableMP program execution is based on the Single Program Multiple
Data (SPMD) model, where each node starts execution from the same main routine,
and continues to execute the same code independently (i.e. asynchronously) until it
encounters an XcalableMP construct (Fig. 2).
A set of nodes that executes a procedure, statement, loop, a block, etc. is referred
to as its executing node set, and is determined by the innermost task, loop, or
array directive surrounding it dynamically, or at runtime. The current executing
node set is an executing node set of the current context, which is managed by the
XcalableMP runtime system on each node.
The current executing node set at the beginning of the program execution, or
entire node set, is a node set that contains all the available nodes, which can be
specified in an implementation-defined way (e.g. through a command-line option).
When a node encounters at runtime either a loop, array, or task construct,
and is contained by the node set specified (explicitly or implicitly) by the on clause

Fig. 2 Execution model of

XMP
4 H. Murai et al.

of the directive, it updates the current executing node set with the specified one
and executes the body of the construct, after which it resumes the last executing
node set and proceeds to execute the subsequent statements.
In particular, when a node in the current executing node set encounters a loop
or an array construct, it executes the loop or the array assignment in parallel with
the other nodes, so that each iteration of the loop or element of the assignment is
independently executed by the node in which the specified data element resides.
When a node encounters a synchronization or a communication directive,
synchronization or communication occurs between it and the other nodes. That is,
such global constructs are performed collectively by the current executing nodes.
Note that neither synchronization nor communication occurs unless these constructs
are specified.

1.3 Data Model

There are two classes of data in XcalableMP: global data and local data. Data
declared in an XcalableMP program are local by default.
Global data are distributed onto a node set by the align directive (see
Sect. 2.4). Each fragment of distributed global data is allocated in the local memory
of a node in the node set.
Local data comprises all data that are not global. They are replicated within the
local memory of each of the executing nodes.
A node can access directly only local data and sections of global data that reside
in its local memory. To access data in remote memory, explicit communication
must be specified in such ways as global communication constructs and coarray
assignments (Fig. 3).

Fig. 3 Data model of XMP

XcalableMP Programming Model and Language 5

1.4 Programming Models

1.4.1 Partitioned Global Address Space

XMP can be classified as a partitioned global address space (PGAS) language, such
as Co-Array Fortran [1], Unified Parallel C [2], and Chapel [3].
In such PGAS languages, multiple executing entities (i.e. threads, processes, or
nodes in XMP) share a part of their address space, which is, however, partitioned
and a portion of which is local to each executing entity.
The two programming models, global-view and local-view, that XMP supports
to achieve high performance and productivity on PGAS are explained below.

1.4.2 Global-View Programming Model

The global-view programming model is useful when, starting from a serial version
of a program, the programmer parallelizes it in a data-parallel style by adding
directives with minimum modification. Based on this model, the programmer
specifies the distribution of data among nodes using the data distribution directives.
The loop construct assigns each iteration of a loop to the node at which the
computed data is located. The global-view communication directives are used to
synchronize nodes, maintain the consistency of shadow areas of distributed data,
and move sections of distributed data globally. Note that the programmer must
specify explicitly communication to make all data references in their program local
using appropriate directives.
In many cases, the XcalableMP program following the global-view programming
model is based on a serial program, and it can produce the same result, regardless
of the number of nodes (Fig. 4).
There are three groups of directives for this model:
• Data mapping, which specifies the data distribution and mapping to nodes
• Work mapping (parallelization), which specifies the work distribution and
mapping to nodes.
• Communication and synchronization, which specify how a node communicates
and synchronizes with the other nodes.
Because these directives are ignored as a comment by the compilers of base
languages (Fortran and C), an XcalableMP program can usually be compiled by
them to ensure that they run properly.

1.4.3 Local-View Programming Model

The local-view programming model is suitable for programs that implement an

algorithm and a remote data reference that are to be executed by each node (Fig. 5).
6 H. Murai et al.

Fig. 4 Parallelization based on the global-view programming model

Fig. 5 Local-view programming model

For this model, some language extensions and directives are provided. The
coarray notation, which is imported from Fortran 2008, is one such extension,
and can be used to explicitly specify data on which node is to be accessed. For
example, the expression of A(i)[N] in XcalableMP Fortran is used to access an
XcalableMP Programming Model and Language 7

array element of A(i) located on the node N. If the access is a reference, then
a one-sided communication to read the value from the remote memory (i.e. the
get operation) is issued by the executing node. If the access is a definition, then
a one-sided communication to write the value to the remote memory (i.e. the put
operation) is issued by the executing node.

1.4.4 Mixture of Global View and Local View

In the global-view model, nodes are used to distribute data and works. In the local-
view model, nodes are used to address remote data in the coarray notation. In
application programs, the programmers should choose an appropriate data model
according to the characteristics of their program. Figure 6 illustrates the global view
and the local view of data.
Data can have both a global view and a local view, and can be accessed in both of
the views. XcalableMP provides a directive to give the local name (alias) to global
data declared in the global-view programming model to enable them to also be
accessed in the local-view programming model. This feature is useful to optimize
a certain part of a program by using explicit remote data access in the local-view
programming model.

1.5 Base Languages

The XcalableMP language specification is defined on the basis of Fortran and C as

the base languages. More specifically, the base language of XcalableMP Fortran is
Fortran 90 or later, and that of XcalableMP C is ISO C90 (ANSI C89) or later with
some extensions (see below).

1.5.1 Array Section in XcalableMP C

In XcalableMP C, the base language C is extended so that a part of an array, that is,
an array section or subarray, can be put in an array assignment statement, which
is described in Sect. 1.5.2, and some XcalableMP constructs. An array section is
built from a subset of the elements of an array, which is specified by a sequence of
square-bracketed integer expressions or triplets, which are in the form of:
[ base ] : [ length ] [ : step ]
When step is positive, the triplet specifies a set of subscripts that is a regularly
spaced integer sequence of length length beginning with base and proceeding in
increments of step up to the largest. The same applies to negative step too.
When base is omitted, it is assumed to be 0. When length is omitted, it is assumed
to be the number of remainder elements of the dimension of the array. When step is
omitted, it is assumed to be 1.
8 H. Murai et al.

Fig. 6 Global view and local view

Assuming that an array A is declared by the following statement,

int A[100];
some array sections can be specified as follows:
A[10:10] array section of 10 elements from A[10] to A[19]
A[10:] array section of 90 elements from A[10] to A[99]
A[:10] array section of 10 elements from A[0] to A[9]
A[10:5:2] array section of 5 elements from A[10] to A[18] by step 2
A[:] array section of the whole of A
XcalableMP Programming Model and Language 9

1.5.2 Array Assignment Statement in XcalableMP C

In XcalableMP C, the base language C is also extended so that it supports array

assignment statements just as Fortran does.
With such statement, the value of each element of the result of the right-hand side
expression is assigned to the corresponding element of the array section on the left-
hand side. When an operator or an elemental function is applied to array sections in
the right-hand side expression, it is evaluated to an array section that has the same
shape as that of the operands or arguments, and each element of which is the result
of the operator or function applied to the corresponding element of the operands or
arguments. A scalar object is assumed to be an array section that has the same shape
as that of the other array section(s) in the expression or on the left-hand side, and
where each element has its value.
Note that an array assignment is a statement, and therefore cannot appear as an
expression in any other statements.
In the example below, an array assignment statement in the fourth line copies the
five elements from B[0] to B[4] into the elements from A[5] to A[9].
XcalableMP C
int A[10];
int B[5];
...
A[5:5] = B[0:5];

1.6 Interoperability

Most of the existing parallel applications are written with MPI. It is not realistic to
port them over to XMP because each of them consists of millions of lines.
Because XMP is interoperable with MPI, users can develop an XMP application
by modifying a part of an existing one instead of rewriting it totally. Besides, when
developing a parallel application from scratch, it is possible to use XMP to write a
complicated part of, for example, domain decomposition while they use MPI, which
could be faster than XMP, to write a hot-spot part that need to be tuned carefully. In
addition, XMP is interoperable with OpenMP and Python (see Chap. 5).
It might be difficult to develop an application with just one programming
language or framework since it generally has its own strong and weak points. Thus,
an XMP program is interoperable with those in other languages to provide both high
productivity and performance.
10 H. Murai et al.

2 Data Mapping

2.1 nodes Directive

The nodes directive declares a node array, which is an array-like arrangement of

nodes in a node set. A node array can be multi-dimensional.
XcalableMP C
#pragma xmp nodes p[4]
XcalableMP Fortran
!$xmp nodes p(4)

The nodes directive declares a one-dimensional node array p that includes four
nodes. In XMP/C, it is zero-based and consists of p[0], p[1], p[2], and p[3].
In XMP/Fortran, it is one-based and consists of p(1), p(2), p(3), and p(4).
XcalableMP C
#pragma xmp nodes p[2][3]
XcalableMP Fortran
!$xmp nodes p(3,2)

The nodes directive declares two-dimensional node array p that includes six
nodes. In XMP/C, it consists of p[0][0], p[0][1], p[0][2], p[1][0],
p[1][1], and p[1][2]. In XMP/Fortran, it consists of p(1,1), p(2,1),
p(3,1), p(1,2), p(2,2), and p(3,2).

Note The ordering of the elements in a node array follows that of a normal array
in the base language, C or Fortran.

XcalableMP C
#pragma xmp nodes p[*]
XcalableMP Fortran
!$xmp nodes p(*)

An asterisk can be specified as the size in the nodes directive to declare a

dynamic node array. In the above code, one-dimensional dynamic node array p
is declared with an asterisk as the size. The actual size of a dynamic node array is
determined at runtime to fit the size of the current executing node set. For example,
when the programmer runs the sample code with three nodes, the node array p
includes three nodes.
The programmer can also declare multi-dimensional dynamic node arrays with
an asterisk.
XcalableMP C
#pragma xmp nodes p[*][3]
XcalableMP Programming Model and Language 11

XcalableMP Fortran
!$xmp nodes p(3,*)

When the programmer runs the sample code with 12 nodes, the node array p
has a shape of 4 × 3, in C, or 3 × 4, in Fortran.

Note The programmer can put an asterisk only in the last dimension, in XMP/For-
tran, or the first dimension, in XMP/C, of the node array.

Hint The dynamic node array may interfere with compiler optimizations. In
general, programs with static ones achieve better performance.

The programmer can declare a node subarray derived from an existing node array.
Node subarrays can be used, for example, to optimize inter-node communication by
reducing the number of nodes participating in the communication.
XcalableMP C
#pragma xmp nodes p[16]
#pragma xmp nodes q[8]=p[0:8]
#pragma xmp nodes r[4][2]=p[8:8]
XcalableMP Fortran
!$xmp nodes p(16)
!$xmp nodes q(8)=p(1:8)
!$xmp nodes r(2,4)=p(9:16)

In line 1, a node array p including 16 nodes is declared. In line 2, a node

subarray q corresponding to the first half of p is declared. In line 3, a two-
dimensional node subarray r corresponding to the latter half of p is declared.
The programmer can declare an n-dimensional node subarray derived from an
m-dimensional one (Fig. 7).
XcalableMP C
#pragma xmp nodes p[4][2]
#pragma xmp nodes row[4]=p[:][*]
#pragma xmp nodes col[2]=p[*][:]
XcalableMP Fortran
!$xmp nodes p(2,4)
!$xmp nodes row(4)=p(*,:)
!$xmp nodes col(2)=p(:,*)

In line 1, a two-dimensional node array p including 4 × 2 nodes is declared. In

line 2, a node subarray row derived from a single row of p is declared. In line 3, a
node subarray col derived from a single column of p is declared.
12 H. Murai et al.

Fig. 7 Node subarrays

A colon represents a triplet which indicates all possible indices in the dimension.
An asterisk indicates the index of the current executing node in the dimension.
For example, col[2] corresponds to p[0][0:2] on nodes p[0][0] and
p[0][1], and to p[1][0:2] on nodes p[1][0] and p[1][1] in XMP/C.
Similarly, col(2) corresponds to p(1:2,1) on nodes p(1,1) and p(2,1),
and to p(1:2,2) on nodes p(1,2) p(2,2) in XMP/Fortran.
In XMP/C, row[0] corresponds to p[0][0] and p[0][1] on p[:][0]
and p[:][1], respectively; col[0] corresponds to p[0][0], p[1][0],
p[2][0], and p[3][0] on p[0][:], p[1][:], p[2][:], p[3][:],
respectively. In XMP/Fortran, row(1) corresponds to p(1,1) and p(2,1) on
p(1,:) and p(2,:), respectively; col(1) corresponds to p(1,1), p(1,2),
p(1,3), and p(1,4) on p(:,1), p(:,2), p(:,3), p(:,4), respectively.

Note The semantics of an asterisk in a node reference is different from that in a

declaration.
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
XcalableMP Programming Model and Language 13

2.2 template Directive

The template directive declares a template, which is a virtual array that is used
as a “template” of parallelization in the programs and to be distributed onto a node
array.
XcalableMP C
#pragma xmp template t[10]
XcalableMP Fortran
!$xmp template t(10)

This template directive declares a one-dimensional template t having ten

elements. Templates are indexed in the similar manner to arrays in the base
languages. For the above examples, the template t is indexed from zero to nine (i.e.
t[0] · · · t[9]), in XMP/C, or one to ten (i.e. t(1) · · · t(10)), in XMP/Fortran.

Hint In many cases, a template should be declared to have the same shape as your
target array.

XcalableMP C
#pragma xmp template t[10][20]
XcalableMP Fortran
!$xmp template t(20,10)

The template directive declares a two-dimensional template t that has 10 ×

20 elements. In XMP/C, t is indexed from t[0][0] to t[9][19], and, in XMP/Fortran,
from t(1,1) to t(20,10).
XcalableMP C
#pragma xmp template t[:]
XcalableMP Fortran
!$xmp template t(:)

In the above examples, a colon instead of an integer is specified as the size to

declare a one-dimensional dynamic template t. The colon indicates that the size
of the template is not fixed and to be fixed at runtime by the template_fix
construct (Sect. 2.6).

2.3 distribute Directive

The distribute directive specifies a distribution of the target template. Either of

block, cyclic, block-cyclic, or gblock (i.e. uneven block) can be specified to distribute
a dimension of a template.
14 H. Murai et al.

2.3.1 Block Distribution

XcalableMP C
#pragma xmp distribute t[block] onto p
XcalableMP Fortran
!$xmp distribute t(block) onto p

The target template t is divided into contiguous blocks and distributed among
nodes in the node array p (Fig. 8). Let’s suppose that the size of the template is
N and the number of nodes is K. If N is divisible by K, a block of size N/K is
assigned to each node; otherwise, a block of size ceil(N/K) is assigned to each of
N/ceil(N/K) nodes, a block of size mod(N, K) to one node, and no block to (K −
N/ceil(N/K) − 1) nodes. The block distribution is useful for regular computations
such as a stencil one.

Note The function ceil(x) returns a minimum integer value greater than or equal to
x, and mod(x, y) returns x modulo y.

XcalableMP C
#pragma xmp nodes p[3]
#pragma xmp template t[22]
#pragma xmp distribute t[block] onto p
XcalableMP Fortran
!$xmp nodes p(3)
!$xmp template t(22)
!$xmp distribute t(block) onto p

Since ceil(22/3) is 8, eight elements are allocated on each of p[0] and p[1],
and the remaining six elements are allocated on p[2].

XMP/C
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

p[0], p(1)
XMP/Fortran
p[1], p(2)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
p[2], p(3)

Fig. 8 Block distribution

XcalableMP Programming Model and Language 15

XMP/C
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

p[0], p(1)
XMP/Fortran
p[1], p(2)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
p[2], p(3)

Fig. 9 Cyclic distribution

2.3.2 Cyclic Distribution

XcalableMP C
#pragma xmp distribute t[cyclic] onto p
XcalableMP Fortran
!$xmp distribute t(cyclic) onto p

The target template t is divided into chunks of size one and distributed among
nodes in the node array p in a round-robin manner (Fig. 9). The cyclic distribution
is useful for the case where the load on each element of the template is not balanced.
XcalableMP C
#pragma xmp nodes p[3]
#pragma xmp template t[22]
#pragma xmp distribute t[cyclic] onto p
XcalableMP Fortran
!$xmp nodes p(3)
!$xmp template t(22)
!$xmp distribute t(cyclic) onto p

2.3.3 Block-Cyclic Distribution

XcalableMP C
#pragma xmp distribute t[cyclic(w)] onto p
XcalableMP Fortran
!$xmp distribute t(cyclic(w)) onto p

The target template t is divided into chunks of size w and distributed among
nodes in the node array p in a round-robin manner (Fig. 10). The block-cyclic
distribution is useful for the case where the load on each element of the template is
not balanced but the locality of the elements is required.
16 H. Murai et al.

XMP/C
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

p[0], p(1)
XMP/Fortran
p[1], p(2)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
p[2], p(3)

Fig. 10 Block-cyclic distribution

XcalableMP C
#pragma xmp nodes p[3]
#pragma xmp template t[22]
#pragma xmp distribute t[cyclic(3)] onto p
XcalableMP Fortran
!$xmp nodes p(3)
!$xmp template t(22)
!$xmp distribute t(cyclic(3)) onto p

2.3.4 Gblock Distribution

XcalableMP C
#pragma xmp distribute t[gblock(W)] onto p
XcalableMP Fortran
!$xmp distribute t(gblock(W)) onto p

The target template t is divided into contiguous blocks of size W[0], W[1], · · · ,
in XMP/C, or W(1), W(2), · · · , in XMP/Fortran, and distributed among nodes in
the node array p (Fig. 11). The array W is called a mapping array. The programmer
can specify irregular (uneven) block distribution with the gblock format.
XcalableMP C
#pragma xmp nodes p[3]
#pragma xmp template t[22]
int W[3] = {6, 11, 5};
#pragma xmp distribute t[gblock(W)] onto p
XcalableMP Fortran
!$xmp nodes p(3)
!$xmp template t(22)
integer, parameter :: W(3) = (/6,11,5/)
!$xmp distribute t(gblock(W)) onto p
XcalableMP Programming Model and Language 17

XMP/C
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

p[0], p(1)
XMP/Fortran
p[1], p(2)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
p[2], p(3)

Fig. 11 Gblock distribution

The programmer can specify an asterisk instead of a mapping array in the

gblock distribution to defer fixing the actual distribution. In such a case, the actual
distribution will be fixed at runtime by using the template_fix construct.

2.3.5 Distribution of Multi-Dimensional Templates

The programmer can distribute a multi-dimensional template onto a node array.

XcalableMP C
#pragma xmp nodes p[2][2]
#pragma xmp template t[10][10]
#pragma xmp distribute t[block][block] onto p
XcalableMP Fortran
!$xmp nodes p(2,2)
!$xmp template t(10,10)
!$xmp distribute t(block,block) onto p

The distribute directive declares the distribution of a two-dimensional

template t onto a two-dimensional node array p. Each dimension of the template
is divided in a block manner and each of the rectangular region is assigned to a node
(Fig. 12).
The programmer can specify a different distribution format in each of the
dimension of a template (Fig. 13).
XcalableMP C
#pragma xmp nodes p[2][2]
#pragma xmp template t[10][10]
#pragma xmp distribute t[block][cyclic] onto p
XcalableMP Fortran
!$xmp nodes p(2,2)
!$xmp template t(10,10)
!$xmp distribute t(cyclic,block) onto p
Random documents with unrelated
content Scribd suggests to you:
were worked by canoes, which would take one end out into mid
harbour, the other being made fast, and sweep the great kupenga
round the shoals of fish making their way up the harbour with the
flood tide. Huge quantities of sharks and other fish were caught in
this manner, a fishing fashion which was only possible under the old
tribal system, when the whole strength of the hapu was available for
such tasks.
Lest readers should question the dimensions of that quarter of a
mile seine, let me say that in quite recent times, up to within the last
thirty-five years or so, an old Ngati-Pikiao chief, the late Pokiha
Taranui (better known as Major Fox), had a net nearly a mile in
length, which was used on special occasions, such as the gathering
of food for native meetings; the locality was Maketu village, on the
shores of the Bay of Plenty. But those enormous kupengas will never
again be hauled through the fish-teeming waters.
From Tikao, too, we hear something of the poetic legend and
nature-myth that steep those swart hills above Rapaki. Yon savage
Poho of rhyolite, and the peaked and pinnacled cores of old
volcanoes that break through the grassy hills for mile after mile, all
have their tales of pre-pakeha years, of which we shall chronicle
something again. Just now we may content ourselves with the
gentler scenes in Rapaki valley, where the kowhai has shed its
showers of gold and the pakeha fruit trees in blossom sweeten the
soft air deliciously. That patch of ploughed land behind the
settlement before long will show the first shoots of kaanga , or
maize; there is not much grown in this island except by the Natives.
The water-front, in spite of its smart new jetty for the launches, is
lonelier than in the old days; for there was a time, long after the
war-canoe era, when three long whaleboats were hauled up on the
sand where the boulders are piled aside yonder and often these
could be seen pulling down to Lyttelton laden with potatoes, corn
and fruit. That bit of beach is over-rough; but a little way to the
north, under the lee of a wild bit of a rocky headland thick with
beautiful light bush, is a gem of a white beach, clean and hard and
shining, a sandy alcove that must have been made for picnicking.
And from this hillside turn in the road where we get our first glimpse
of Rapaki, we may also most fitly take our sunset farewell of the
kainga of an artist’s dream.
The sun is over the range, and Tamatea’s gloomy peak is outlined in
sharp symmetry against the burning west. In the deep gullies
between the spear-head and the ridge of the Sacred Fire the smoky-
blue mists are already forming, and wreathing and creeping around
the tangled shrubberies of bush that have escaped the general
massacre. The harbour lies a sheet of scarcely moving tender
turquoise, just a shade lighter than the face of the famous Tikitapu,
inaccurately called the Blue Lake by the pakeha ; high beyond the
shark’s-teeth of the peaks that someone has named the Seven
Sleepers are drawn in soft blue upon the rose of the heaven above,
their feet are bathed in violet, and the purple mists swim wraith-like
from their hidden hollows. The sun goes, and the delicacy, the
tenderness of colour, the fading of landscape details into a haze like
the camp-fire smokes of the legendary Patu-paiarehe , weave a veil
of faery over the valley and the darkening sea. The little boys and
girls of the settlement are still at play around the meeting-hall, and
every call and every laugh come clearly through the velvet soft air.
Down in a nearer dwelling, a girl is rehearsing a poi -dance song,
and the lilt is familiar, the half-sad chant that begins,

“Hoki-hoki tonu mai,

Te wairua o te tau.”
(“Return, return again to me,
The spirit of my love.”)

It is the song crooned by the women and the children in every

kainga that, like slumbrous little Rapaki down yonder, sent its young
men to use rifle and bayonet beside their pakeha brothers-in-arms
on the thundering battlefields half a world away.
 Chapter V.
THE “AHI-A-TAMATEA.”
HOW THE SACRED FIRE CAME TO WITCH HILL.
As we travel northward along the Summit Track from the Poho-o-
Tamatea, observing from this commanding height—a thousand feet
above little Rapaki village, lying in its grassy nest below—how that
great spearhead of rock has in reality an almost level top, we come
to a remarkable broken wall of grey lava moss-crusted and shrub-
tufted, protruding from the grassy flanks of the craggy knoll called
Witch Hill. It is in fact a huge dyke of once-molten lava, cutting wall-
like through the old lava flows and trending southward across the
shoulder of its parent hill. From the very crest of the range it shoots
in a palisade of frost-shattered rock, towering thirty feet and more in
places above the stone-strewn tussocks, and it stretches some
distance in irregular cyclopean steps down the steep slope towards
Rapaki. There is an old stone quarry on the face of Witch Hill, where
this dyke juts out like a broken castle wall of the Patu-paiarehe , the
Dim People of long ago, and just there the old waggon track goes
down over the smoothed-out hills to the Canterbury Plains and the
green banks of the Opaawaho.
 Witch Hill.
The Giants’ Causeway some fanciful pakeha has named this wall of
lava. To the Maori it is the “Ahi-a-Tamatea” or “the Fire of Tamatea”,
and a strange nature legend there is thereto; a folk tale in which
fable and geologic myth are curiously blended. It is a volcano myth
which closely resembles, and is no doubt a copy of, the North Island
legend of Ngatoro-i-Rangi and the origin of the Ngauruhoe volcano.
The march downhill of this curious upstanding dyke of lava, grey
against the more sombre tints of the range, may be traced in the
masses of lava boulders through which the Rapaki water-course has
cut its way in its lustier days. And if you turn your face southward
and look far across the upper part of the harbour, near the western
slopes of Mount Herbert, you will plainly see what seems a
continuation of the remarkable wall of lava which welled ages ago
white-hot from the cauldrons of Ruaimoko. Those grey-white
parapets of fire-made rock are the Ashes of the Fires of Tamatea,
and this is the wonderful tale of Tamatea and his magic fire, a tale of
old which brings in, too, our great arrowhead peak, towering yonder
on Rapaki’s western side, yon huge upjut lording it sentry-wise over
the Maori kainga . The peak of “Tamatea’s Breast,” is one of the very
few landscape saliencies in these parts which still remain in the
hands of the people of Takitimu descent. A venerable name this, for
it is quite six hundred years old and is a connecting link with the
greatest land-explorer in Maori-Polynesian story, a prototype of the
adventurous “Kai-ruri,” the white surveyor of pakeha pioneering
days.
Tamatea seems to have been possessed of the true “wander-
hunger,” for when he arrived on the shores of the North Island in his
ocean-going canoe the Takitimu (or Takitumu) from his Eastern
Pacific home—Tahiti or one of the neighbouring islands—via
Rarotonga—his restless heart impelled him to more adventure. First
making the land in the far north, he voyaged on down the East
Coast, sailing and paddling from bay to bay, leaving here and there
some of his sea-weary crew, who intermarried with the inhabitants,
and he did not cease his sailorly enterprise until he had reached the
foggy shore of Murihiku, the “Tail of the Land” which the white man
calls Southland. Here the Takitimu was hauled ashore and if one is to
accept literally the legend of the Maori, there she is to be seen now,
metamorphosed in marvellous fashion into mountain form, the lofty
blue range of the Takitimu—mis-spelled “Takitimos” on the maps and
locally spoken of as “the Takitimos.” It is curious that this isolated
range—fairy-haunted in native legend—lifting abruptly on all sides
from the tussocky plains that slope to Lake Manapouri, from more
than one point bears a resemblance to the form of a colossal canoe
upturned. Legend, too, says that Tamatea settled awhile at the foot
of the Canoe Mountains, and that he had a camp village at the lower
end of Lake Te Anau, where eels and birds were abundant.
 The Summit Road, overlooking Governor’s Bay.
C. Beken, photo
Then, wearying for the trail and the pikau , he set out on a march
which was nothing less than heroic, from Southland to the newly-
settled homes of his people in the North. With a number of
companions and food bearers the barefooted explorer trudged
across country, through the unpeopled tussock prairies of Otago and
the plains now known
as Canterbury, fording or swimming the rivers or crossing them on
rude rafts (mokihi ) made of bundles of flax-stalks or of dried raupo ,
until he reached the hills that wall in Lyttelton Harbour. He travelled
along the range-top, as was the way of the Maori explorer, until he
neared the dip in the sharp ridge at the back of Rapaki, over which
the Maoris and the pioneer white men made a track.
Now, Tamatea had carried with him, in a section of a hollow rata log,
as was the fashion of the Maori, a smouldering fire for his nightly
camps. No common fire this; it was an ahi-tipua , a “magic blaze,” a
sacred fire directly kindled from that trebly-tapu fire which Uenuku,
the great high priest of far Hawaiki, had sent with his canoe
voyagers. The Takitimu, being a tapu canoe, carried no cooked food,
and the only plants the people brought in her were ornamental ones,
for scent and beauty and sweet flowers. She was a great double
canoe, and could carry two hundred people. Her consort was the
canoe Arai-te-uru, which carried “Te Ahi-a-Uenuku—Uenuku’s Fire”—
and all kinds of food plants, even, says Tikao, a grain which is said
to have resembled the pakeha’s wheat. Coming round Te Matau-a-
Maui, Hawke’s Bay, the Arai-te-Uru nearly capsized; she went over
on her side, and continued in that attitude until she finally
overturned at Matakaea Point, near Oamaru, where she still lies—
turned to stone! The sacred fire was saved and it was taken by the
chiefs up the Waitaki River and placed there in the ground; and
there until about forty-five years ago it was still actually burning,
issuing from the earth in a little undying flame, and it was called “Te
Ahi a Uenuku.” (A seam of lignite is said to have been found burning
in the locality by the early settlers and explorers, and this the Maoris
identified with the Hawaikian sorcerer’s magic blaze.)
But it seems that Tamatea’s fire so carefully tended by the gods
went out as he travelled slowly up the hills from Otago, and none
had been kindled again when he reached the Port Hills. And as he
and his companions came out of the bush and passed out on to the
summit of the mountain above Rapaki, a great storm of wind and
rain, followed by hail and snow, burst upon them from the south and
they were like to perish from the cold. It was freezing cold, and
Tamatea was without fire or the means of making one, for no fire-
making timbers grew in that spot. In his extremity he stood upon
the tall crag-top yonder—the one that now is called “The Breast of
Tamatea”—and called aloud and karakia’d and made incantations for
sacred fire to be sent from the North Island, the land of active
volcanoes, to save the lives of himself and his companions. He called
to his elder relative, Ngatoro-i-Rangi, and to the guardians of the
Ahi-Tipua, the volcanic fires.
And the chief’s fervent prayer was answered in a moment. The fire,
sent by the gods in the heart of the North Island, burst forth from
Tongariro and speeding down the rift of the Wanganui River valley it
touched a spot near Nelson, and again it touched Motu-nau—the
small island close to the Canterbury Coast—and then it appeared
again in a magic blaze on the side of Witch Hill, and the Maori
explorers warmed their frozen limbs and were saved. The fire did
not stop here in its wonderful flight, for it went on across the
harbour, and the white chute of rock, like a huge sheep-dip trough,
shining yonder above the bay of Waiaki is the last of the sacred
flames of Tamatea.
And when the chief left the spot next day to continue his journey he
said, “Let this place be called The Place where Tamatea’s Fire-Ashes
Lie”; and so to-day the rocky wall which the white man has named
the Giant’s Causeway is to the Maori “Te Whaka-takanga-o-te-
Karehu-a-Tamatea.” And the volcanic fire itself is “Te Ahi-a-Tamatea.”
Such is the story of Tamatea’s Fire as told by Hone Taare Tikao, in
reciting the legends learned from the long-dead elders of his tribe. A
legend embalming a distinct perception of the geological history of
these hills. The Kaumatua truly says that the magical walls of
Tamatea’s Fire Ashes are of later origin than the volcanic crags and
hills which lie about them, and across which they cut. The Wanganui
River, down which the sacred fire came from the crater of
Ngauruhoe, was then dry; it was a rift opened for it by the Volcano
Gods. Tikao speaks of a time when the lower part of Lyttelton
Harbour was not in eruption, but when the upper part was; from the
southern side of Quail Island to the head of the bay was a furnace;
there was no water in it at that immeasurably distant day. There are
lava rocks, he says, like those of the Ahi-a-Tamatea on the shore of
Quail Island; and then there are the always wonderful cliff of Pari-
mataa and the wall of Otarahaka on the Mount Herbert side of the
upper harbour.
Of the lava dykes Professor R. Speight writes in his account of the
geology of the Port Hills that they radiate from the harbour as a
centre and form, as it were, the ribs of the mountain, holding it
firmly together and helping it to resist the enormous strains to which
it was exposed before and during eruptions. “Judging from the
persistent nature of these dykes,” he adds, “it is clear that the
mountain must have been split at times from top to bottom, and the
liquid material which welled from the fissures must have looked at
night like a red-hot streak across the country.”

Te Poho-o-Tamatea, or Tamatea’s Breast, in the middle distance.

W. A. Taylor, photo
And so, by way of the long Maori story, the one solitary example of
volcano-legendry the writer has heard from the South Island, we
come to the Rapaki peak’s guardian name, Te Poho-o-Tamatea. The
explorer bestowed his name upon the height, tapa-ing it after
himself and his sacred breast, much as a present-day traveller justly
might claim the right to map-name some peak after himself.
Tamatea’s travels led him far after he turned his back on the Port
Hills. He marched up on the coast to Kaikoura, and there or further
north he built a canoe, or, what is more probable, borrowed or
forcibly appropriated one belonging to the tangata whenua who
must already have been living in the Marlborough country, and
crossed to the North Island. He canoed up the Wanganui River—
there is a curious rock, the end of which the natives used to paint
red with kokowai ochre, projecting from the eastern bank many
miles above Pipi-riki, still known as Tamatea’s Rock—and crossed the
central plateau to Lake Taupo, and thence went on to the East
Coast. There are innumerable stories illustrating his genius for
exploration, and from end to end of New Zealand place
nomenclature memorises his travels and justifies the name by which
he is known in history, Tamatea-pokai-whenua, or “Tamatea-who-
travels-through-the-land.” He was a true type of the pioneer and the
path-finder, the Fremont of Aotea-roa and the Wai-pounamu, the
man who heard the “one everlasting Whisper”—
Something hidden. Go and find it. Go and look behind the Ranges—
Something lost behind the Ranges. Lost and waiting for you. Go!
 Chapter VI.
HILLS OF FAERY.
THE LITTLE PEOPLE OF THE MIST.
Our talk turned one day to the poetic legendary past of the Port Hills
above us and the Banks Peninsula peaks yonder, hemming in
Lyttelton Harbour in an annular line of dead fire crags. They were
the hills of an artist’s dream this golden afternoon, all their asperities
of crag and bare bluff softened and gilded by the mellow light of a
calm, bright, windless day. There was just the faintest of hazy films
drawing over the nearer hills from the waters. The sea below us at
Rapaki lay in unbreathing quiet, as soft, as bright and blue as
Kipling’s Indian Ocean. The rich purple of distance painted the most
remote of the Peninsula peaks, and here and there a wispy tail of
mist floated about the head of a gully that cleft the crumpled hills
swelling up into the rocky summits of the Pohue and the Ahu-patiki.
Just where the land went steeply up from the head of the bay to the
shoulders of Te Ahu-Patiki, otherwise Mount Herbert, we could see
the curious light grey rock wall of the Tarahaka, above the Pari-
mataa, “Obsidian Cliff,” like a great chute down the mountain side,
gleaming in the sun, the tipua rock of the fire gods. Nearer, in the
middle of the picture, were the black volcanic cliffs, and the green
slopes and pine-groves of the Lepers’ Isle, the island which the
Maoris of old called Otamahua, mapped by the pakeha as Quail
Island; it was one tragic spot in the picture, a place of living death.
The Kaumatua spoke of the hilltop homes of the Patu-paiarehe , the
fairies, whose craggy castles defended by the thick dark woods and
the fogs and mists, ringed all this harbour round and made the high
places of the Peninsula an uncanny land, given up to all manner of
enchantments. Not that he reposed implicit faith in the fairy stories
himself, he told them as he had heard them from his elders in the
days of long ago. “We are half pakeha ourselves now, we Ngai-
Tahu,” he said, “and our young people deride these notions about
the Patu-paiarehe . Yet—these hills were different in my young days,
when the mists came down and the fog enveloped the little streams
tumbling down all the valleys from the gloomy places. We went
birding into the forest, and we made clearings for cultivations and
cut firewood in the bush for sale to the pakeha. Sometimes, when
the mists came down and the fog enveloped the hills, on still, calm
days our old men and women would say the fairies were out, the
sun-shunning Patu-paiarehe , and it were well not to venture up the
range. On brooding quiet days our people could hear the thin voices
of the little folk—they were small, fair people, as all the elders said—
crying out to each other and singing fairy songs and playing little
songs on their wooden or bone flutes, their koauau and putorino .
This Poho-o-Tamatea, the high peak behind the kainga of Rapaki,
was one of their homes, their pas , but there were others, the high
places of the fairies, all around these hills, from my peak above
there right away along the peak-tops touched by the Summit Road,
right round to Cooper’s Knobs and then on to the Peninsula, even to
Otoki, the haunted mountain which the pakeha calls Brazenose, on
the southern side of Akaroa town.”
 Rhodes’s Monument (Home of the Fairies), to the east of Mt.
Herbert, taken from Lyttelton.
W. A. Taylor, photo
The Kaumatua swept his finger across the southern sky-line, the wild
and broken peaks of Banks Peninsula, rising up in powerful slants
from the soft blue of the harbour, creased with the gullies of water-
courses. “Over yonder,” he said, “are the chief pas of the Patu-
paiarehe , which I shall list for you: There is the rock of Te Pohue,
which pakehas call the Monument between Purau and Port Levy;
there is Hukuika Peak, on the hill road between Pigeon Bay and Little
River; there is the mountain-top of Te U-Kura, which commands all
the hill-country of the Peninsula—it is just at the back of the
stopping-place called Hilltop on the road from Little River to Akaroa
town. Also there are the high rocky peaks which overlook Akaroa
Harbour—Pu-Waitaha, or French Hill, between Wainui settlement
and Buchanan’s; Otehore, a rocky flat-topped height above French
Farm, on the upper part of the harbour; the summit heights above
Akaroa town, Purple Peak, Mount Berard and Brazenose—these we
call O-te-Patatu, Tara-te-rehu, and Otoki; and lastly Tuhiraki, the
sky-pencilling peak, which the French named Mount Bossu, on the
western side of the harbour. All these were the mountain pas of the
fairies.
“Now, amongst the fairy pas ,” the word-of-mouth historian went on,
“there are two places of particular enchantment. One is that lofty
palisade of a peak which you call Brazenose, and the level-topped
hill above French Farm. Otoki—the Place of the Axe—is the name for
Brazenose, but there is an ancient burying-ground there, upon the
misty mountain immediately above the little Maori village of Onuku,
on the beach side, which is called Otehore; the Wahi-tapu , or
cemetery, makes it doubly sacred. This place and the height above
French Farm were both called Otehore, and the people of the mists
loved them well. On the small piece of level ground on that Otehore
which stands above French Farm the fairy hapu of those parts had
their pa , and there they would gather at night for their fairy
meetings, and gather by day also in the foggy weather when no
Maori eye could see them; and then would be heard their sweet fairy
songs, their waiatas , and the tunes they played upon their flutes,
sounding faintly from the cloudy mountain.”
Indeed, on dim and foggy days, and when the wet vapours becloud
the long-dead volcano land, it is not difficult to enter into the spirit
of the Maori fancy, and in imagination people all these craggy
pinnacles with the Patu-paiarehe , and the wooded gulches with
Maeroero , the wild men of the bush. When the mists steal down on
the rhyolite knobs of the fire-fused Port Hills and the Peninsula,
craggy beyond description, peaked in a hundred fantastic shapes,
with great black and grey nipples of lava protruding from the grassy
slopes and the woody ridges, the land has an eerie look, fit
playground for the Forgetful People, as the old Irish would have
called them. In the early morning too, when the fogs are lifting from
the bays, and here and there a black broken thumb of a peak juts
out above the trailing vapours, or the mist-wreaths drape in torn
veilings the huge rampart-like scarps, and when the first rosy
sunbeams glorify the half-revealed mountains, the poetic mind can
well conceive of this region as one of fairy wizardry and all manner
of dusk-time magic. One day when it rained and blew, we were
motoring over the hills from the Plains to Akaroa, scarcely able to
see twenty yards in front of the car when we ran cautiously round to
Hilltop. Suddenly the dense masses of fog were rent aside by the
wind, and right above us, with a fore-ground of blackened tree-
stumps, remnant of the ruined forest, we caught a glimpse of the
black, jagged tor which the Maori called Te Puha, thrust up like a
great spearhead by Ruaimoko, the Fire-God; in the light of the
Kaumatua’s stories it was easy to imagine it a sentry-tower of the
fairies, or of such a place of witchery as the Ben Bulben of which we
read in W. B. Yeats’s “Celtic Twilight.”
There is another rugged, dark volcanic crag, wooded about its base,
crowning one of these Peninsula ridges, of which the Old Man of
Rapaki has a poetic story. This is the pinnacle above and to the
north of the Hilltop Hotel, on the divide where the traveller gets his
first sight of Akaroa lying more than a thousand feet below. The
Kaumatua gave it as one of his fairy peaks, its Native name is Te U-
Kura, and as far as I have been able to ascertain, it bears no official
pakeha name. Te U-Kura, he says, is an exceedingly ancient
designation; it was a name given by the fairies. It means “The
Resting Place of the Ruddy Light.” In his younger days, when he
lived at Tikao Bay, on the shores of Akaroa harbour, he frequently
observed for himself the peculiar fitness of the Hilltop name. Often
at sunset a cloud-cap would rest lightly on the dark skull of the fairy
crag, and to this cloud the declining sun would impart a bright red
glow, the kura of the Maori. This was a sign to the weather-wise that
a nor’-wester was blowing on the Canterbury Plains—a tohu , or
token, that was seldom astray. “U” means to rest, as a canoe upon
the beach. This surely is a name that the pakeha should imprint
upon his Peninsula maps. It lends itself to more than one
euphonious variant in the pakeha -tongue, as “Red Cloud’s Rest,” or
“Place of the Sunset’s Glow.”
Behind the town of Akaroa, the grassy hills, lightly wooded here and
there like a beautiful wild park, culminate in a craggy skyline, more
than a thousand feet above the fields and orchards on the town
outskirts. Dark tors of rhyolite, grassed to the base of

Cockayne’s Cairn Cass Peak

Kennedy’s Bush.
W. A. Taylor, photo
their nippled upswells, dreamy on sunny days in their delicate wash
of hazy blue. Here near the Stony Bay road saddle, is the legend-
haunted O-te-Patatu, the old-time ridge of the fairies and the titi .
The titi or muttonbird has long since disappeared from the Port and
Peninsula hills, but in the Maori days it was plentiful on many such
high places. It loved to build on lofty hills, where the soil was soft
enough to enable it to make its burrows. There was a cliff at O-te-
Patatu, says the Kaumatua , where the titi lived and bred in great
numbers in the long ago. The Maori tribes who lived on the beaches
and on the trenched and parapeted promontories made expeditions
to the hilltop in the season when the young birds were fat, and
caught, cooked and preserved them in kelp baskets and pottles, just
as the Foveaux Strait and Stewart Island Maoris do at the present
time. But the Patu-paiarche who lived on the crags of O-te-Patatu
and Tara-te-rehu also hunted these titi , and slaughtered them in
such numbers, in a most unfairylike manner, that presently these
fishy petrels became extinct in that locality. A woman of the Ngai-
Tahu, or Ngati-Mamoe, at Akaroa, who was said to be in love with a
fairy, but who no doubt also loved the palatable muttonbird, chanted
this little waiata , a song of fairy times which is still to be heard at
Rapaki and other Native villages:—

Titi whakatai aro rua

E hoki ra koe
Ki O-te-Patatu,
Ki te pa whakatangi
Ki te koauau,
Ki tauwene ai
E raro i au-e!

(Translation)

O titi, bird of the sea,

Bird of the hilltop cave,
Come back to O-te-Patatu,
To the lofty dwelling
Where the sweet sounds are heard,
The sound of the faery flute,
The music of the mountains
That thrilled me through and through!

The soft and plaintive flute song of the fair-skinned folk who lived on
these misty mountains seems to have appealed to the Native heart,
for it is described as sweeter by far than the whakatangitangi , the
“making sound upon sound,” of the ordinary Maori flute-players.
Distance, aided by imagination, no doubt lent it additional
enchantment. And the Patu-paiarehe men appear often to have
made themselves agreeable to the wahine Maori, for stories are told
of women being taken as wives by the fairy chiefs, and of girls from
the Maori villages wandering away into the woods to meet their fairy
lovers. The offspring of such unions were always known by their
extremely fair skins, unnaturally pale, and their light flaxen hair; they
were korako or albinos.
And there was another people of the wilds. Sometimes on dark
nights the Maori villagers would see the light of torches moving
about in bays across Lyttelton Harbour, then the Bay of Raupo,
where they knew there was no settlement of their tribe, and they
would say to each other, “See! The Maeroero are out, spearing
patiki.” The Maeroero were the wild men of the woods, fierce hairy
giants who sometimes captured Maori women and carried them off
to be their wives in the bush of the Port Hills. The Maeroero are
described as having very long and sharp finger-nails, so long that
they were great claws, and it was with these long talons that they
speared the flatfish and caught the birds in the forests.
 The Summit Road pathway, through the Devil’s Staircase, or The
Remarkable Dykes (on the way to Kaituna)
To the matter-of-fact pakeha and the modernised Maori there is a
very simple explanation of these fairy tales. The Patu-paiarehe and
the Maero were simply the remnants of aboriginal tribes, such as the
Ngati-Mamoe or the Waitaha or their predecessors, driven away into
the heart of the mountains and the forests, where they lived a wild,
secluded life, existing on the foods of the wilderness. The old English
and Scottish belief in fairy people arose in much the same way, the
very word “pixy” comes from the name of the Picts, who were driven
into the hills and caves. Nevertheless, for those who like to preserve
their Peter Pan fancies and illusions this theory may cheerfully be
disregarded, and we may still, on days of mist and cloudy calm,
imagine the little tribes of the rocks flitting out from their caves and
hollow trees and raising as of old their thin voices in their waiatas
and piping their fairy sweet koauau music on the level hilltop of
Otehore or the dark rock of Tamatea’s Breast.
MOUNT PLEASANT AND ITS “TAPU.”
Now the Old Man’s memories take him back seawards across the
Port Hills to the familiar knolls of Tauhinu-Korokio. This is the Native
name of Mount Pleasant, the great grassy upswell of land lifting
sixteen hundred feet above sea-level and looking down serenely on
the vast curving sand-line of Pegasus Bay. The Maori name is a
combination of the names of two Native shrubs which were very
plentiful on that portion of the hills in former times and specimens of
which may still be seen there. Tauhinu is a heath-like plant found all
over New Zealand; it is the Pomaderis phylicaefolia of the botanists.
It grows two or three feet high, or sometimes higher, and it is so
plentiful and so prone to spread that it has been placed on the
Agricultural Department’s index expurgatorious as a noxious weed,
in spite of all the sweet heathery perfume of its blossoms which the
bees love. The korokio is a small bushy black-branched growth which
the tauhinu , as the Maoris say, often embraces and smothers, as
the rata eventually smothers the tree which it clasps. Tauhinu-
korokio was an ancient pa of Ngati-Mamoe which stood on the site
of Major Hornbrook’s old homestead and the present house of
 Hinekura.
J. Cowan, photo
stay for visitors, just below a remnant of the ancient bush on the
northern and sunny slope of the green mountain. There was a good
spring of water close by, and this important fact no doubt
determined the situation of the Ngati-Mamoe hillmen’s village. In
Ngai-Tahu times, after Te Rangi-Whakaputa had conquered this pa in
the course of his subjugation of the Whaka-Raupo aborigines, the
vicinity was found by the Ngai-Tahu to be a very suitable spot for
such vegetable foods as the korau and the pora , or pohata, now
extinct; the roots, which were sweet, were dried in the sun and
stored in ruas , or underground pits and earthed-in storehouses, as
potatoes and kumara are now. Kumara and taro , the tropic foods of
the north, did not flourish in the hill country of the south, though
they did well enough, with care, on favoured parts of the Canterbury
Plains and the Coast.
There is in Maori belief an exceedingly tapu spot on Tauhinu-korokio,
close to where the old Ngati-Mamoe pa stood. It was probably either
a tuahu or a burying-place; the tuahu was the spot where the tribal
gods, or, rather, their symbols in wood and stone, were kept, and
where the wise men, the tohungas, resorted for incantation and
occult ceremonies and the black art of the makutu, by which
enemies might be slain though they were at a great distance, by
thought transmission and the malignant projection of the will. It is
not well to camp there even now, should you have Maori blood in
your veins. The flax-clumps and the tauhinu bushes still murmur the
name of Ngati-Mamoe, and though pakeha sheep have long grazed
over the site of Tauhinu-korokio and pakeha voices make lively the
mountain side, the soil holds the mysterious spell of the tapu. Maoris
who have camped on the side of the track which goes over the hills
there have spent a night of inexplicable discomfort, inexplicable, that
is, but for the presence of the tapu and the unseen spirits of
Maoridom. Taare Tikao himself says that many years ago, when he
was shearing at Major Hornbrook’s, he was taken suddenly and
mysteriously ill, and that the illness was probably the effect of the
local tapu. However, these fancies need not trouble the pakeha,
whose constitution is not affected by even the most virulent of Maori
bedevilments.

Printed by
Whitcombe & Tombs Limited
Christchurch
*** END OF THE PROJECT GUTENBERG EBOOK MAORI FOLK-TALES
OF THE PORT HILLS, CANTERBURY, NEW ZEALAND ***

Updated editions will replace the previous one—the old editions

will be renamed.

Creating the works from print editions not protected by U.S.

copyright law means that no one owns a United States
copyright in these works, so the Foundation (and you!) can copy
and distribute it in the United States without permission and
without paying copyright royalties. Special rules, set forth in the
General Terms of Use part of this license, apply to copying and
distributing Project Gutenberg™ electronic works to protect the
PROJECT GUTENBERG™ concept and trademark. Project
Gutenberg is a registered trademark, and may not be used if
you charge for an eBook, except by following the terms of the
trademark license, including paying royalties for use of the
Project Gutenberg trademark. If you do not charge anything for
copies of this eBook, complying with the trademark license is
very easy. You may use this eBook for nearly any purpose such
as creation of derivative works, reports, performances and
research. Project Gutenberg eBooks may be modified and
printed and given away—you may do practically ANYTHING in
the United States with eBooks not protected by U.S. copyright
law. Redistribution is subject to the trademark license, especially
commercial redistribution.

START: FULL LICENSE

THE FULL PROJECT GUTENBERG LICENSE
 PLEASE READ THIS BEFORE YOU DISTRIBUTE OR USE THIS WORK

To protect the Project Gutenberg™ mission of promoting the

free distribution of electronic works, by using or distributing this
work (or any other work associated in any way with the phrase
“Project Gutenberg”), you agree to comply with all the terms of
the Full Project Gutenberg™ License available with this file or
online at www.gutenberg.org/license.

Section 1. General Terms of Use and

Redistributing Project Gutenberg™
electronic works
1.A. By reading or using any part of this Project Gutenberg™
electronic work, you indicate that you have read, understand,
agree to and accept all the terms of this license and intellectual
property (trademark/copyright) agreement. If you do not agree
to abide by all the terms of this agreement, you must cease
using and return or destroy all copies of Project Gutenberg™
electronic works in your possession. If you paid a fee for
obtaining a copy of or access to a Project Gutenberg™
electronic work and you do not agree to be bound by the terms
of this agreement, you may obtain a refund from the person or
entity to whom you paid the fee as set forth in paragraph 1.E.8.

1.B. “Project Gutenberg” is a registered trademark. It may only

be used on or associated in any way with an electronic work by
people who agree to be bound by the terms of this agreement.
There are a few things that you can do with most Project
Gutenberg™ electronic works even without complying with the
full terms of this agreement. See paragraph 1.C below. There
are a lot of things you can do with Project Gutenberg™
electronic works if you follow the terms of this agreement and
help preserve free future access to Project Gutenberg™
electronic works. See paragraph 1.E below.
1.C. The Project Gutenberg Literary Archive Foundation (“the
Foundation” or PGLAF), owns a compilation copyright in the
collection of Project Gutenberg™ electronic works. Nearly all the
individual works in the collection are in the public domain in the
United States. If an individual work is unprotected by copyright
law in the United States and you are located in the United
States, we do not claim a right to prevent you from copying,
distributing, performing, displaying or creating derivative works
based on the work as long as all references to Project
Gutenberg are removed. Of course, we hope that you will
support the Project Gutenberg™ mission of promoting free
access to electronic works by freely sharing Project Gutenberg™
works in compliance with the terms of this agreement for
keeping the Project Gutenberg™ name associated with the
work. You can easily comply with the terms of this agreement
by keeping this work in the same format with its attached full
Project Gutenberg™ License when you share it without charge
with others.

1.D. The copyright laws of the place where you are located also
govern what you can do with this work. Copyright laws in most
countries are in a constant state of change. If you are outside
the United States, check the laws of your country in addition to
the terms of this agreement before downloading, copying,
displaying, performing, distributing or creating derivative works
based on this work or any other Project Gutenberg™ work. The
Foundation makes no representations concerning the copyright
status of any work in any country other than the United States.

1.E. Unless you have removed all references to Project

Gutenberg:

1.E.1. The following sentence, with active links to, or other

immediate access to, the full Project Gutenberg™ License must
appear prominently whenever any copy of a Project
Gutenberg™ work (any work on which the phrase “Project
Gutenberg” appears, or with which the phrase “Project
Gutenberg” is associated) is accessed, displayed, performed,
viewed, copied or distributed:

This eBook is for the use of anyone anywhere in the United

States and most other parts of the world at no cost and
with almost no restrictions whatsoever. You may copy it,
give it away or re-use it under the terms of the Project
Gutenberg License included with this eBook or online at
www.gutenberg.org. If you are not located in the United
States, you will have to check the laws of the country
where you are located before using this eBook.

1.E.2. If an individual Project Gutenberg™ electronic work is

derived from texts not protected by U.S. copyright law (does not
contain a notice indicating that it is posted with permission of
the copyright holder), the work can be copied and distributed to
anyone in the United States without paying any fees or charges.
If you are redistributing or providing access to a work with the
phrase “Project Gutenberg” associated with or appearing on the
work, you must comply either with the requirements of
paragraphs 1.E.1 through 1.E.7 or obtain permission for the use
of the work and the Project Gutenberg™ trademark as set forth
in paragraphs 1.E.8 or 1.E.9.

1.E.3. If an individual Project Gutenberg™ electronic work is

posted with the permission of the copyright holder, your use and
distribution must comply with both paragraphs 1.E.1 through
1.E.7 and any additional terms imposed by the copyright holder.
Additional terms will be linked to the Project Gutenberg™
License for all works posted with the permission of the copyright
holder found at the beginning of this work.

1.E.4. Do not unlink or detach or remove the full Project

Gutenberg™ License terms from this work, or any files
 containing a part of this work or any other work associated with
Project Gutenberg™.

1.E.5. Do not copy, display, perform, distribute or redistribute

this electronic work, or any part of this electronic work, without
prominently displaying the sentence set forth in paragraph 1.E.1
with active links or immediate access to the full terms of the
Project Gutenberg™ License.

1.E.6. You may convert to and distribute this work in any binary,
compressed, marked up, nonproprietary or proprietary form,
including any word processing or hypertext form. However, if
you provide access to or distribute copies of a Project
Gutenberg™ work in a format other than “Plain Vanilla ASCII” or
other format used in the official version posted on the official
Project Gutenberg™ website (www.gutenberg.org), you must,
at no additional cost, fee or expense to the user, provide a copy,
a means of exporting a copy, or a means of obtaining a copy
upon request, of the work in its original “Plain Vanilla ASCII” or
other form. Any alternate format must include the full Project
Gutenberg™ License as specified in paragraph 1.E.1.

1.E.7. Do not charge a fee for access to, viewing, displaying,

performing, copying or distributing any Project Gutenberg™
works unless you comply with paragraph 1.E.8 or 1.E.9.

1.E.8. You may charge a reasonable fee for copies of or

providing access to or distributing Project Gutenberg™
electronic works provided that:

• You pay a royalty fee of 20% of the gross profits you derive
from the use of Project Gutenberg™ works calculated using the
method you already use to calculate your applicable taxes. The
fee is owed to the owner of the Project Gutenberg™ trademark,
but he has agreed to donate royalties under this paragraph to
the Project Gutenberg Literary Archive Foundation. Royalty
 payments must be paid within 60 days following each date on
which you prepare (or are legally required to prepare) your
periodic tax returns. Royalty payments should be clearly marked
as such and sent to the Project Gutenberg Literary Archive
Foundation at the address specified in Section 4, “Information
about donations to the Project Gutenberg Literary Archive
Foundation.”

• You provide a full refund of any money paid by a user who

notifies you in writing (or by e-mail) within 30 days of receipt
that s/he does not agree to the terms of the full Project
Gutenberg™ License. You must require such a user to return or
destroy all copies of the works possessed in a physical medium
and discontinue all use of and all access to other copies of
Project Gutenberg™ works.

• You provide, in accordance with paragraph 1.F.3, a full refund of

any money paid for a work or a replacement copy, if a defect in
the electronic work is discovered and reported to you within 90
days of receipt of the work.

• You comply with all other terms of this agreement for free
distribution of Project Gutenberg™ works.

1.E.9. If you wish to charge a fee or distribute a Project

Gutenberg™ electronic work or group of works on different
terms than are set forth in this agreement, you must obtain
permission in writing from the Project Gutenberg Literary
Archive Foundation, the manager of the Project Gutenberg™
trademark. Contact the Foundation as set forth in Section 3
below.

1.F.

1.F.1. Project Gutenberg volunteers and employees expend

considerable effort to identify, do copyright research on,
transcribe and proofread works not protected by U.S. copyright
law in creating the Project Gutenberg™ collection. Despite these
efforts, Project Gutenberg™ electronic works, and the medium
on which they may be stored, may contain “Defects,” such as,
but not limited to, incomplete, inaccurate or corrupt data,
transcription errors, a copyright or other intellectual property
infringement, a defective or damaged disk or other medium, a
computer virus, or computer codes that damage or cannot be
read by your equipment.

1.F.2. LIMITED WARRANTY, DISCLAIMER OF DAMAGES - Except

for the “Right of Replacement or Refund” described in
paragraph 1.F.3, the Project Gutenberg Literary Archive
Foundation, the owner of the Project Gutenberg™ trademark,
and any other party distributing a Project Gutenberg™ electronic
work under this agreement, disclaim all liability to you for
damages, costs and expenses, including legal fees. YOU AGREE
THAT YOU HAVE NO REMEDIES FOR NEGLIGENCE, STRICT
LIABILITY, BREACH OF WARRANTY OR BREACH OF CONTRACT
EXCEPT THOSE PROVIDED IN PARAGRAPH 1.F.3. YOU AGREE
THAT THE FOUNDATION, THE TRADEMARK OWNER, AND ANY
DISTRIBUTOR UNDER THIS AGREEMENT WILL NOT BE LIABLE
TO YOU FOR ACTUAL, DIRECT, INDIRECT, CONSEQUENTIAL,
PUNITIVE OR INCIDENTAL DAMAGES EVEN IF YOU GIVE
NOTICE OF THE POSSIBILITY OF SUCH DAMAGE.

1.F.3. LIMITED RIGHT OF REPLACEMENT OR REFUND - If you

discover a defect in this electronic work within 90 days of
receiving it, you can receive a refund of the money (if any) you
paid for it by sending a written explanation to the person you
received the work from. If you received the work on a physical
medium, you must return the medium with your written
explanation. The person or entity that provided you with the
defective work may elect to provide a replacement copy in lieu
of a refund. If you received the work electronically, the person
or entity providing it to you may choose to give you a second
opportunity to receive the work electronically in lieu of a refund.