DFSMStvs Planning and Operating Guide
DFSMStvs Planning and Operating Guide
SC26-7348-00
z/OS
SC26-7348-00
Note
Before using this information and the product it supports, be sure to read the general information under “Notices” on
page 127.
Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Appendix B. Accessing data sets that have retained locks or lost locks 121
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Programming interface information . . . . . . . . . . . . . . . . . 128
Trademarks. . . . . . . . . . . . . . . . . . . . . . . . . . 128
Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Contents v
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
For information about accessibility features of z/OS and z/OS.e, for users who have
a physical disability, see Appendix C, “Accessibility”, on page 125.
To use this document effectively, you should also be familiar with the following IBM
products, programs, and components:
v CICS® (Customer Information Control System)
DFSMStvs extends the availability of CICS by enabling multiple batch jobs and
CICS to share access to the same data sets to update data as well as to read it.
DFSMStvs and CICS can use the same log of logs, a log stream that provides
information for forward recovery programs.
v CICS VSAM Recovery (CICSVR)
CICSVR is a forward recovery program. It can use written to forward recovery
logs by CICS or DFSMStvs to recover VSAM data sets.
v Data Facility Storage Management Subsystem data facility product (DFSMSdfp™)
DFSMSdfp provides functions for storage management, data management,
program management, device management, and distributed data access. For
information about DFSMSdfp, see z/OS DFSMSdfp Storage Administration
Reference.
v Hierarchical Storage Manager (DFSMShsm™)
DFSMShsm provides automatic space management and availability functions
through a hierarchy of storage devices. For information about DFSMShsm, see
z/OS DFSMShsm Storage Administration Guide.
v Recoverable resource management services (RRMS)
RRMS provides the context and unit-of-recovery management under which
DFSMStvs participates as a recoverable resource manager. Part of the operating
system, RRMS comprises registration services, context services, and recoverable
resource services (RRS). For information about RRMS, see z/OS MVS
Programming: Resource Recovery.
v System logger
DFSMStvs uses the services of the system logger for logging. For information
about the system logger, see z/OS MVS Setting Up a Sysplex.
From this Web site, you can link directly to the z/OS softcopy documents by
selecting the Library icon. You can also link to IBM Direct to order printed
documents.
You can access LookAt from the Internet at this Web site:
http://www.ibm.com/eserver/zseries/zos/bkserv/lookat/
To use LookAt as a TSO/E command, you must have LookAt installed on your host
system. You can obtain the LookAt code for TSO/E from a disk in z/OS Collection
(SK3T-4269) or from the News section on the LookAt Web site.
Some messages have information in more than one document. For those
messages, LookAt displays a list of documents in which the message appears.
Licensed documents are available only to customers with a z/OS or z/OS.e license.
Access to these documents requires an IBM Resource Link user ID and password
and a key code. With your z/OS or z/OS.e order, you should have received a Memo
to Licensees (GI10-0671) that includes this key code.
To obtain your IBM Resource Link user ID and password, log on to the following
Web site: http://www.ibm.com/servers/resourcelink
To register for access to the z/OS licensed documents, follow these steps:
1. Log on to Resource Link, using your Resource Link user ID and password.
2. Select User Profiles, located on the left-hand navigation bar.
You cannot access the z/OS licensed documents unless you have registered for
access to them and received an e-mail confirmation informing you that your request
has been processed.
You can use the PDF format on either z/OS Licensed Product Library CD-ROM or
IBM Resource Link to print licensed documents.
Transaction processing provides functions that coordinate work flow and the
processing of individual tasks for the same data sets. VSAM record-level sharing
and DFSMStvs provide key functions that enable multiple batch update jobs to run
concurrently with CICS access to the same data sets, while maintaining integrity
and recoverability.
Terminology
This document introduces several new terms or uses terms with definitions that are
specific to either VSAM record-level sharing (RLS) or DFSMStvs.
Atomic
When an application changes data in multiple resource managers as a
single transaction, and all of the changes are accomplished through a
single commit request by a syncpoint manager, the transaction is called
atomic. If the transaction is successful, all the changes will be committed. If
any piece of the transaction is not successful, then all of the changes will
be backed out. An atomic instant occurs when the syncpoint manager in a
two-phase commit process logs a commit record for the transaction.
Backout
A request to remove all changes to resources since the last commit or
backout or for the first unit of recovery, since the beginning of the
application. Backout is also called rollback or abort. Any of the following
events can initiate backout:
v A user request
v An inability of a resource manager to commit resource changes
v An abnormal end of a user task while a transaction is in-flight
Transaction processing
Transaction processing provides data sharing of recoverable resources and ensures
that data is synchronized. In z/OS, transaction processing means that each logical
unit of work runs as a single transaction. Products such as CICS, IMS, DB2, and
DFSMStvs provide a transaction-processing environment. An application that
maintains a database must be capable of managing transactions. The transactions
can arrive virtually simultaneously from clients.
For example, consider a context in which changes are made to two different
records in a recoverable VSAM data set. A field in one record is decremented from
200 to 100. A field in the other record item is incremented from 700 to 800.
Transactional recovery ensures that either both changes are made or neither
change is made. When the application requests that the changes be committed,
both changes are made atomically.
Coordination of recovery
DFSMStvs requires three key programs and three key functions to coordinate the
recovery of protected resources.
Key programs: The following three programs work together to coordinate the
recovery of protected resources and the records within a recoverable VSAM data
set:
Application program
The application program accesses protected resources and requests
changes to the resources.
Resource manager
A resource manager controls and manages access to a resource. A
resource manager is an authorized program that provides an application
programming interface (API) that enables the application program to read
and change a protected resource. The resource manager, in conjunction
with the syncpoint manager, controls whether changes to resources are
committed or backed out. For DFSMStvs, the resources consist of records
in recoverable VSAM data sets.
Syncpoint manager
A syncpoint manager uses a two-phase commit protocol to coordinate
changes to protected resources, so that either all changes are made or
none of the changes are made. Recoverable resource services (RRS)
functions as the syncpoint manager.
Key functions: The three programs above work in concert to provide the three
basic functions needed to implement transactional recovery:
Resource locking
This function provides serialized access to changed resources.
Resource recovery logging
This function enables a resource manager to keep a record of the changes
made to recoverable resources by a transaction.
Two-phase commit processing
This function provides the services that ensure that a transaction appears
as an atomic operation. A two-phase commit protocol and backout
Resource locking
Locking serializes access to the records that are being changed. Locking at a more
granular level, such as record level rather than at a control interval (CI) level, is
done to minimize the need to wait for locks. Record locks are generally released at
syncpoint time, when commit processing occurs and the unit of recovery ends. A
transaction might be required to wait if it requests a change to a record that is
locked by another transaction. This wait generally lasts until the lock-holding
transaction reaches a sync point or the transaction reaches a timeout.
During the prepare phase, the commit coordinator, RRS, invokes the prepare exits
of each of the resource managers. The resource managers determine whether they
can make the changes permanent during the commit phase. This means that they
must keep their backout information until the commit phase in case they are told to
back out the changes. If any of the resource managers are unable to perform the
prepare function, RRS directs all of the resource managers to back out the
transaction.
During the commit phase, the resource managers commit the changes represented
by a unit of recovery and release any resource serialization. When all resource
managers have completed the task, the application is notified, the unit of recovery
is completed, and the two-phase commit process ends.
With VSAM RLS, multiple CICS systems can directly access a shared VSAM data
set, eliminating the need to ship functions between the application-owning regions
and file-owning regions. CICS provides the logging, commit, and backout functions
for VSAM recoverable data sets. VSAM RLS provides record-level serialization and
cross-system caching. CICSVR provides a forward recovery utility.
VSAM RLS uses a coupling facility to perform data set-level locking, record locking,
and data caching. VSAM RLS uses the conditional write and cross-invalidate
functions of the coupling facility cache structure, thereby avoiding the need for
control interval (CI) level locking. VSAM RLS uses the coupling facility caches as
store-through caches. When a control interval of data is written, it is written to both
the coupling facility cache and to direct access storage device (DASD). This
ensures that problems occurring with a coupling facility cache do not result in the
loss of VSAM data.
Figure 1 illustrates the AOR, FOR, and VSAM request flow. CICS AORs function
ship VSAM requests to access a specific data set to the CICS FOR that owns that
data set. This distributed access form of data sharing has existed in CICS for some
time.
VSAM
VSAM CICS
FOR
VSAM
MVS 1 MVS n
Figure 2 illustrates how VSAM RLS enables multiple CICS transaction server AORs
to access VSAM data sets for update while preserving data integrity. The updating
regions can reside in any MVS™ image within a Parallel Sysplex. Each MVS image
has one SMSVSAM server. Each AOR that needs to access the VSAM data in RLS
mode connects automatically to the SMSVSAM server address space. VSAM RLS
stores locking information in a coupling-facility structure that SMSVSAM owns.
SMSVSAM SMSVSAM
Address space Address space
MVS 1 MVS 32
The CICS file-control program provides commit, backout, and forward recovery
logging functions for VSAM recoverable data sets. With VSAM RLS, CICS
continues to provide the transactional functions. VSAM RLS itself does not provide
the transactional functions but does provide record locking based on a coupling
facility as well as coupling-facility data caching.
The CICS file-control program supports recoverable and nonrecoverable data sets.
The data set definition includes a recoverability attribute. You can specify the
following attribute options:
LOG(NONE)
Nonrecoverable.
This option declares the data set to be nonrecoverable. CICS does not
perform any logging of changes for a data set that has this attribute. Neither
backout nor forward recovery is provided.
LOG(UNDO)
Recoverable.
This option declares the data set to be backward recoverable.
LOG(ALL)
Recoverable.
This option declares the data set to be both backward recoverable and
forward recoverable.
In addition to the logging and recovery functions provided for backout, CICS records
the image of changes to the data set, after they were made. The forward recovery
You can use CICS recoverable file attributes that correspond to VSAM data set
attributes. The file definition must match the data set attributes in the catalog. You
can use the IDCAMS DEFINE and ALTER (LOG(xxx)) commands to set the data
set attributes, either by running an IDCAMS job or by defining them in the SMS
DATACLAS parameter. You can also use the BWO and LOGSTREAMID parameters
in the IDCAMS commands.
For CICS and DFSMStvs access, you can define VSAM data sets as recoverable
with the backup-while-open (BWO) parameter set to TYPECICS. Both CICS and
DFSMStvs can open a recoverable data set for output.
For more information about the IDCAMS DEFINE and ALTER commands, see z/OS
DFSMS Access Method Services for Catalogs.
You can open a non-RLS data set in nonshared resources (NSR), local shared
resources (LSR), or global shared resources (GSR) mode. When you open a data
set in NSR, LSR, or GSR access mode, the recoverable attributes of the data set
do not apply and are ignored. Thus, existing programs that do not use VSAM RLS
access are not impacted by the recoverable data set rules. For more information
regarding NSR, LSR, and GSR access mode, see z/OS DFSMS: Using Data Sets.
The CICS backout function removes changes made to the recoverable data sets by
a transaction. When a transaction abnormally ends, CICS performs a backout
implicitly.
The commit and backout functions protect an individual transaction from changes
that other transactions make to a recoverable data set (or other recoverable
resource).
The SMSVSAM server owns two data spaces. One data space is called the
SMSVSAM data space. It contains some VSAM RLS control blocks and a
system-wide buffer pool. The other data space, named MMFSTUFF, is used by
VSAM RLS to collect statistical information that is used to produce VSAM RLS
system management facilities (SMF) 42 subtype 16-19 records.
VSAM RLS can ensure read integrity across splits. It uses the cross-invalidate
function of the coupling facility to invalidate copies of data and index control
intervals in buffer pools other than the writer’s buffer pool. This invalidation ensures
that all VSAM RLS readers, CICS and non-CICS, are able to see any records
moved by a concurrent CI or CA split. On each GET request, VSAM RLS tests the
validity of the buffers, and if invalid, refreshes the buffers from the coupling facility
or DASD.
VSAM RLS provides record locking and buffering across the CICS and non-CICS
read or write sharers of nonrecoverable data sets. VSAM RLS releases the record
lock when the buffer that contains the change is written to the coupling-facility
cache and DASD. This record locking differs from the case in which a CICS
transaction modifies VSAM RLS recoverable data sets. In that case, the
corresponding locks are held until the end of a recoverable unit of work. For a CICS
transaction, however, the locks are not held until the end of a transaction in either
of the following cases:
v The transaction is made up of multiple units of work.
v The locks are released at a sync point (at the end of each logical unit of work).
For sequential and skip-sequential processing, VSAM RLS does not write a
modified CI until one of the following actions occurs:
v The processing moves to another CI.
v The application issues an ENDREQ.
Recommendation: Be careful about quiesce with a path name; always use a base
name.
Share options
For non-RLS access, VSAM uses the share options settings to determine the type
of sharing allowed. VSAM provides full read and write integrity for the VSAM RLS
users, but does not provide read integrity for the non-RLS user. You cannot specify
a non-RLS open-for-output request when the data set is already opened for VSAM
RLS.
VSAM RLS provides read sharing and write sharing for multiple users; it does not
use share option settings to determine levels of sharing. When you request an RLS
open and the data set is already open for non-RLS input, VSAM does check the
cross-region setting. If the share option is 2, the data set can be opened in RLS
mode. The open fails if you use any other share option, or if the data set is opened
for non-RLS output.
Locking
Non-RLS access provides local locking (within the scope of a single buffer pool) at
the VSAM CI level. A locking contention can result in an exclusive control conflict
error response to a VSAM record management request.
All SMSVSAM servers (one per MVS image) use the coupling facility for
locking.When contention occurs on a VSAM record, the request that encountered
the contention waits for the contention to be removed. The lock manager provides
deadlock detection. When a lock request is in deadlock, the request is rejected
resulting in the VSAM record management request completing with a deadlock error
response.
VSAM RLS supports a timeout value that can be specified through the request
parameter list (RPL), in the PARMLIB member, or in the JCL. CICS provides two
places where you can supply a timeout value:
v By transaction, you can specify the DTIMOUT value in the transaction definition.
v By CICS region, you can use the FTIMOUT system initialization value.
CICS uses DTIMOUT when it is supplied; otherwise, it uses FTIMOUT. This
ensures that a transaction does not wait indefinitely for a lock to become available.
VSAM RLS uses a timeout function of the lock manager.
Lock Retention: VSAM RLS uses share and exclusive record locks to control
access to the shared data. An exclusive lock ensures that a single user is updating
a specific record. The exclusive lock causes a read-with-integrity request for the
record by another user, such as a CICS transaction or non-CICS application, to wait
until the update is complete and the lock is released.
Failure conditions can delay completion of an update to a recoverable data set. This
occurs when a CICS transaction enters in-doubt status. This means that CICS waits
for a distributed unit of work to complete before CICS can back out or commit the
Exclusive and retained locks are not available to other users. When another user
encounters a lock contention with an exclusive lock, the user’s lock request waits.
When another user encounters a lock contention with a retained lock, the lock
request is immediately rejected with a retained lock error response. This causes the
VSAM record management request, which produced the lock request, to fail.
Non-RLS access while retained locks exist: Retained locks are created when a
failure occurs. The locks need to remain until completion of the corresponding
recovery. The retained locks have meaning only for RLS access. Lock requests
issued by RLS access requests can encounter the retained locks. Non-RLS access
does not perform record locking and does not encounter the retained locks.
To ensure integrity of a recoverable sphere, VSAM does not permit non-RLS update
access to the sphere while retained locks exist for that sphere. An installation might
need to run a non-CICS application that requires non-RLS update access to the
sphere. VSAM RLS provides an IDCAMS command that sets the status of a sphere
to enable non-RLS update access to a recoverable sphere while retained locks
exist. This command does not release the retained locks. VSAM informs the CICS
transaction servers that hold the retained locks when they later open the sphere
with RLS.
For information about non-RLS access to data sets that have retained locks, see
Appendix B, “Accessing data sets that have retained locks or lost locks”, on
page 121.
The VSAM RLS record management request task must be the same task that
opened the ACB, or the task that opened the access method control block (ACB)
must be in the task hierarchy.
DFSMStvs overview
Without DFSMStvs, a CICS system might have been available only during normal
business hours. After business hours, the CICS system or application would be shut
down for the supporting batch work to run. As soon as CICS stopped, backups
were taken of key data sets as a point of recovery. Batch jobs could then be
scheduled to run. If several jobs updated the same data set, they ran in sequence
because they could not update the data set at the same time. After the updates
were complete, a job would read the updated data and produce reports, and
another backup would be taken. Finally, CICS could be restarted and become
active again. However, the need to extend the availability of CICS has increased
due to growing business volume.
DFSMStvs supports data sets that are defined as recoverable. That is, the log
attribute for the data set is either UNDO (backout logging only) or ALL (backout and
If you use DFSMStvs, you can run some batch jobs without taking your data sets
offline. Most likely, you need to make changes to enable batch applications to use
DFSMStvs. For details about application changes, see “Coding an application to
use DFSMStvs” on page 64.
DFSMStvs also provides batch program support for the repeatable read option. This
option enables a transaction to reread a record and see the same data the second
time. This is because a share record lock is obtained for this type of access, which
prohibits other transactions from making updates until the transaction is committed
or backed out.
Recovery coordination
The records within a recoverable VSAM data set are protected resources. The
application program, the resource manager, and the syncpoint manager work
together to protect resources.
These three components work in concert to provide the three basic functions
necessary for implementing transactional recovery:
v Resource locking
This function provides serialized access to changed resources. VSAM RLS and
DFSMStvs use a locking function based on a coupling facility, and the locking is
done at the VSAM record level.
v Resource recovery logging
This function enables a resource manager to keep a record of the changes that a
transaction makes recoverable resources. The system logger is the service that
is used to accomplish this task.
v Two-phase commit and backout protocols
This function provides the services that ensure that a transaction appears as an
atomic operation. It is the responsibility of two-phase commit and backout
protocols to ensure that all of the changes made by a transaction are either
VSAM RLS support enables batch jobs to access recoverable data sets in RLS
mode, but only for reading; batch jobs cannot write to these data sets in RLS mode.
With DFSMStvs, batch jobs are able to read and write to recoverable data sets
while CICS is concurrently processing those same data sets. A non-CICS
application must use RRS to partition the updates done to recoverable VSAM data
sets into units of recovery. The application must invoke the RRS commit and
backout interfaces to cause points of synchronization. The points of synchronization
delineate units of recovery.
If a batch job attempts to communicate with RRS and the system logger goes
down, RRS suspends the job until the logger comes back up. If this happens, you
cannot cancel the job. To make RRS release the job, you need to bring the system
logger back up or take RRS down.
Transaction processing
When an application uses DFSMStvs to process a transaction, DFSMStvs and RLS
perform the following tasks:
v Resource locking
v Logging the updates to records in recoverable data sets and logging the image of
records before they are updated. This allows the updates to be backed out, if
necessary. Optionally, DFSMStvs also records the images of records after they
are updated so that forward recovery programs can use the records.
VSAM RLS uses the coupling facility to provide locking for read and write integrity
and caching. DFSMStvs uses the same locking structure and cache structure as
other users of VSAM RLS. This allows applications that use DFSMStvs to read and
write recoverable data sets concurrently.
VSAM RLS obtains the exclusive lock at GET UPD time, not when you change the
record. DFSMStvs does backout logging at this time because GET UPD implies that
the you are going to update the record, and a complete before image of the record
is available then. Spanned and compressed records are not necessarily available
any more when you get to the PUT or ERASE.
Because of the locking and logging overhead, GET UPD is not an efficient way to
browse a data set when you use DFSMStvs. If you are browsing and updating only
one record in ten or so, use GET NUP instead. If you need read integrity, you can
specify CR or CRE. When you find a record that you want to update, do a GET
UPD followed by PUT UPD or ERASE.
Log records can also be written to a forward recovery log. The forward recovery log
contains copies of records after they were changed. The records are written to the
system logger log streams. Forward recovery logs are shared between systems;
these logs are shared across all CICS and DFSMStvs instances. This provides
sysplex-wide merged log streams that can be used as input to forward recovery
products such as CICSVR.
Initiating transactions
VSAM record management requests initiate transactions. DFSMStvs calls RRS to
register an interest in a unit of recovery, which creates the unit of recovery and
associates it with the task control block (TCB) or context.
Sync points are initiated by calling RRS interfaces for commit and backout
processing.
The two-phase commit protocol is a set of actions. These actions are used to
ensure that an application program makes either all changes to the resources
The first phase of the commit process is called prepare, the second phase is called
commit. The phases of commit processing are described as follows:
Phase 1, prepare
During the prepare phase, the commit coordinator, RRS, invokes the
prepare exits of each of the participating resource managers. In these exits,
each of the resource managers determine whether they can make the
changes permanent during the commit phase. This means that they must
keep their backout information until the commit phase, in case they are told
to back out the changes.
The process of making the changes permanent is called committing. If all of
the resource managers are able to commit their changes, processing
continues with the second phase of commit processing. If any of the
resource managers is unable to perform the commit function, RRS directs
all of the resource managers to back out the unit of recovery.
For data sets that have the forward recovery attribute, backout creates
compensating forward recovery records. After a media failure and the
restoration of a dated backup copy of the data, forward recovery reapplies
all updates since that backup.
Phase 2, commit
In the second phase, the resource managers either commit or back out the
changes represented by a unit of recovery, and they release the held locks.
When all resource managers have completed the task, the following
processing occurs:
v The application is notified.
v The unit of recovery is completed.
v The locks are released.
v The two-phase commit protocol ends.
Backout: DFSMStvs logs all updates to recoverable data sets that are modified in
a unit of recovery. In this way, if a unit of recovery fails, the changes that were
made as part of the unit of recovery can be reversed.
Each instance of DFSMStvs has a private undo log. This log contains the status of
the units of recovery in a DFSMStvs instance and the backout records required to
back out changes that were made to VSAM recoverable data sets by these units of
recovery. Any of the following events can trigger a backout request:
v A backout request from the application
v An abnormal end of the unit of recovery or the TCB with which it is associated
v Detection of an error that DFSMStvs cannot fix, such as an I/O error on one of
the data sets
If DFSMStvs detects such an error, it votes against committing the unit of recovery
when the application calls prepare.
Records added to the end of an ESDS are not backed out by DFSMStvs because
no such function as ERASE exists for an ESDS. You cannot actually delete a
record that has been written over in an ESDS. CICS has an exit that can modify the
record to imply that it has been deleted.
Restriction: When you are running DFSMStvs concurrently CICS and bad input
was used for a batch job, you cannot restore the data set, bring CICS up, and then
and rerun the batch job later with good input as you could with just CICS. The
forward recovery logs do not identify job names, which you would need to identify
the specific records updated by the batch job with the bad input. If other batch jobs
or CICS updated the same records, backing out the changes is not possible.
Figure 3 on page 20 is an example of how DFSMStvs can be used with CICS and
with batch jobs. DFSMStvs works with RRS to commit the VSAM data set changes
and release the corresponding VSAM locks.
CICS AOR
Batch Job
CICS AOR VSAM RLS & Transactional
VSAM Batch Job
DA0TV400
While VSAM RLS provides multisystem data sharing for CICS, DFSMStvs is
intended for non-CICS applications. From a logical perspective, CICS and
DFSMStvs do the same kind of work. DFSMStvs and CICS complement one
another by using the same locking protocols and the same type of commit and
backout protocol. If both DFSMStvs and CICS want to update the same group of
records, one application has to complete its current unit of recovery before the other
can begin.
Native contexts
A native context is the automatically occurring context of the application program,
and the protected resources associated with a work request. A native context is
always associated with a single application task. This context is associated with a
specific MVS dispatchable unit (task) and always exists.
Every task in the system has an associated context, thus, for a given task there is
always a context. When a task is created, context services provide the native
context for the task. Resource managers can create privately managed contexts
and associate them with a specific task. The privately managed context then
becomes the current context. The native context still exists, but it is not current. If
the resource manager later disassociates the privately managed context from the
task, the native context would again become current. The native context would also
become current if the privately managed context ends but the task with which it is
associated does not. DFSMStvs does not create or switch contexts; it does,
however, allow others to do so. DFSMStvs runs under a privately managed context
only if another resource manager has created one. That context must be the current
context associated with a task when DFSMStvs receives control.
Units of recovery
Using the two-phase commit protocols, applications use the commit service of the
syncpoint manager to make changes permanent, and the backout service to back
out the changes. When the syncpoint manager receives a commit request, the
syncpoint manager synchronizes commitment of resources.
A context has only one active unit of recovery at a time. This unit of recovery must
complete before another unit of recovery begins. The syncpoint manager knows
Unit-of-recovery states
The unit of recovery changes from state to state throughout the course of the
two-phase commit protocol. Figure 5 shows the flow of the two-phase commit
process. In the first in-reset state, the application is at the start of its unit of
recovery and has not used any protected resources. When protected resources are
accessed for add, delete or update, the unit of recovery moves into the in-flight
state with the resource manager expressing interest in the unit of recovery. Once
the application issues its commit, the in-prepare state is entered. Phase one begins
when the syncpoint manager tells each resource manager to prepare its resources
to move forward or backward. When all resource managers reply positively, the unit
of recovery transitions to the in-commit state and phase 2 begins. The syncpoint
manager tells each resource manager to make its changes permanent.
When all resource managers have completed the task, the application is notified,
the unit of recovery is completed, and the two-phase commit protocol ends. The
application might have altered protected resources using multiple resource
managers (for example, DB2, IMS, and DFSMStvs) within a recoverable unit of
work.
The protocol also has provisions if a resource manager replies negatively to the
prepare notification. In this case, the in-commit state becomes an in-backout state,
and resource managers are told to back out any changes.
For example, when multiple resource managers are involved, an application might
do some DB2 work, and then some DFSMStvs work, and finally some IMS work. All
of the work can be interrelated so that it is all under the umbrella of a single unit of
recovery that RRS coordinates. The application completes its transactions and then
calls RRS to commit or back out those transactions. In this way, the work within this
unit of recovery spans several different resource managers. A unit of recovery can
be distributed not only across multiple resource managers but across systems and
networks.
Planning tasks
Because DFSMStvs builds on VSAM RLS, many of the planning steps for the
implementation of VSAM RLS are common to a DFSMStvs implementation. You
should perform the following tasks to plan for using DFSMStvs:
v Determine the hardware and availability requirements for coupling facilities. For
DFSMStvs, the coupling facilities not only support lock tables and caching of data
but should also be used for logging.
v Determine which applications need the ability to update recoverable data sets
concurrent with CICS usage of those data sets, and then determine which of
them can use the DFSMStvs support.
v Determine the size and number of coupling-facility cache structures and their
connectivity.
v Determine the size of the coupling-facility lock structure. With the use of
DFSMStvs, batch jobs update recoverable data sets while CICS is also using
them, which results in increased locking activity.
v Determine the requirements for the SMS configuration, including storage-class
specifications of coupling-facility structures, connectivity, and modifications to the
ACS routines. If you have already established storage classes for VSAM RLS
and coupling-facility cache structures, you can use those for both VSAM RLS
and DFSMStvs. You might need to make the structures larger, however, due to
increased usage.
v Add new parameters to the IGDSMSxx member of SYS1.PARMLIB for
DFSMStvs initialization, quiesce time out, and activity keypointing.
v Convert batch jobs for use of DFSMStvs to update recoverable data sets:
– Modify each job to recognize sync points and to take the appropriate action,
whether commit or backout, by calling z/OS RRS services.
Recommendation: Do not perform a commit or backout through an RRS
panel.
– Modify the programs or JCL to specify that DFSMStvs is to be used.
– Examine each job to ensure that the use of multiple request-parameter lists
within a single program will not cause lock contention within that program.
– Code each job to handle the loss of positioning that occurs at commit or
backout for unpaired requests (an unpaired request consists of a GET UPD
not followed by a PUT UPD).
– Examine each job to ensure that it handles any new return and reason codes
from VSAM appropriately.
Coupling-facility planning
VSAM record-level sharing and DFSMStvs use coupling facilities to hold cached
copies of individual records and to hold lock and log structures. Planning the use of
coupling facilities is important to ensure that you meet your availability and
performance objectives. This section introduces some of the coupling-facility
functions and provide information to help you decide the type and number of
coupling facilities and how much storage you need to implement DFSMStvs.
For more information about coupling facilities, see z/OS MVS Setting Up a Sysplex.
Coupling facilities
A coupling facility provides locking, caching, and list services between
coupling-capable z/OS processors. Coupling-facility links connect a coupling facility
to the coupling-capable processors. The coupling-facility control code (CFCC)
provides the coupling-facility functions. The CFCC can run in a processor
resource/systems manager (PR/SM™) logical partition (LP) in the following
processors
v IBM Enterprise System/9000® (ES/9000®) 9021 711-based processor
v An IBM S/390 Parallel Enterprise Server™ 9672 (IBM 9672)
v An IBM S/390® coupling facility 9674 (IBM 9674)
You can use the Integrated Coupling Migration Facility (ICMF) for migration and for
testing software that requires a coupling facility. ICMF provides the CFCC functions
without coupling-facility links and is limited to z/OS systems that run in an LP on a
single physical system. ICMF is available on the 9021 711-based processors, 9121
511-based processors, and IBM 9672s.
DFSMStvs is one of many exploiters of coupling facilities. You might already have
coupling facilities in use for one or more of these functions:
v Enhanced catalog sharing
v Tape drive sharing
v JES2 checkpoint
v Operator log
v Global resource serialization
v VTAM® generic resources
v SmartPipes
v Resource recovery services (RRS)
v CICS logs, temporary storage, and data tables
v DB2 locks and buffer pools
v IMS short message queue, locks and buffer pools
v VSAM record-level sharing (RLS) for caching and locking
While it is possible to set a coupling facility up as an LPAR, this is not the ideal
configuration. If you set the coupling facility up that way and the machine goes
down, you lose both the coupling facility and the SMSVSAM server that was
running in another LPAR. If there is only one coupling facility, this results in the loss
of the lock structure as well as the SMSVSAM server. Any data sets it might have
been accessing are now in lost locks throughout the Parallel Sysplex until the
SMSVSAM image comes back up and does lost locks recovery.
For information about non-RLS access to data sets that have lost locks, see
Appendix B, “Accessing data sets that have retained locks or lost locks”, on
page 121.
You can configure an internal coupling facility (ICF) in either of these processors:
v IBM Enterprise System/9000 (ES/9000) 9021 711-based processor
v IBM S/390 Parallel Enterprise Server 9672 (IBM 9672)
The 9672 G3 servers (or later models) support an ICF, in which one of the
central processors is used only to run CFCC, the licensed internal code of the
coupling facility.
Both z/OS images and a coupling facility could fail at the same time in the event of
a processor complex failure. In this situation, some structures cannot be rebuilt in
another coupling facility. In general, an internal coupling facility should be used as
backup for a production coupling facility if you do not also use coupling-facility
duplexing.
The batteries last for a few minutes if the coupling facility continues to run. You can
use the internal batteries to supply power for the power-saver state, in which they
keep only the memory active during a power outage.
In general, you should put lock structures and list structures for the system logger in
nonvolatile coupling facilities.
If you maintain the lock structure in a volatile coupling facility, a power outage could
cause a failure and loss of information in the coupling-facility lock structure. Should
that happen, all outstanding recovery (CICS restart and backout) for any affected
data sets must be completed before new sharing work is allowed for those data
sets.
For more information about log streams, see Chapter 4, “Setting up DFSMStvs
logging”, on page 47.
Coupling-facility size
The size of a coupling facility is defined as the total amount of storage it uses. Total
storage includes both the control areas that the coupling-facility control code
(CFCC) requires and data areas that applications use. The coupling-facility
allocation rules and coupling-facility allocation-increment size, a function of the level
of CFCC, affect the size of the coupling facility.
The storage in a coupling facility is divided into distinct objects called structures.
The majority of coupling-facility storage is used for structures. The Coupling Facility
Resource Management (CFRM) policy defines these structures.
To determine the size of the system logs, look at the individual batch jobs that you
plan to use with DFSMStvs and estimate how much space they are likely to require.
Determine how many batch jobs there are and how much data they are likely to
update per transaction.
An estimate of the amount of storage that you need in a coupling facility for
DFSMStvs should include an estimate of the space required for the primary and
secondary system logs (undo log and shunt log) and the additional space required
in the log of logs and forward recovery logs. The undo log and shunt log need to be
large enough for backout logging from all the batch jobs that you plan to use with
DFSMStvs, based on how much data that the jobs are likely to update for each
transaction. You also need to make the log of logs and forward recovery logs large
enough to account for the extra activity from the batch jobs; if these logs already
exist, you need to increase their sizes. These logs might need to be twice as large
if the batch activity would be roughly equivalent to CICS activity and you do not
plan to change your frequency of backups or of clearing out space in the forward
recovery logs.
For information about other structures, see Parallel Sysplex Configuration, Volume
2: Cookbook.
Lock-structure sizing
You define the coupling facility lock structure, IGWLOCK00, in the CFRM policy,
using the IXCMIAPU utility. The following formula is for estimating the size of
IGWLOCK00 (in megabytes) for VSAM RLS. You can use this formula and then add
additional space for the extra locking activity by batch jobs that use DFSMStvs.
10 MB * number_of_systems * lock_entry_size
Use the information in Table 1 to determine the actual lock-entry size for different
MAXSYSTEM values.
Table 1. Effect of MAXSYSTEM value on lock-table-entry size
MAXSYSTEM value Lock-entry size
7 or fewer 2 bytes
> = 8 and < 24 4 bytes
> = 24 and < 56 8 bytes
Use these estimates as rough initial values. Other factors influence the size of the
lock structure, such as false contentions and retained locks. For more information
about the sizing of the lock structure, see z/OS DFSMSdfp Storage Administration
Reference.
For more information about the MAXSYSTEM parameter, see z/OS MVS Setting Up
a Sysplex.
Cache-structure sizing
You need to define cache structures large enough for the the activity of VSAM RLS
as well as the additional activity of batch jobs that use DFSMStvs. The rest of this
section discusses cache-structure sizing for VSAM RLS.
For the best performance with VSAM RLS buffering without DFSMStvs, the total of
all of the coupling-facility cache-structure sizes that you define (the coupling-facility
cache) should be the total of the VSAM local-shared-resource (LSR) buffer-pool
sizes used in non-RLS mode. The VSAM LSR buffer-pool size is the sum of the
LSR pool size and if used, the corresponding Hiperspace™ pool size.
You can run VSAM RLS with less coupling-facility cache storage than this, but the
coupling-facility cache must be large enough for the coupling-facility cache
directories to contain an entry for each of the VSAM RLS local buffers across all
Performance should improve when the coupling-facility cache is larger than the sum
of the local VSAM LSR buffer pool sizes. When the coupling-facility cache is
smaller, performance depends on the dynamics of the data references among the
systems involved. In some cases, you might want to consider increasing the size of
a very small (2 MB to 10 MB) coupling-facility cache.
For more information about CICS TS structures, see the CICS Transaction Server
for z/OS Migration Guide.
List-structure sizing
The size of list structures is calculated by determining the sum of the log stream
sizes. To estimate the total size of the log streams, see “Log stream sizing” on
page 36
Processor-capacity planning
Before you can use DFSMStvs functions, you must have implemented a Parallel
Sysplex and VSAM record-level sharing (RLS). In the same way that it is difficult to
estimate the resources needed by coupling facilities, it is difficult to estimate the
additional processor use for DFSMStvs applications.
Software-configuration planning
If you have multiple z/OS images in a Parallel Sysplex, you can get near-continuous
availability of DFSMStvs. If one z/OS system fails, the others can continue to run.
When you stop one of the systems for hardware or software maintenance, you do
not need to stop all the systems. If you have an environment with multiple z/OS
JES2 systems, you should use JES2 Multiple Access Spool (JES/MAS). If you use
JES2 MAS, batch jobs can be executed on any of the systems. In the case of a
system failure, ARM supports started task and job restart on the other systems.
If your current software configuration has only a single z/OS image, you can still
use DFSMStvs if you are not running it in LOCAL mode and the system is set up as
a single-system sysplex. You can share VSAM data sets between CICS systems
and batch programs, or among batch programs. By using sharing, you can aim for
24-hour availability for your online CICS system, and you can minimize or even
eliminate your batch window. For batch jobs alone, DFSMStvs offers you the ability
to run jobs in parallel against the same data.
System-logger planning
The system logger provides a sysplex-wide logging mechanism for VSAM, CICS,
and RRS. An integrated MVS function designed to support system and subsystem
components in a Parallel Sysplex, the system logger implements a set of services
that enables applications to write, browse, and delete log data. The system logger
exploit coupling facility technology.
You can use the system logger services to merge data from multiple instances of an
application, including from different systems across a Parallel Sysplex. The system
logger significantly reduces the complexity of managing multiple logs (which would
otherwise be required) and allows a single-system view of the log data in a Parallel
Sysplex environment.
Log streams
The log data area that subsystems use to write log entries is called a log stream.
DFSMStvs, SMSVSAM, RRS, and CICS use log streams to write logs.
Primary system log: The primary system log provides a separate backout log
stream for each instance of DFSMStvs, called the undo log. The name of the undo
log must be as follows:
IGTWTVnnn.IGWLOG.SYSLOG
The undo log must be allocated before you start DFSMStvs. This log stream is
connected during DFSMStvs initialization and disconnected when the DFSMStvs
instance ends.
A DFSMStvs instance logs its backout records in its undo log stream. These undo
log records are used to back out the uncommitted changes made by a unit of
recovery and are therefore written before the data is changed. An application can
initiate backout, or it can occur as a result of an error, such as an abnormal end of
the application.
Secondary system log: The secondary system log stream for DFSMStvs is called
the shunt log. The name of the shunt log must be as follows:
IGTWTVnnn.IGWSHUNT.SHUNTLOG
The shunt log must be allocated before you start DFSMStvs. This log stream is
connected during DFSMStvs initialization and disconnected when the DFSMStvs
instance ends.
Resource manager data log: This log includes the information about the
resource managers using RRS services.
Restart log: This log includes the information about incomplete units of recovery
needed during restart. This information enables a functioning RRS instance to take
over incomplete work left over from an RRS instance that failed.
Main UR state log: This log includes the state of active units of recovery. RRS
periodically moves this information into the RRS delayed unit-of-recovery state log
when unit-of-recovery completion is delayed.
Delayed UR state log: This log includes the state of active units of recovery,
when unit-of-recovery completion is delayed.
Archive log: This log includes the information about completed units of recovery.
This log is optional but recommended.
Forward recovery logs: The forward recovery logs compose a set of user-defined
log streams that is connected when the first data set that uses it is opened and
disconnected when the last data set that uses it is closed.
These log streams are shared across all instances of DFSMStvs. The forward
recovery log stream used for recording the redo records for a data set is specified
by the log stream ID attribute of the data set in the catalog entry. You can define
the log stream ID at a data set level. The minimum number of log streams is one,
and the maximum number is the same as the number of forward recoverable data
sets. You can determine the number of forward recovery log streams and assign the
log streams to the recoverable data sets.
Because redo log records contain changed images of records, they are written after
the changed records are written to the data set. One exception to this is the case
where sequential I/O is used and the records are buffered prior to being written out.
The redo log stream might also contain compensating records, which are written
during backout processing to indicate that records were restored to their original
state.
DFSMStvs and CICS write to forward recovery log streams, and forward recovery
programs, such as CICSVR, use them.
Log of logs: This log stream is used only if one is specified in the IGDSMSxx
member of SYS1.PARMLIB or the CICS definition. Definition of the log of logs is
optional. If you define a log of logs, it must be present; otherwise, DFSMStvs jobs
would fail. It contains copies of the tie-up records and file-close records written to
forward recovery logs and is used by forward recovery utilities, such as CICSVR, to
automate the forward recovery process.
The CICS system log is a single logical stream made up of two separately defined
log streams: the primary system log and the secondary system log. If CICS uses
recoverable data sets, one primary system log and one secondary system log are
needed for each CICS instance.
CICS and DFSMStvs should share the log of logs so that forward recovery can be
done correctly.
Recommendation: Put the same kinds of log streams in the same structure.
For example, in a cloned CICS environment, all the primary system logs of the
application owning regions have the same attributes, so they can be put in the
same structures. In contrast, the primary system log of an application owning region
and the primary system log of a TOR have completely different characteristics, so
they should not be put in the same structure.
Recommendation: For DFSMStvs log streams, do not put the primary system log
and the secondary system log in the same structure. You can put all primary system
logs in the same structures. Similarly, you can put all secondary system logs in the
same structures.
For RRS log streams, the main UR state log, the delayed UR state log, and the
archive log can be put in one structure. The resource-manager data log and the
restart log should be put in another structure.
Both a DASD-only log stream and a coupling facility log stream can have data in
multiple DASD log data sets; as a log stream fills log data sets on DASD, the
system logger automatically allocates new ones for the log stream.
If you use DFSMStvs in a single-z/OS environment, you can use DASD-only log
streams for all log streams. Even if you run in a multiple z/OS environment, the
following log streams can be placed in DASD-only log streams because they are
not shared:
v Primary system log for DFSMStvs
v Secondary system log for DFSMStvs
v Primary system log for CICS
v Secondary system log for CICS
If you use DASD-only log streams, performance problems can easily occur. In
general, you should use the coupling facility for all production log streams because
you must set up the coupling facility for DFSMStvs locking anyway.
Recommendation: Do not use DASD-only log streams for the log of logs or for
forward recovery logs in a multisystem sysplex environment because they cannot
be shared. The first system that opened a data set would be able to use the logs,
but any subsequent opens on other systems would fail because they could not
connect to the forward recovery logs.
You can use the DFHLSCU utility, which CICS supplies, to help calculate the
amount of coupling-facility space that you need and the average buffer size of your
log streams. For information about how to use the DFHLSCU utility, see CICS
Operations and Utilities Guide.
For more information about estimating the size of log streams, see Parallel Sysplex
Configuration, Volume 2: Cookbook.
Recommendation: Do not set up your sysplex with two LPARs, your MVS image
and your coupling facility, on one machine.
If you want to protect against this double failure, you can keep the log data on disk
in DASD staging data sets. The performance for log writes can degrade, however, if
you choose DASD staging data sets, which IBM does
Recommendation: Do not use DASD staging sets for DFSMStvs logs except to
protect against a double failure of MVS and the coupling facility.
If you decide to use DASD staging data sets, you need one per log stream. A
DASD staging data set is a VSAM linear data set, and it should be managed by
SMS. The following recommendations apply to defining log streams:
v Define STG_DUPLEX(YES) and DUPLEXMODE(COND) for log streams that are
associated with the system log. This ensures that the system logger
automatically duplexes to DASD staging data sets if it detects that the coupling
facility is not failure independent.
v If you are operating with only a single coupling facility, define
STG_DUPLEX(YES) and DUPLEXMODE(UNCOND) for log streams that are
associated with the system log.
v Define STG_DUPLEX(YES) for log streams that are associated with forward
recovery logs. If you do not and a failure causes loss of data from the log
stream, you would need to take a new image copy of the associated VSAM data
sets. There would be a consequent period of time until this was complete when
the data sets would not be fully protected.
v Define each DASD staging data set to be at least the same size as the
associated log stream’s share of the coupling facility, but round the average block
size up to 4 K.
The recommended value for HIGHOFFLOAD is 85 percent for the primary system
log.
Log trimming
DFSMStvs deletes most of the log stream entries in the undo log and shunt log
automatically. For example, SMSVSAM periodically requests that the system logger
delete old entries in the system log. This activity is called log trimming.
Log trimming does not apply to the log of logs or to any forward recovery log. Your
installation is responsible for the maintenance of these logs.
Excessively large units of recovery might defeat the process of log trimming. Units
of recovery might become excessively large if you run a batch job that does not
issue commit or backout calls. Such programs would cause performance problems
because they could force log offloads that would otherwise be unnecessary. They
could also interfere with your online applications by holding a large number of locks.
Recommendation: Modify your batch jobs to issue periodic sync points before you
convert them to use DFSMStvs.
Sizing
In general, the primary and secondary system logs should not be offloaded because
you need high performance access. So if you have enough storage in the coupling
facility for the system logger log streams, you do not need to spend much time
estimating the capacity needed for DASD log data sets.
In contrast, you should estimate the capacity for DASD log data sets needed by
forward recovery log streams and the RRS archive log stream because you will
keep the log information until you think it is no longer needed to perform a forward
recovery, probably because you have taken one or more backup copies. When you
decide that you no longer need the forward recovery log entries, you need to
archive or delete them yourself.
For example, if you take a backup copy of your VSAM data sets once a week, you
might choose to delete the forward recovery log stream in DASD log data sets one
week after taking the backup. In this case, you would estimate the capacity needed
by the DASD log data sets for your redo logs for one week.
Recovery procedures
DFSMStvs gives you the ability to do both forward recovery and backward recovery
using a separate program such as CICSVR. You can either select both of them or
just backward recovery at a VSAM data set level.
If you select forward recovery, the cost of running applications increases because of
the additional logging activity. You also need operational procedures for log
management and forward recovery.
Basically, backups of VSAM data sets are not used on their own. You perform
forward recovery to return the data sets right up to the point of a failure.
Reorganization
If you need to reorganize your VSAM data sets to remove the effects of control
interval and control area splits or to increase the space allocated, you will still need
to perform these tasks while nobody else is using the data sets.
ARM is a recovery function that provides improved availability for batch jobs and
started tasks by automatically restarting them after they unexpectedly end. In a
Parallel Sysplex, ARM provides additional benefit through its ability to restart
registered clients on another MVS system image in the Parallel Sysplex if the MVS
system image they were originally using fails. Transaction managers, resource
managers, and restartable subsystems can be restarted automatically. Any system
that require automatic restart must register with ARM. Systems affected by a failure
are usually restarted on the same MVS system image, or on a different one if the
MVS system image itself has failed.
The cost of computing, productivity, and availability are improved because shared
resources and transactions in progress can be recovered and lost function and
services restored. The following points apply to the use of ARM:
v ARM provides job and started task recovery. Transaction or database recovery is
the responsibility of the restarted applications.
v ARM does not start the applications initially (that is, at the first IPL or subsequent
IPLs after failures). This is the function of automation or production control
products. Interface points are provided through exits, the Event Notification
Facility (ENF), and macros.
v The MVS system image or Parallel Sysplex must have sufficient spare capacity
to guarantee a successful restart.
v Elements that are on an MVS system image that fails are restarted on another
MVS system image.
ARM support is part of the overall task of planning for and installing Parallel
Sysplex support. For more information about ARM, see z/OS MVS Setting Up a
Sysplex.
If the SMSVSAM address space fails, all in-flight units of recovery that were using
the DFSMStvs instance at the time of failure are backed out. The backouts for
these units of recovery are not performed at the time of the failure by the failed
DFSMStvs instance. The backouts are performed either at the time of the failure or
later by a peer recovery instance of DFSMStvs.
However, in the case of a z/OS system failure, restart can take a long time in
comparison, and the process is not automatic. So in-flight units of recovery can
remain for a long time. To prevent this situation, DFSMStvs provides a function
known as peer recovery.
When peer recovery occurs, another SMSVSAM instance in the sysplex (a peer)
performs the backout for in-flight units of recovery that were being processed by the
SMSVSAM address space that failed. Peer recovery will start automatically if there
is an ARM policy (created using IXCMIAPU) which includes DFSMStvs and any
other resource managers which might be involved in a unit of recovery in a restart
group.
If you do not use ARM, you should change your operational procedures to issue the
following command in another system as soon as possible in case of a system
failure:
VARY SMS,TRANSVSAM(tvsname),PEERRECOVERY,ACTIVE
Installation of DFSMStvs
For information about enabling DFSMStvs on your z/OS or z/OS.e system, see
″Installation Information″ in the ZOS V1R4 PSP upgrade, subsets ZOSGEN and
DFSMS.
For information about enabling DFSMStvs on your z/OS or z/OS.e system, see
″Installation Information″ in the ZOSV1R4 PSP upgrade, subsets ZOSGEN and
DFSMS. For migration considerations, see the migration chapter of z/OS DFSMStvs
Administration Guide.
http://www-1.ibm.com/servers/eserver/zseries/pso/
Because DFSMStvs writes to forward recovery log streams, the space required for
these log streams might increase. Similarly, because DFSMStvs writes to the
primary and secondary system log streams (undo and shunt log streams), the
demand increases for space in the coupling-facility structure. DFSMStvs moves
long-running transactions and transactions that become shunted (due to backout
failure) from the undo log stream to the shunt log stream.
The size of the coupling-facility structure for the primary log stream can affect
performance. If the structure is too small, the system logger moves log buffers off to
DASD spill data sets. While the system logger is moving log buffers, it is unable to
service write requests from DFSMStvs.
One dedicated coupling facility plus a coupling facility LPAR provides the next
safest environment for normal logging and lock-structure use. This environment has
the same advantages of rebuilding with minimal impact to DFSMStvs instances that
are running. Furthermore, MVS detects that the LPAR coupling facility is not in a
failure-independent domain and causes the system logger to write log stream data
to staging data sets for extra security.
The DFSMS cache structures are store-through caches, and no changed data
resides in them. Store-through caching means that a user writes changed data to
the cache structure and to permanent storage at the same time and under the
same serialization. This way, the data in the cache structure always matches the
data in permanent storage.
Because there is no chance of data loss, the DFSMS cache structures do not need
to be in a nonvolatile coupling facility. The DFSMS lock structure has persistent
connections and is a persistent structure. Data loss in the lock table requires a
double failure. For example, a double failure occurs at the loss of the
coupling-facility structure and the failure of one system in the sysplex at the same
time. VSAM RLS also has a sharing-control data set to protect the integrity of the
data even if there is a double failure. If this protection is not enough, the DFSMS
lock structure can be placed in a nonvolatile coupling facility.
DFSMStvs caches all of its data in DFSMS cache structures that you define. The
DFSMS lock structure and the system logger do all the logging for DFSMStvs. The
IGWLOCK00 coupling-facility lock structure controls all VSAM RLS and DFSMStvs
locks. DFSMStvs defines the names of its undo and shunt log streams, and you
define the DFSMStvs log of logs and forward recovery log streams. RRS, which
must be active for DFSMStvs to function, needs five log streams to be defined.
If you run with a single coupling facility, its failure would cause the system logger
and any other users of the coupling facility to suspend normal operation until
access to the coupling facility is restored. DFSMStvs would, effectively, be unusable
in such a situation.
If a batch job attempts to communicate with RRS and the system logger goes
down, RRS suspends the job until the logger comes back up. You cannot cancel
the batch job because RRS has suspended it. If you cancel RRS or bring the
system logger back up, however, RRS releases the batch job, which then ends;
otherwise, the job remains suspended.
Unless you specify that the system logger is to use staging data sets, the recovery
of log stream data depends on the MVS images remaining active. This is so that
the system logger can use copies of log records that are held in storage to
repopulate the coupling facility when it is again available. If you must run with a
single coupling facility, specify the DUPLEXMODE(UNCOND) command to force the
use of staging data sets.
You can specify whether duplexing to staging data sets is either conditional or
unconditional, depending on whether or not the coupling facility is in an independent
failure domain. If conditional, the system logger chooses either the data space
option or a staging data set; if unconditional, duplexing to a staging data set is
forced.
MVS normally keeps a second copy of the data written to the coupling facility in a
data space. The second copy is used to build a coupling-facility log in the event of
an error. This is satisfactory as long as the coupling facility is failure-independent (in
a separate central processor complex (CPC) and nonvolatile) from MVS.
The coupling facility is in the same CPC or uses volatile storage, the system logger
supports staging data sets for copies of log stream data. That log stream data
would otherwise be vulnerable to failures that impact both the coupling facility and
the MVS images.
For the syntax and descriptions of these fields, see z/OS MVS Initialization and
Tuning Reference.
Recommendations:
v Share the IGDSMSxx parmlib member across all systems in the sysplex.
v After a cold start, take the following steps for any data set for which recovery
was not complete:
1. Recover the data set manually (without using forward recovery).
2. Take a backup of the data set and of any other data set that uses the same
forward recovery log.
3. Delete and redefine the data set’s forward recovery log.
If recovery was not complete for a data set, it is most likely in a damaged state,
and you must recovered it manually. If the data set is forward recoverable, then
its forward recovery log might also be damaged.
The DFSMStvs instance names must be unique. Because some parameters are
positional, ensure that you specify the parameters in the proper order.
Recommendation: You should cold start DFSMStvs only when the log has been
damaged and cannot be trusted to use for backouts. In any other situation, you
should always warm start DFSMStvs because you would leave your data sets
damaged if you did a cold start. Before a cold start, print the log so that you have
some information about what might be damaged.
The recoverability of the data set determines the amount of logging that DFSMStvs
does:
LOG(NONE)
The data set is nonrecoverable, and no logging is done. Nonrecoverable
data sets are not normally accessed in DFSMStvs mode, but are accessed
in DFSMStvs mode if CRE is specified.
LOG(UNDO)
The data set is recoverable and receives only backout logging. Syncpoint
processing is done on its behalf, but the data set is not forward
recoverable.
LOG(ALL)
The data set is recoverable and receives both backout and forward
recovery logging.
STRUCTURE NAME(OPERLOG_STR )
SIZE(60512)
PREFLIST(CFSVTX,CFSVTY,CFSVTZ)
STRUCTURE NAME(LOG_DFHLOG_001 )
SIZE(10240)
PREFLIST(CFSVTX,CFSVTY,CFSVTZ)
STRUCTURE NAME(LOG_DFHLOG_002 )
SIZE(10240)
PREFLIST(CFSVTX,CFSVTY,CFSVTZ)
STRUCTURE NAME(LOG_DFHLOG_003 )
SIZE(10240)
PREFLIST(CFSVTX,CFSVTY,CFSVTZ)
STRUCTURE NAME(LOG_DFHLOG_004 )
SIZE(10240)
PREFLIST(CFSVTX,CFSVTY,CFSVTZ)
STRUCTURE NAME(LOG_DFHSHUNT_001)
SIZE(10240)
PREFLIST(CFSVTX,CFSVTY,CFSVTZ)
STRUCTURE NAME(LOG_DFHSHUNT_002)
SIZE(10240)
PREFLIST(CFSVTX,CFSVTY,CFSVTZ)
STRUCTURE NAME(LOG_DFHSHUNT_003)
SIZE(10240)
PREFLIST(CFSVTX,CFSVTY,CFSVTZ)
STRUCTURE NAME(LOG_DFHSHUNT_004)
SIZE(10240)
PREFLIST(CFSVTX,CFSVTY,CFSVTZ)
STRUCTURE NAME(LOG_USERJRNL_001)
SIZE(10240)
PREFLIST(CFSVTX,CFSVTY,CFSVTZ)
STRUCTURE NAME(LOG_USERJRNL_002)
SIZE(10240)
PREFLIST(CFSVTX,CFSVTY,CFSVTZ)
STRUCTURE NAME(LOG_USERJRNL_003)
SIZE(10240)
PREFLIST(CFSVTX,CFSVTY,CFSVTZ)
STRUCTURE NAME(LOG_USERJRNL_004)
SIZE(10240)
PREFLIST(CFSVTX,CFSVTY,CFSVTZ)
STRUCTURE NAME(LOG_GENERAL_001)
SIZE(10240)
PREFLIST(CFSVTX,CFSVTY,CFSVTZ)
/*
Each log stream is allocated a proportion of the structure space, based on the
number of currently connected log streams (up to the limit specified in LOGSNUM).
For example, a structure can be defined to contain a maximum of 30 log streams. If
only 10 log streams are connected, each log stream can use one tenth of the space
in the structure. As other log streams are connected and disconnected, the system
logger adjusts the proportion of space to each log stream used.
The examples above define one structure per log. Multiple log streams can be
within a single structure so defining this many log structures is optional.
You must define the log streams for the DFSMStvs undo log and shunt log to the
system logger before accessing a recoverable VSAM data set in DFSMStvs mode.
For each log stream that is in use, DFSMStvs needs two buffers. Each buffer can
be a maximum block size of 64 KB for the log stream, and approximately 600 bytes
of state data, all of which are above the 16 MB line. The SMSVSAM server address
space contains the buffers.
A log stream is a sequence of data blocks, with each log stream identified by its
own log stream identifier, which is called the log stream name. The DFSMStvs
system logs, log of logs, and forward recovery logs map to specific MVS log
streams. Each log stream is a sequence of blocks of user data, which the system
logger internally partitions over three different types of storage:
Primary storage
This is a structure within a coupling facility that holds the most recent
records written to the log stream. Log data written to the coupling facility is
also copied to either a data space or a staging data set.
Secondary storage
When the primary storage structure for a log stream becomes full, the older
records automatically spill into secondary storage. The secondary storage
structure consists of data sets that are managed by the storage
management subsystem. Each log stream, identified by its log stream
name, is written to its own log data sets.
The system logger provides access to these logs. DFSMStvs has access to the log
stream data during recovery operations, when it can browse the data, read forward
and backward, or read any block directly. Figure 8 on page 53, Figure 9 on page 58,
and Figure 10 on page 59 show you how you can define log streams for DFSMStvs
to use. In these examples, each of the log streams has its own structure; your log
streams can share structures.
Recommendations:
v Specify DIAG(YES) for the DFSMStvs system logs. This parameter indicates that
special logger-diagnostic system activity is allowed for this log stream. This is
especially useful when certain system logger errors occur, such as an X'804' type
condition, which indicates that some data might have been lost.
v Log stream and log stream staging data sets are single extent VSAM linear data
sets that require SHAREOPTIONS(3,3). The default is SHAREOPTIONS(1,3).
Recommendation: Explicitly specify the SHAREOPTIONS values.
Recommendation: Take frequent sync points to minimize the amount of data being
held in the coupling facility.
Generally, the volume of data that DFSMStvs keeps in the primary system log at
any one time covers between two and three activity keypoints. The activity keypoint
frequency, which is measured in blocks of log data, and defined by the AKP
parameter in the IGDSMSxx member of SYS1.PARMLIB, determines this volume.
The way that DFSMStvs manages the system log data makes the frequency of
activity keypointing an important factor when planning the size of the primary log
stream. Use the AKP parameter to specify keypointing frequency. Review the
activity keypoint (AKP) frequency defined for each DFSMStvs instance. The larger
the value, the more coupling facility space you need for the system logs. But do not
set AKP so low that transactions last longer than an activity keypoint interval.
DFSMStvs manages the size of the system log by deleting old, completed units of
work (log tail deletion). If you need long-term data retention, then you might want to
copy the data from the log stream into alternative archive storage. For more
information about the AKP parameter, see z/OS MVS Initialization and Tuning
Reference.
A log tail is the oldest end of the log. At each activity keypoint, DFSMStvs deletes
the tail of the system log by establishing a point on the system log before which all
older data blocks can be deleted. If the oldest ″live″ unit of work is in data block x,
all data blocks older than x (x-1 and older) can be deleted. DFSMStvs keeps the
two most recent, complete activity keypoints on the primary system log and deletes
data from complete units of work older than this.
The system log is designed to ensure the availability of logged data following a
system or DFSMStvs failure, thereby maintaining data integrity. Do not use the
system log for any other purpose. Forward recovery logs cannot be written to
system log streams.
If DFSMStvs fails or abnormally ends, all in-flight units of recovery that were using
DFSMStvs at the time of its failure are backed out. A unit of recovery that is in-flight
is one that has made a change to a recoverable resource but has not yet
committed or backed out that change. The backouts for these units of recovery are
not performed at the time of the failure by the failed DFSMStvs instance. The
backouts are performed either at the time of the failure or later by a peer recovery
In the event of a DFSMStvs failure during the backout, the recovery or restart
processing repeats the entire backout. Backout processing tolerates duplicate
records and attempts to delete nonexistent record conditions, which arise from the
attempt to repeat a backout operation that was previously completed.
The backout records for in-doubt and long-running units of recovery present
space-management problems within undo log streams. Ideally, the backout records
in an undo log stream have a short life cycle. This enables the obsolete portion of
the log stream to be deleted. Also, the system logger does not need to off-load the
log data from the coupling facility to DASD data sets. Units of recovery that do not
reach an end-of-unit of recovery status within a short period of time interfere with
this space management algorithm.
If too much old data is left in the log, there are two conditions that can occur when
attempting to write records to a log stream. First, the system logger could return a
return-and-reason code that indicates that the coupling facility storage limit was
reached. Second, the system logger can return a return-and-reason code that
indicates that the staging data set storage limit was reached. In either case, the
system logger offloads data to DASD. DFSMStvs cannot write any further
information to its log streams until the problem has been resolved.
DFSMStvs uses a secondary log, called a shunt log. This log tracks units of
recovery for which DFSMStvs is unable to complete processing, for example due to
an I/O error or unavailability of a resource, such as a volume or a cache.
Backout records are not written to the undo log for a PUT UPD or ERASE request
because these operations modify existing records that are first obtained using GET
UPD.
DFSMStvs logs the forward recovery records to system-logger log streams. These
log streams are shared across DFSMStvs instances and provide sysplex-wide log
streams that you can use as input to forward recovery products, for example,
CICSVR.
For data sets with LOG(ALL) specified, DFSMStvs backout processing creates the
forward recovery log records. These records, due to their more recent time stamps,
supersede the corresponding forward recovery records generated during earlier
processing of the unit of recovery. The log records are in the order in which the
changes are made. When forward recovery is needed, the forward recovery product
reads through the records and actually makes the change, and then find the
compensating record and undoes the change. The records do not supersede one
another; both changes are applied.
You can use one forward recovery log stream for multiple data sets, so you do not
need to define a log stream for each forward recoverable data set. Consider the
following criteria before you decide to share a forward recovery log stream:
v Your installation’s ability to balance transaction performance
v Rapid recovery
v The work involved in managing a large number of log streams
The system logger can store all the forward recovery log records from multiple
DFSMStvs instances in a shared forward recovery log.
You can have a separate forward recovery log by data set, rather than by job. A
separate forward recovery log by data set has these benefits:
v Recovery can be more straightforward with a one-to-one relationship between
logs and the data sets to be recovered.
v Shared logs are related to multiple jobs and units of work using the same data
set.
The log of logs provides a summary of which recoverable VSAM data sets
DFSMStvs has used, when the data sets were used, and to which log stream the
forward recovery log records were written. If you have a forward recovery product
that can utilize the log of logs, ensure that all DFSMStvs instances sharing the
recoverable data sets write to the same log of logs. Do not share the log of logs
between test and production DFSMStvs instances. Sharing the log of logs could
compromise the contents of production data sets during a restore operation.
Figure 9 on page 58 gives an example of defining forward recovery log streams for
DFSMStvs and CICS.
The log of logs provides data set recovery products such as CICSVR with the
information required to control forward recovery. If you use both DFSMStvs and
CICS, use the same log of logs for both. When an application opens a file to
access a data set, a tie-up record records the association between the file and the
data set in the forward recovery log.
If you do not want DFSMStvs to write to a log of logs, omit the LOG_OF_LOGS
parameter from the IGDSMSxx member of SYS1.PARMLIB.
If you use DFSMStvs in a sysplex and you use a log of logs, all DFSMStvs
instances that access the same recoverable data sets should share the log of logs
as a single log stream. Figure 10 gives an example of defining a log of logs for
DFSMStvs and CICS.
Bypassing the checking step also applies to the checking done for the CHKAUTH
operand on the RACROUTE REQUEST=DEFINE macro instruction. All other RACF
processing occurs as usual.
Bypassing the checking step also applies to the checking done for the CHKAUTH
operand on the RACROUTE REQUEST=DEFINE macro instruction. All other RACF
processing occurs as usual.
If the VSAM RLS server address space is neither PRIVILEGED nor TRUSTED,
grant the SMSVSAM server the appropriate access authorization for log stream
profiles. Specify the authorization in the RACF LOGSTRM general resource class
for the user ID of the VSAM RLS address space, SMSVSAM. First you must
associate SMSVSAM with an RACF-defined user ID, such as SYSTASK, in the
started procedures table. Do not use a user ID of SMSVSAM.
DFSMStvs has no facility for controlling LOGSTRM security checks. These are
controlled when the MVS security administrator activates the LOGSTRM general
resource class by means of the SETROPTS command. If the LOGSTRM resource
class is active, DFSMStvs needs update authority for the log stream profiles for the
logs to which it writes. Use the user ID of the VSAM RLS address space,
SMSVSAM, when you grant access. Define the log streams to MVS.
Permit read access to users who need to read the DFSMStvs system log streams,
but not update access. Only DFSMStvs instances should have update access to the
primary system log stream (undo log stream) and secondary system log stream
(shunt log stream).
DFSMStvs also writes to forward recovery log streams and a log of logs that is
used to optimize forward recovery. To protect these log streams, code the
appropriate RDEFINE commands and PERMIT commands for each of them.
For all forward recoverable data sets that DFSMStvs accesses, grant DFSMStvs
access to these logs:
v The log of logs
v The forward recovery log streams
Each of these log streams requires update authority for each of the other log
streams. For example, issue the commands shown in Figure 12:
For peer recovery to be possible, grant the VSAM RLS server the authority to read
and write to the log streams of the DFSMStvs instances on other systems. Issue
the commands shown in Figure 13 on page 62 to grant the VSAM RLS server
authority:
Figure 13. Example of RACF commands to grant VSAM RLS authority to read and write log
streams
Use care if a batch job uses multiple RPLs. Using different RPLs to access the
same record can cause lock contention within a unit of recovery. Because sync
points (commits or backouts) cause all locks to be released, the application cannot
depend on locking across a sync point.
Defining transactions
You should not simply modify an existing batch job to use DFSMStvs with no further
changes. This would cause the entire job to be seen as a single transaction and
locks would be held. The log records would need to exist for the entire life of the
Batch jobs cannot open recoverable data sets for output in RLS mode; only CICS
can do that. In an application that shares a recoverable data set, a batch job must
use DFSMStvs to access the data set for output. The application must also support
transactional recovery.
VSAM data sets are stored on direct access storage devices (DASD). VSAM
divides its data set storage into control areas, which are further divided into control
intervals. Control intervals are the unit of data transmission between virtual and
auxiliary storage. Each one is of fixed size and, in general, contains a number of
records. A KSDS or ESDS can have records that extend over more than one control
interval. These are called spanned records.
Table 3 shows the type of open resulting from the parameters specified for a VSAM
data set opened by a batch job (non-CICS). The left column shows the type of data
set and type of open. Column headings indicate the RLS option specified.
Table 3. Opening recoverable and nonrecoverable VSAM data sets from batch jobs
Data set type and NRI CR CRE
type of open
Recoverable open for RLS RLS DFSMStvs
input
Recoverable open for DFSMStvs DFSMStvs DFSMStvs
output
Nonrecoverable open RLS RLS DFSMStvs
for input
Nonrecoverable open RLS RLS DFSMStvs
for output
The results of opening different types of VSAM data sets with different read integrity
options (NRI, CR, or CRE) depend on whether the batch job opens the data set for
input or output, as follows:
1. If you open a recoverable data set for output from a batch job (non-CICS) and
you specify RLS with any option (NRI, CR, or CRE), you get a DFSMStvs
open.
2. If you attempt to open a recoverable data set for output from a batch job
(non-CICS) and you do not specify RLS, the open will fail (unless the open
attempt is during your batch window processing and you have disabled access
by CICS to the data set).
3. If you open a recoverable data set for input from a batch job (non-CICS) and
you specify RLS with any option (NRI, CR, or CRE), you get an RLS open.
4. If you open a recoverable data set for input from a batch job (non-CICS) and
you do not specify RLS, you get whatever type of open you requested: NSR,
LSR, or GSR (provided share option 2 is in effect).
5. If you open a nonrecoverable data set for output from a batch job (non-CICS)
and you specify RLS as either NRI or CR, you get an RLS open.
6. If you open a nonrecoverable data set for output from a batch job (non-CICS)
and you specify RLS as CRE, you get a DFSMStvs open.
7. If you open a nonrecoverable data set for input from a batch job (non-CICS)
and you specify RLS as either NRI or CR, you get an RLS open.
8. If you open a nonrecoverable data set for input from a batch job (non-CICS)
and you specify RLS as CRE, you get a DFSMStvs open.
Do not rely on these implicit sync points. Performing syncpoint processing in this
manner can cause units of recovery to be much longer than necessary and
adversely affect system performance.
There is no need for an application that is using DFSMStvs services to provide the
URID when invoking VSAM. When VSAM invokes DFSMStvs services, DFSMStvs
determines the URID by using the RRS unit of recovery under which the VSAM
request was issued. VSAM builds the lock name using the unit of work identifier
(UOW ID) or URID, the record key, and the name of the CICS APPLID or
DFSMStvs instance.
If a request is made to update a recoverable data set, the associated exclusive lock
must remain held until the next sync point. This ensures that the resource remains
protected until a decision is made to commit or back out the request. If DFSMStvs
fails, VSAM continues to hold the lock until DFSMStvs restarts.
A VSAM request obtains the record lock and the lock is held until released by
DFSMStvs during commit or backout processing.
KEY HASH
18888888888888888 8888888888888888
18888888888888885 8888888888888885
28888888888888885 8888888888888885
The first two keys have different hash values but the last two have the same value.
If UR1 is accessing key 18888888888888885 and UR2 tries to access key
28888888888888885, UR2 gets an indication that it has contention with UR1. Since
they are trying to lock different records, this is false contention. The program would
treat it as a real contention, back out the unit of recovery and try again.
Avoiding deadlocks
When applications access data sets in VSAM RLS or DFSMStvs mode, deadlocks
can occur within a single instance. They can also arise between two instances of
DFSMStvs running under different MVS images. VSAM RLS performs deadlock
detection and resolution across systems, within its own resources. If it detects a
deadlock condition, VSAM RLS fails one of the requests to break the deadlock
cycle. DFSMStvs writes messages that identify the members of the deadlock chain.
Because timeouts are the only way VSAM RLS has for resolving cross-resource
deadlocks, you need to specify a timeout value for DFSMStvs requests. You can
specify the timeout value in the RLSTMOUT parameter in the IGDSMSxx member
of SYS1.PARMLIB, in the JCL, in the ACB, or in the RPL request. If you specify
different timeout values, the RPL specification overrides the ACB, which overrides
A unit of recovery should be entirely logically independent, not merely with regard to
protected resources but also with regard to execution flow.
Instead, the batch application must have a built-in method of tracking its processing
position within its string or series of transactions. One possible method is to use a
VSAM recoverable data set or a job-processing position file to track the commit
position for the job. This is a nonshared VSAM recoverable data set in which the
application records its progress. Every time it reaches a sync point, the application
makes a note of it in the job processing position file. Because it is a recoverable
file, if the unit of recovery is committed, so is the job processing position file.
Likewise, if the unit of recovery is backed out, the job processing position file is
backed out and correctly indicates that the last successful unit of recovery was the
prior one.
When the application fails, any uncommitted changes are backed out. The already
committed changes cannot be backed out because they are already visible to other
jobs and transactions. In fact, the records that were changed by previously
For this reason, it is important that jobs and applications using DFSMStvs be written
to execute as a string of transactions. They must use a commit point tracking
mechanism for restart. When they restart, they should skip any units of recovery
that were already completed and begin from the first unit of recovery that was either
not attempted or backed out.
One method of tracking the position of a job within the input stream is with a VSAM
recoverable data set. As the application executes its units of recovery, the
application writes information indicating which units of recovery have been
processed to the position file. At commit time, the position file and the application
data sets are in sync. If the unit of recovery is backed out, the change to the
position file is also backed out. It is not necessary for each input record to
correspond to a commit point, but the position file must have a record of each
commit or backout request.
Use a private (nonshared) recoverable data set to track the input stream of the job
at commit time. Use the following steps as a guide in setting up your application to
determine the restart point:
1. When the job is rerun, position the input stream to the position shown in the
private data set.
2. Read the next record from the input stream.
3. Update the private data set to reflect the new position.
4. Perform the indicated processing.
5. Issue the commit at the end of the transaction.
The position file and the application data sets are in sync.
For a list of the positioning states for reason codes for sequential, direct, and
skip-sequential processing, see the z/OS DFSMStvs Administration Guide.
It is possible for an application to close a data set that has uncommitted changes
(that is, a unit of recovery is in-flight). It is also possible for a data set to be deleted
or renamed before the unit of recovery completes. The data sets that have retained
locks and shunted log records associated with them could be deleted or renamed.
Recommendation: Do not delete or rename a data set unless you are doing so in
preparation for forward recovery (in which case, locks should first be unbound).
Deleting or renaming a data set can result in loss of any association between the
data set and its log records and locks.
The problem occurs because the access method services DELETE command
upgrades the enqueue for the data set MY.TVS.KSDS from shared to exclusive.
Various levels of authorization are required to use the SHCDS parameters. For
information about this autorization, see z/OS DFSMS Access Method Services for
Catalogs.
After a data set has been deleted or renamed, it is possible for another data set to
be created with the name. If any shunted log records related to the original data set
are not purged using the access method services SHCDS PURGE command, they
could be applied to the new data set in error.
Although users might not be using a data set, the data set could be in use if
DFSMStvs restart processing is in progress. If this is the case, it is not be possible
to delete or rename the data set until DFSMStvs is finished.
Because DFSMStvs automatically retries transactions that are in the shunt log, you
need to make sure that the log does not contain anything that cannot be retried. For
example, if you delete a data set for which recovery is owed, VSAM RLS releases
all of the locks associated with the data set. Suppose you then allocate a data set
with the same name as the deleted data set. If the shunt log still contains
outstanding work for the deleted data set, DFSMS might automatically retry this
work on the new data set of the same name, without any locks.
Recommendation: Before you delete a data set, use the the SHCDS PURGE
command to rid the shunt log of any outstanding work for the data set.
Various levels of authorization are required to use the SHCDS parameters. For
information about this autorization, see z/OS DFSMS Access Method Services for
Catalogs.
Alternatively, if you have units of recovery that were shunted due to backout
failures, you can resolve those units of recovery by following these steps:
1. Use the access method services SHCDS LISTSHUNTED(dsname) command to
list all the shunted entries for a particular data set.
2. Make certain that the problem that caused the failure has been corrected. This
might involve ensuring that the volume is available or that the data set has been
recovered.
3. Determine whether the shunted entries should be retried.
If you used the access method services SHCDS RESETLOCKS command to reset
the locks for a data set, the reset causes any attempt to take action on shunted log
entries for the data set to fail.
The same is true if you delete the data set without unbinding the locks. If you used
the access method services SHCDS FRUNBIND command, followed by the access
method services SHCDS RFDELETEUNBOUNDLOCKS command to reset the
locks for a data set, it causes any attempt to take action on shunted log entries for
the data set to fail.
Both problems cause DFSMStvs to see a unit of recovery that has log records but
no locks, which can be referred to as suspended state. A unit of recovery that is in
a suspended state cannot be retried because it is not safe to do so without the
locks. Use the access method services SHCDS command to purge the unit of
recovery.
You need to convert batch jobs that use DFSMStvs to update recoverable data
sets. This involves the following tasks:
v Modifying jobs to use sync points and take appropriate action (commit or
backout)
v Modifying jobs to use RRS services to invoke commit and backout processing
v Modifying jobs to specify that DFSMStvs is to be used
v Examining jobs to ensure that multiple RPLs in use within a single application
does not cause lock contention within a unit of recovery
v Coding jobs to handle loss of positioning. Loss of positioning might occur at
commit or backout for unpaired requests (a GET UPD request is not followed by
a PUT UPD request, or an IDALKADD request not followed by a PUT NUP
request)
v Examining jobs to ensure that any new reason codes are handled appropriately.
v If any applications act as work dispatchers, examining jobs to ensure that all
work intended to be part of a single unit of recovery is handled under the same
context
Another reason to use more advanced techniques would be if you would like to split
a batch process into several jobs. By splitting a batch process, originally written to
be a single threaded job, you can use DFSMStvs and run the jobs in parallel.
Another reason to use more advanced techniques would be if you have a batch
process that was originally written to be a single threaded job, but using DFSMStvs,
you would like to split it into several jobs that run in parallel.
A work manager, which takes work requests and parcels them out to tasks, might
create privately managed contexts. These contexts and the units of recovery
associated with them can be moved around, unlike the default contexts that every
TCB in the system has. So, if the work manager knows that a piece of work it is
about to submit belongs to a particular unit of recovery, the work manager can take
that unit of recovery’s privately managed context and attach it to the TCB before
giving that TCB the work to do.
Record management requests get the locks; commit and backout processing
release the locks. For example, GET UPDATE, PUT ADD requests get locks.
ENDREQ, PUT UPDATE and DELETE requests do not release locks. Commit and
backout processing release the locks unless there is a problem, in which case the
locks are retained.
Multitasking
DFSMStvs can be used within an application that uses multitasking. That is, the
application can consist of a mother task with multiple daughter tasks to which work
is allotted. For example, Figure 15 on page 79 shows a mother task and three
subtasks.
ATTACH
DA0TV800
One use for multitasking might be for an application that functions as a transaction
dispatcher. The following rules apply; if they are not followed, unpredictable results
can occur. The application must follow these rules because DFSMStvs has no way
to detect violations or to enforce these rules.
1. When DFSMStvs is invoked, it uses the current context. Normally, this is the
RRS native context that is associated with a dispatchable unit of work. A context
is a representation of a work request, and a unit of recovery is the work done by
or on behalf of the context between one point of consistency and another.
Therefore, all work within a single unit of recovery (transaction) must come
under a single context. When using native contexts, a single unit of recovery
cannot cross task boundaries. In the example above, subtask 1 and subtask 2
cannot do work on behalf of the same unit of recovery.
2. If an application wishes to share work for a single unit of recovery across task
boundaries, it must create one or more privately managed contexts. The
application is responsible for managing those contexts and ensuring that the
correct context is associated with a dispatchable unit when DFSMStvs is
invoked. This ensures that the work being done is associated with the correct
unit of recovery. Because privately managed contexts take precedence over
native contexts, associating a privately managed context with a dispatchable
unit causes it to become the current context.
3. Applications must ensure that the same context is used for operations that are
part of the same unit of recovery. If an application issued a GET UPD request
then switched contexts before issuing the corresponding PUT UPD request
unpredictable results could occur.
4. DFSMStvs allows context switching provided that your application completes all
asynchronous I/O. You can use the CHECK macro to wait for I/O associated
with an RPL to complete.
The additional work that DFSMStvs performs to obtain locks, to perform two-phase
commits, and to log updates consumes additional resources and affects
performance. This chapter describes how to monitor performance and how to tune
the DFSMStvs environment for the best performance.
Monitoring performance
You can use the following information and tools to monitor DFSMStvs performance:
v SMF record type 42 (hexadecimal 2A)
v SMF record type 88 (hexadecimal 58)
v RMF™ post-processor reports
v RMF monitor III
v CICS monitoring tools
v System messages
v Operator commands
For more information about SMF records, see z/OS MVS System Management
Facilities (SMF).
For more information about SMF records, see z/OS MVS System Management
Facilities (SMF).
The data set by job summary, DSNJ, might help you to identify how individual data
sets are being used.
The RMF component of z/OS contains additional Monitor III displays for RLS.
The CICS statistics titled File: requests information give you information about the
types of accesses to files open in CICS. You can produce reports with this
information by using the DFHSTUP program supplied with CICS.
You can also use CICS Monitoring Facility data to break down transaction response
time into its components, including I/O time.
System messages
Message IGW859I is issued to the system console when an individual unit of
recovery is holding more unique locks than the limit specified in the MAXLOCKS
parameter in SYS1.PARMLIB(IGDSMSxx). This message can help you identify a job
that is not issuing commits frequently enough or that has a logic problem, which is
probably causing performance problems for other users of DFSMStvs.
Operator commands
New options for the DISPLAY SMS command show you the status of these items:
v DFSMStvs facilities
v A specific job that is using DFSMStvs
v One or more units of recovery
v Entries in the shunt logs
v Log streams in use by DFSMStvs
v A data set being accessed in DFSMStvs mode
For information about these options, see z/OS MVS System Commands.
You can use the VARY SMS command to change the state of DFSMStvs, log
streams that DFSMStvs uses, and data sets that DFSMStvs processes. For
information about the VARY SMS command, see z/OS MVS JCL Reference.
Recommendation: Be careful about quiesce with a path name; always use a base
name.
Table 5 describes the effects of undo log states, shunt log states, log-of-log states,
and forward recovery log states.
Table 5. Effects of log states on DFSMStvs processing
Log type Log state Effect
Undo or shunt Enabled DFSMStvs services are available for use both by
in-progress units of recovery and new units of recovery.
Undo or shunt Quiescing DFSMStvs allows requests by existing units of recovery to
be processed but does not allow new units of recovery to
start. Quiescing the undo or shunt log is equivalent to
quiescing DFSMStvs.
Undo or shunt Quiesced There are no units of recovery using DFSMStvs services
and new units of recovery cannot start
Undo or shunt Disabling All DFSMStvs requests are failed, including those
submitted by in-progress units of recovery. Units of
recovery are neither committed nor backed out and cannot
be completed until DFSMStvs restart processing is run.
Disabling the undo or shunt log is equivalent to disabling
DFSMStvs.
Undo or shunt Disabled There are no units of recovery using DFSMStvs services
and new units of recovery cannot start. All data sets that
were open for DFSMStvs processing have been closed,
and DFSMStvs has unregistered with RRS and the lock
manager and disconnected from its logs.
Log of logs Enabled DFSMStvs writes tie-up records and file-close records to
the log of logs so that they can be used to optimize
forward recovery processing.
Log of logs Quiescing DFSMStvs writes file-close records to the log of logs for
any forward recoverable data sets that are currently open
but does not write any additional tie up records to the log
of logs.
Log of logs Quiesced All paired tie-up records and file-close records were written
to the log of logs, but no additional records are being
written to the log of logs.
Log of logs Disabling DFSMStvs runs without the log of logs. If forward recovery
is needed for any forward recoverable data sets, the
forward recovery utility will not be able to use the
optimization provided by the log of logs.
Event Effect
VARY command to enable The command is rejected. DFSMStvs is already initializing.
DFSMStvs
VARY command to quiesce The command waits until DFSMStvs finishes initializing to
DFSMStvs quiesce DFSMStvs processing and change its state to
quiescing.
VARY command to disable The command waits until DFSMStvs finishes initializing to
DFSMStvs disable DFSMStvs processing and change its state to
disabling.
RRS becomes unavailable DFSMStvs makes a note that RRS is unavailable.
DFSMStvs begins disabling itself and sets its state to
disabling when initialization reaches a point where it can do
so.
A system log I/O error occurs DFSMStvs makes a note that the I/O error occurred.
DFSMStvs begins quiescing itself and sets its state to
quiescing when initialization reaches a point where it can do
so.
Event Effect
VARY command to enable Command waits until DFSMStvs finishes initializing and then
DFSMStvs stacks behind the quiesce command.
VARY command to quiesce Command waits until DFSMStvs finishes initializing and is
DFSMStvs then rejected because such a command was already
issued.
VARY command to disable Command waits until DFSMStvs finishes initializing and then
DFSMStvs stacks behind the quiesce command.
RRS becomes unavailable DFSMStvs makes a note that this occurred. DFSMStvs
begins disabling itself and sets its state to disabling when
initialization reaches a point where it can do so.
A system log I/O error occurs DFSMStvs makes a note that this occurred. DFSMStvs will
begin quiescing itself and will set its state to quiescing when
initialization reaches a point where it can do so.
3. When the DFSMStvs state is initializing, a system log I/O error occurred, and
an operator command has been issued to quiesce the previous instance of
DFSMStvs.
Event Effect
VARY command to enable Command waits until DFSMStvs finishes initializing and then
DFSMStvs is rejected because DFSMStvs is already initialized.
VARY command to quiesce Command waits until DFSMStvs finishes initializing and then
DFSMStvs DFSMStvs begins quiescing its processing and sets its state
to quiescing.
VARY command to disable Command waits until DFSMStvs finishes initializing and then
DFSMStvs DFSMStvs begins disabling its processing and sets its state
to disabling.
RRS becomes unavailable DFSMStvs makes a note that this occurred. DFSMStvs
begins disabling itself and sets its state to disabling when
initialization reaches a point where it can do so.
A system log I/O error occurs This cannot occur. In this state, a cold start must be done,
and DFSMStvs would not be writing to the system log
during a cold start.
4. The DFSMStvs state is initializing, a system log I/O error occurred, and an
operator command has been issued to quiesce the previous instance of
DFSMStvs processing.
In this state, DFSMStvs would not be initializing unless a cold start has been
requested to clear the system log I/O error. All opens and record management
requests are failed with nonzero return and reason codes. All closes are
allowed; the opens with which the closes are associated probably belonged to
a previous instance of DFSMStvs.
Recommendation: After a cold start, any data sets for which recovery was not
completed are most likely left in a damaged state and must be recovered
manually. If the data sets are forward recoverable, their forward recovery logs
might also be damaged. Manually recover the data sets (without using forward
recovery), take backups of them and of any other data sets that use the
forward recovery log, and then delete and redefine the forward recovery log.
Event Effect
VARY command to enable Command waits until DFSMStvs finishes initializing and then
DFSMStvs stacks behind the previous quiesce command.
VARY command to quiesce Command waits until DFSMStvs finishes initializing and then
DFSMStvs is rejected because such a command was already issued.
VARY command to disable Command waits until DFSMStvs finishes initializing and then
DFSMStvs stacks behind the previous quiesce command.
Event Effect
VARY command to enable The command is rejected. DFSMStvs is already available.
DFSMStvs
VARY command to quiesce DFSMStvs begins the process of quiescing itself and sets
DFSMStvs its state to quiescing.
VARY command to disable DFSMStvs begins the process of disabling itself and sets its
DFSMStvs state to disabling.
RRS becomes unavailable DFSMStvs begins the process of disabling itself and sets its
state to disabling.
A system log I/O error occurs DFSMStvs begins the process of quiescing itself and sets
its state to quiescing.
6. DFSMStvs has initialized and its state is quiescing because a VARY command
was issued to quiesce DFSMStvs processing.
In this state, opens and closes are all processed normally. Any record
management requests for units of recovery that are already in progress are
also processed normally. Record management requests for new units of
recovery are rejected.
Recommendation: After a cold start, any data sets for which recovery was not
completed are most likely left in a damaged state and must be recovered
manually. If the data sets are forward recoverable, their forward recovery logs
might also be damaged. Manually recover the data sets (without using forward
recovery), take backups of them and of any other data sets that use the
forward recovery log, and then delete and redefine the forward recovery log.
Event Effect
VARY command to enable DFSMStvs sets its state to initializing and reinitializes itself.
DFSMStvs
VARY command to quiesce The command is rejected. DFSMStvs is already in the
DFSMStvs process of quiescing itself.
VARY command to disable DFSMStvs begins the process of disabling itself and sets its
DFSMStvs state to disabling.
RRS becomes unavailable DFSMStvs begins the process of disabling itself and sets its
state to disabling.
A system log I/O error occurs The DFSMStvs state does not change, because DFSMStvs
is already quiescing. However, DFSMStvs makes a note of
the error, because a cold start will be required to correct the
problem.
Event Effect
VARY command to enable The command is rejected. A cold start is required to
DFSMStvs recovery from the system log I/O error.
VARY command to quiesce DFSMStvs is already in the process of quiescing itself, but
DFSMStvs make a note that this occurred so that it will not come back
up until another VARY command is issued to tell it to
ENABLE itself.
VARY command to disable DFSMStvs begins the process of disabling itself and sets its
DFSMStvs state to disabling.
RRS becomes unavailable DFSMStvs begins the process of disabling itself and sets its
state to disabling.
A system log I/O error occurs There is no operational change because such an error has
already occurred.
8. DFSMStvs has initialized and its state is quiescing because a system log I/O
error occurred and because a VARY command was issued to quiesce
DFSMStvs processing.
In this state, opens and closes are all processed normally. Any record
management requests for units of recovery that are already in progress are
also processed normally. Record management requests for new units of
recovery are rejected.
Event Effect
VARY command to enable DFSMStvs makes a note that it was told to enable, but it
DFSMStvs issues a message to indicate that it is still quiescing due to
the system log I/O error.
VARY command to quiesce The command is rejected. DFSMStvs is already quiescing.
DFSMStvs
VARY command to disable DFSMStvs begins the process of disabling itself and sets its
DFSMStvs state to disabling.
RRS becomes unavailable DFSMStvs begins the process of disabling itself and sets its
state to disabling.
A system log I/O error occurs There is no operational change because such an error has
already occurred.
Event Effect
VARY command to enable DFSMStvs begins the process of reinitializing itself and sets
DFSMStvs its state to initializing.
VARY command to quiesce The command is rejected. DFSMStvs is already quiesced.
DFSMStvs
VARY command to disable The command is rejected. DFSMStvs is already quiesced.
DFSMStvs
RRS becomes unavailable DFSMStvs unregisters with RRS and RLS, disconnects from
its logs, and sets its state to disabled.
A system log I/O error occurs This cannot happen. DFSMStvs processing is quiesced;
therefore, DFSMStvs cannot write to its system logs.
10. The DFSMStvs state is quiesced because a system log I/O error occurred.
DFSMStvs does not reach this state until the last data set that was open for
DFSMStvs processing is closed. In this state, all opens are failed with nonzero
return and reason codes. All closes and record management requests are
rejected with X'0F4' abends because no closes or record management
requests can occur after the last data set open for DFSMStvs processing is
closed.
Recommendation: After a cold start, any data sets for which recovery was not
completed are most likely left in a damaged state and must be recovered
manually. If the data sets are forward recoverable, their forward recovery logs
might also be damaged. Manually recover the data sets (without using forward
recovery), take backups of them and of any other data sets that use the
forward recovery log, and then delete and redefine the forward recovery log.
Event Effect
VARY command to enable The command is rejected. A cold start is required to recover
DFSMStvs from the system log I/O error. The SMSVSAM server must
be recycled.
VARY command to quiesce DFSMStvs makes a note that the command was issued so
DFSMStvs that it will not come up automatically if the SMSVSAM
server happens to recycle with TV_START_TYPE set to
COLD.
VARY command to disable DFSMStvs makes a note that the command was issued so
DFSMStvs that it will not come up automatically if the SMSVSAM
server happens to recycle with TV_START_TYPE set to
COLD.
RRS becomes unavailable DFSMStvs unregisters with RRS and RLS, disconnects from
its logs, and sets its state to disabled.
A system log I/O error occurs This cannot happen. DFSMStvs processing is quiesced;
therefore, DFSMStvs cannot write to its system logs.
11. The DFSMStvs state is quiesced because a VARY command was issued to
quiesce DFSMStvs processing and a system log I/O error occurred.
DFSMStvs does not reach this state until the last data set that was open for
DFSMStvs processing is closed. In this state, all opens are failed with nonzero
return and reason codes. All closes and record management requests are
Event Effect
VARY command to enable DFSMStvs resets the indicator that a command was issued
DFSMStvs to quiesce but does not reinitialize. A cold start is required to
recover from the system log I/O error. The SMSVSAM
server must be recycled.
VARY command to quiesce The command is rejected. DFSMStvs is already quiesced.
DFSMStvs
VARY command to disable The command is rejected. DFSMStvs is already quiesced.
DFSMStvs
RRS becomes unavailable DFSMStvs unregisters with RRS and RLS, disconnects from
its logs, and sets its state to disabled.
A system log I/O error occurs This cannot happen. DFSMStvs processing is quiesced,
therefore, DFSMStvs cannot write to its system logs.
12. The DFSMStvs state is disabling because RRS became unavailable, and then
RRS became available again.
In this state, DFSMStvs must complete the disable process before it can
reinitialize because RRS reinitialized, which requires that DFSMStvs reinitialize
as a new instance. All opens are failed with nonzero return and reason codes.
All closes are processed normally. All record management requests are failed
with nonzero return and reason codes.
Recommendation: After a cold start, any data sets for which recovery was not
completed are most likely left in a damaged state and must be recovered
manually. If the data sets are forward recoverable, their forward recovery logs
might also be damaged. Manually recover the data sets (without using forward
recovery), take backups of them and of any other data sets that use the
forward recovery log, and then delete and redefine the forward recovery log.
Event Effect
VARY command to enable The command is rejected. DFSMStvs must complete the
DFSMStvs disable process before can reinitialize. It will do so
automatically once the disable process completes.
VARY command to quiesce DFSMStvs makes a note that the command was issued.
DFSMStvs When RRS comes back up, DFSMStvs will reinitialize in a
quiescing state long enough to clean up any incomplete
units of recovery. It will not allow new work to begin.
VARY command to disable DFSMStvs makes a note that the command was issued so
DFSMStvs that it will not automatically reinitialize when the disable
completes.
RRS becomes unavailable DFSMStvs makes a note that this occurred and reinitializes
when RRS becomes available again.
13. The DFSMStvs state is disabling because RRS became unavailable, and then
RRS became available again; in addition, an operator command was issued to
quiesce DFSMStvs processing.
In this state, DFSMStvs must complete the disable process before it can
reinitialize because RRS reinitialized, which requires that DFSMStvs reinitialize
as a new instance. All opens are failed with nonzero return and reason codes.
All closes are processed normally. All record management requests are failed
with nonzero return and reason codes.
Recommendation: After a cold start, any data sets for which recovery was not
completed are most likely left in a damaged state and must be recovered
manually. If the data sets are forward recoverable, their forward recovery logs
might also be damaged. Manually recover the data sets (without using forward
recovery), take backups of them and of any other data sets that use the
forward recovery log, and then delete and redefine the forward recovery log.
Event Effect
VARY command to enable DFSMStvs makes a note that this occurred. It will reinitialize
DFSMStvs automatically once the disable process completes.
VARY command to quiesce The command is rejected. Such a command was previously
DFSMStvs issued.
VARY command to disable DFSMStvs makes a note that the command was issued so
DFSMStvs that it will not automatically reinitialize when the disable
completes.
RRS becomes unavailable DFSMStvs makes a note that this occurred. It does not
reinitialize automatically when RRS becomes available
again because a command was issued to quiesce it.
A system log I/O error occurs DFSMStvs makes a note that this error occurred so that it
does not automatically reinitialize until DFSMStvs is cold
started.
14. The DFSMStvs state is disabling because an operator command was issued to
disable DFSMStvs processing.
In this state, all opens are failed with nonzero return and reason codes. All
closes are processed normally. All record management requests are failed with
nonzero return and reason codes.
Recommendation: After a cold start, any data sets for which recovery was not
completed are most likely left in a damaged state and must be recovered
manually. If the data sets are forward recoverable, their forward recovery logs
might also be damaged. Manually recover the data sets (without using forward
recovery), take backups of them and of any other data sets that use the
forward recovery log, and then delete and redefine the forward recovery log.
Event Effect
VARY command to enable DFSMStvs begins the process of reinitializing itself and sets
DFSMStvs its state to initializing.
VARY command to quiesce The command is rejected. Quiescing is a less restrictive
DFSMStvs state than disabling. As a result, transitioning from disabling
to quiescing is not allowed.
15. The DFSMStvs state is disabling because RRS became unavailable and then
became available again; in addition, a system log I/O error occurred.
In this state, DFSMStvs must complete the disable process before it can
reinitialize because RRS reinitialized, which requires that DFSMStvs reinitialize
as a new instance. All opens are failed with nonzero return and reason codes.
All closes are processed normally. All record management requests are failed
with nonzero return and reason codes.
Recommendation: After a cold start, any data sets for which recovery was not
completed are most likely left in a damaged state and must be recovered
manually. If the data sets are forward recoverable, their forward recovery logs
might also be damaged. Manually recover the data sets (without using forward
recovery), take backups of them and of any other data sets that use the
forward recovery log, and then delete and redefine the forward recovery log.
Event Effect
VARY command to enable The command is rejected. A cold start must be done to
DFSMStvs recover from the system log I/O error, which requires
recycling the server.
VARY command to quiesce DFSMStvs makes a note that this occurred. When the
DFSMStvs SMSVSAM server recycles with TV_START_TYPE set to
COLD, DFSMStvs reinitializes in a quiescing state long
enough to clean up any incomplete units of recovery. It will
not allow new work to begin.
VARY command to disable DFSMStvs makes a note that this occurred so that it does
DFSMStvs not automatically reinitialize if the SMSVSAM server
recycles with TV_START_TYPE set to COLD.
RRS becomes unavailable DFSMStvs makes a note that this occurred. It does not
reinitialize automatically when RRS becomes available
again because of the system log I/O error.
A system log I/O error occurs There is no operational change because such an error has
already occurred.
Event Effect
VARY command to enable The command is rejected. DFSMStvs will automatically
DFSMStvs reinitialize when the disable completes and RRS becomes
available.
VARY command to quiesce DFSMStvs makes a note that the command was issued.
DFSMStvs When RRS comes back up, DFSMStvs reinitializes in a
quiescing state long enough to clean up any incomplete
units of recovery. It does not allow new work to begin.
VARY command to disable DFSMStvs makes a note that this occurred so that it does
DFSMStvs not automatically reinitialize when RRS becomes available.
RRS becomes available DFSMStvs finishes disabling and then reinitializes itself
automatically.
A system log I/O error occurs DFSMStvs makes a note that this occurred. A cold start is
required to recover from the error.
17. The DFSMStvs state is disabling because an operator command was issued to
disable DFSMStvs processing and a system log I/O error occurred.
In this state, all opens are failed with nonzero return and reason codes. All
closes are processed normally. All record management requests are failed with
nonzero return and reason codes.
Recommendation: After a cold start, any data sets for which recovery was not
completed are most likely left in a damaged state and must be recovered
manually. If the data sets are forward recoverable, their forward recovery logs
might also be damaged. Manually recover the data sets (without using forward
recovery), take backups of them and of any other data sets that use the
forward recovery log, and then delete and redefine the forward recovery log.
Event Effect
VARY command to enable The command is rejected. Quiescing is a less restrictive
DFSMStvs state than disabling. As a result, transitioning from disabling
to quiescing is not allowed.
VARY command to quiesce DFSMStvs switches from disabling to quiescing and sets its
DFSMStvs state to quiescing.
VARY command to disable The command is rejected. DFSMStvs is already disabling.
DFSMStvs
RRS becomes unavailable DFSMStvs makes a note that this occurred. It does not
automatically reinitialize when RRS becomes available
because an operator command was issued to disable it.
A system log I/O error occurs There is no operational change because such an error has
already occurred.
18. The DFSMStvs state is disabling because RRS is unavailable and because an
operator command was issued to disable DFSMStvs processing
In this state, DFSMStvs must complete the disable process before it can
reinitialize because RRS reinitialized, which requires that DFSMStvs reinitialize
as a new instance. All opens are failed with nonzero return and reason codes.
All closes are processed normally. All record management requests are failed
with nonzero return and reason codes.
Recommendation: After a cold start, any data sets for which recovery was not
completed are most likely left in a damaged state and must be recovered
manually. If the data sets are forward recoverable, their forward recovery logs
Event Effect
VARY command to enable DFSMStvs makes a note that this occurred. It will reinitialize
DFSMStvs automatically when RRS becomes available.
VARY command to quiesce DFSMStvs makes a note that this occurred, but the state
DFSMStvs remains disabling because of the RRS failure. When RRS
comes back up, DFSMStvs will reinitialize in a quiescing
state long enough to clean up any incomplete units of
recovery. It will not allow new work to begin.
VARY command to disable The command is rejected. DFSMStvs is already disabling.
DFSMStvs
RRS becomes available DFSMStvs makes a note that this occurred. It does not
reinitialize itself automatically because an operator
command was issued to disable DFSMStvs processing.
A system log I/O error occurs DFSMStvs makes a note that this occurred. A cold start will
be required to recover from the error.
Event Effect
VARY command to enable DFSMStvs makes a note that this occurred. It will reinitialize
DFSMStvs automatically when RRS becomes available.
VARY command to quiesce The command is rejected. Such a command was already
DFSMStvs issued.
VARY command to disable DFSMStvs makes a note that this occurred.
DFSMStvs
RRS becomes available DFSMStvs makes a note that this occurred. It will not
reinitialize automatically when RRS becomes available
again because an operator command was issued to quiesce
DFSMStvs processing.
A system log I/O error occurs DFSMStvs makes a note that this occurred. A cold start will
be required to recover.
Event Effect
VARY command to enable The command is rejected. A cold start is required to recover
DFSMStvs from the system log I/O error.
VARY command to quiesce DFSMStvs makes a note that this occurred. When the
DFSMStvs SMSVSAM server recycles with TV_START_TYPE set to
COLD, DFSMStvs will reinitialize in a quiescing state long
enough to clean up any incomplete units of recovery. It will
not allow new work to begin.
VARY command to disable DFSMStvs makes a note that this occurred so that it will not
DFSMStvs automatically reinitialized if the SMSVSAM server recycles
with TV_START_TYPE set to COLD.
RRS becomes available DFSMStvs makes a note that this occurred. It will not
reinitialize automatically because a cold start is required to
recover from the system log I/O error.
A system log I/O error occurs There is no operational change because such an error has
already occurred.
Event Effect
VARY command to enable DFSMStvs makes a note that this occurred. It will not
DFSMStvs automatically reinitialize when RRS becomes available
because of the system log I/O error.
VARY command to quiesce DFSMStvs makes a note that this occurred. When the
DFSMStvs SMSVSAM server recycles with TV_START_TYPE set to
COLD, DFSMStvs will reinitialize in a quiescing state long
enough to clean up any incomplete units of recovery. It will
not allow new work to begin.
VARY command to disable The command is rejected. Such a command was already
DFSMStvs issued.
Event Effect
VARY command to enable DFSMStvs makes a note that this occurred. It will not
DFSMStvs automatically reinitialize when RRS becomes available
because of the system log I/O error.
VARY command to quiesce The command is rejected. Such a command was already
DFSMStvs issued.
VARY command to disable DFSMStvs makes a note that this occurred.
DFSMStvs
RRS becomes available DFSMStvs makes a note that this occurred. It will not
reinitialize automatically because a cold start is required to
recover from the system log I/O error.
A system log I/O error occurs There is no operational change because such an error has
already occurred.
23. The DFSMStvs state is disabling because RRS became unavailable a then
became available again, and an operator command was issued to disable
DFSMStvs; in addition, a system log I/O error occurred.
In this state, DFSMStvs must complete the disable process before it can
reinitialize because RRS reinitialized, which requires that DFSMStvs reinitialize
as a new instance. All opens are failed with nonzero return and reason codes.
All closes are processed normally. All record management requests are failed
with nonzero return and reason codes.
Recommendation: After a cold start, any data sets for which recovery was not
completed are most likely left in a damaged state and must be recovered
manually. If the data sets are forward recoverable, their forward recovery logs
might also be damaged. Manually recover the data sets (without using forward
recovery), take backups of them and of any other data sets that use the
forward recovery log, and then delete and redefine the forward recovery log.
24. The DFSMStvs state is disabled because RRS became unavailable an then
became available again.
When DFSMStvs reaches this state, it begins the process of reinitialize itself
and sets its state to initializing, as long as no commands are issued to the
contrary. In this state, all opens are failed with nonzero return and reason
codes. All closes and record management requests are rejected with X'0F4'
abends because no data sets should be open for DFSMStvs processing if
DFSMStvs is disabled.
Event Effect
VARY command to enable The command is rejected. DFSMStvs is about to reinitialize
DFSMStvs on its own.
VARY command to quiesce DFSMStvs makes a note that the command was issued.
DFSMStvs When RRS comes back up, DFSMStvs will reinitialize in a
quiescing state long enough to clean up any incomplete
units of recovery. It will not allow new work to begin.
VARY command to disable DFSMStvs makes a note that this occurred. The state
DFSMStvs remains DISABLE and DFSMStvs does not reinitialize.
RRS becomes unavailable DFSMStvs makes a note that this occurred. It begins the
process of reinitializing itself and waits for RRS to become
available.
A system log I/O error occurs This cannot occur when DFSMStvs is disabled because
there is no DFSMStvs activity.
25. The DFSMStvs state is disabled because RRS became unavailable an then
became available again, and an operator command was issued to quiesce
DFSMStvs processing before DFSMStvs could begin to reinitialize.
In this state, all opens are failed with nonzero return and reason codes. All
closes and record management requests are rejected with X'0F4' abends
because no data sets should be open for DFSMStvs processing if DFSMStvs
is disabled.
Event Effect
VARY command to enable DFSMStvs begins the process of reinitializing itself and sets
DFSMStvs its state to initializing.
VARY command to quiesce The command is rejected. Such a command was already
DFSMStvs issued.
VARY command to disable DFSMStvs makes a note that this occurred.
DFSMStvs
26. The DFSMStvs state is disabled because an operator command was issued to
disable DFSMStvs processing.
In this state, all opens are failed with nonzero return and reason codes. All
closes and record management requests are rejected with X'0F4' abends
because no data sets should be open for DFSMStvs processing if DFSMStvs
is disabled.
Event Effect
VARY command to enable DFSMStvs begins the process of reinitializing itself and sets
DFSMStvs its state to initializing.
VARY command to quiesce The command is rejected. DFSMStvs is already disabled.
DFSMStvs
VARY command to disable The command is rejected. DFSMStvs is already disabled.
DFSMStvs
RRS becomes unavailable DFSMStvs makes a note that this occurred. It will not
reinitialize itself automatically when RRS becomes available
again because an operator command was issued to disable
DFSMStvs processing.
A system log I/O error occurs This cannot occur when DFSMStvs is disabled because
there is no DFSMStvs activity.
27. The DFSMStvs state is disabled because an operator command was issued to
disable DFSMStvs processing; in addition, a system log I/O error occurred. A
subsequent VARY command might also have been issued to change the state
of DFSMStvs, as follows:
v Enable DFSMStvs, but DFSMStvs could not reinitialize because of the
system log I/O error
v Quiesce DFSMStvs, which made a note of this, but the state did not change
because DFSMStvs was already disabled
In this state, all opens are failed with nonzero return and reason codes. All
closes and record management requests are rejected with X'0F4' abends
because no data sets should be open for DFSMStvs processing if DFSMStvs
is disabled.
Recommendation: After a cold start, any data sets for which recovery was not
completed are most likely left in a damaged state and must be recovered
manually. If the data sets are forward recoverable, their forward recovery logs
might also be damaged. Manually recover the data sets (without using forward
recovery), take backups of them and of any other data sets that use the
forward recovery log, and then delete and redefine the forward recovery log.
Event Effect
VARY command to enable The command is rejected. A cold start is required to recover
DFSMStvs from the system log I/O error.
Event Effect
VARY command to enable DFSMStvs begins the process of reinitializing itself and sets
DFSMStvs its state to initializing.
VARY command to quiesce DFSMStvs makes a note that the command was issued.
DFSMStvs When RRS comes back up, DFSMStvs will reinitialize in a
quiescing state long enough to clean up any incomplete
units of recovery. It will not allow new work to begin.
VARY command to disable DFSMStvs makes a note that this occurred. It will not
DFSMStvs automatically reinitialize when RRS becomes available.
RRS becomes available DFSMStvs begins the process of reinitializing itself and sets
its state to initializing.
A system log I/O error occurs This cannot occur when DFSMStvs is disabled because
there is no DFSMStvs activity.
29. The DFSMStvs state is disabled because RRS is unavailable and an operator
command was issued to disable DFSMStvs processing.
In this state, all opens are failed with nonzero return and reason codes. All
closes and record management requests are rejected with X'0F4' abends
because no data sets should be open for DFSMStvs processing if DFSMStvs
is disabled.
Event Effect
VARY command to enable DFSMStvs begins the process of reinitializing itself and sets
DFSMStvs its state to initializing.
VARY command to quiesce DFSMStvs makes a note that this occurred. It will not
DFSMStvs automatically reinitialize when RRS becomes available.
VARY command to disable The command is rejected. DFSMStvs is already disabled.
DFSMStvs
RRS becomes available DFSMStvs makes a note that this occurred. It does not
automatically reinitialize itself because an operator
command was issued to disable DFSMStvs processing.
A system log I/O error occurs This cannot occur when DFSMStvs is disabled because
there is no DFSMStvs activity.
Event Effect
VARY command to enable DFSMStvs begins the process of reinitializing itself and sets
DFSMStvs its state to initializing.
VARY command to quiesce The command is rejected. Such a command was already
DFSMStvs issued.
VARY command to disable DFSMStvs makes a note that this occurred.
DFSMStvs
RRS becomes available DFSMStvs makes a note that this occurred. It does not
automatically reinitialize itself because an operator
command was issued to quiesce DFSMStvs processing.
A system log I/O error occurs This cannot occur when DFSMStvs is disabled because
there is no DFSMStvs activity.
Event Effect
VARY command to enable The command is rejected. A cold start is required to recover
DFSMStvs from the system log I/O error.
VARY command to quiesce DFSMStvs makes a note that this occurred. When the
DFSMStvs SMSVSAM server recycles with TV_START_TYPE set to
COLD, DFSMStvs will reinitialize in a quiescing state long
enough to clean up any incomplete units of recovery. It will
not allow new work to begin.
VARY command to disable DFSMStvs makes a note that this occurred so that it will not
DFSMStvs automatically reinitialize if the SMSVSAM server recycles
with TV_START_TYPE set to COLD.
RRS becomes available DFSMStvs makes a note that this occurred. It does not
automatically reinitialize itself because the system log I/O
error requires a cold start.
A system log I/O error occurs This cannot occur when DFSMStvs is disabled because
there is no DFSMStvs activity.
Event Effect
VARY command to enable DFSMStvs makes a note that this occurred. It does not
DFSMStvs reinitialize. A cold start is required to recover from the
system log I/O error.
VARY command to quiesce DFSMStvs makes a note that this occurred. When the
DFSMStvs SMSVSAM server recycles with TV_START_TYPE set to
COLD, DFSMStvs will reinitialize in a quiescing state long
enough to clean up any incomplete units of recovery. It will
not allow new work to begin.
VARY command to disable DFSMStvs makes a note that this occurred so that it will not
DFSMStvs automatically reinitialize if the SMSVSAM server recycles
with TV_START_TYPE set to COLD.
RRS becomes available DFSMStvs makes a note that this occurred. It does not
automatically reinitialize itself because the system log I/O
error requires a cold start.
A system log I/O error occurs This cannot occur when DFSMStvs is disabled because
there is no DFSMStvs activity.
For sequential processing, the way that locks are held and released is predictable.
Locks are obtained and released in sequence through the data set as the job
progresses.
Sequential update applications that use large buffers can have the greatest
performance difference when moved to DFSMStvs, which caches data in the
coupling facility. DFSMStvs does not attempt to read ahead for sequential access.
Unless a record is found in the SMSVSAM buffer pool, each VSAM request results
in the transfer of one control interval. This is a lower degree of buffering than you
can do for sequential nonshared access to VSAM data. So, the elapsed time of
sequential updates increases because the data transfer is less efficient.
You should plan to review logging activity using the SMF type 88 records produced
by the system logger.
Recommendation: Define the system log streams with DIAG(YES). This forces the
system logger to create memory dump in the event of a lost data condition.
A dump of XCF and system logger address spaces from all systems are useful in
diagnosing such problems. Issue the following series of MVS commands as shown
in Figure 16:
Figure 16. Example of MVS commands to produce a dump of XCF and system logger
address spaces
For more information about how to code the commands, see z/OS MVS Diagnosis:
Tools and Service Aids.
If you suspect a problem with the coupling facility structure, use the
R xx,STRLIST=(STRNAME=structure,(LISTNUM=ALL),ACC=NOLIM),CONT
instruction.
Error records written to the log data set might also be useful.
D XCF,CPL,TYPE=LOGR
Figure 17. Example of a command to display system logger couple data set status
D XCF,CPL,TYPE=LOGR
IXC358I 14.47.51 DISPLAY XCF 391
LOGR COUPLE DATA SETS
PRIMARY DSN: SYS1.SYSPLEX2.SEQ26.PLOGR
VOLSER: P2SS05 DEVN: 230D
FORMAT TOD MAXSYSTEM
12/20/95 09:25:48 8
ALTERNATE DSN: SYS1.SYSPLEX2.SEQ26.ALOGR
VOLSER: P2SS06 DEVN: 2C10
FORMAT TOD MAXSYSTEM
12/20/95 09:27:45 8
LOGR IN USE BY ALL SYSTEMS
Figure 18. Example of a normal response from a command to display system logger couple
data set status
If the response shows that LOGR is not in use by all systems, this could be a sign
of a problem you need to investigate further. Look for IXCxxx messages that might
indicate the cause of the problem, and issue the command as shown in Figure 19
to attempt reconnection to the couple data set:
SETXCF CPL,TYPE=(LOGR),PCOUPLE=(couple_dataset_name)
D XCF,STR,STRNM=*,STATUS=FPCONN
Figure 20. Example of a command to display all structures with Failed_Persistent connections
You might also see latch set name SYS.IXGLOGER_MISC. The system logger
should resolve any failed connections.
D GRS,C
D GRS,RES=(SYSZLOGR,*)
D GRS,C
ISG020I 12.06.49 GRS STATUS 647
NO ENQ CONTENTION EXISTS
NO LATCH CONTENTION EXISTS
D GRS,RES=(SYSZLOGR,*)
ISG020I 14.04.28 GRS STATUS 952
NO REQUESTORS FOR RESOURCE SYSZLOGR *
GRS,C
ISG020I 12.06.31 GRS STATUS 619
LATCH SET NAME: SYS.IXGLOGER_LCBVT
CREATOR JOBNAME: IXGLOGR CREATOR ASID: 0202
LATCH NUMBER: 7
REQUESTOR ASID EXC/SHR OWN/WAIT
IXGLOGR 0202 EXCLUSIVE OWN
IXGLOGR 0202 SHARED WAIT
D GRS,RES=(SYSZLOGR,*)
ISG020I 19.58.33 GRS STATUS 374
S=STEP SYSZLOGR 91
SYSNAME JOBNAME ASID TCBADDR EXC/SHR OWN/WAIT
MV26 MSLDELC1 002F 008F6370 EXCLUSIVE OWN
S=STEP SYSZLOGR 93
SYSNAME JOBNAME ASID TCBADDR EXC/SHR OWN/WAIT
MV26 MSLWRTC1 002E 008DED90 EXCLUSIVE OWN
MV26 MSLWRTC1 002E 008DB990 EXCLUSIVE WAIT
MV26 MSLWRTC1 002E 008DB700 EXCLUSIVE WAIT
MV26 MSLWRTC1 002E 008F60C8 EXCLUSIVE WAIT
S=SYSTEMS SYSZLOGR LPAYROL.TESTLOG.TLOG1
SYSNAME JOBNAME ASID TCBADDR EXC/SHR OWN/WAIT
MV27 IXGLOGR 0011 008F7398 EXCLUSIVE OWN
MV26 IXGLOGR 0011 008F7398 EXCLUSIVE WAIT
This example shows which tasks have exclusive enqueues on the log streams and
which tasks are waiting for them. It is quite normal for enqueues and latches to be
obtained, occasionally with contention. They are indications of a problem only when
they last for more than a minute or so.
Long term enqueuing on the SYSZLOGR resource can be a sign of problems even
if there is no contention.
D GRS,RES=(SYSDSN,*)
Figure 24. Example of a GRS command to display log streams with an exclusive enqueue
A typical response to this command looks like the example in Figure 25:
Figure 25. Example of output from a GRS command to display log streams with an exclusive
enqueue
The system logger monitors its allocation and HSM recall service tasks for delays
and provides a mechanism (through WTORs) to interrupt these delayed requests.
To the system logger, the interruption is an error condition for the current request.
Removal of a delayed request enables the processing of other log stream resource
requests. Messages IXG271I and IXG272E are issued if the system logger detects
that a delay of a service request is inhibiting other log stream resource requests.
Use the FAIL option only if you cannot determine why the request is not completing.
Replying FAIL might cause undesirable results. This option is meant to keep the
rest of the system logger applications running, at the expense of one hung
application.
Examine any other error messages from the system logger or from any exploiter of
the affected log stream. If you reply FAIL, the system logger might cause other
components such as allocation to enter their recovery, create memory dumps, or
issue various messages. If you reply FAIL to message IXG272E, you might see a
dump in media manager, dynamic allocation, or catalog, depending on what type of
I/O delay that the system logger was experiencing at the time IXG272E was issued.
There is an unexpected dump that results in abend 0E0 during SVCDMP
processing.
Resolving waits
For a batch job wait that you think might be related to DFSMStvs, diagnosis starts
with a dump of the batch job. First examine the linkage stack of the batch job to
determine with which component the batch job might have a cross-memory
problem. Then perform a dump of that component. Many components can cause
hangs, including DFSMStvs, VSAM RLS, a catalog, the system logger, and RRS.
If you encounter a wait that you think is related to logging, check the MVS console
for messages that have the prefix IXG. This prefix is for system logger messages.
These messages might provide more information about the cause of the wait. The
MVS console might also reveal evidence of resource contention within MVS.
Various levels of authorization are required to use the SHCDS parameters. For
information about this authorization, see z/OS DFSMS Access Method Services for
Catalogs.
Recommendation: Do not cold start DFSMStvs unless the DFSMStvs system logs
have been damaged. Any recovery processing that was to be done for recoverable
data sets is not done after a cold start of DFSMStvs. Any data sets for which
recovery was not complete are most likely left in a damaged state. If the data sets
are forward recoverable, then their forward recovery logs might also be damaged.
You should manually recover the data sets (without using forward recovery), make a
backup of those data sets and any other data sets that use the same forward
recovery log, and then delete and redefine the forward recovery log.
Issue the VARY SMS command on another system. That system then runs peer
recovery for the instance specified on the command, as well as any instances for
which that instance was performing peer recovery.
To use ARM, you must create an ARM policy that groups instances of DFSMStvs
with other resource managers that might have a shared instance in a unit of
recovery. ARM supports this method with the administrative policy
DFSMStvs registers with ARM as an abstract resource (one that is not associated
with a job or started task) when it initializes. DFSMStvs requests that it be restarted
only when the system on which it is running fails unexpectedly by specifying an
element ending type of SYSTERM and an element bind of CURSYS. DFSMStvs
provides its restart method by supplying the text of the required VARY SMS
command as its start text when it registers with ARM. If DFSMStvs ends, it
deregisters with ARM.
If the installation is not using ARM or if ARM is unavailable, you can initiate peer
recovery manually by using the VARY SMS command. For example, issue the
following command:
VARY SMS,TRANVSAM(001),PEERRECOVERY,ACTIVE
For example, DFSMStvs instance IGWTV001 was running peer recovery for
instances IGWTV002 and IGWTV003. If the system fails, and peer recovery is
initiated for IGWTV001 on another system, that system performs peer recovery for
IGWTV001, IGWTV002, and IGWTV003. However, starting peer recovery for
IGWTV002 or IGWTV003 on another system would start peer recovery for the
specified instance, and only for the specified instance.
Recommendation: Be careful about quiesce with a path name; always use a base
name.
Unlike CICS, DFSMStvs does not control the open requests and close requests of
data sets that it accesses. DFSMStvs supports the ability to quiesce and close
ACBs in a limited manner. If a data set is not currently open for DFSMStvs access,
the data set is quiesced; otherwise, the quiesce of the data set is rejected. To
quiesce a data set completely, ensure that it is not currently open in DFSMStvs
mode; take these steps
Various levels of authorization are required to use the SHCDS parameters. For
information about this authorization, see z/OS DFSMS Access Method Services for
Catalogs.
To prevent damage to the data set, DFSMStvs defers the decision to back out a
specific record to an installation-provided exit that is a batch override exit. This is an
optional exit that DFSMStvs calls. It uses the exit to back out a unit of recovery
involving a data set that a PERMITNONRLSUPDATE command impacts. DFSMStvs
calls the exit once for each affected undo log record in the data set. The purpose of
the exit is to return to DFSMStvs an indication of whether or not the undo log
record should be applied. The input is an undo log record and a data set name. The
output is a Boolean response of whether or not to do the backout.
Recommendation: Although the exit can perform other processing, it should not
attempt to update any recoverable resources.
If you do not provide the installation exit and DFSMStvs encounters backouts that a
PERMITNONRLSUPDATE command impacts, DFSMStvs issues a message. It
does not apply the backout records. Use the access method services SHCDS
RETRY or PURGE command to clean up the backout records.
Recommendation: Be careful about quiesce with a path name; always use a base
name.
Restrictions:
1. The exit does not receive a save area and must not attempt to use the value in
register 13 or the exit abends. The storage pointed to by register 13 is in a
different key than the key in which the exit receives control. As a result, the exit
should not attempt to use register 13 to save or restore the registers. It is not
necessary for the exit to restore the registers before returning. This is because it
receives control through the SYNCHX (SVC 12) exit, and the SYNCHX exit
restores the registers.
2. It is difficult for the exit to obtain an autodata area because it runs in key 8
problem state, which is not authorized. This limits the subpools it can use to
those subpools that obtain storage in the TCB key. The TCB key is normally key
5, which prevents the exit from referring to the storage. Therefore, the exit
receives passed a pointer to an area that can be used as an autodata area in
The name of this exit must be IGW8PNRU. DFSMStvs loads the module, which
must reside in LINKLIB or LPALIB. If the load fails (for example, no exit is found),
DFSMStvs issues a message. If the following conditions exist, DFSMStvs shunts
the unit of recovery:
v No exit is found.
v One of the data sets was accessed through PERMITNONRLSUPDATE.
To fix a code error or enhance the function of the exit, restart DFSMStvs to enable
the new exit. The exit must be loadable from any system that might perform peer
recovery for another system. The exit can issue SVC instructions.
If this problem occurs, examine the dump to investigate why the exit abnormally
ended.
Appendix B. Accessing data sets that have retained locks or lost locks 123
124 z/OS V1R4.0 DFSMStvs Planning and Operating Guide
Appendix C. Accessibility
Accessibility features help a user who has a physical disability, such as restricted
mobility or limited vision, to use software products successfully. The major
accessibility features in z/OS enable users to do these tasks:
v Use assistive technologies such as screen readers and screen-magnifier
software
v Operate specific or equivalent features using only the keyboard
v Customize display attributes such as color, contrast, and font size
Assistive technologies
Assistive technology products, such as screen readers, function with the user
interfaces in z/OS or z/OS.e. Consult the assistive technology documentation for
specific information when using assistive technology to access z/OS or z/OS.e
interfaces.
IBM may have patents or pending patent applications covering subject matter
described in this document. The furnishing of this document does not give you any
license to these patents. You can send license inquiries, in writing, to:
IBM Director of Licensing
IBM Corporation
North Castle Drive
Armonk, NY 10504-1785
U.S.A.
For license inquiries regarding double-byte (DBCS) information, contact the IBM
Intellectual Property Department in your country or send inquiries, in writing, to:
IBM World Trade Asia Corporation Licensing
2-31 Roppongi 3-chome, Minato-ku
Tokyo 106, Japan
The following paragraph does not apply to the United Kingdom or any other
country where such provisions are inconsistent with local law:
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS
PUBLICATION “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE. Some states do not allow disclaimer of express or
implied warranties in certain transactions, therefore, this statement may not apply to
you.
IBM may use or distribute any of the information you supply in any way it believes
appropriate without incurring any obligation to you.
The licensed program described in this information and all licensed material
available for it are provided by IBM under terms of the IBM Customer Agreement,
IBM International Program License Agreement, or any equivalent agreement
between us.
This information unit also documents information that is NOT intended to be used
as Programming Interfaces of DFSMStvs. This information is identified where it
occurs by an introductory statement to a chapter or section.
Trademarks
The following terms are trademarks of the International Business Machines
Corporation in the United States, other countries, or both:
UNIX is a registered trademark in the United States and other countries licensed
exclusively through X/Open Company Limited.
Other company, product, and service names might be trademarks or service marks
of others.
addressed-direct access. In VSAM, the retrieval or automatic class selection (ACS) routine. A
storage of a data record identified by its relative byte procedural set of ACS language statements. Based on
address (RBA), independent of the record’s location a set of input variables, the ACS language statements
relative to the previously retrieved or stored record. generate the name of a predefined SMS class, or a list
of names of predefined storage groups, for an MVS
addressed-sequential access. In VSAM, the retrieval data set.
or storage of a data record in its entry sequence relative
to the previously retrieved or stored record. automatic restart manager (ARM). A z/OS recovery
function that can automatically restart a batch job,
addressing mode (AMODE). An attribute of an entry started task, or abstract resource after it ends
point in a program that identifies the addressing range unexpectedly or after the system on which it is running
in virtual storage that the module is capable of goes down unexpectedly.
addressing. In 24-bit addressing mode, only 24-bit
addresses can be used.
B
AKP. See activity keypoint.
backout. A request to remove all changes to
alias. An alternative name for a catalog, a non-VSAM resources since the last commit or backout or for the
data set, or a member of a partitioned data set (PDS) or first unit of recovery, since the beginning of the
partitioned data set extended (PDSE). application. Backout is also called rollback or abort.
alias entry. An entry that relates an alias to the real backout log. See undo log.
entry name of a user catalog or non-VSAM data set.
binder. The DFSMS program that processes the
allocation. (1) Generically, the entire process of output of language translators and compilers into an
obtaining a volume and unit of external storage, and executable program (load module or program object). It
setting aside space on that storage for a data set. (2) replaces the linkage editor and batch loader in in z/OS.
The process of connecting a program to a data set or
devices. blocking. The process of combining two or more
records into one block.
alternate index. A key-sequenced data set that
contains index entries organized by the alternate keys block size. The number of records, words, or
of its associated base data records. It provides an characters in a block; usually specified in bytes.
alternate means of locating records in the data
component of a cluster on which the alternate index is
based.
C
alternate key. One or more characters within a data CA. See control area.
record used to identify the data record or to control its
catalog. A data set that contains extensive information
use. Unlike the primary key, the alternate key can
required to locate other data sets, to allocate and
identify more than one data record. An alternate key is
deallocate storage space, to verify the access authority
used to build an alternate index or to locate one or
of a program or operator, and to accumulate data set
more base data records through an alternate index. See
usage statistics. (A) (I)
also generic key, key, and key field.
central processor complex (CPC). A physical
application owning region. A CICS address space
collection of hardware that consists of main storage,
whose primary purpose is to manage application
one or more central processors, timers, and channels.
programs.
CI. See control interval.
application. The use to which an access method is
put or the end result that it serves, contrasted to the CICS. Customer Information Control System.
internal operation of the access method.
CICSVR. Customer Information Control System VSAM
ARM. See automatic restart manager. Recovery, a forward recovery utility, which can can
perform forward recovery for DFSMStvs and others as
atomic. Pertaining to a transaction’s changes to the
well as for CICS.
state of resources: either all changes happen or none
happen. It would not be possible for some updates to class. See SMS class.
configuration. (1) The arrangement of a computer coupling facility (CF). The hardware that provides
system as defined by the characteristics of its functional high-speed caching, list processing, and locking
units. (2) See SMS configuration. functions in a Parallel Sysplex.
consistent read. A level of read integrity that VSAM coupling facility (CF) cache structure. The CF
RLS obtains for a share lock on the record that is hardware that provides a data cache.
accessed by a GET or POINT request. Consistent read
ensures that the reader does not see an uncommitted coupling facility (CF) lock structure. The CF
change made by another transaction. hardware that supports Parallel Sysplex-wide locking.
consistent read explicit. A level of read integrity that CPC. See central processor complex.
is the same as consistent read, except that VSAM RLS
keeps the share lock on the record until the end of the D
transaction. This option is available only to CICS
transactions and to DFSMStvs. VSAM does not data class. A collection of allocation and space
recognize the end of the transaction for usage other attributes, defined by the storage administrator, that are
than by CICS or DFSMStvs. This capability is often used to create a data set.
referred to as repeatable read.
data extent block (DEB). A control block that
context. Sometimes called a work context, a context is describes the physical attributes of the data set.
a representation of a work request, or part of a work
request, in an application. A context might have a series Data Facility Storage Management Subsystem
of units of recovery associated with it. See also native (DFSMS). An operating environment that helps
context and privately managed context. automate and centralize the management of storage. To
manage storage, SMS provides the storage
control area (CA). (1) A group of control intervals administrator with control over data class, storage class,
used as a unit for formatting a data set before adding management class, storage group, and automatic class
records to it. selection routine definitions.
(2) In a key-sequenced data set, the set of control
intervals, pointed to by a sequence-set index record, Data Facility Storage Management Subsystem data
that is used for distributing free space and for placing a facility product (DFSMSdfp). A DFSMS functional
sequence-set index record adjacent to its data. component or base element of z/OS that provides
functions for storage management, data management,
control blocks in common (CBIC). A facility that program management, device management, and
allows a user to open a VSAM data set so the VSAM distributed data access.
control blocks are placed in the common service area
(CSA) of the MVS operating system. This provides the
Glossary 131
Glossary
Data Facility Storage Management Subsystem data address according to the beginning of the data set.
set services (DFSMSdss™). A DFSMS functional Normally accessed via BDAM.
component or base element of z/OS that is used to
copy, move, dump, and restore data sets and volumes. directly-allocated printer. A printer that is allocated to
the application program.
Data Facility Storage Management Subsystem
Transactional VSAM Services (DFSMStvs). An IBM dynamic buffering. A user-specified option that
licensed program for running batch VSAM processing requests that the system handle acquisition,
concurrently with CICS online transactions. DFSMStvs assignment, and release of buffers.
users can run multiple batch jobs and online
transactions against VSAM data, in data sets defined as
recoverable, with concurrent updates. DFSMStvs is a
E
licensed component of DFSMS entry-sequenced data set. A data set whose records
are loaded without respect to their contents, and whose
data record. A collection of items of information from
relative byte addresses (RBAs) cannot change. Records
the standpoint of its use in an application, as a user
are retrieved and stored by addressed access, and new
supplies it to the system storage. Contrast with index
records are added at the end of the data set.
record.
ESA. See Enterprise Systems Architecture.
data security. Prevention of access to or use of data
or programs without authorization. As used in this exclusive control. A way of preventing multiple
publication, the safety of data from unauthorized use, write-add BDAM requests from updating the same
theft, or purposeful destruction. dummy record or writing over the same available space
on a track. When specified by the user, the exclusive
data set control block (DSCB). A control block in the
control lock requests that the system prevent the data
VTOC that describes data set characteristics.
block that is about to be read from being modified by
data synchronization. The process by which the other requests; it is specified in a read macro and
system ensures that data previously given to the system released in a write or relex macro. When a write-add
via WRITE, CHECK, PUT, and PUTX macros is written request is about to be processed, the system
to some form of nonvolatile storage. automatically gets exclusive control of either the data
set or the track.
device number. The reference number assigned to
any external device. extended format. The format of a data set that has a
data set name type (DSNTYPE) of EXTENDED, for
DFSMS. See Data Facility Storage Management example, extended format and extended key-sequenced
Subsystem. data sets. Data sets in extended format can be striped
or compressed. Data in an extended format VSAM
DFSMSdfp. See Data Facility Storage Management KSDS can be compressed.
Subsystem data facility product.
extended format data set. A sequential data set that
DFSMSdss. See Data Facility Storage Management is structured logically the same as a physical sequential
Subsystem data set services. data set but that is stored in a different physical format.
Extended format data sets consist of one or more
DFSMStvs. See Data Facility Storage Management stripes and can take advantage of the sequential data
Subsystem Transactional VSAM Services. striping access technique. See also striping and stripe.
dictionary. A table that associates words, phrases, or extent. A continuous space on a DASD volume
data patterns to shorter tokens. The tokens are used to occupied by a data set or portion of a data set.
replace the associated words, phrases, or data patterns
when a data set is compressed.
F
direct access. The retrieval or storage of data by a
reference to its location in a data set rather than relative field. In a record or control block, a specified area
to the previously retrieved or stored data. See also used for a particular category of data or control
addressed-direct access. information.
direct access device space management (DADSM). file-owning region (FOR). A data-owning region, a
A DFP component used to control space allocation and CICS address space whose primary purpose is to
deallocation on DASD. manage files and databases.
direct data set. A data set whose records are in file system. In the z/OS UNIX hierarchical file system
random order on a direct access volume. Each record is (HFS) environment, the collection of files and file
stored or retrieved according to its actual address or its management structures on a physical or logical mass
format-DS. ASCII variable-length, spanned records. hierarchical file system (HFS) data set. A data set
that contains a POSIX-compliant file system, which is a
format-F. Fixed-length records. collection of files and directories organized in a
hierarchical structure, that can be accessed using z/OS
format-FB. Fixed-length, blocked records.
UNIX System Services. See also file system.
format-FBS. Fixed-length, blocked, standard records.
hierarchical file system (HFS). A POSIX-compliant
format-FS. Fixed-length, standard records. file system, which is a collection of files and directories
organized in a hierarchical structure, that can be
format-U. Undefined-length records. accessed using z/OS UNIX System Services. HFS
enables an application written in a high-level language
format-V. Variable-length records. to create, store, retrieve, and manipulate data on a
storage device. The view of the data to the end user is
format-VB. Variable-length, blocked records. a hierarchical directory structure similar to IBM DOS.
See also file system.
format-VBS. Variable-length, blocked, spanned
records.
I
format-VS. Variable-length, spanned records.
index record. A collection of data-record pointers
Forward recoverable data set. A data set that was retrieved and stored together. Contrast with index record
defined with the LOG(ALL) attribute option.
instance. (1) The code and control blocks that
forward recovery. A process used to recover a lost represent access to VSAM data sets through
data set. The data is recovered from a backup copy and DFSMStvs. An instance of DFSMStvs starts when
all the changes that were made after the backup copy DFSMStvs is initialized as part of SMSVSAM address
was taken are applied. The forward recovery process space initialization or enabled by operator command.
requires a log of the changes made to a data set, The instance ends when DFSMStvs enters a quiesced
together with a date and time stamp. The log of or disabled state or when the SMSVSAM address space
changes is called the forward recovery log. ends. (2) A peer recovery instance of DFSMStvs serves
to recover from the failure of some other DFSMStvs
forward recovery log. A log that contains copies of
instance that ran on another system within a sysplex
records after they were changed. The forward recovery
when the system on which the other DFSMStvs
log records are used by forward recovery programs and
instance was running failed. The peer recovery instance
products such as CICS VSAM Recovery (CICSVR) to
shares the SMSVSAM address space, and certain
reconstruct the data set in the event of hardware or
control blocks, with a ″native″ DFSMStvs instance.
software damage to the data set.
Automatic restart manager (ARM) can start a peer
free space. Space reserved within the control intervals recovery instance automatically when a system in a
of a key-sequenced data set for inserting new records sysplex fails. A peer recovery instance can also be
into the data set in key sequence or for lengthening started manually by operator command. The instance
records already there; also, whole control intervals ends when it completes its peer recovery process, when
reserved in a control area for the same purpose. it is stopped by operator command and enters a
quiesced state, or when the SMSVSAM address space
ends.
G
in-backout. The state of a unit of recovery when one
gigabyte. 1 073 741 824 bytes. or more resource managers reply negatively to a
Glossary 133
Glossary
commit request. The syncpoint manager tells each
resource manager to back out the changes. The
K
resources are returned to the values they had before
key-sequenced data set (KSDS). A VSAM data set
the unit of recovery was processed. When all the
whose records are loaded in ascending key sequence
resource managers have backed out the changes, the
and controlled by an index. Records are retrieved and
syncpoint manager notifies the application.
stored by keyed access or by addressed access, and
in-commit. The state of a unit of recovery when all new records are inserted in key sequence because of
resource managers reply positively to a commit request. free space allocated in the data set. Relative byte
The syncpoint manager tells each resource manager to addresses of records can change because of control
make its changes permanent. When all resource interval or control area splits.
managers have made the changes, the syncpoint
keyed-sequential access. In VSAM, the retrieval or
manager notifies the application.
storage of a data record in its key or relative-record
in-completion. The state of a unit of recovery when sequence, relative to the previously retrieved or stored
any enabled completion exit routines run. After this record as defined by the sequence set of an index.
phase completes, RRS passes a return code to the
kilobyte. 1024 bytes.
application indicating that the changes have been
committed or backed out.
L
in-doubt. For a distributed request, the state of a unit
of recovery on the originating system from the end of library. Synonym for partitioned data set. See
the prepare phase of the two-phase commit until the partitioned data set.
distributed syncpoint resource manager (DSRM) returns
a commit or backout request. linear data set (LDS). A VSAM data set that contains
data but no control information. A linear data set can be
in-end. The state of a unit of recovery when the accessed as a byte-addressable string in virtual storage.
resource managers have responded to the syncpoint
manager that commit or backout is complete. The unit load module. The output of the linkage editor; a
of recovery is logically complete. program in a format ready to load into virtual storage for
execution. Contrast with program object.
in-flight. The state of a unit of recovery when an
application accesses protected resources. The resource local shared resources (LSR). Resources in the local
managers express interest in the unit of recovery. resource pool.
in-forget. The state of a unit of recovery for a locate mode. A transmittal mode in which a pointer to
distributed request. The unit of recovery has completed, a record is provided instead of moving the record.
but RRS is waiting for the server distributed syncpoint Contrast with move mode.
resource manager (SDSRM) to indicate how to process
the log records for the unit of recovery. log of logs. A log that DFSMStvs and CICS write to
provide information to forward recovery programs such
in-only-agent. The state of a unit of recovery when as CICS VSAM Recovery (CICSVR). The log of logs is
only one resource manager has expressed an interest a form of user journal that contains copies of the tie-up
in the unit of recovery. RRS invokes the ONLY_AGENT records that DFSMStvs or CICS has written to forward
exit routine to tell the resource manager to process the recovery logs. This log provides a summary of which
commit immediately. recoverable VSAM data sets that DFSMStvs or CICS
has used, when they were used, and to which log
in-prepare. The state of a unit of recovery when the stream the forward recovery log records were written.
application has issued a commit request and the
If you have a forward recovery product that can utilize
syncpoint manager tells each resource manager to
the log of logs, ensure that all CICS regions that share
prepare its resources for commit or backout.
the recoverable data sets write to the same log-of-logs
in-reset. The state of a unit of recovery before an log stream.
application program has used any protected resources.
log stream. A log stream is a collection of data in log
in-state-check. The state of a unit of recovery when blocks that reside in the coupling facility or on DASD.
the application has issued a commit request and the
log tail. In DFSMStvs, the oldest log record of interest.
resource managers check if their resources are in the
Log tail deletion is the process of deleting unneeded
correct state.
records that are older than the oldest record of interest
to DFSMStvs.
M P
management class. A collection of management Parallel Sysplex. A sysplex that uses one or more
attributes, defined by the storage administrator, used to coupling facilities.
control the release of allocated but unused space; to
control the retention, migration, and back up of data partitioned data set (PDS). A data set on direct
sets; to control the retention and back up of aggregate access storage that is divided into partitions, called
groups, and to control the retention, back up, and class members, each of which can contain a program, part of
transition of objects. a program, or data.
member. A partition of a partitioned data set or PDSE. partitioned data set extended (PDSE). An
system-managed data set that contains an indexed
move mode. A transmittal mode in which the record to directory and members that are similar to the directory
be processed is moved into a user work area. and members of partitioned data sets. A PDSE can be
used instead of a partitioned data set.
MVS. Multiple Virtual Storage.
path. A named, logical entity composed of one or
N more clusters (an alternate index and its base cluster,
for example).
native context. The automatically occurring context of
PDS directory. A set of records in a partitioned data
a work request. A native context is associated with a
set (PDS) used to relate member names to their
single task. This context always exists.
locations on a DASD volume.
non-VSAM data set. A data set allocated and
peer recovery. A recovery process that occurs when
accessed using one of the following methods: BDAM,
an application fails. Peer users can perform recovery
BPAM, BISAM, BSAM, QSAM, QISAM.
and clean up resources.
nonrecoverable data set. A data set for which no
peer recovery instance. See instance.
changes are logged because its LOG parameter is
either undefined or set to NONE. Neither backout nor pointer. An address or other indication of location. For
forward recovery is provided for a nonrecoverable data example, an RBA is a pointer that gives the relative
set. location of a data record or a control interval in the data
set to which it belongs.
nonshared resources. A data set that does not use
shared resources. primary key. One or more characters within a data
record used to identify the data record or control its use.
NSR. See nonshared resources.
A primary key must be unique.
Glossary 135
Glossary
program library. A type of PDSE which contains resource manager (RM). A subsystem or component,
program objects only. A PDSE from which programs are such as CICS, IMS, or DB2, or DFSMStvs, that
loaded into memory for execution by the operating manages resources that can be involved in transactions.
system. There are three types of resource managers: work
managers, data resource managers, and communication
program object. All or part of a computer program in resource managers.
a form suitable for loading into virtual storage for
execution. Program objects are stored in PDSE program reusable data set. A VSAM data set that can be
libraries and have fewer restrictions than load modules. reused as a work data set, regardless of its old
Program objects are produced by the binder. contents. It must not be a base cluster of an alternate
index.
protected resource. A local or distributed resource
that can be changed in a synchronized manner during RLS. See VSAM record-level sharing (VSAM RLS).
processing coordinated by a syncpoint manager, such
as RRS. Databases, conversations between two
communications managers, or product-specific
S
resources can all be protected resources. A protected
scheduling. The ability to request that a task set
resource is also often called a recoverable resource.
should be started at a particular interval or on
occurrence of a specified program interrupt.
R
secondary space allocation. Amount of additional
random access. See direct access. space requested by the user for a data set when
primary space is full. Contrast with primary space
record definition field (RDF). A field stored as part of allocation.
a stored record segment; it contains the control
information required to manage stored record segments secondary system log. See shunt log.
within a control interval.
security. See data security.
record-level sharing. See VSAM record-level sharing
(VSAM RLS). sequence checking. The process of verifying the
order of a set of records relative to some field’s collating
recoverable data set. A data set that can be sequence.
recovered using backout or forward recovery
processing, defined with the LOG parameter set to UNDO sequential access. The retrieval or storage of a data
or ALL. See also protected resource. record in: its entry sequence, its key sequence, or its
relative record number sequence, relative to the
recoverable resource. A data set that can be previously retrieved or stored record. See also
recovered using commit, backout, or forward recovery addressed-sequential access and keyed-sequential
processing because its LOG parameter is set to UNDO or access.
ALL.
sequential data set. A data set whose records are
register. An internal computer component capable of organized on the basis of their successive physical
storing a specified amount of data and accepting or positions, such as on magnetic tape. Contrast with
transferring this data rapidly. direct data set.
relative byte address (RBA). The displacement of a service request block (SRB). A system control block
data record or a control interval from the beginning of used for dispatching tasks.
the data set to which it belongs; independent of the
manner in which the data set is stored. shared lock. A lock that several tasks can hold.
relative record data set (RRDS). A type of VSAM shared resources. A set of functions that permit the
data set whose records have fixed or variable lengths, sharing of a pool of I/O-related control blocks, channel
and are accessed by relative record number. programs, and buffers among several VSAM data sets
open at the same time.
residence mode (RMODE). The attribute of a load
module that identifies where in virtual storage the shunt. The process of moving failed work or
program will reside (above or below 16 megabytes). long-running work from the primary system log to the
secondary system log. If a unit of work fails, it is
resource. A database, a conversation between two removed (shunted) from the primary system log to the
systems, or a product-specific item. A resource can be secondary system log, pending recovery from the
local (residing on the current system) or distributed failure.
(residing on another system). A resource is protected
when it can be changed in a synchronized manner.
Glossary 137
Glossary
K M
keyboard 125 main UR state log 34
multitasking 78
L
licensed documents xiii N
list structures 29 native contexts 20
size 31 non-CICS applications 20
lock structure 28 non-RLS
size 29 access to VSAM data sets 11
locking access with retained locks 13
non-RLS 12 locking 12
retained locks 12 nonrecoverable data set 2
serializing resources 16 nonrecoverable data sets
locking, resource 15 read and write sharing 11
log 2, 3, 18
backout 17, 18
forward recovery 17 O
system offloading 54
primary 2 Overview of DFSMStvs 14
secondary 2
log of logs 2, 34, 59
log stream 47, 51
forward recovery 56
P
parallel sysplex environment 41
log streams
peer recovery 51, 114
access to 60
interference 116
CICS 35
SMSVSAM failures during 116
DASD log data sets 37
starting 115
DASD staging data sets 37
system failures during 116
DASD-only 35
performance delays 47
definition examples 52
planning for DFSMStvs
DFSMStvs 33
coupling-facility planning 26
forward recovery logs 34
overview 25
log of logs 34
planning tasks 25
overview 33
processor capacity 31
primary system log 2, 33
software configuration 32
RRS 34
system logger 32
secondary system log 2, 33
VSAM operations planning 38
shunt 3
Index 141
spanned records 67 units of recovery 21
staging data sets 44 in-doubt 55
storage types long-running 55
primary 51
secondary 51
tertiary 51 V
sync point 3 VSAM data sets 67
syncpoint manager non-RLS access 11
coordinating recovery 4 RLS access 12
definition 3 share options 12
system log VSAM operations
storage 54 ERASE 55
system log streams PUT 55
names 53 VSAM operations planning
system logger 47 Automatic Restart Manager (ARM) 39
availability 107 forward recovery operation 38
list structures 29 overview 38
logging flow 32 recovery procedures 38
planning 32 reorganization 39
VSAM record management requests 17
VSAM RLS
T Read integrity options 10
task control block (TCB) 66, 78 VSAM RLS (record-level sharing) 6, 14, 19
transaction processing access mode
batch job 15 GSR 9
description 3 LSR 9
DFSMStvs and RLS tasks 16 NSR 9
overview 1 and CICS 6
transactional recovery 4 data set types
transactional recovery 15 ESDS 6, 67
CICS 9 KSDS 6, 67
CICS file-control program 19 RRDS 6, 67
coordinating recovery 4 VRRDS 6
description 4 enabling
nonrecoverable data sets 11 MACRF=RLS 10
overview 1 execution mode requirements 13
programs options not supported 13
application program 4 processing 6
resource manager 4 read integrity options 10
syncpoint manager 4 read sharing integrity 11
resource locking 4, 5 recoverable data set 6
resource recovery logging 5 recoverable data sets 8
two-phase commit processing 5 request 13
transactions retained locks 12
defining 64 share options 12
two-phase commit 15, 19, 22 SMSVSAM
processing 3, 5 server 9
two-phase commit processing 4, 5 using the coupling facility 6
VSAM sharing
cache structures 28
U lock structure 28
undo log 3
backout logging 54
unit of recovery 3, 20 W
unit of work 3 waits 113
unit-of-recovery states
in-backout state 22
in-commit 22
in-flight 22
in-prepare 22
in-reset 22
Overall, how satisfied are you with the information in this book?
How satisfied are you that the information in this book is:
When you send comments to IBM, you grant IBM a nonexclusive right to use or distribute your comments in any
way it believes appropriate without incurring any obligation to you.
Name Address
Company or Organization
Phone No.
___________________________________________________________________________________________________
Readers’ Comments — We’d Like to Hear from You Cut or Fold
SC26-7348-00 Along Line
_ _ _ _ _ _ _Fold
_ _ _and
_ _ _Tape
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _Please
_ _ _ _ _do
_ _not
_ _ staple
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _Fold
_ _ _and
_ _ Tape
______
NO POSTAGE
NECESSARY
IF MAILED IN THE
UNITED STATES
_________________________________________________________________________________________
Fold and Tape Please do not staple Fold and Tape
Cut or Fold
SC26-7348-00 Along Line
Printed in U.S.A.
SC26-7348-00