Note 365481 - Block corruptions
Note Language: English Version: 50 Validity: Valid from 18.12.2007
Summary
Symptom
You have a corrupt block and want to clean it up.
The following Oracle error messages indicate a corrupt block:
ORA-01410: Invalid ROWID
ORA-01498: Block check failure - see trace file
ORA-01499: Table/index cross reference failure - see trace file
ORA-01578: ORACLE data block corrupted
ORA-08102: Index key not found
ORA-08103: Object no longer exists
ORA-00600 [x]:
This error message states that a few arguments indicate corruptions, for
example x=12700 or x=6002. In general Ora-00600 does not indicate
corruption. However, you must determine the cause of Ora-00600.
More Terms
ora-1578, ora-1499, ora-1498, corruption, corrupt block
[12700] [6002]
Cause and Prerequisites
It is very likely that the corruption is caused by problems at a level
below that of the database (hardware, operating system, firmware, and so
on).
In rare cases Oracle bugs may cause corruptions. Since Restore/Recovery can
generally not repair this type of corruption, we have released a Hot News
note for this.
Under certain conditions, Oracle may indicate that blocks of objects are
corrupt during a recovery process since changes to these objects were not
written to the online redo log. This may occur, for example, in a BW system
when you create indexes with the 'nologging' option. The following are
clear indications of blocks deliberately marked as software-corrupt: After
a recovery (1), a large number (2) of corrupt blocks occur in a relatively
small number (3) of indexes (4). The corrupt blocks appear consecutively in
sequences (5), for example, the blocks 5,6,7,8,9,10,11 and 127,128,129,130
and 1120,1121,1122 are reported as corrupt. You cannot correct this type of
corruption using the restore and recovery function, as the corruption
actually occurred during a recovery operation. The only solution here is to
recreate the indexes. For more details, see Note 547464.
Solution
In addition to the procedures for cleaning up a corrupt block described
below, you should ALWAYS:
o See FAQ Note 540463 for further information about consistency
checks and block corruptions. This note is essential to help you
understand the activities described in the following. In addition,
this note describes the option to recover corrupt blocks with
almost zero downtime, if there is a non-corrupt back-up.
09.02.2008 Page 1 of 10
Note 365481 - Block corruptions
o Check your hardware (or have it checked) because defective hardware
is the cause for corruption in most cases.
o Check your ENTIRE database according to Note 23345.
o For all steps that you execute, always ensure that the initial
status can be restored (for example, always copy the files you want
to replace beforehand - always rename the indexes of corrupt
tables, do not drop them).
o Log EXACTLY which actions were carried out and in which sequence.
In the following section, the database user is referred to as SAPR3.
Replace this name as appropriate to your situation with SAP<SID>, SAPSR3
and so on.
1. Identify the object that belongs to this corrupt block
Depending on the consistency check procedures, you will receive the
object (table or index) name directly or simply the number of the
corrupt block and the name of the data file that contains the corrupt
block. Under unfavorable circumstances, the system does not record a
table name anywhere if a corrupt block is identified during running
operation. Check your ENTIRE database according to Note 23345.
Oracle distinguishes between relative and absolute file numbers.
Although these are often the same, this is not necessarily always the
case. Unless otherwise indicated, 'file number' refers to the absolute
file number in all cases below. Within the context of this note,
relative file numbers only play a role in the specification of rowids
(see below). The relative file number is also indicated in dbv
messages that refer to a relative dba (database block address), for
example:
Corrupt block relative dba: 0x24400b33 (file 145, block 2867)
For all selects on dba_extents, v$datafile, when you dump blocks or
during brrestore of individual files, you must always specify the
absolute file number.
It is possible to map between absolute file numbers, relative file
numbers and file names using the following statement:
select file#, rfile#, name from v$datafile;
If you have only file names and block numbers (=page numbers), execute
the following SQL queries in sqlplus or servermanager:
o select file# from v$datafile where name = '<Filename>';
=> file number of the file that contains the corrupt block
o select segment_name, partition_name, segment_type, block_id, blocks
from dba_extents
where (<corrupted block> between
block_id and (block_id + blocks - 1))
and file_id = <Filenummer from the first statement> and rownum < 2;
=> Object name, object type
09.02.2008 Page 2 of 10
Note 365481 - Block corruptions
If this SELECT needs to be executed for several blocks, it may be
worth while to copy the dba_extents completely using 'create table
as select', create an index for file_id, block_id, and to execute
the statement on the copy. Be aware that the copy requires a
considerable amount of memory, and that you should delete it
immediately after the queries in order to avoid the risk of reading
obsolete data in future SELECT statements.
If this query does not return any object, it is a question of:
- a corrupt block that does not belong to any object (this is
probably the case).
These types of blocks are only identified by the DB Verify. If
Oracle wants to use this block for an object, the system
reformats it and then no longer reports it as being corrupt.
You can ignore these blocks or preferably delete them using one
of the following methods:
a) Use brspace to reorganize the tablespaces in question
(including data files) OR
b) Create a dummy object (dummy table) that uses this block,
fill this dummy object and then delete it. The procedure is
described in detail in Note 354293.
- a block that belongs to an object that is located in a LMTS,
the header block of which is in a data file that is currently
set offline (this is rarely the case).
To get an overview of all of the objects that are not displayed
in dba_extents, execute the following statement:
select
s.segment_name,
s.partition_name,
s.segment_type,
s.header_block,
s.header_file
from
dba_tablespaces t,
dba_segments s,
dba_data_files dd,
v$datafile vd
where
vd.status not in ('SYSTEM','ONLINE') and
vd.file# = dd.file_id and
dd.tablespace_name = t.tablespace_name and
t.extent_management = 'LOCAL' and
t.tablespace_name = s.tablespace_name and
vd.file# = s.header_file;
2. Is it an object of type Cache?
A cache object may become corrupt if the database was upgraded to
Oracle 9 and installed on Oracle 6. From now on, dbverify identifies
blocks of this object as being corrupt. This is not critical, but it
09.02.2008 Page 3 of 10
Note 365481 - Block corruptions
is inconvenient.
To eliminate THIS TYPE of corruption, open a message on the BC-DB-ORA
component and send it directly to Oracle Development Support,
referring to this note. Development Support will repair the underlying
dictionary inconsistency. To your message, attach the trace file
created by the following command in the saptrace/usertrace:
alter system dump datafile <file_id> block min <lowest block_id>
block max <highest block_id>;
In addition, attach the output of the following command:
select owner, segment_name, header_file, header_block, relative_fno,
blocks from dba_segments where segment_type = 'CACHE';
3. Does the database object belong to the SYS database user or is it
located in the SYSAUX tablespace?
General rule: The object only belongs to the SYS user if it is in the
system tablespace.
If yes:
Save the corrupt data file and transfer a backup of this file from a
backup that does not contain the corruption. Recover this file up to
the current time. If you have no such backup, your consistency
check/backup concept is inadequate.
The approach described in Note 748434 may be of further help. Since it
selects data from the dictionary, the system may return corruption
messages when you follow this procedure, which make it impossible to
proceed (Note 53716 may also be helpful).
If the procedures described are not successful, contact SAP Support
for more help. See Note 1116190 for information.
4. Is it a rollback segment of the SAPR3 user?
If yes:
Proceed according to Note 4190.
5. Is it an undosegment (you use new Undo Management as of Oracle 9)?
If yes:
Check whether all the following conditions have been met:
o the DB can be stopped with 'shutdown normal' or 'shutdown
immediate'
o there are no entries written during stopping/starting in the alert
log that are connected to the undo corruption
o v$fast_start_transaction is empty after each restart of the DB
o no rollback segment-specific underscore parameters were used
09.02.2008 Page 4 of 10
Note 365481 - Block corruptions
=> The corruption is either in the free space of the undo table or in
an area of the undo segment that is no longer required for a rollback
of an open transaction. If all the prerequisites have been met, the
undo tablespace can be dropped and created again. Otherwise, open a
problem message.
6. Is it a user SAPR3 index?
If yes:
You can repair corrupt indexes by using brspace to carry out a
reorganization (delete and recreate the index), or by performing a
restore/recovery on the corrupt files. Brspace supports Oracle's
rebuild option when structuring indexes, but this requires a readable
index, which makes this option unsuitable. In most cases, an index
reorganization is considerably faster than a restore/recovery unless
you QUICKLY have a CHECKED backup available as well as ALL archive
logs that have arisen since you created the backup and only a few
changes were carried out in the tablespace in question that must be
traced. If you are not sure which method is better for you, select the
index reorganization, provided that you do not have additional table
corruptions.
Index corruptions are not critical regarding data loss because you can
always set up indexes again. However, during index reorganization, the
table is locked against change accesses, which is why you should
perform this during a period of little or no system activity.
If you want to reorganize a unique index, you MUST either stop the R/3
system in order to avoid duplicate keys or use the Hide option during
reorganization.
You can recognize unique indexes if the following select returns a
'UNIQUE' (otherwise 'NONUNIQUE'):
select INDEX_NAME, UNIQUENESS from dba_indexes where index_name =
'<OBJECTNAME>';
7. Is it a user sapr3 table?
If yes:
If you have not done so already, use dbverify to analyze all the
tablespace files that contain the table, to check whether the table
still has more corrupt blocks.
Analyze the table (if you have not done this already) using 'analyze
table validate structure cascade'. Note: During this time, the table
is locked against changes. If necessary, terminate the Analyze, for
example if your WP update hangs with an update on the table. As of
Oracle 9, you can carry out an 'analyze table validate structure
09.02.2008 Page 5 of 10
Note 365481 - Block corruptions
cascade online' that no longer blocks the table.
Do not try to reorganize the table after a successful export. Certain
corruptions can be exported and errors only occur during the import
process (IMP-00020).
a) dbverify without errors/analyze without errors
You probably do not have a corruption. Repeat the access that
indicated the corruption. If this is successful this time AND if
you can read the table using a
select * from sapr3."<TABLE>";
AND if none of the procedures described in Note 23345 reports a
problem, it is very likely that a temporary corruption existed in
the memory but not on the hard disk. The block (which was only
corrupt in the main memory) was not reimported to the hard disk
because it was not changed. Always check your hardware in this
case. Note 786645 contains additional reasons for temporary
corruption.
b) dbverify with errors/analyze without errors
The corruption is probably located in a block of the table that has
never contained data. Execute the following command:
alter table <tablename> deallocate unused;
The corruption should then appear in the free space and should
therefore be deleted as described there.
Furthermore, in tables with LOB columns, the corruption may be
located in the lob segment. The analysis is not capable of
recognizing this type of corruption. In this case, proceed as
described under "dbverify with errors/analyze with errors".
c) dbverify with errors/analyze with errors
Save the corrupt data file and transfer a backup of this file from
a backup that does not contain the corruption. Recover this file up
to the current time.
If you do not have a backup that you can definitely say does not
contain the corruption, you can restore a file from a backup that
is still available in A DIRECTORY THAT DOES NOT BELONG TO THE DB
and check this backup using dbv. If a corruption is not reported,
you have subsequently verified your backup because, on the one
hand, dbverify can recognize the corruption and, on the other, your
backup does not have any corruptions.
To import a file back into another directory using brrestore, use
the following command:
Abstract:
brrestore -b <log> -m <number of file>=<Restore Directory>
For example:
brrestore -b bddcbjcq.afd -m 59=c:\temp
d) dbverify without errors/analyze with errors
This case differs from the previous case in that dbverify does not
recognize an existing corruption that an analysis finds. If you do
not have a backup that you can definitely say does not contain the
corruption, you can check this easily at a later stage.
After you have made a backup of the corrupt file, you can reimport
and recover the backup of the file without checking it and hope
that the corruption was not yet contained in the backup OR
you can create a temporary check system using a (partial) DB copy
09.02.2008 Page 6 of 10
Note 365481 - Block corruptions
from this backup (consisting of the rollback, system and tablespace
that contains the corruption) and analyze the table on this check
system. If this is possible without errors, in retrospect you have
verified your backup and you can restore and recover this.
If all the backups contain the corruption, your consistency
check/backup concept is inadequate.
Create a message and provide the following information:
Information 1)
Determine the total number of table records using the primary index. To do
this, first determine the data type of the first primary index column
(called <column1> in the following). This is almost always a character
field.
Abstract:
describe sapr3."<TABLE>";
For example:
describe sapr3."RESB";
=>
Column Name Null? Type
------------------------------ -------- ----
MANDT NOT NULL VARCHAR2(3)
RSNUM NOT NULL VARCHAR2(10)
...
All of the following restrictions of the type
<column1> >= ' ' or <column1> < ' '
in the WHERE conditions require that the 'CHAR' string must be contained in
the 'Type' column in the FIRST row. This is almost always the case - also
in the example.
If a 'NUMBER' is contained instead of 'CHAR', you must ALWAYS change
<column1> >= ' ' or <column1> < ' '
to
<column1> >= 0 or <column1> < 0
Even if the condition appears meaningless, it forces Oracle to make a
primary index access. The hint alone is not sufficient.
Now count the entries:
Abstract:
select /*+ INDEX("<TABLE>" "<TABLE>~0") */ count(*)
from sapr3."<TABLE>"
where <column1> >= ' ' or <column1> < ' ';
For example:
Table RESB is corrupt
select /*+ INDEX("RESB" "RESB~0") */ count(*)
from sapr3."RESB"
a where mandt >= ' ' or mandt < ' ';
Include the call and command output of the commands in your message
(Information 1)
Information 2)
09.02.2008 Page 7 of 10
Note 365481 - Block corruptions
Determine the primary keys of the data records in the FIRST corrupt block
of the table by using the following statement:
Abstract:
select /*+ INDEX("<TABLE>" "<TABLE>~0") */ <keyfields of TABLE>
from "<TABLE>"
where dbms_rowid.rowid_to_restricted(rowid,1) like '<rowid>'
and <column1> >= ' ' or <column1> < ' ';
OR
select /*+ INDEX("<TABLE>" "<TABLE>~0") */ <keyfields of table>
from "<TABLE>"
where dbms_rowid.rowid_block_number(rowid) = <corrupted block> and
dbms_rowid.rowid_to_absolute_fno(rowid,'SAPR3','<TABLE>')
= <file# of the corrupted block>
and <column1> >= ' ' or <column1> < ' ';
For example:
The file that contains the corrupt block has the relative file number 59
(hex: 003B), the corrupt block has number 118882 (hex: 0001D062), and the
table containing this block (determined in accordance with 1)) is RESB.
select /*+ INDEX("RESB" "RESB~0") */ MANDT,RSNUM,RSPOS,RSART
from "RESB"
where dbms_rowid.rowid_to_restricted(rowid,1) like '0001D062.%.003B'
and mandt >= ' ' or mandt < ' ';
block.row.file
OR
the file that contains the corrupt block has the absolute file number 59
(hex: 003B), the corrupt block has number 118882 (hex: 0001D062) Table
that contains this block (determined in accordance with 1)) is RESB.
select /*+ INDEX("RESB" "RESB~0") */ MANDT,RSNUM,RSPOS,RSART
from "RESB"
where dbms_rowid.rowid_block_number(rowid) = 118882 and
dbms_rowid.rowid_to_absolute_fno(rowid,'SAPR3','RESB') = 59
and mandt >= ' ' or mandt < ' ';
Pay attention to upper and lower case for the terms in quotation marks.
Both commands must return the same results.
Include the call and command output of the commands in your message
(Information 2)
Information 3) - on request only
Dump the corrupt block using:
Abstract:
Alter system dump datafile <file# of the corrupted block> block
<corrupted block>;
For example:
Alter system dump datafile 59 block 118882;
Save the relevant file in the
$SAPDATA_HOME/saptrace/usertrace
09.02.2008 Page 8 of 10
Note 365481 - Block corruptions
directory and set it on sapserv3 as described in Note 40024.
As part of remote consulting, SAP Support will now clarify to what extent
this is redundant data or how you can read corrupt blocks. In this case,
this represents a partial loss of data that cannot be avoided. Under no
circumstances try to clean up the corruptions in SAP tables yourself by
reading the corrupt blocks because only SAP knows the dependencies between
the tables. Otherwise, although you can delete the DB corruptions,
application inconsistencies will occur as a result.
By setting certain parameters in init<SID>.ora, you can force Oracle to
read corruptions. Consequently, you can discover whether other corrupt
blocks still exist in a table. These corrupt blocks are skipped and logged
in the saptrace/usertrace directory. You can then analyze these as
described under Information 2) and Information 3). You can execute these
steps during productive operation, provided that the corruption is not so
serious that you cannot work productively anyway. For the time being, only
use the parameter in the session that you use for analysis.
After you have completed the analysis, you can activate the parameters for
the entire database and reorganize the table. Since these parameters may
result in very serious inconsistencies in the database, they are only
issued by request or if the tables that contain corruptions are specified.
It is ESSENTIAL that they are removed again before you include the
production operation again after you have reorganized the table. It is YOUR
responsibility whether to use the parameters or not. If you already know
the parameter, NEVER use it without prior consultation with SAP.
After this type of action, the remote consulting can try to restore the
lost data or return dependent tables to a consistent status again (this
does not apply to customer tables whose query dependencies only you, the
customer, know).
The following is a list of tables that can probably be restored from the
application support (secondary index tables):
BSAD, BSAK, BSAS, BSID, BSIK, BSIM, BSIS, BSIW, BSIX
Fully indexed tables (all columns of the table appear in the primary
index):
SCPARTICIP, USGRP
Tables that can be created empty and are filled automatically when the
system is stopped (if only one of the tables is affected, all tables of the
relevant row must nevertheless be emptied for consistency reasons):
D010L, D010Q, D010Y, D010LINF
D020L, D020LINF, D021L, D021LINF
D346T, D342L
DDFTX
D344L
REPOLOAD
DYNPTXTLD, DYNPLOAD
DDLOG
09.02.2008 Page 9 of 10
Note 365481 - Block corruptions
Header Data
Release Status: Released for Customer
Released on: 18.12.2007 10:30:04
Priority: Recommendations/additional info
Category: Consulting
Main Component BC-DB-ORA Oracle
The note is not release-dependent.
Related Notes
Number Short Text
1116190 Handling of database corruptions by the SAP Support
764015 Segment corruption in ASSM tablespace
724972 ORA-00200: Block, dba <address>, already marked corrupted
605062 FAQ: Restore and recovery
591600 Error due to inappropriate column values
546006 Problems with Oracle due to operating system errors
540463 FAQ: Consistency Checks + Block Corruptions
434645 Point-in-time recovery: prevention and effects
354293 DBVerify reports corrupt block in freespace area
96296 CC-ERROR: Database problem in client copy
31496 CC-INFO: Client deleted by mistake
23345 Consistency check of ORACLE database
4190 Recovery for defective rollback segments
4161 Complete Recovery
Attributes
Attribute Value
Database system ORACLE
09.02.2008 Page 10 of 10
