2021-02-16
Incident: 111126 / 2021 - BP1 : READ activation failure
due to "SQL message: ORA-00060: deadlock detected
while waitin for r
ID 002075129400001111262021
Customer 647545 - Wipro Limited
Installation 0020804815 - SAP NETWEAVER
System BP1 - BW Production system
Component Oracle (BC-DB-ORA)
Status Customer Action
Priority High
Estimated Automatic Confirmation Date 09.03.2021
Communication
15.02.2021 08:37:22 UK - Problem Description: Sthitaprajna Das (S0018064212)
Product Area selected: SAP NetWeaver > Business Warehouse
Component selected (Manual Selection): BC-DB-ORA
--- Steps to Reproduce ---
Note to SAP SUPPORT: I consent to SAP Support reproducing the described issue while connected remotely to my
non-production environments, using any steps provided by me in the description below. If it is necessary to do this in a
production environment, SAP Support will contact me specifically before proceeding.
--- Description ---
Dear SAP,
We have upgraded our BW system (NetWeaver 7.5 SP-17, Oracle - 19c)recently. Post upgrade, we have
encountered below deadlock issues while accessing tables /BIC/ADUK_READ00 & /BIC/ADNU_PAAL00
Database error text: "SQL message: ORA-00060: deadlock detected while waiting for resource"
Please be informed, we had set this parameter to fix this issue earlierfor oracle-11g. Now database has been
upgraded to 19c and in spite of having the value as 20, deadlocks are getting created in the table which
cause application job failure. Hence, to resolve the issue, we have tried to increase the INITRANS parameter
multiple times (to 50, 100 and even to 150) but issue still persists.
Also we could not find any specific pattern for the issue interms of load in the system.
Due to this issue, read activation jobs are getting failed in production which is having following impact to our
business.
> Delay of Data Enquiry Refresh.
> DES users will not be able to view the reads submitted previous day into the Xoserve. This can lead to
© 2021 SAP SE or an SAP affiliate company. All rights reserved 1 of 8
2021-02-16
incidents from users asking for updated data.
> Delay in completion of daily batch loads and reports
> Manual intervention of restart of the job.
Could you please look into the issue immediately and provide us the solution to resolve this deadlock issue?
Thanks,
Sthitaprajna Das
SAP Basis Team
15.02.2021 08:37:23 UK - Business Impact: Sthitaprajna Das (S0018064212)
Due to this issue, read activation jobs are getting failed in production which is having following impact to our
business.
> Delay of Data Enquiry Refresh.
> DES users will not be able to view the reads submitted previous day into the Xoserve. This can lead to
incidents from users asking for updated data.
> Delay in completion of daily batch loads and reports
> Manual intervention of restart of the job
15.02.2021 09:03:34 UK - Call to Customer: SAP
Contact Person: Mr. Sthitaprajna Das
Call time: 2021.02.15 / 10:03:34 / CET
Contact Number: +44-7770143933
Customer reached Y/N: y
Call back Y/N:
Subject of Call:looks an application deadlock
15.02.2021 09:56:52 UK - Reply: SAP
Dear Customer,
I have attached the shortdump occured at 15.02.2021 06:11:29
which refers to a deadlock occured to table /BIC/ADNU_PAAL00.
Please attach the corresponsding deadlock trace ( I am not able to reach that )
As we discussed on phone, deadlock is application type.
Since the issue did not occur on earlier oracle release (was it 12.1 or 12.2? ) the issue isnore complicated
then described oin note 84348 and can be not purely application origin.
I can see deadlocks for several tables like /BIC/ADNU_PAAL00, /BIC/ADUK_READ00.
These are ompressed tables with different sizes:
select table_name , INI_TRANS , compression, compress_for from dba_tables where table_name IN ( '/BIC/ADNU_PAAL00',
'/BIC/ADUK_READ00');
/BIC/ADNU_PAAL00 50 ENABLED ADVANCED (848 MB)
/BIC/ADUK_READ00 150 ENABLED ADVANCED ( Size[Mb] 290467 )
Would it be possible for you to decompress the smaller table /BIC/ADNU_PAAL00
and check if the deadlock reoccurs for that table afterwards ( as a test).
I would like to ask you the following:
- attach the oracle trace with the deadlock graph produced at 15.02.2021 06:11:29
- decompress /BIC/ADNU_PAAL00 as a test and tell us if deadlocks for that table still occur.
- confirm which was oracle release before upgrade and if deadlocks were there too with old oracle release
Best Regards,
Péter Mihálffy
SAP Product Support
15.02.2021 10:45:49 UK - Info for SAP: Sthitaprajna Das (S0018064212)
Dear Peter,
© 2021 SAP SE or an SAP affiliate company. All rights reserved 2 of 8
2021-02-16
As discussed, we have found the SQL statement which is creating this deadlock situation. However, we cant
avoid these statements to be run in parallel due to business requirement and to meet our SLAs.
Please be informed, same was working fine before upgrade and post upgrade of BW system to 7.5, we have
encountered this issue. Hence we would like to have a resolution for this issue.
We have attached the alert logs (traces) which contains the error details in the attachment section.
Trace Logs:-
FOR READ TABLE:-
2021-02-15 03:56:32.390*:ksq.c@13169:ksqdld_hdr_dump():
DEADLOCK DETECTED ( ORA-00060 )
See Note 60.1 at My Oracle Support for Troubleshooting ORA-60 Errors
[Transaction Deadlock]
The following deadlock is not an ORACLE error. It is a
deadlock due to user error in the design of an application
or from issuing incorrect ad-hoc SQL. The following
information may aid in determining the deadlock:
Deadlock graph:
------------Blocker(s)----------- ------------Waiter(s)------------
Resource Name process session holds waits serial process session holds waits serial
TX-00D6001B-0017EC2F-00000000-00000000 1174 5566 X 62572 753 1957 X 45844
TX-00AB0000-001D140A-00000000-00000000 753 1957 X 45844 1174 5566 X 62572
----- Information for waiting sessions -----
Session 5566:
sid: 5566 ser: 62572 audsid: 26814931 user: 44/SAPBWH
flags: (0x1000041) USR/- flags2: (0x40009) -/-/INC
flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
pid: 1174 O/S info: user: orabp1, term: UNKNOWN, ospid: 51364
image: oracle@T3PRPDB002
client details:
O/S info: user: bp1adm, term: pts/0, ospid: 24130
machine: T3PRVAP033 program: dw.sapBP1_D41@T3PRVAP033 (TNS V1-V3)
application name: GP00O2TRQCYLMI6HDAJB6636C9H, hash value=3237768036
action name: 21465, hash value=2374629301
current SQL:
UPDATE "/BIC/ADUK_READ00" SET
"RECORDMODE"=:A0,"/BIC/ICUKCLASS"=:A1,"/BIC/ICNUCNRFN"=:A2,"/BIC/ICUK_TTZ"=:A3,"/BIC/ICUK_MSN"=:A
_RR"=:A6,"/BIC/ICUK_MR"=:A7,"/BIC/ICUKPMP"=:A8,"/BIC/ICUKRECT"=:A9,"/BIC/ICNUMMIMI"=:A10,"/BIC/ICUKPM
/ICUK_CUR"=:A14,"/BIC/ICUKCTTZ"=:A15,"/BIC/ICUKPID"=:A16,"/BIC/IKREJEG"=:A17,"/BIC/ICUKDMSP"=:A18,"AE
:A22,"/BIC/ICUKSHARE"=:A23,"/BIC/ICUKSTS"=:A24,"/BIC/ICUKBEDT"=:A25,"/BIC/ICUKMRSC"=:A26,"UCSRVIDE
ICUKRCDT"=:A30,"/BIC/ICNURUPDA"=:A31,"/BIC/ICNUPRDT"=:A32,"/BIC/ICNUNXPDT"=:A33,"FC_COUNT"=:A34,
A37,"/BIC/IKNURC
-------------------------------------------------------------------------------------------------------------------------------------
© 2021 SAP SE or an SAP affiliate company. All rights reserved 3 of 8
2021-02-16
FOR PAAL TABLE:-
The following deadlock is not an ORACLE error. It is a
deadlock due to user error in the design of an application
or from issuing incorrect ad-hoc SQL. The following
information may aid in determining the deadlock:
Deadlock graph:
------------Blocker(s)----------- ------------Waiter(s)------------
Resource Name process session holds waits serial process session holds waits serial
TX-009D0011-001DAA2C-00000000-00000000 997 2202 X 8049 826 6259 X 46073
TX-00840014-0026BA3D-00000000-00000000 826 6259 X 46073 997 2202 X 8049
client details:
O/S info: user: bp1adm, term: pts/0, ospid: 12103
machine: T3PRVAP034 program: dw.sapBP1_D42@T3PRVAP034 (TNS V1-V3)
application name: GP00O2TRP86BUD6S4N6397XGKX0, hash value=321329047
action name: 7960, hash value=3501717678
current SQL:
UPDATE "/BIC/ADNU_PAAL00" SET
"RECORDMODE"=:A0,"/BIC/ICNUOPFLO"=:A1,"/BIC/ICNUALMTD"=:A2,"/BIC/ICNUSTFLG"=:A3,"/BIC/ICNUPRFLG
ON"=:A6,"CHANGEDBY"=:A7,"CHANGEDATE"=:A8,"/BIC/ICNUCUSCD"=:A9,"/BIC/ICNUMTRID"=:A10,"/BIC/ICNUM
:A14 WHERE "/BIC/ICNUSERID"=:A15 AND "/BIC/ICNUACTNU"=:A16 AND "/BIC/ICNUGASDY"=:A17 AND
"/BIC/ICNUPERID"=:A18 AND "/BIC/ICNUSPFUN"=:A19
-----------------------------------------------------------------------------------------------------------------------------
Answer for your queries:-
- attach the oracle trace with the deadlock graph produced at 15.02.2021 06:11:29 - Attached decompress
/BIC/ADNU_PAAL00 as a test and tell us if deadlocks for that table still occur. - It is not possible for us to
straight away decompress a table in Prod environment without knowing the impact to our current
business process
- confirm which was oracle release before upgrade and if deadlocks were there too with old oracle release -
we were in 11g and upgraded it to 19c on 30th Jan 2021. It has never happened in earlier releases.
Thanks,
Sthitaprajna Das
15.02.2021 13:30:23 UK - Info for Customer: SAP
Dear customer,
I'd like to inform you that I have forwarded your incident to the development support team and a colleague from that
team will be contacting you as soon as possible.
Best regards,
SAP Product Support
15.02.2021 14:57:12 UK - Reply: SAP
Dear customer,
your incident is now processed in Oracle Development Support.
© 2021 SAP SE or an SAP affiliate company. All rights reserved 4 of 8
2021-02-16
You see multiple deadlock situations with ODS activation since using Oracle 19c.
Please see SAP Note 3024449 for explanation and solution.
mit freundlichen Grüßen / kind regards
Bernhard Baumann
Oracle Platforms
External | On behalf of SAP Development Support
15.02.2021 16:15:36 UK - Info for SAP: Sthitaprajna Das (S0018064212)
Dear SAP,
Thanks for the note and suggestions.
Could you please help us to answer below questions?
1. We are facing this issue for these 2 tables as of now. If it happens for few more larger tables in future, then
what would be the enduring solution as decompress and compress options involves lot of manual work and
analysis in terms of impact to existing processes?
2. What are the risks associated with decompressing the table as we can see this SAP Note: 3024449 is
released today and no side effect has been mentioned in the note? Also enabling ADO policies mentioned in
the SAP Note does not look convincing to implement in Production environment as it suggests to create
tablespace and shift the table to new tablespace to enable the ADO policies.
3. Though it is clear from the SAP Note that, this is a bug after oracle upgrade to 19c and it was never an
issue in 12.1 or earlier as the compression of blocks has worked completely different before 12.2, could you
please confirm the same?
4. Is there any permanent fix available/planned in your roadmap of Oracle 19c?
Thanks,
Sthitaprajna Das
16.02.2021 07:42:08 UK - Reply: SAP
Dear Sthitaprajna Das,
to answer your questions:
1. We are facing this issue for these 2 tables as of now. If it happens for few more larger tables in future, then
what would be the enduring solution as decompress and compress options involves lot of manual work and
analysis in terms of impact to existing processes?
A: do proactive change all your ODS tables to be using ADO instead of OLTP compression as recommended
in note 3024449.
2. What are the risks associated with decompressing the table as we can see this SAP Note: 3024449 is
released today and no side effect has been mentioned in the note? Also enabling ADO policies mentioned in
the SAP Note does not look convincing to implement in Production environment as it suggests to create
tablespace and shift the table to new tablespace to enable the ADO policies.
A: there are no risks known.
© 2021 SAP SE or an SAP affiliate company. All rights reserved 5 of 8
2021-02-16
The existing blocks of table are NOT decompressed physically, rather we only change the flag for handling
new blocks no longer be OLTP compressed. When ADO kicks in after some time (by policy) then blocks not
touched for the given time in policy will be compressed at a time where is not much system load using
database background processes.
Using ADO is supported and recommended in SAP production systems too.
Please refer to the ADO whitepaper how it works and what one have to do to activate.
Its recommended to have ADO policies at tablespace level, that means either you create a new tablespace
for ADO and move the ODS tables over, or you activate it at the existing tablespace where ODS currently
located.
The recommendation is to use a new tablespace.
This SAP note was created by me yesterday, summarizing a solution we have given to some other customers
during the past weeks, facing the same issue.
3. Though it is clear from the SAP Note that, this is a bug after oracle upgrade to 19c and it was never an
issue in 12.1 or earlier as the compression of blocks has worked completely different before 12.2, could you
please confirm the same?
A: The clarify: This is not a bug.
BUG 31901675 - TX LOCK HELD BY ANOTHER SESSION NOT UPDATING THE SAME ROW
Status: 32 - Not a Bug. To Filer
we got the explanation in this bug entry and the information that it is a design constraint of OLTP
compression, what is not changeable.
That explanation was given to SAP by Note 3024449.
4. Is there any permanent fix available/planned in your roadmap of Oracle 19c?
A: the permanent solution is given in SAP Note 3024449.
As mentioned in 3), its not a bug and so we will not get another solution.
mit freundlichen Grüßen / kind regards
Bernhard Baumann
Oracle Platforms
External | On behalf of SAP Development Support
Contacts
Secondary
Role Name Time Zone Primary Phone E-Mail
Phone
Sthitaprajna Das +44- sthitaprajna.das
Reporter UK +44-7770143933
(S0018064212) 447770143933 @wipro.com
© 2021 SAP SE or an SAP affiliate company. All rights reserved 6 of 8
2021-02-16
Attachments
File Name Description File Type File Size Created By Created On
dump_deadlock.tx dump_deadlock.tx 15.02.2021
TXT 216.1 KB SAP
t t 09:32:05 UK
BP1_ora_46620.tr Trace files PAAL Sthitaprajna Das 15.02.2021
TRC 1.4 MB
c table 6:11AM (S0018064212) 10:45:49 UK
BP1_ora_51364.tr Trace file READ Sthitaprajna Das 15.02.2021
TRC 282.0 KB
c table 3:56AM (S0018064212) 10:45:49 UK
KBA/SAP Notes
KBA/SAP Note Number Note Text
Multiple deadlocks thrown during ODS activation after
3024449
updating to 19c
Action Log
Changed On Changed At Changed By Action Old Value New Value
Monday Sthitaprajna Das
08:37:22 UK Component BC-DB-ORA
15.02.2021 (S0018064212)
0020751294
Sthitaprajna Das
08:37:22 UK Incident created 0000111126
(S0018064212)
2021
Sthitaprajna Das
08:37:22 UK Priority changed Medium High
(S0018064212)
Sthitaprajna Das
08:37:22 UK Status Not Sent to SAP Sent to SAP
(S0018064212)
Memo/Text
09:03:34 UK SAP Call to Customer
changed
10:06:26 UK SAP Status Sent to SAP Customer Action
Memo/Text
10:06:26 UK SAP Reply
changed
Sthitaprajna Das
10:45:49 UK Status Customer Action Sent to SAP
(S0018064212)
Sthitaprajna Das Memo/Text
10:45:49 UK Info for SAP
(S0018064212) changed
© 2021 SAP SE or an SAP affiliate company. All rights reserved 7 of 8
2021-02-16
Memo/Text Info for
13:30:23 UK SAP
changed Customer
14:59:14 UK SAP Status Sent to SAP Customer Action
SAP Note
14:59:14 UK SAP 3024449
add./rem.
Memo/Text
14:59:14 UK SAP Reply
changed
Sthitaprajna Das
16:15:36 UK Status Customer Action Sent to SAP
(S0018064212)
Sthitaprajna Das Memo/Text
16:15:36 UK Info for SAP
(S0018064212) changed
Tuesday
07:45:12 UK SAP Status Sent to SAP Customer Action
16.02.2021
Memo/Text
07:45:12 UK SAP Reply
changed
© 2021 SAP SE or an SAP affiliate company. All rights reserved 8 of 8
