blog.tanelpoder.com 1
Advanced	
  Oracle	
  Troubleshooting	
  preparation	
  session:
Tanel’s	
  command	
  line	
  setup	
  &	
  tools
Tanel	
  Põder
http://blog.tanelpoder.com/seminar
blog.tanelpoder.com 2
Intro:	
  About	
  me
• Tanel	
  Põder
Oracle	
  Database	
  Performance	
  geek
Exadata	
  Performance	
  geek
Linux	
  Performance	
  geek
Hadoop	
  Performance	
  geek
Expert	
  Oracle	
  Exadata	
  
book
(with	
  Kerry	
  Osborne	
  and	
  
Randy	
  Johnson	
  of	
  Enkitec)
blog.tanelpoder.com 3
All	
  scripts	
  and	
  tools	
  are	
  downloadable	
  and	
  free
• Download	
  the	
  scripts	
  and	
  tools	
  from:
• http://blog.tanelpoder.com/files/
• Updates	
  (since	
  I	
  delivered	
  this	
  class	
  in	
  2010):
• I	
  have	
  shut	
  the	
  PlanViz website	
  down	
  so	
  ignore	
  some	
  slides
• I	
  haven't	
  updated	
  the	
  PerfSheet v3	
  tool	
  since	
  2010
• Luca	
  Canali has	
  built	
  a	
  new	
  PerfSheet v4:
• http://canali.web.cern.ch/canali/resources.htm
blog.tanelpoder.com 4
Productivity	
  and	
  speed	
  of	
  action
• Talking	
  to	
  Oracle
• SQLPLUS	
  still	
  rocks	
  J
• …especially	
  in	
  cases	
  where	
  you	
  need	
  to	
  deviate	
  from	
  the	
  usual	
  path
• You’ll	
  have	
  a	
  (searchable)	
  history	
  of	
  output	
  of	
  your	
  last	
  commands
• On	
  the	
  screen	
  or	
  in	
  a	
  spool/logfile…
• You	
  can	
  run	
  your scripts
• It’s	
  more	
  “pure”	
  than	
  GUI	
  development	
  tools	
  (which	
  try	
  to	
  be	
  clever,	
  do	
  extra	
  
stuff,	
  don’t	
  fetch	
  all	
  data	
  etc)
blog.tanelpoder.com 5
SQLPLUS	
  on	
  Windows
• Configuration
• Do	
  NOT	
  use	
  sqlplusw.exe	
  -­‐>
• This	
  is	
  what	
  you	
  get	
  when
running	
  sqlplus	
  from	
  Start
menu
• Luckily	
  sqlplusw.exe	
  is
gone	
  since	
  11g!
• Use	
  CMD.exe	
  and	
  run
sqlplus.exe	
  -­‐>
Much	
  better!!!
blog.tanelpoder.com 6
• CMD.exe	
  needs	
  some	
  tweaking…
• You	
  should	
  check	
  QuickEdit	
  mode,	
  Increase	
  Screen	
  Buffer	
  size	
  etc
SQLPLUS	
  on	
  Windows	
  – CMD.exe	
  settings
1. 2. 3.
blog.tanelpoder.com 7
Setting	
  SQLPLUS	
  Window	
  title	
  automatically
• Set	
  cmd	
  window	
  title	
  to	
  include	
  database	
  session	
  info
• SQL>	
  host	
  title	
  “test”
• @i.sql	
  queries	
  session,	
  instance	
  info	
  from	
  V$	
  views	
  and	
  sets	
  title	
  accordingly
• i.sql	
  is	
  automatically
called	
  by	
  login.sql
if	
  its	
  in	
  SQLPATH
directory
blog.tanelpoder.com 8
SQLPLUS	
  on	
  Windows
• Command	
  line	
  history
• Up/Down	
  arrow	
  -­‐ navigate	
  between	
  commands
• CTRL+Left/Right	
  arrow	
  – jump	
  one	
  word	
  left/right
• F7 – Display	
  a	
  popup	
  box	
  with	
  command	
  line	
  history
• Use	
  up/down	
  arrows	
  to	
  navigate
• ENTER	
  to	
  execute	
  command
• Right	
  arrow	
  to	
  put	
  the	
  command	
  to	
  command	
  line	
  for	
  editing
• First	
  letter	
  of	
  command	
  to	
  jump	
  between	
  commands	
  starting	
  with	
  that	
  letter	
  
(case	
  sensitive)
• F8 – Retrieve	
  command	
  from	
  history
• Type	
  a	
  beginning	
   of	
  the	
  command	
  (for	
  example	
  “sel”)	
  and	
  press	
  F8
• This	
  will	
  jump	
  through	
  all	
  commands	
  starting	
  with	
  “sel”	
  (case	
  sensitive)
• List	
  commands	
  from	
  history
• SQL>	
  host	
  doskey	
  /history	
  /exename=sqlplus.exe
• @h <text>	
  – history	
  script	
  showing	
  old	
  commands	
  with	
  text	
  int	
  them
• @ha	
  – history	
  all
blog.tanelpoder.com 9
SQLPLUS	
  on	
  Windows
• Output	
  formatting
• @HTML
• Uses	
  SET	
  MARKUP	
  ON	
  HTML
• Runs	
  the	
  query	
  in	
  SQLPLUS	
  buffer	
  and	
  spools	
  the	
  output	
  to	
  .HTML	
  file
• Launches	
  that	
  .HTML	
  file	
  with	
  appropriate	
  OS	
  command
• @XLS
• Same	
  as	
  above,	
  but	
  spools	
  to	
  an	
  .XLS	
  file	
  and	
  launches	
  it	
  :-­‐)
• @CSV
• Uses	
  SET	
  COLSEP	
  “,”	
  and	
  spools	
  to	
  a	
  .CSV	
  file
blog.tanelpoder.com 10
SQLPLUS	
  on	
  Unix	
  /	
  Linux	
  /	
  MacOS
• Configuration
• Setting	
  xterm	
  title
• From	
  shell
– echo	
  -­‐ne	
  "033]0;	
  your	
  text	
  here 007”
• From	
  SQLPLUS
– host	
  echo	
  -­‐ne	
  "033]0;	
  your	
  text	
  here 007”
• From	
  vi
– set	
  title
– set	
  titlelen=20
• Decent	
  terminal	
  emulators
• Putty	
  on	
  Windows	
  (small,	
  lightweight	
  and	
  works	
  well!)
– http://www.chiark.greenend.org.uk/~sgtatham/putty/
• Terminator	
  on	
  MacOS,	
  Unix,	
  Linux	
  (and	
  apparently	
  Windows)
– http://software.jessies.org/terminator/
blog.tanelpoder.com 11
SQLPLUS	
  on	
  Unix	
  /	
  Linux	
  /	
  MacOS
• Output	
  formatting
• Same	
  as	
  on	
  Windows
• Launching	
  the	
  HTML	
  file	
  is	
  done	
  with	
  a	
  different	
  command:
• MacOS – open command	
  launches	
  the	
  program	
  associated	
  with	
  .HTML	
  extension
• Unix/Linux	
  – I	
  just	
  run	
  firefox	
  by	
  default,	
  but	
  you	
  can	
  change	
  it	
  in	
  the	
  script
• Terminal	
  with	
  left-­‐right	
  scrolling	
  capability?
• I’m	
  using	
  Terminator
• (Not	
  the	
  one	
  you	
  can	
  install	
  from	
  Ubuntu	
  repositories)
• http://software.jessies.org/terminator/
• However,	
  it	
  needs	
  to	
  install	
  its	
  own	
  terminfo file	
  to	
  function	
  properly!
– This	
  will	
  be	
  an	
  issue	
  if	
  you	
  need	
  to	
  SSH	
  into	
  a	
  lot	
  of	
  servers	
  in	
  an	
  enterprise
– Good	
  for	
  demoing	
  stuff	
  though!	
  J
• SQL>	
  set	
  truncate	
  on
• Sqlplus	
  allows	
  you	
  to	
  truncate	
  the	
  columns	
  which	
  don’t	
  fit	
  into	
  the	
  linesize
instead	
  of	
  wrapping	
  
blog.tanelpoder.com 12
SQLPLUS	
  on	
  Unix	
  /	
  Linux	
  /	
  MacOS	
  X
• Command	
  line	
  history	
  using	
  RLWRAP
• Rlwrap	
  (readline	
  wrapper)	
  adds	
  GNU	
  readline capability	
  for	
  any	
  text	
  mode	
  
application
• BASH	
  shell	
  has	
  the	
  same	
  functionality	
   built	
  in
• Just	
  run	
  rlwrap	
  sqlplus instead	
  of	
  just	
  sqlplus	
  (or	
  any	
  other	
  utility)
• Rlwrap	
  gives	
  also	
  tab	
  completion!
• If	
  you	
  tell	
  it	
  to…
• It	
  can	
  use	
  a	
  wordfile (with	
  Oracle	
  keywords,	
  view,	
  column	
  names)
– rlwrap	
  -­‐f
• And	
  it	
  can	
  remember every	
  word	
  it	
  sees	
  on	
  the	
  screen!
– rlwrap	
  -­‐r
blog.tanelpoder.com 13
SQLPLUS	
  with	
  RLWRAP	
  – Keyboard	
  shortcuts
• Keyboard	
  shortcuts	
  (Emacs	
  mode):
• Up/down	
  arrows	
  – navigate	
  command	
  line	
  history
• CTRL+A	
  – go	
  to	
  beginning	
  of	
  line
• CTRL+E	
  – go	
  to	
  end	
  of	
  line
• CTRL+R	
  – search	
  command	
  history	
  backwards
• press	
  CTRL+R	
  again	
  to	
  move	
  on	
  to	
  next	
  match
• CTRL+S – search	
  command	
  history	
  forward
• go	
  “back”	
  from	
  too	
  many	
  CTRL+R’s
• ALT+F	
  /	
  ALT+B	
  – Move	
  Forward	
  /	
  Backward	
  by	
  one	
  word
• This	
  works	
  if	
  ALT	
  is	
  used	
  as	
  the	
  META	
  key	
  for	
  emacs	
  (check	
  your	
  terminal	
  
emulator	
  preferences)
• CTRL+K – erase	
  (kill)	
  line	
  from	
  current	
  position	
  to	
  the	
  end	
  of	
  line
• CTRL+X,	
  BACKSPACE	
  – erase	
  line	
  from	
  current	
  position	
  to	
  the	
  beginning
• CTRL+W or	
  ALT+BACKSPACE	
  – erase	
  one	
  word	
  at	
  a	
  time
• CTRL	
  +	
  _	
  -­‐ undo	
  command	
  edit!	
  (or	
  CTRL+X,	
  u)
blog.tanelpoder.com 14
Installing	
  RLWRAP
• Linux
• RHEL/OEL/CentOS	
  – Get	
  from	
  here:	
  http://fedoraproject.org/wiki/EPEL
• Ubuntu:	
  apt-­‐get	
  install	
  rlwrap
• MacOS	
  X	
  – Get	
  it	
  from	
  an	
  open	
  source	
  package	
  site	
  for	
  MacOS	
  
• MacPorts	
  -­‐ http://www.macports.org/
• DarwinPorts	
  -­‐ http://darwinports.com/
• Fink	
  -­‐ http://www.finkproject.org/
• Solaris	
  (SPARC	
  and	
  Intel)
• http://www.sunfreeware.com/
• On	
  HP-­‐UX	
  there’s	
  a	
  similar	
  built	
  in	
  command:	
  ied
• Should	
  also	
  work	
  on	
  Windows	
  with	
  Cygwin
• http://blog.tanelpoder.com/2010/05/07/flexible-­‐sqlplus-­‐command-­‐line-­‐history-­‐with-­‐rlwrap/
blog.tanelpoder.com 15
Setting	
  up	
  TPT	
  scripts	
  -­‐ 1
• Tanel’s	
  Performance	
  Tuning	
  scripts	
  :-­‐)
• 500+	
  scripts	
  which	
  I	
  use	
  “every	
  day”
• Most	
  scripts	
  do	
  one	
  thing	
  only	
  (such	
  as	
  show	
  parameter	
  values	
  etc)
• Also	
  latest	
  versions	
  of	
  the	
  more	
  sophisticated	
  scripts	
  (Snapper,	
   LatchProfX	
  etc)	
  
are	
  in	
  there
• Download	
  the	
  .zip	
  file	
  and	
  unzip	
  to	
  a	
  directory:
• http://blog.tanelpoder.com/files/
• Set	
  SQLPATH	
  variable	
  to	
  point	
  to	
  the	
  script	
  directory
• Unix:
• export	
  SQLPATH=$HOME/tpt
• Windows:
• set	
  SQLPATH=C:userstaneltpt
• Or	
  Right	
  click	
  on	
  My	
  Computer	
  -­‐>	
  Properties	
  -­‐>	
  Advanced	
  Settings	
  -­‐>	
  Environment	
  
Variables;	
  and	
  add	
  it	
  there
blog.tanelpoder.com 16
Setting	
  up	
  TPT	
  scripts	
  -­‐ 2
• Edit	
  init.sql	
  (init.sql	
  sets	
  up	
  various	
  sqlplus	
  variables)
• Uncomment	
  the	
  appropriate	
  parameters	
  in	
  the	
  SETUP	
  section
• It	
  tells	
  sqlplus	
  whether	
  you’re	
  running	
  it	
  on	
  Windows,	
  Unix	
  or	
  MacOS
-- SETUP SECTION –
-------------------------------------------------------------------
-- Uncomment the following lines if you run your sqlplus on Windows
-------------------------------------------------------------------
-- def SQLPATH=%SQLPATH%
-- def _start=start
-- def _delete=del
-- host doskey /exename=sqlplus.exe desc=set lines 80 sqlprompt ""$Tdescribe $*$Tset lines 299
sqlprompt "SQL> "
-------------------------------------------------------------------
• Edit	
  i.sql	
  (i.sql	
  is	
  the	
  “who	
  am	
  I”	
  script	
  which	
  also	
  sets	
  window	
  title)
• Uncomment	
  the	
  appropriate	
  title	
  command	
  in	
  the	
  setup	
  section	
  (EOF)
host title &_i_user@&_i_conn [sid=&mysid ser#=&_i_serial spid=&_i_spid …
-- host echo -ne "033]0;&_i_user@&_i_inst sid=&mysid,&_i_serial007"
blog.tanelpoder.com 17
Database	
  Performance	
  Tools
blog.tanelpoder.com 18
What	
  is	
  Oracle	
  Database	
  activity	
  about?
1. A	
  bunch	
  of	
  sessions
• You	
  need	
  a	
  session	
  for	
  doing	
  anything	
  in	
  Oracle
1. Applications	
  use	
  these	
  sessions	
  to	
  execute	
  (PL)SQL	
  cursors
• Any	
  database	
  call	
  requires	
  use	
  of	
  a	
  cursor
• Including	
  PL/SQL	
  stored	
  procedure	
  calls
1. The	
  cursor	
  execution	
  will	
  do	
  some	
  work
• During	
  the	
  execution,	
  the	
  processes	
  either:
• Work	
  on	
  CPU
– Or	
  at	
  least	
  want	
  to	
  work	
  on	
  CPU,	
  but	
  OS	
  scheduler	
  doesn’t	
  allow	
  them
• Do	
  not	
  Work	
  on	
  CPU
– They	
  have	
  been	
  put	
  to	
  sleep	
  by	
  OS	
  because	
  of	
  a	
  blocking	
  system	
  call
– They	
  have	
  gone	
  to	
  sleep	
  voluntarily,	
  waiting	
  for	
  some	
  event	
  completion
blog.tanelpoder.com 19
Sampling	
  session	
  activity	
  details	
  over	
  time	
  – Which	
  SQL?
SELECT	
  A
SELECT	
  X
idle SELECT	
  B UPDATE	
  C
Time
SID	
  10
SID	
  11
SID	
  12
idle idle
SELECT	
  Y INSERT	
  Zidle
SELECT	
  N SELECT	
  N UPDATE	
  KSELECT	
  N idle SELECT	
  N SEL
SELECT	
  A idle SELECT	
  B UPDATE	
  Cidle idle
SELECT	
  Aidle UPDATE	
  C idleidle COMMIT
SID	
  13
SID	
  14
SID	
  15
2 x	
  SELECT	
  A
1 x	
  SELECT	
  X
1	
  x	
  SELECT	
  Y
1 x	
  SELECT	
  N
Sample	
  session	
  
activity	
  over	
  
time	
  -­‐>	
  
(v$session)
3 x	
  SELECT	
  A
1 x	
  SELECT	
  X
1	
  x	
  SELECT	
  Y
1 x	
  SELECT	
  N
…
3 x	
  UPDATE	
  C
1 x	
  SELECT	
  X
1 x	
  SELECT	
  Y
Count the	
  number	
  of	
  
(active)	
  sessions	
  each	
  
sample,	
  
group	
  by	
  SQL	
  ID
blog.tanelpoder.com 20
Sampling	
  session	
  activity	
  – Working	
  or	
  Waiting?
SELECT	
  A
SELECT	
  X
idle SELECT	
  B UPDATE	
  C
Time
SID	
  10
SID	
  11
SID	
  12
idle idle
SELECT	
  Y INSERT	
  Zidle
SELECT	
  N SELECT	
  N UPDATE	
  KSELECT	
  N idle SELECT	
  N SEL
SELECT	
  A idle SELECT	
  B UPDATE	
  Cidle idle
SELECT	
  Aidle UPDATE	
  C idleidle COMMIT
SID	
  13
SID	
  14
SID	
  15
Commit
User	
  IO
CPU
Locks
Time
Session
count
blog.tanelpoder.com 21
Understanding	
  Database	
  Performance
• Performance	
  diagnosis	
  data	
  should	
  have	
  at	
  least	
  these	
  details
• Which	
  Sessions?
• Which	
  SQL?
• Work	
  or	
  wait?
• When?
• This	
  data	
  can	
  be	
  grouped	
  and	
  summarized	
  by	
  any	
  dimension	
  
depending	
  on	
  the	
  “performance	
  question”	
  asked!
• The	
  sampled	
  session	
  activity	
  dataset	
  is	
  like	
  a	
  FACT	
  table	
  in	
  DW’s
• The	
  existence of	
  a	
  session	
  sample	
  shows	
  that	
  the	
  session	
  was	
  active
• The	
  dimension	
  attributes	
  show	
  a	
  large	
  variety	
  of	
  extra	
  details
• Which	
  SQL	
  was	
  running,	
   Working	
   or	
  Waiting,	
  Which	
  event,	
  P1,P2..	
  	
  	
  	
  
-­‐>	
  v$session.sid
-­‐>	
  v$session.sql_id,	
  plsql_(sub)program_id
-­‐>	
  v$session.state,	
  v$session.event,	
  p1,p2,p3
-­‐>	
  Time	
  when	
  samples	
  were	
  taken
blog.tanelpoder.com 22
A	
  selective	
  list	
  of	
  dimensions	
  for	
  performance	
  analysis
SQL> desc v$session
Name Null? Type
------------------------------- -------- ----------------------------
1 SADDR RAW(8)
2 SID NUMBER
7 USERNAME VARCHAR2(30)
12 STATUS VARCHAR2(8)
21 PROGRAM VARCHAR2(48)
25 SQL_ID VARCHAR2(13)
37 PLSQL_OBJECT_ID NUMBER
38 PLSQL_SUBPROGRAM_ID NUMBER
39 MODULE VARCHAR2(48)
41 ACTION VARCHAR2(32)
45 ROW_WAIT_OBJ# NUMBER
61 CLIENT_IDENTIFIER VARCHAR2(64)
64 BLOCKING_SESSION NUMBER
70 EVENT VARCHAR2(64)
71 P1TEXT VARCHAR2(64)
72 P1 NUMBER
74 P2TEXT VARCHAR2(64)
75 P2 NUMBER
77 P3TEXT VARCHAR2(64)
78 P3 NUMBER
82 WAIT_CLASS VARCHAR2(64)
85 STATE VARCHAR2(19)
Status	
  is	
  ACTIVE	
  for	
  
sessions	
   running	
  a	
  
database	
  call	
  (background	
  
sessions	
   are	
  always	
  in	
  a	
  
call)
Easy	
  way	
  to	
  get	
  TOP	
  SQL,	
  
just	
  group	
  and	
  count	
  the	
  
samples	
  by	
  SQL_ID
From	
  10.2.0.3	
  onwards	
  
you	
  can	
  get	
  easy	
  TOP	
  
PL/SQL,	
  just	
  group	
  by	
  
PL_SQL_OBJECT_ID/SUBP
ROGRAM_ID	
  and	
  join	
  to	
  
same	
  columns	
  in	
  
DBA_PROCEDURES
Very	
  important	
  to	
  
remember:	
  If	
  wait	
  STATE	
  
!=	
  WAITING,	
  the	
  session	
  
is	
  NOT	
  WAITING (It’s	
  
working	
  on	
  CPU	
  or	
  tries	
  to	
  
get	
  onto	
  CPU)
If	
  you	
  want	
  the	
  TOP	
  
BLOCKER,	
  just	
  group	
  the	
  
samples	
  by	
  
blocking_session
For	
  TOP	
  WAITs,	
  just	
  
group	
  the	
  samples	
  by	
  
EVENT	
  where	
  
STATE=WAITING
blog.tanelpoder.com 23
Instance	
  Activity	
  overview
• What	
  is	
  the	
  database	
  (instance)	
  doing	
  right	
  now?
• @a.sql – Instance	
  Activity	
  SQL
• It	
  takes	
  one	
  ASH	
  style	
  sample	
  of	
  V$SESSION	
  and	
  shows	
  TOP	
  sql/wait
• I	
  wrote	
  it	
  to	
  show	
  how	
  easy	
  it	
  is	
  to	
  get	
  an	
  overview	
  of	
  instance	
  activity
• But	
  it’s	
  also	
  very	
  short	
  to	
  type	
  – I	
  actually	
  use	
  it	
  regularly	
  :-­‐)
• I	
  just	
  run	
  it	
  couple	
  of	
  times	
  in	
  a	
  row!
select
count(*)
, sql_id
, case state when 'WAITING' then 'WAITING' else 'ON CPU' end state
, case state when 'WAITING' then event else 'On CPU / runqueue' end event
from
v$session
where
status='ACTIVE’
and wait_class != 'Idle'
and sid != (select sid from v$mystat where rownum=1)
group by
sql_id
, case state when 'WAITING' then 'WAITING' else 'ON CPU' end
, case state when 'WAITING' then event else 'On CPU / runqueue' end
order by
count(*) desc
blog.tanelpoder.com 24
Snapper	
  -­‐ Overview
• Major	
  improvement	
  – ASH	
  style	
  session	
  activity	
  sampling!
• From	
  V$SESSION	
  /	
  V$SESSION_WAIT	
  (no	
  separate	
  licenses	
  needed)
• @snapper	
   ash <total_seconds>	
  	
  <sampling_interval>	
  	
  <SIDs>
• Still	
  works	
  all	
  the	
  way	
  from	
  Oracle	
  9.2	
  to	
  latest	
  Oracle	
  version
• Still	
  NO	
  installation/database	
  changes	
  are	
  needed!
• It’s	
  just	
  an	
  anonymous	
  PL/SQL	
  block,	
  parsed	
  on	
  the	
  fly…
• Look	
  into	
  the	
  code	
  to	
  see	
  some	
  cool	
  challenges	
  involved!!!
• Still	
  free	
  and	
  still	
  has	
  a	
  picture	
  of	
  a	
  real	
  Snapper	
  (fish)	
  in	
  it!	
  ;-­‐)
• …and	
  you	
  can	
  still	
  run	
  it	
  directly	
  from	
  web	
  if	
  you	
  feel	
  adventurous
• SQL>	
  @http://blog.tanelpoder.com/files/scripts/snapper.sql .....
blog.tanelpoder.com 25
Snapper	
  – Session	
  Activity	
  Sampling
• Example	
  1a:	
  Single	
  Session	
  troubleshooting:
SQL> @snapper ash,gather=n 5 1 32
-- Session Snapper v3.01 by Tanel Poder ( http://www.tanelpoder.com )
------------------------------- Sampled Session Activity Profile -----------------------
-- Act% | Wait Event | Wait Class | SQL_ID --
----------------------------------------------------------------------------------------
84% | db file scattered read | User I/O | g723q6scvr9h8
11% | ON CPU | ON CPU | g723q6scvr9h8
3% | db file sequential read | User I/O |
3% | ON CPU | ON CPU | c0j6cx9kzjf7g
SQL> @sqlid g723q6scvr9h8
HASH_VALUE CH# SQL_TEXT
---------- ---- -----------------------------------------------------------------------
431728136 0 /* SQL Analyze(0) */ select /*+ full(t) no_parallel(t)
no_parallel_index(t) dbms_stats cursor_sharing_exact use_weak_name_resl
dynamic_sampling(0) no_monitoring no_substrb_pad */to_char(count
("SYS_NC00009$")),to_char(substrb(dump(min("SYS_NC00009$"),16,0,32),
As	
  we	
  measure	
  a	
  single	
  
session,	
   then	
  the	
  max
Activity%	
  can	
  be	
  100%	
  (allow	
  
for	
  some	
  rounding	
  errors).
If	
  the	
  activity	
  is	
  measured	
  less	
  
than	
  100%,	
  it	
  means	
  that	
  the	
  
session	
   was	
  not	
  active	
  the	
  
remaining	
  time	
  (it	
  was	
  idle).
blog.tanelpoder.com 26
Snapper	
  – Session	
  Activity	
  and	
  Performance	
  Counters
• Example	
  1b:	
  Single	
  Session	
  troubleshooting	
  with	
  perf	
  counters
SQL> @snapper ash,stats 5 1 146
-- Session Snapper v3.01 by Tanel Poder ( http://www.tanelpoder.com )
--------------------------------------------------------------------------------------------------------
SID, USERNAME , TYPE, STATISTIC , DELTA, HDELTA/SEC, %TIME,
--------------------------------------------------------------------------------------------------------
146, SYS , STAT, session logical reads , 309970, 61.99k,
146, SYS , STAT, consistent gets , 309966, 61.99k,
146, SYS , STAT, consistent gets from cache , 309967, 61.99k,
146, SYS , STAT, consistent gets from cache (fastpath) , 309968, 61.99k,
146, SYS , STAT, calls to kcmgcs , 1660, 332,
146, SYS , STAT, no work - consistent read gets , 308388, 61.68k,
146, SYS , STAT, table scans (short tables) , 332, 66.4,
146, SYS , STAT, table scan rows gotten , 25006755, 5M,
146, SYS , STAT, table scan blocks gotten , 308415, 61.68k,
146, SYS , TIME, DB CPU , 3960000, 792ms, 79.2%
146, SYS , TIME, sql execute elapsed time , 4059947, 811.99ms, 81.2%
146, SYS , TIME, DB time , 4059947, 811.99ms, 81.2%
-- End of snap 1, end=2010-03-01 11:31:10, seconds=5
---------------------------------- Sampled Session Activity Profile ------------------------------
-- Act% | Wait Event | Wait Class | SQL_ID --
--------------------------------------------------------------------------------------------------
100% | ON CPU | ON CPU | 7vxd09448zcrx
blog.tanelpoder.com 27
Snapper	
  – Session	
  Activity	
  Sampling
• Example	
  2:	
  Whole	
  instance	
  overview
• You	
  can	
  write	
  any	
  query	
  instead	
  of	
  the	
  list	
  of	
  SIDs:
SQL> @snapper ash,stats,gather=n 30 1 "select sid from v$session"
-- Session Snapper v3.01 by Tanel Poder ( http://www.tanelpoder.com )
------------------- Sampled Session Activity Profile ----------------------------
-- Act% | Wait Event | Wait Class | SQL_ID --
---------------------------------------------------------------------------------
164% | read by other session | User I/O | 3jbwa65aqmkvm
99% | ON CPU | ON CPU | 7vxd09448zcrx
99% | ON CPU | ON CPU | bt4txmkqf7u89
98% | db file scattered read | User I/O | 3jbwa65aqmkvm
98% | direct path read | User I/O | 3jbwa65aqmkvm
61% | db file scattered read | User I/O | dw2zgaapax1sg
41% | control file parallel wri | System I/O
34% | db file async I/O submit | System I/O
27% | log file parallel write | System I/O
24% | db file sequential read | User I/O | 0bzhqhhj9mpaa
As	
  we	
  are	
  measuring	
  more	
  
than	
  one	
  session,	
   then	
  it’s	
  ok	
  
to	
  see	
  over	
  100%	
  of	
  activity.	
  
100%	
  stands	
  for	
  all	
  response	
  
time	
  of	
  a	
  single	
  session!
blog.tanelpoder.com 28
Snapper	
  – Performance	
  Counters
• Example	
  3a	
  -­‐ Find	
  top	
  REDO	
  geneators
SQL> @snapper ash,stats,gather=s,sinclude=redo_size 5 1 "select sid from v$session”
-- Session Snapper v3.01 by Tanel Poder ( http://www.tanelpoder.com )
------------------------------------------------------------------------------------------------
SID, USERNAME , TYPE, STATISTIC , DELTA, HDELTA/SEC,
------------------------------------------------------------------------------------------------
20, SYS , STAT, redo size , 13376076, 2.68M,
31, SYS , STAT, redo size , 18232, 3.65k,
143, SYS , STAT, redo size , 26796, 5.36k,
-- End of snap 1, end=2010-03-01 11:52:47, seconds=5
--------------------------------- Sampled Session Activity Profile -----------------------------
-- Act% | Wait Event | Wait Class | SQL_ID --
------------------------------------------------------------------------------------------------
69% | log file parallel write | System I/O
50% | ON CPU | ON CPU | crwqs334cz65x
27% | log buffer space | Configuration | crwqs334cz65x
12% | log buffer space | Configuration | f6f6wxb0st0yp
8% | ON CPU | ON CPU
8% | db file sequential read | User I/O | gvt8zu1k1tsff
blog.tanelpoder.com 29
Snapper	
  – Performance	
  Counters
• Example	
  3b:
• Find	
  top	
  IO	
  throughput	
  (MB)	
  users
SQL> @snapper stats,gather=s,sinclude=physical%bytes 5 1 "select sid from v$session"
-- Session Snapper v3.01 by Tanel Poder ( http://www.tanelpoder.com )
-----------------------------------------------------------------------------------------------
SID, USERNAME , TYPE, STATISTIC , DELTA, HDELTA/SEC,
-----------------------------------------------------------------------------------------------
6, (CKPT) , STAT, physical read total bytes , 16384, 3.28k,
6, (CKPT) , STAT, physical write total bytes , 32768, 6.55k,
6, (CKPT) , STAT, cell physical IO interconnect bytes , 49152, 9.83k,
20, SYS , STAT, physical read total bytes , 29360128, 5.87M,
20, SYS , STAT, cell physical IO interconnect bytes , 29360128, 5.87M,
20, SYS , STAT, physical read bytes , 29360128, 5.87M,
139, SOE , STAT, physical read total bytes , 1835008, 367k,
139, SOE , STAT, cell physical IO interconnect bytes , 1835008, 367k,
139, SOE , STAT, physical read bytes , 1835008, 367k,
-- End of snap 1, end=2010-03-01 12:00:35, seconds=5
• You	
  can	
  use	
  the	
  same	
  trick	
  with	
  any	
  performance	
  counter
• IOPS	
  – @snappiops
• CPU	
  – @snapcpu
blog.tanelpoder.com 30
Snapper	
  – Write	
  output	
  to	
  a	
  tracefile
• Example	
  4:
• SQL>	
  @snapper	
  ash,trace	
  30	
  999999	
  all
• Uses	
  DBMS_SYSTEM.KSDWRT()	
  for	
  writing	
  to	
  the	
  tracefile
• In	
  above	
  example,	
  a	
  Snapper	
  snapshot	
  is	
  written	
  to	
  a	
  tracefile	
  every	
  30	
  
seconds	
  (and	
  999999	
  times	
  in	
  a	
  row	
  unless	
  you	
  cancel)
• You	
  can	
  run	
  tail	
  -­‐f in	
  OS	
  for	
  realtime	
  Snapper	
  output	
  monitoring	
  	
  
• “all”	
  option	
  instead	
  of	
  a	
  SID	
  stands	
  for	
  all	
  sessions	
  in	
  the	
  database
• Note	
  that	
  gathering	
  V$SESSTAT	
  stats	
  for	
  a	
  lot	
  of	
  sessions frequently	
  is	
  CPU-­‐
expensive
• That’s	
  why	
  the	
  “all”	
  option	
  disables	
  V$SESSTAT	
  gathering	
  in	
  Snapper	
  v3.0	
  by	
  
default
• You	
  can	
  still	
  enable	
  it	
  by	
  adding	
  the	
  gather=…	
  option	
  to	
  collect	
  the	
  stats	
  you	
  want	
  
(read	
  the	
  script	
  header	
  for	
  documentation)
blog.tanelpoder.com 31
Snapper	
  – Advanced	
  Performance	
  Counters
• 1178 V$SESSTAT	
  performance	
  counters	
  in	
  Oracle	
  12.1.0.2!
• One	
  separate	
  set	
  for	
  each	
  session!
• Where	
  are	
  the	
  performance	
  counters	
  documented?
• Oracle	
  Database	
  Reference	
  – “Statistics	
  Descriptions”
• http://download.oracle.com/docs/cd/E14072_01/server.112/e10820/stats002.htm
• Note	
  that	
  the	
  above	
  URL	
  may	
  change	
  when	
  Oracle	
  releases	
  new	
  docs
blog.tanelpoder.com 32
Case	
  study	
  – Bad	
  index	
  build	
  performance	
  -­‐ 1
• An	
  index	
  build	
  was	
  taking	
  too	
  long	
  time	
  
• Function	
  based	
  index	
  build	
  on	
  a	
  busy	
  table
• Build,	
  not	
  re-­‐build
• Ran	
  with	
  ONLINE	
  option
• Executed	
  in	
  parallel
• The	
  parallel	
  slaves	
  were	
  doing	
  the	
  full	
  table	
  scan
• But	
  they	
  were	
  waiting	
  for	
  db	
  file	
  sequential	
  read for	
  over	
  80%	
  of	
  their	
  time
• Why	
  would	
  be	
  the	
  parallel	
  slaves	
  (which	
  did	
  a	
  full	
  table	
  scan)	
  be	
  doing	
  single	
  
block	
  reads	
  so	
  much?
blog.tanelpoder.com 33
Case	
  study	
  – Bad	
  index	
  build	
  performance	
  -­‐ 2
---------------------------------------------------------------------------------
-- SID, TYPE, STATISTIC , DELTA, HDELTA/SEC
---------------------------------------------------------------------------------
3768, STAT, session logical reads , 11522, 677.76
3768, STAT, physical read total IO requests , 4188, 246.35
3768, STAT, physical read total multi block requests, 877, 51.59
3768, STAT, physical read total bytes , 73924608, 4.35M
3768, STAT, consistent gets , 11559, 679.94
3768, STAT, consistent gets from cache , 11556, 679.76
3768, STAT, consistent gets - examination , 615, 36.18
3768, STAT, physical reads , 8988, 528.71
3768, STAT, physical reads cache , 8988, 528.71
3768, STAT, physical read IO requests , 4197, 246.88
3768, STAT, physical read bytes , 73637888, 4.33M
3768, STAT, physical reads cache prefetch , 4752, 279.53
3768, STAT, table scan rows gotten , 28079, 1.65k
3768, STAT, table scan blocks gotten , 7399, 435.24
3768, STAT, table fetch continued row , 3335, 196.18
3768, STAT, PX local messages sent , 192, 11.29
3768, WAIT, db file sequential read , 14393221, 846.66ms
3768, WAIT, db file scattered read , 1040277, 61.19ms
Total	
  single	
  block	
  IO	
  	
  requests	
  
=	
  
total	
  IO	
  requests	
  –
total	
  multi	
  block	
  requests	
  =	
  
246	
  – 51	
  =	
  195
physical	
  reads is	
  a	
  misleading	
  
name	
  – it’s	
  really	
  the	
  number	
  
of	
  blocks read	
  via	
  PIOs
single	
  block	
  reads	
  due	
  
chained	
  /	
  migrated	
  rows!
blog.tanelpoder.com 34
Snapper	
  -­‐ Summary
• Oracle	
  Session	
  Snapper	
  is	
  a	
  powerful	
  ad-­‐hoc	
  session-­‐level	
  
performance	
  diagnosis	
  tool	
  which	
  does	
  not	
  require	
  any	
  installation	
  
or	
  database	
  changes!
• Features:
• ASH	
  style	
  session	
  activity	
  sampling
• Reports	
  top	
  SQL,	
  top	
  Waits,	
  etc
• Very	
  useful	
  for	
  drilling	
  down	
  to	
  details	
  when	
  the	
  plain	
  wait	
  interface	
  and	
  SQL	
  
trace	
  do	
  not	
  give	
  enough	
  details
• Takes	
  snapshots	
  of	
  V$	
  views	
  (V$SESSTAT,	
  etc)	
  and	
  reports	
  deltas
• Report	
  all stats	
  of	
  some sessions	
  (like	
  the	
  advanced	
  diagnosis	
  example)
• Report	
  some stats	
  of	
  all sessions	
  (like	
  the	
  redo	
  or	
  IO	
  example)
• Flexible!

Tanel Poder - Scripts and Tools short

  • 1.
    blog.tanelpoder.com 1 Advanced  Oracle  Troubleshooting  preparation  session: Tanel’s  command  line  setup  &  tools Tanel  Põder http://blog.tanelpoder.com/seminar
  • 2.
    blog.tanelpoder.com 2 Intro:  About  me • Tanel  Põder Oracle  Database  Performance  geek Exadata  Performance  geek Linux  Performance  geek Hadoop  Performance  geek Expert  Oracle  Exadata   book (with  Kerry  Osborne  and   Randy  Johnson  of  Enkitec)
  • 3.
    blog.tanelpoder.com 3 All  scripts  and  tools  are  downloadable  and  free • Download  the  scripts  and  tools  from: • http://blog.tanelpoder.com/files/ • Updates  (since  I  delivered  this  class  in  2010): • I  have  shut  the  PlanViz website  down  so  ignore  some  slides • I  haven't  updated  the  PerfSheet v3  tool  since  2010 • Luca  Canali has  built  a  new  PerfSheet v4: • http://canali.web.cern.ch/canali/resources.htm
  • 4.
    blog.tanelpoder.com 4 Productivity  and  speed  of  action • Talking  to  Oracle • SQLPLUS  still  rocks  J • …especially  in  cases  where  you  need  to  deviate  from  the  usual  path • You’ll  have  a  (searchable)  history  of  output  of  your  last  commands • On  the  screen  or  in  a  spool/logfile… • You  can  run  your scripts • It’s  more  “pure”  than  GUI  development  tools  (which  try  to  be  clever,  do  extra   stuff,  don’t  fetch  all  data  etc)
  • 5.
    blog.tanelpoder.com 5 SQLPLUS  on  Windows • Configuration • Do  NOT  use  sqlplusw.exe  -­‐> • This  is  what  you  get  when running  sqlplus  from  Start menu • Luckily  sqlplusw.exe  is gone  since  11g! • Use  CMD.exe  and  run sqlplus.exe  -­‐> Much  better!!!
  • 6.
    blog.tanelpoder.com 6 • CMD.exe  needs  some  tweaking… • You  should  check  QuickEdit  mode,  Increase  Screen  Buffer  size  etc SQLPLUS  on  Windows  – CMD.exe  settings 1. 2. 3.
  • 7.
    blog.tanelpoder.com 7 Setting  SQLPLUS  Window  title  automatically • Set  cmd  window  title  to  include  database  session  info • SQL>  host  title  “test” • @i.sql  queries  session,  instance  info  from  V$  views  and  sets  title  accordingly • i.sql  is  automatically called  by  login.sql if  its  in  SQLPATH directory
  • 8.
    blog.tanelpoder.com 8 SQLPLUS  on  Windows • Command  line  history • Up/Down  arrow  -­‐ navigate  between  commands • CTRL+Left/Right  arrow  – jump  one  word  left/right • F7 – Display  a  popup  box  with  command  line  history • Use  up/down  arrows  to  navigate • ENTER  to  execute  command • Right  arrow  to  put  the  command  to  command  line  for  editing • First  letter  of  command  to  jump  between  commands  starting  with  that  letter   (case  sensitive) • F8 – Retrieve  command  from  history • Type  a  beginning   of  the  command  (for  example  “sel”)  and  press  F8 • This  will  jump  through  all  commands  starting  with  “sel”  (case  sensitive) • List  commands  from  history • SQL>  host  doskey  /history  /exename=sqlplus.exe • @h <text>  – history  script  showing  old  commands  with  text  int  them • @ha  – history  all
  • 9.
    blog.tanelpoder.com 9 SQLPLUS  on  Windows • Output  formatting • @HTML • Uses  SET  MARKUP  ON  HTML • Runs  the  query  in  SQLPLUS  buffer  and  spools  the  output  to  .HTML  file • Launches  that  .HTML  file  with  appropriate  OS  command • @XLS • Same  as  above,  but  spools  to  an  .XLS  file  and  launches  it  :-­‐) • @CSV • Uses  SET  COLSEP  “,”  and  spools  to  a  .CSV  file
  • 10.
    blog.tanelpoder.com 10 SQLPLUS  on  Unix  /  Linux  /  MacOS • Configuration • Setting  xterm  title • From  shell – echo  -­‐ne  "033]0;  your  text  here 007” • From  SQLPLUS – host  echo  -­‐ne  "033]0;  your  text  here 007” • From  vi – set  title – set  titlelen=20 • Decent  terminal  emulators • Putty  on  Windows  (small,  lightweight  and  works  well!) – http://www.chiark.greenend.org.uk/~sgtatham/putty/ • Terminator  on  MacOS,  Unix,  Linux  (and  apparently  Windows) – http://software.jessies.org/terminator/
  • 11.
    blog.tanelpoder.com 11 SQLPLUS  on  Unix  /  Linux  /  MacOS • Output  formatting • Same  as  on  Windows • Launching  the  HTML  file  is  done  with  a  different  command: • MacOS – open command  launches  the  program  associated  with  .HTML  extension • Unix/Linux  – I  just  run  firefox  by  default,  but  you  can  change  it  in  the  script • Terminal  with  left-­‐right  scrolling  capability? • I’m  using  Terminator • (Not  the  one  you  can  install  from  Ubuntu  repositories) • http://software.jessies.org/terminator/ • However,  it  needs  to  install  its  own  terminfo file  to  function  properly! – This  will  be  an  issue  if  you  need  to  SSH  into  a  lot  of  servers  in  an  enterprise – Good  for  demoing  stuff  though!  J • SQL>  set  truncate  on • Sqlplus  allows  you  to  truncate  the  columns  which  don’t  fit  into  the  linesize instead  of  wrapping  
  • 12.
    blog.tanelpoder.com 12 SQLPLUS  on  Unix  /  Linux  /  MacOS  X • Command  line  history  using  RLWRAP • Rlwrap  (readline  wrapper)  adds  GNU  readline capability  for  any  text  mode   application • BASH  shell  has  the  same  functionality   built  in • Just  run  rlwrap  sqlplus instead  of  just  sqlplus  (or  any  other  utility) • Rlwrap  gives  also  tab  completion! • If  you  tell  it  to… • It  can  use  a  wordfile (with  Oracle  keywords,  view,  column  names) – rlwrap  -­‐f • And  it  can  remember every  word  it  sees  on  the  screen! – rlwrap  -­‐r
  • 13.
    blog.tanelpoder.com 13 SQLPLUS  with  RLWRAP  – Keyboard  shortcuts • Keyboard  shortcuts  (Emacs  mode): • Up/down  arrows  – navigate  command  line  history • CTRL+A  – go  to  beginning  of  line • CTRL+E  – go  to  end  of  line • CTRL+R  – search  command  history  backwards • press  CTRL+R  again  to  move  on  to  next  match • CTRL+S – search  command  history  forward • go  “back”  from  too  many  CTRL+R’s • ALT+F  /  ALT+B  – Move  Forward  /  Backward  by  one  word • This  works  if  ALT  is  used  as  the  META  key  for  emacs  (check  your  terminal   emulator  preferences) • CTRL+K – erase  (kill)  line  from  current  position  to  the  end  of  line • CTRL+X,  BACKSPACE  – erase  line  from  current  position  to  the  beginning • CTRL+W or  ALT+BACKSPACE  – erase  one  word  at  a  time • CTRL  +  _  -­‐ undo  command  edit!  (or  CTRL+X,  u)
  • 14.
    blog.tanelpoder.com 14 Installing  RLWRAP •Linux • RHEL/OEL/CentOS  – Get  from  here:  http://fedoraproject.org/wiki/EPEL • Ubuntu:  apt-­‐get  install  rlwrap • MacOS  X  – Get  it  from  an  open  source  package  site  for  MacOS   • MacPorts  -­‐ http://www.macports.org/ • DarwinPorts  -­‐ http://darwinports.com/ • Fink  -­‐ http://www.finkproject.org/ • Solaris  (SPARC  and  Intel) • http://www.sunfreeware.com/ • On  HP-­‐UX  there’s  a  similar  built  in  command:  ied • Should  also  work  on  Windows  with  Cygwin • http://blog.tanelpoder.com/2010/05/07/flexible-­‐sqlplus-­‐command-­‐line-­‐history-­‐with-­‐rlwrap/
  • 15.
    blog.tanelpoder.com 15 Setting  up  TPT  scripts  -­‐ 1 • Tanel’s  Performance  Tuning  scripts  :-­‐) • 500+  scripts  which  I  use  “every  day” • Most  scripts  do  one  thing  only  (such  as  show  parameter  values  etc) • Also  latest  versions  of  the  more  sophisticated  scripts  (Snapper,   LatchProfX  etc)   are  in  there • Download  the  .zip  file  and  unzip  to  a  directory: • http://blog.tanelpoder.com/files/ • Set  SQLPATH  variable  to  point  to  the  script  directory • Unix: • export  SQLPATH=$HOME/tpt • Windows: • set  SQLPATH=C:userstaneltpt • Or  Right  click  on  My  Computer  -­‐>  Properties  -­‐>  Advanced  Settings  -­‐>  Environment   Variables;  and  add  it  there
  • 16.
    blog.tanelpoder.com 16 Setting  up  TPT  scripts  -­‐ 2 • Edit  init.sql  (init.sql  sets  up  various  sqlplus  variables) • Uncomment  the  appropriate  parameters  in  the  SETUP  section • It  tells  sqlplus  whether  you’re  running  it  on  Windows,  Unix  or  MacOS -- SETUP SECTION – ------------------------------------------------------------------- -- Uncomment the following lines if you run your sqlplus on Windows ------------------------------------------------------------------- -- def SQLPATH=%SQLPATH% -- def _start=start -- def _delete=del -- host doskey /exename=sqlplus.exe desc=set lines 80 sqlprompt ""$Tdescribe $*$Tset lines 299 sqlprompt "SQL> " ------------------------------------------------------------------- • Edit  i.sql  (i.sql  is  the  “who  am  I”  script  which  also  sets  window  title) • Uncomment  the  appropriate  title  command  in  the  setup  section  (EOF) host title &_i_user@&_i_conn [sid=&mysid ser#=&_i_serial spid=&_i_spid … -- host echo -ne "033]0;&_i_user@&_i_inst sid=&mysid,&_i_serial007"
  • 17.
  • 18.
    blog.tanelpoder.com 18 What  is  Oracle  Database  activity  about? 1. A  bunch  of  sessions • You  need  a  session  for  doing  anything  in  Oracle 1. Applications  use  these  sessions  to  execute  (PL)SQL  cursors • Any  database  call  requires  use  of  a  cursor • Including  PL/SQL  stored  procedure  calls 1. The  cursor  execution  will  do  some  work • During  the  execution,  the  processes  either: • Work  on  CPU – Or  at  least  want  to  work  on  CPU,  but  OS  scheduler  doesn’t  allow  them • Do  not  Work  on  CPU – They  have  been  put  to  sleep  by  OS  because  of  a  blocking  system  call – They  have  gone  to  sleep  voluntarily,  waiting  for  some  event  completion
  • 19.
    blog.tanelpoder.com 19 Sampling  session  activity  details  over  time  – Which  SQL? SELECT  A SELECT  X idle SELECT  B UPDATE  C Time SID  10 SID  11 SID  12 idle idle SELECT  Y INSERT  Zidle SELECT  N SELECT  N UPDATE  KSELECT  N idle SELECT  N SEL SELECT  A idle SELECT  B UPDATE  Cidle idle SELECT  Aidle UPDATE  C idleidle COMMIT SID  13 SID  14 SID  15 2 x  SELECT  A 1 x  SELECT  X 1  x  SELECT  Y 1 x  SELECT  N Sample  session   activity  over   time  -­‐>   (v$session) 3 x  SELECT  A 1 x  SELECT  X 1  x  SELECT  Y 1 x  SELECT  N … 3 x  UPDATE  C 1 x  SELECT  X 1 x  SELECT  Y Count the  number  of   (active)  sessions  each   sample,   group  by  SQL  ID
  • 20.
    blog.tanelpoder.com 20 Sampling  session  activity  – Working  or  Waiting? SELECT  A SELECT  X idle SELECT  B UPDATE  C Time SID  10 SID  11 SID  12 idle idle SELECT  Y INSERT  Zidle SELECT  N SELECT  N UPDATE  KSELECT  N idle SELECT  N SEL SELECT  A idle SELECT  B UPDATE  Cidle idle SELECT  Aidle UPDATE  C idleidle COMMIT SID  13 SID  14 SID  15 Commit User  IO CPU Locks Time Session count
  • 21.
    blog.tanelpoder.com 21 Understanding  Database  Performance • Performance  diagnosis  data  should  have  at  least  these  details • Which  Sessions? • Which  SQL? • Work  or  wait? • When? • This  data  can  be  grouped  and  summarized  by  any  dimension   depending  on  the  “performance  question”  asked! • The  sampled  session  activity  dataset  is  like  a  FACT  table  in  DW’s • The  existence of  a  session  sample  shows  that  the  session  was  active • The  dimension  attributes  show  a  large  variety  of  extra  details • Which  SQL  was  running,   Working   or  Waiting,  Which  event,  P1,P2..         -­‐>  v$session.sid -­‐>  v$session.sql_id,  plsql_(sub)program_id -­‐>  v$session.state,  v$session.event,  p1,p2,p3 -­‐>  Time  when  samples  were  taken
  • 22.
    blog.tanelpoder.com 22 A  selective  list  of  dimensions  for  performance  analysis SQL> desc v$session Name Null? Type ------------------------------- -------- ---------------------------- 1 SADDR RAW(8) 2 SID NUMBER 7 USERNAME VARCHAR2(30) 12 STATUS VARCHAR2(8) 21 PROGRAM VARCHAR2(48) 25 SQL_ID VARCHAR2(13) 37 PLSQL_OBJECT_ID NUMBER 38 PLSQL_SUBPROGRAM_ID NUMBER 39 MODULE VARCHAR2(48) 41 ACTION VARCHAR2(32) 45 ROW_WAIT_OBJ# NUMBER 61 CLIENT_IDENTIFIER VARCHAR2(64) 64 BLOCKING_SESSION NUMBER 70 EVENT VARCHAR2(64) 71 P1TEXT VARCHAR2(64) 72 P1 NUMBER 74 P2TEXT VARCHAR2(64) 75 P2 NUMBER 77 P3TEXT VARCHAR2(64) 78 P3 NUMBER 82 WAIT_CLASS VARCHAR2(64) 85 STATE VARCHAR2(19) Status  is  ACTIVE  for   sessions   running  a   database  call  (background   sessions   are  always  in  a   call) Easy  way  to  get  TOP  SQL,   just  group  and  count  the   samples  by  SQL_ID From  10.2.0.3  onwards   you  can  get  easy  TOP   PL/SQL,  just  group  by   PL_SQL_OBJECT_ID/SUBP ROGRAM_ID  and  join  to   same  columns  in   DBA_PROCEDURES Very  important  to   remember:  If  wait  STATE   !=  WAITING,  the  session   is  NOT  WAITING (It’s   working  on  CPU  or  tries  to   get  onto  CPU) If  you  want  the  TOP   BLOCKER,  just  group  the   samples  by   blocking_session For  TOP  WAITs,  just   group  the  samples  by   EVENT  where   STATE=WAITING
  • 23.
    blog.tanelpoder.com 23 Instance  Activity  overview • What  is  the  database  (instance)  doing  right  now? • @a.sql – Instance  Activity  SQL • It  takes  one  ASH  style  sample  of  V$SESSION  and  shows  TOP  sql/wait • I  wrote  it  to  show  how  easy  it  is  to  get  an  overview  of  instance  activity • But  it’s  also  very  short  to  type  – I  actually  use  it  regularly  :-­‐) • I  just  run  it  couple  of  times  in  a  row! select count(*) , sql_id , case state when 'WAITING' then 'WAITING' else 'ON CPU' end state , case state when 'WAITING' then event else 'On CPU / runqueue' end event from v$session where status='ACTIVE’ and wait_class != 'Idle' and sid != (select sid from v$mystat where rownum=1) group by sql_id , case state when 'WAITING' then 'WAITING' else 'ON CPU' end , case state when 'WAITING' then event else 'On CPU / runqueue' end order by count(*) desc
  • 24.
    blog.tanelpoder.com 24 Snapper  -­‐Overview • Major  improvement  – ASH  style  session  activity  sampling! • From  V$SESSION  /  V$SESSION_WAIT  (no  separate  licenses  needed) • @snapper   ash <total_seconds>    <sampling_interval>    <SIDs> • Still  works  all  the  way  from  Oracle  9.2  to  latest  Oracle  version • Still  NO  installation/database  changes  are  needed! • It’s  just  an  anonymous  PL/SQL  block,  parsed  on  the  fly… • Look  into  the  code  to  see  some  cool  challenges  involved!!! • Still  free  and  still  has  a  picture  of  a  real  Snapper  (fish)  in  it!  ;-­‐) • …and  you  can  still  run  it  directly  from  web  if  you  feel  adventurous • SQL>  @http://blog.tanelpoder.com/files/scripts/snapper.sql .....
  • 25.
    blog.tanelpoder.com 25 Snapper  –Session  Activity  Sampling • Example  1a:  Single  Session  troubleshooting: SQL> @snapper ash,gather=n 5 1 32 -- Session Snapper v3.01 by Tanel Poder ( http://www.tanelpoder.com ) ------------------------------- Sampled Session Activity Profile ----------------------- -- Act% | Wait Event | Wait Class | SQL_ID -- ---------------------------------------------------------------------------------------- 84% | db file scattered read | User I/O | g723q6scvr9h8 11% | ON CPU | ON CPU | g723q6scvr9h8 3% | db file sequential read | User I/O | 3% | ON CPU | ON CPU | c0j6cx9kzjf7g SQL> @sqlid g723q6scvr9h8 HASH_VALUE CH# SQL_TEXT ---------- ---- ----------------------------------------------------------------------- 431728136 0 /* SQL Analyze(0) */ select /*+ full(t) no_parallel(t) no_parallel_index(t) dbms_stats cursor_sharing_exact use_weak_name_resl dynamic_sampling(0) no_monitoring no_substrb_pad */to_char(count ("SYS_NC00009$")),to_char(substrb(dump(min("SYS_NC00009$"),16,0,32), As  we  measure  a  single   session,   then  the  max Activity%  can  be  100%  (allow   for  some  rounding  errors). If  the  activity  is  measured  less   than  100%,  it  means  that  the   session   was  not  active  the   remaining  time  (it  was  idle).
  • 26.
    blog.tanelpoder.com 26 Snapper  –Session  Activity  and  Performance  Counters • Example  1b:  Single  Session  troubleshooting  with  perf  counters SQL> @snapper ash,stats 5 1 146 -- Session Snapper v3.01 by Tanel Poder ( http://www.tanelpoder.com ) -------------------------------------------------------------------------------------------------------- SID, USERNAME , TYPE, STATISTIC , DELTA, HDELTA/SEC, %TIME, -------------------------------------------------------------------------------------------------------- 146, SYS , STAT, session logical reads , 309970, 61.99k, 146, SYS , STAT, consistent gets , 309966, 61.99k, 146, SYS , STAT, consistent gets from cache , 309967, 61.99k, 146, SYS , STAT, consistent gets from cache (fastpath) , 309968, 61.99k, 146, SYS , STAT, calls to kcmgcs , 1660, 332, 146, SYS , STAT, no work - consistent read gets , 308388, 61.68k, 146, SYS , STAT, table scans (short tables) , 332, 66.4, 146, SYS , STAT, table scan rows gotten , 25006755, 5M, 146, SYS , STAT, table scan blocks gotten , 308415, 61.68k, 146, SYS , TIME, DB CPU , 3960000, 792ms, 79.2% 146, SYS , TIME, sql execute elapsed time , 4059947, 811.99ms, 81.2% 146, SYS , TIME, DB time , 4059947, 811.99ms, 81.2% -- End of snap 1, end=2010-03-01 11:31:10, seconds=5 ---------------------------------- Sampled Session Activity Profile ------------------------------ -- Act% | Wait Event | Wait Class | SQL_ID -- -------------------------------------------------------------------------------------------------- 100% | ON CPU | ON CPU | 7vxd09448zcrx
  • 27.
    blog.tanelpoder.com 27 Snapper  –Session  Activity  Sampling • Example  2:  Whole  instance  overview • You  can  write  any  query  instead  of  the  list  of  SIDs: SQL> @snapper ash,stats,gather=n 30 1 "select sid from v$session" -- Session Snapper v3.01 by Tanel Poder ( http://www.tanelpoder.com ) ------------------- Sampled Session Activity Profile ---------------------------- -- Act% | Wait Event | Wait Class | SQL_ID -- --------------------------------------------------------------------------------- 164% | read by other session | User I/O | 3jbwa65aqmkvm 99% | ON CPU | ON CPU | 7vxd09448zcrx 99% | ON CPU | ON CPU | bt4txmkqf7u89 98% | db file scattered read | User I/O | 3jbwa65aqmkvm 98% | direct path read | User I/O | 3jbwa65aqmkvm 61% | db file scattered read | User I/O | dw2zgaapax1sg 41% | control file parallel wri | System I/O 34% | db file async I/O submit | System I/O 27% | log file parallel write | System I/O 24% | db file sequential read | User I/O | 0bzhqhhj9mpaa As  we  are  measuring  more   than  one  session,   then  it’s  ok   to  see  over  100%  of  activity.   100%  stands  for  all  response   time  of  a  single  session!
  • 28.
    blog.tanelpoder.com 28 Snapper  –Performance  Counters • Example  3a  -­‐ Find  top  REDO  geneators SQL> @snapper ash,stats,gather=s,sinclude=redo_size 5 1 "select sid from v$session” -- Session Snapper v3.01 by Tanel Poder ( http://www.tanelpoder.com ) ------------------------------------------------------------------------------------------------ SID, USERNAME , TYPE, STATISTIC , DELTA, HDELTA/SEC, ------------------------------------------------------------------------------------------------ 20, SYS , STAT, redo size , 13376076, 2.68M, 31, SYS , STAT, redo size , 18232, 3.65k, 143, SYS , STAT, redo size , 26796, 5.36k, -- End of snap 1, end=2010-03-01 11:52:47, seconds=5 --------------------------------- Sampled Session Activity Profile ----------------------------- -- Act% | Wait Event | Wait Class | SQL_ID -- ------------------------------------------------------------------------------------------------ 69% | log file parallel write | System I/O 50% | ON CPU | ON CPU | crwqs334cz65x 27% | log buffer space | Configuration | crwqs334cz65x 12% | log buffer space | Configuration | f6f6wxb0st0yp 8% | ON CPU | ON CPU 8% | db file sequential read | User I/O | gvt8zu1k1tsff
  • 29.
    blog.tanelpoder.com 29 Snapper  –Performance  Counters • Example  3b: • Find  top  IO  throughput  (MB)  users SQL> @snapper stats,gather=s,sinclude=physical%bytes 5 1 "select sid from v$session" -- Session Snapper v3.01 by Tanel Poder ( http://www.tanelpoder.com ) ----------------------------------------------------------------------------------------------- SID, USERNAME , TYPE, STATISTIC , DELTA, HDELTA/SEC, ----------------------------------------------------------------------------------------------- 6, (CKPT) , STAT, physical read total bytes , 16384, 3.28k, 6, (CKPT) , STAT, physical write total bytes , 32768, 6.55k, 6, (CKPT) , STAT, cell physical IO interconnect bytes , 49152, 9.83k, 20, SYS , STAT, physical read total bytes , 29360128, 5.87M, 20, SYS , STAT, cell physical IO interconnect bytes , 29360128, 5.87M, 20, SYS , STAT, physical read bytes , 29360128, 5.87M, 139, SOE , STAT, physical read total bytes , 1835008, 367k, 139, SOE , STAT, cell physical IO interconnect bytes , 1835008, 367k, 139, SOE , STAT, physical read bytes , 1835008, 367k, -- End of snap 1, end=2010-03-01 12:00:35, seconds=5 • You  can  use  the  same  trick  with  any  performance  counter • IOPS  – @snappiops • CPU  – @snapcpu
  • 30.
    blog.tanelpoder.com 30 Snapper  –Write  output  to  a  tracefile • Example  4: • SQL>  @snapper  ash,trace  30  999999  all • Uses  DBMS_SYSTEM.KSDWRT()  for  writing  to  the  tracefile • In  above  example,  a  Snapper  snapshot  is  written  to  a  tracefile  every  30   seconds  (and  999999  times  in  a  row  unless  you  cancel) • You  can  run  tail  -­‐f in  OS  for  realtime  Snapper  output  monitoring     • “all”  option  instead  of  a  SID  stands  for  all  sessions  in  the  database • Note  that  gathering  V$SESSTAT  stats  for  a  lot  of  sessions frequently  is  CPU-­‐ expensive • That’s  why  the  “all”  option  disables  V$SESSTAT  gathering  in  Snapper  v3.0  by   default • You  can  still  enable  it  by  adding  the  gather=…  option  to  collect  the  stats  you  want   (read  the  script  header  for  documentation)
  • 31.
    blog.tanelpoder.com 31 Snapper  –Advanced  Performance  Counters • 1178 V$SESSTAT  performance  counters  in  Oracle  12.1.0.2! • One  separate  set  for  each  session! • Where  are  the  performance  counters  documented? • Oracle  Database  Reference  – “Statistics  Descriptions” • http://download.oracle.com/docs/cd/E14072_01/server.112/e10820/stats002.htm • Note  that  the  above  URL  may  change  when  Oracle  releases  new  docs
  • 32.
    blog.tanelpoder.com 32 Case  study  – Bad  index  build  performance  -­‐ 1 • An  index  build  was  taking  too  long  time   • Function  based  index  build  on  a  busy  table • Build,  not  re-­‐build • Ran  with  ONLINE  option • Executed  in  parallel • The  parallel  slaves  were  doing  the  full  table  scan • But  they  were  waiting  for  db  file  sequential  read for  over  80%  of  their  time • Why  would  be  the  parallel  slaves  (which  did  a  full  table  scan)  be  doing  single   block  reads  so  much?
  • 33.
    blog.tanelpoder.com 33 Case  study  – Bad  index  build  performance  -­‐ 2 --------------------------------------------------------------------------------- -- SID, TYPE, STATISTIC , DELTA, HDELTA/SEC --------------------------------------------------------------------------------- 3768, STAT, session logical reads , 11522, 677.76 3768, STAT, physical read total IO requests , 4188, 246.35 3768, STAT, physical read total multi block requests, 877, 51.59 3768, STAT, physical read total bytes , 73924608, 4.35M 3768, STAT, consistent gets , 11559, 679.94 3768, STAT, consistent gets from cache , 11556, 679.76 3768, STAT, consistent gets - examination , 615, 36.18 3768, STAT, physical reads , 8988, 528.71 3768, STAT, physical reads cache , 8988, 528.71 3768, STAT, physical read IO requests , 4197, 246.88 3768, STAT, physical read bytes , 73637888, 4.33M 3768, STAT, physical reads cache prefetch , 4752, 279.53 3768, STAT, table scan rows gotten , 28079, 1.65k 3768, STAT, table scan blocks gotten , 7399, 435.24 3768, STAT, table fetch continued row , 3335, 196.18 3768, STAT, PX local messages sent , 192, 11.29 3768, WAIT, db file sequential read , 14393221, 846.66ms 3768, WAIT, db file scattered read , 1040277, 61.19ms Total  single  block  IO    requests   =   total  IO  requests  – total  multi  block  requests  =   246  – 51  =  195 physical  reads is  a  misleading   name  – it’s  really  the  number   of  blocks read  via  PIOs single  block  reads  due   chained  /  migrated  rows!
  • 34.
    blog.tanelpoder.com 34 Snapper  -­‐Summary • Oracle  Session  Snapper  is  a  powerful  ad-­‐hoc  session-­‐level   performance  diagnosis  tool  which  does  not  require  any  installation   or  database  changes! • Features: • ASH  style  session  activity  sampling • Reports  top  SQL,  top  Waits,  etc • Very  useful  for  drilling  down  to  details  when  the  plain  wait  interface  and  SQL   trace  do  not  give  enough  details • Takes  snapshots  of  V$  views  (V$SESSTAT,  etc)  and  reports  deltas • Report  all stats  of  some sessions  (like  the  advanced  diagnosis  example) • Report  some stats  of  all sessions  (like  the  redo  or  IO  example) • Flexible!