oracle中Frequent generate a lot of cdmp* directories contain *bucket trace in bdump问题分析
最近有套库突然目录使用告警，发现TRACE 目录使用较高，发现在bdump 目录中每分钟都生成一个cdmp*目录，且每个目录中含大量的文件，这是一套oracle 11.2.0.3 rac on hpux 11.31的数据库， 如果没有在系统中启用dump event ，那就可能是bug ，这里简单的记录。
oracle@anbob2:/oracle/app/oracle/diag/rdbms/weejar/weejar2/trace> ls
alert_weejar2.log cdmp_20160408221002 cdmp_20160409050508 cdmp_20160427102501 cdmp_20160503133006 cdmp_20160506124003
cdmp_20160407113747 cdmp_20160408221101 cdmp_20160409050609 cdmp_20160427102602 cdmp_20160503133105 cdmp_20160506124103
cdmp_20160407113801 cdmp_20160408221202 cdmp_20160409050708 cdmp_20160427103304 cdmp_20160503133205 cdmp_20160506124202
...
oracle@anbob2:/oracle/app/oracle/diag/rdbms/weejar/weejar2/trace/cdmp_20160509173502> ls
weejar2_acms_15550_bucket.trc weejar2_ora_16823_bucket.trc weejar2_ora_18001_bucket.trc
weejar2_ora_19458_bucket.trc weejar2_ora_26892_bucket.trc weejar2_ora_29042_bucket.trc
weejar2_acms_15550_bucket.trm weejar2_ora_16823_bucket.trm weejar2_ora_18001_bucket.trm
weejar2_ora_19458_bucket.trm weejar2_ora_26892_bucket.trm weejar2_ora_29042_bucket.trm
...
oracle@anbob2:/oracle/app/oracle/diag/rdbms/weejar/weejar2/trace/cdmp_20160509173502> cat weejar2_ora_8317_bucket.trc
Trace file /oracle/app/oracle/diag/rdbms/weejar/weejar2/trace/cdmp_20160509173502/weejar2_ora_8317_bucket.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /oracle/app/oracle/product/11.2.0.3/dbhome_1
System name: HP-UX
Node name: anbob2
Release: B.11.31
Version: U
Machine: ia64
Instance name: weejar2
Redo thread mounted by this instance: 2
Oracle process number: 531
Unix process pid: 8317, image: oracle@anbob2
*** 2016-05-09 17:35:07.273
*** SESSION ID:(832.4201) 2016-05-09 17:35:07.273
*** 2016-05-09 17:35:07.273
Process diagnostic dump for oracle@anbob2, OS id=8317,
pid: 531, proc_ser: 66, sid: 832, sess_ser: 4201
-------------------------------------------------------------------------------
current sql:
client details:
O/S info: user: weblogic, term: unknown, ospid: 1234
machine: qmweb86 program: JDBC Thin Client
application name: JDBC Thin Client, hash value=2546894660
Current Wait Stack:
0: waiting for 'SQL*Net message from client'
driver id=0x74637000, #bytes=0x1, =0x0
wait_id=55 seq_num=56 snap_id=1
wait times: snap=4.499210 sec, exc=4.499210 sec, total=4.499210 sec
wait times: max=infinite, heur=4.499210 sec
wait counts: calls=0 os=0
in_wait=1 iflags=0x1a0
Wait State:
fixed_waits=0 flags=0x22 boundary=0x0000000000000000/-1
Session Wait History:
elapsed time of 0.000031 sec since current wait
0: waited for 'SQL*Net message to client'
driver id=0x74637000, #bytes=0x1, =0x0
wait_id=54 seq_num=55 snap_id=1
wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000012 sec of elapsed time
1: waited for 'SQL*Net message from client'
driver id=0x74637000, #bytes=0x1, =0x0
wait_id=53 seq_num=54 snap_id=1
wait times: snap=0.001062 sec, exc=0.001062 sec, total=0.001062 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000027 sec of elapsed time
2: waited for 'SQL*Net message to client'
driver id=0x74637000, #bytes=0x1, =0x0
wait_id=52 seq_num=53 snap_id=1
wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000016 sec of elapsed time
3: waited for 'SQL*Net message from client'
driver id=0x74637000, #bytes=0x1, =0x0
wait_id=51 seq_num=52 snap_id=1
wait times: snap=0.001342 sec, exc=0.001342 sec, total=0.001342 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000526 sec of elapsed time
4: waited for 'SQL*Net message to client'
driver id=0x74637000, #bytes=0x1, =0x0
wait_id=50 seq_num=51 snap_id=1
wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000173 sec of elapsed time
5: waited for 'SQL*Net message from client'
driver id=0x74637000, #bytes=0x1, =0x0
wait_id=49 seq_num=50 snap_id=1
wait times: snap=0.000847 sec, exc=0.000847 sec, total=0.000847 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000003 sec of elapsed time
6: waited for 'SQL*Net message to client'
driver id=0x74637000, #bytes=0x1, =0x0
wait_id=48 seq_num=49 snap_id=1
wait times: snap=0.000002 sec, exc=0.000002 sec, total=0.000002 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000041 sec of elapsed time
7: waited for 'SQL*Net message from client'
driver id=0x74637000, #bytes=0x1, =0x0
wait_id=47 seq_num=48 snap_id=1
wait times: snap=0.029814 sec, exc=0.029814 sec, total=0.029814 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000076 sec of elapsed time
8: waited for 'SQL*Net message to client'
driver id=0x74637000, #bytes=0x1, =0x0
wait_id=46 seq_num=47 snap_id=1
wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000153 sec of elapsed time
9: waited for 'SQL*Net message from client'
driver id=0x74637000, #bytes=0x1, =0x0
wait_id=45 seq_num=46 snap_id=1
wait times: snap=0.002608 sec, exc=0.002608 sec, total=0.002608 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000002 sec of elapsed time
Sampled Session History of session 832 serial 4201
---------------------------------------------------
The sampled session history is constructed by sampling
the target session every 1 second. The sampling process
captures at each sample if the session is in a non-idle wait,
an idle wait, or not in a wait. If the session is in a
non-idle wait then one interval is shown for all the samples
the session was in the same non-idle wait. If the
session is in an idle wait or not in a wait for
consecutive samples then one interval is shown for all
the consecutive samples. Though we display these consecutive
samples in a single interval the session may NOT be continuously
idle or not in a wait (the sampling process does not know).
The history is displayed in reverse chronological order.
sample interval: 1 sec, max history 120 sec
---------------------------------------------------
[121 samples, 17:33:07 - 17:35:07]
idle wait at each sample
-------------------------------------------------------------------------------
Process diagnostic dump actual duration=0.005000 sec
(max dump time=30.000000 sec)
*** 2016-05-09 17:35:07.278
-------------------------------------------------------------------------------
Trace Bucket Dump Begin: default bucket for process 531 (osid: 8317)
TIME(*=approx):SEQ:COMPONENT:FILE@LINE:FUNCTION:SECT/DUMP: [EVENT#:PID:SID] DATA
-------------------------------------------------------------------------------
2016-05-09 17:32:17.785270 :GSIPC:kjctc.c@321:kjctccr(): GSIPC:CONN: Disconnect from inst 1, receiver 2
2016-05-09 17:32:17.785271 :89F5D5D1:db_trace:ksxp.c@4368:ksxpmsgcncl(): [10401:531:0] KSXPMSGCNCL: client 2 tid inst 1 ptid 257 flags 0x100
2016-05-09 17:32:17.785271 :GSIPC:kjctc.c@321:kjctccr(): GSIPC:CONN: Disconnect from inst 1, receiver 3
2016-05-09 17:32:17.785272 :89F5D5D2:db_trace:ksxp.c@4368:ksxpmsgcncl(): [10401:531:0] KSXPMSGCNCL: client 2 tid inst 1 ptid 257 flags 0x100
2016-05-09 17:32:17.785569 :GSIPC:kjcc.c@626:kjccdel(): GSIPC:CLNT: deleted client comm layer structures 2
2016-05-09 17:32:17.785576 :89F5D5D4:db_trace:kss.c@1414:kssdch(): [10809:531:0] kssdch(0xc0000010e1d0bdc8 = process, 2) 2 0 exit
查看系统event dump
SQL> oradebug setmypid
Statement processed.
SQL> oradebug eventdump system
10949 trace name context forever,level 1
28401 trace name context forever,level 1
SQL> oradebug eventdump process
28401 trace name context forever,level 1
10949 trace name context forever,level 1
SQL> oradebug eventdump session
28401 trace name context forever,level 1
10949 trace name context forever,level 1
Mos中查了下定位了一个bug,Bug 17062524 Diagnostic enhancement to write default trace bucket to trace file for database
The default RDBMS tracing bucket is not dumped to the trace file on a
failure – instead it is dumped to a cdmp* file.
Workaround
Find the cdmp* file containing the default tracing bucket’s contents. (This
file is anot included by default when packaging an incident so has to be
collected manually)
fixed:
12.2 (Future Release)
12.1.0.2 (Server Patch Set)
