Running db2 takeover hadr on db on a standby machine followed by
db2 stop force before the takeover command completes
successfully can cause memory to be freed twice. This would lead
to a panic in the system. This APAR can only occur on DB2 9.5
FP7.
2010-11-25-17.42.00.931548-300 I68908668E377 LEVEL:
Warning
PID : 21039 TID : 47382024677696PROC :
db2sysc
INSTANCE: svtdbm NODE : 000
EDUID : 310 EDUNAME: db2hadrp (SAKE)
FUNCTION: DB2 UDB, High Availability Disaster Recovery,
hdrSDoTakeover, probe:47003
MESSAGE : Info: Standby has completed takeover (now primary).
2010-11-25-17.42.00.944158-300 I68909046E342 LEVEL:
Warning
PID : 21039 TID : 47382024677696PROC :
db2sysc
INSTANCE: svtdbm NODE : 000
EDUID : 310 EDUNAME: db2hadrp (SAKE)
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrEduS,
probe:20342
MESSAGE : Info: Standby Finished.
2010-11-25-17.42.00.945189-300 I68909389E553 LEVEL:
Warning
PID : 21039 TID : 47381592664384PROC :
db2sysc
INSTANCE: svtdbm NODE : 000 DB : SAKE
APPHDL : 0-28727 APPID: *LOCAL.svtdbm.101126110514
AUTHID : SVTDBM
EDUID : 630 EDUNAME: db2agent (SAKE)
FUNCTION: DB2 UDB, base sys utilities,
sqeAgentServices::RequestAgent, probe:12
MESSAGE : Coordinator request rejected.
DATA #1 : Hexdump, 4 bytes
0x00002B17E2FE9400 : 0700 0000
....
2010-11-25-17.42.00.948897-300 I68909943E461 LEVEL: Error
PID : 21039 TID : 47381592664384PROC :
db2sysc
INSTANCE: svtdbm NODE : 000 DB : SAKE
APPHDL : 0-28727 APPID: *LOCAL.svtdbm.101126110514
AUTHID : SVTDBM
EDUID : 630 EDUNAME: db2agent (SAKE)
FUNCTION: DB2 UDB, base sys utilities,
sqeAgentServices::RequestAgent, probe:70
MESSAGE : Agent not allocated, sqlcode = -6036
2010-11-25-17.42.00.950313-300 E68910405E495 LEVEL:
Warning
PID : 21039 TID : 47381592664384PROC :
db2sysc
INSTANCE: svtdbm NODE : 000 DB : SAKE
APPHDL : 0-28727 APPID: *LOCAL.svtdbm.101126110514
AUTHID : SVTDBM
EDUID : 630 EDUNAME: db2agent (SAKE)
FUNCTION: DB2 UDB, base sys utilities,
sqeAgentServices::RequestAgent, probe:100
MESSAGE : ADM7519W DB2 could not allocate an agent. The
SQLCODE is "-6036".
2010-11-25-17.42.00.954150-300 I68910901E456 LEVEL: Severe
PID : 21039 TID : 47381592664384PROC :
db2sysc
INSTANCE: svtdbm NODE : 000 DB : SAKE
APPHDL : 0-28727 APPID: *LOCAL.svtdbm.101126110514
AUTHID : SVTDBM
EDUID : 630 EDUNAME: db2agent (SAKE)
FUNCTION: DB2 UDB, base sys utilities,
sqeIcoordCB::DispatchIndependentDBAgent, probe:110
RETCODE : ZRC=0xFFFFE86C=-6036
2010-11-25-17.42.00.955750-300 I68911358E544 LEVEL: Error
PID : 21039 TID : 47381592664384PROC :
db2sysc
INSTANCE: svtdbm NODE : 000 DB : SAKE
APPHDL : 0-28727 APPID: *LOCAL.svtdbm.101126110514
AUTHID : SVTDBM
EDUID : 630 EDUNAME: db2agent (SAKE)
FUNCTION: DB2 UDB, High Availability Disaster Recovery,
hdrSpawnICoord, probe:30100
MESSAGE : HADR ICoord Error:
DATA #1 : Hexdump, 4 bytes
0x00002B17E2FEA9B0 : 6D00 0580
m...
2010-11-25-17.42.00.957140-300 I68911903E533 LEVEL: Error
PID : 21039 TID : 47381592664384PROC :
db2sysc
INSTANCE: svtdbm NODE : 000 DB : SAKE
APPHDL : 0-28727 APPID: *LOCAL.svtdbm.101126110514
AUTHID : SVTDBM
EDUID : 630 EDUNAME: db2agent (SAKE)
FUNCTION: DB2 UDB, High Availability Disaster Recovery,
hdrSpawnICoord, probe:30101
MESSAGE : sqlcode:
DATA #1 : Hexdump, 4 bytes
0x00002B17E2FEA9F0 : 6CE8 FFFF
l...
2010-11-25-17.42.00.958549-300 I68912437E558 LEVEL:
Warning
PID : 21039 TID : 47381592664384PROC :
db2sysc
INSTANCE: svtdbm NODE : 000 DB : SAKE
APPHDL : 0-28727 APPID: *LOCAL.svtdbm.101126110514
AUTHID : SVTDBM
EDUID : 630 EDUNAME: db2agent (SAKE)
FUNCTION: DB2 UDB, High Availability Disaster Recovery,
hdrTakeoverHdrRouteIn, probe:55603
RETCODE : ZRC=0x8005006D=-2147155859=SQLE_CA_BUILT
"SQLCA has been built and saved in component specific
control block."
2010-11-25-17.42.00.954201-300 I68912996E341 LEVEL:
Warning
PID : 21039 TID : 47382024677696PROC :
db2sysc
INSTANCE: svtdbm NODE : 000
EDUID : 310 EDUNAME: db2hadrp (SAKE)
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrEduP,
probe:20301
MESSAGE : Info: Primary Started.
2010-11-25-17.42.00.965391-300 I68913338E389 LEVEL:
Warning
PID : 21039 TID : 47382024677696PROC :
db2sysc
INSTANCE: svtdbm NODE : 000
EDUID : 310 EDUNAME: db2hadrp (SAKE)
FUNCTION: DB2 UDB, High Availability Disaster Recovery,
hdrSetTcpWindowSize, probe:32201
MESSAGE : Info: HADR Socket send buffer size, SO_SNDBUF: 16384
bytes
2010-11-25-17.42.00.966825-300 I68913728E392 LEVEL:
Warning
PID : 21039 TID : 47382024677696PROC :
db2sysc
INSTANCE: svtdbm NODE : 000
EDUID : 310 EDUNAME: db2hadrp (SAKE)
FUNCTION: DB2 UDB, High Availability Disaster Recovery,
hdrSetTcpWindowSize, probe:32251
MESSAGE : Info: HADR Socket receive buffer size, SO_RCVBUF:
87380 bytes
2010-11-25-17.42.00.965075-300 I68914121E1738 LEVEL: Severe
PID : 21039 TID : 47381592664384PROC :
db2sysc
INSTANCE: svtdbm NODE : 000 DB : SAKE
APPHDL : 0-28727 APPID: *LOCAL.svtdbm.101126110514
AUTHID : SVTDBM
EDUID : 630 EDUNAME: db2agent (SAKE)
FUNCTION: DB2 UDB, SQO Memory Management,
sqloDiagnoseFreeBlockFailure, probe:10
MESSAGE : Possible memory corruption detected.
DATA #1 : ZRC, PD_TYPE_ZRC, 4 bytes
0x820F0002
DATA #2 : Corrupt block address, PD_TYPE_CORRUPT_BLK_PTR, 8
bytes
0x00002b1789615100
DATA #3 : Block header, PD_TYPE_BLK_HEADER, 24 bytes
0x00002B17896150E8 : 1596 78B1 0200 B0FA 0000 0000 0000 0000
..x.............
0x00002B17896150F8 : 0000 0000 0000 0000
........
DATA #4 : Data header, PD_TYPE_BLK_DATA_HEAD, 48 bytes
0x00002B1789615100 : 8000 0000 0000 0000 DDDD DDDD DDDD DDDD
................
0x00002B1789615110 : DDDD DDDD DDDD DDDD DDDD DDDD DDDD DDDD
................
CALLSTCK:
[0] 0x00002B1761A3CEEC pdLog + 0x18C
[1] 0x00002B1763F7042A /home/svtdbm/sqllib/lib64/libdb2e.so.1
+ 0x382E42A
[2] 0x00002B1763F70F52 sqlofmblkEx + 0x130
[3] 0x00000000004147B6 _Z9sqlofmblkPv + 0x18
[4] 0x00002B1763EE040A sqlo_acs_fblk + 0x1C
[5] 0x00002B176259A889
_Z26sqlexAgentCleanUserSecInfoP16SQLEX_AUTHINFO_TP5sqlca + 0xA3
[6] 0x00002B1762385917
_ZN14sqeApplication12AppStopUsingEP8sqeAgenthP5sqlca + 0xF67
[7] 0x00002B1762532B3C _Z11sqlesrspWrpP14db2UCinterface +
0x2D8
[8] 0x00002B176253306D sqleUCagentConnectReset + 0x17D
[9] 0x00002B17626B852C /home/svtdbm/sqllib/lib64/libdb2e.so.1
+ 0x1F7652C
2010-11-25-17.42.00.976550-300 I68915860E456 LEVEL: Severe
PID : 21039 TID : 47381592664384PROC :
db2sysc
INSTANCE: svtdbm NODE : 000 DB : SAKE
APPHDL : 0-28727 APPID: *LOCAL.svtdbm.101126110514
AUTHID : SVTDBM
EDUID : 630 EDUNAME: db2agent (SAKE)
FUNCTION: DB2 UDB, SQO Memory Management,
sqloDiagnoseFreeBlockFailure, probe:30
DATA #1 : String, 12 bytes
Double free.
2010-11-25-17.42.00.978148-300 E68916317E928 LEVEL:
Critical
PID : 21039 TID : 47381592664384PROC :
db2sysc
INSTANCE: svtdbm NODE : 000 DB : SAKE
APPHDL : 0-28727 APPID: *LOCAL.svtdbm.101126110514
AUTHID : SVTDBM
EDUID : 630 EDUNAME: db2agent (SAKE)
FUNCTION: DB2 UDB, base sys utilities, sqle_panic, probe:10
MESSAGE : ADM14001C An unexpected and critical error has
occurred: "Panic".
The instance may have been shutdown as a result.
"Automatic" FODC
(First Occurrence Data Capture) has been invoked and
diagnostic
information has been recorded in directory
"/home/svtdbm/sqllib/db2dump/FODC_Panic_2010-11-25-17.42.00.9780
28/".
Please look in this directory for detailed evidence
about what
happened and contact IBM support if necessary to
diagnose the
problem. |
