Prior to the instance down of this problem, the message
"marking fmp as unstable after new thread creation failure:"
at sqlerMasterThreadReq is logged and fmpRowUseCount indicates
1 as db2diag.log as below:
-----
2011-06-02-19.22.51.266000+540 I11423990F453 LEVEL: Severe
PID : 2136 TID : 6228 PROC :
db2syscs.exe
INSTANCE: TOOL NODE : 000 DB : TOOLSDB
APPHDL : 0-49740 APPID: *LOCAL.TOOL.110602102634
AUTHID : SYSTEM
EDUID : 6228 EDUNAME: db2agent (TOOLSDB)
FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadReq,
probe:10
RETCODE : ZRC=0xFFFFFBEE=-1042
2011-06-02-19.22.51.266000+540 E11424445F3409 LEVEL: Severe
PID : 2136 TID : 6228 PROC :
db2syscs.exe
INSTANCE: TOOL NODE : 000 DB : TOOLSDB
APPHDL : 0-49740 APPID: *LOCAL.TOOL.110602102634
AUTHID : SYSTEM
EDUID : 6228 EDUNAME: db2agent (TOOLSDB)
FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadReq,
probe:910
DATA #1 : String, 58 bytes
marking fmp as unstable after new thread creation failure:
DATA #2 : String, 8 bytes
Fmp TID:
DATA #3 : Hexdump, 4 bytes
0x000007FFFDE3CC90 : 0000 0000
....
DATA #4 : String, 8 bytes
Fmp Row:
DATA #5 : sqlerFmpRow, PD_SQLER_TYPE_FMP_ROW, 496 bytes
fmpPid: 5396
fmpPoolList Ptr: 0x0000000000000000 fmpForcedList Ptr:
0x0000000000000000
nextFmpCB Ptr: 0x000007fffdc0f1a0 prevFmpCB Ptr:
0x0000000000000000
fmpIPCList Ptr: 0x000007fffde3cb00
stateFlags: 0x00000012 numFmp32Attaches: 0
numActiveThreads: 0 numPoolThreads: 0
fmpCodePage: 0 fmpRowUseCount: 1
active: 0x01 rowLoaderValidate: 0x00
ipcLatch:
0x000007FFFDE3C004 : 8000 AB01
....
rowLatch:
0x000007FFFDE3C008 : 8000 AC01
....
fmpAgentList:
-----
Then later timing, this instance goes down with below message:
-----
2011-06-02-20.47.51.453000+540 E11740971F1136 LEVEL: Severe
PID : 2136 TID : 6480 PROC :
db2syscs.exe
INSTANCE: TOOL NODE : 000 DB : TOOLSDB
APPHDL : 0-53112 APPID: *LOCAL.TOOL.110602114844
AUTHID : SYSTEM
EDUID : 6480 EDUNAME: db2agent (TOOLSDB)
FUNCTION: DB2 UDB, SQO Memory Management,
sqloDiagnoseFreeBlockFailure, probe:999
MESSAGE : Memory validation failure, diagnostic file dumped.
DATA #1 : String, 28 bytes
Corrupt pool free tree node.
DATA #2 : File name, 29 bytes
2136.6480.mem_diagnostics.txt
CALLSTCK:
[0] 0x0000000180116CDA pdLog + 0x36A
[1] 0x0000000180051873 sqloDirectIOIsRegistryEnable + 0x3DB
[2] 0x000000018007C95D sqloxltc_app + 0x5EFF
[3] 0x00000001800519C3 sqloDirectIOIsRegistryEnable + 0x52B
[4] 0x000000018004A7DA sqlogmblkEx + 0x52A
[5] 0x0000000001F7676E sqlerGetFMPIPC + 0x10A
[6] 0x0000000001CBC386 sqlccGetIPCs + 0xAE6
[7] 0x0000000001F4BA99 sqlerAllocFmpIPC + 0x13B
[8] 0x0000000001F4A8E7 sqlerAddFmpToPool + 0x2D7
[9] 0x0000000001F4981A sqlerGetFmpFromPool + 0x3C0
-----
The associated trap file has a similar trace back as below:
---
<sqloFreeMemChunks> <NotFound:-1>
<sqloDirectIOIsRegistryEnable> <NotFound:-1>
<sqloxltc_app> <NotFound:-1>
<sqloDirectIOIsRegistryEnable> <NotFound:-1>
<sqlogmblkEx> <NotFound:-1>
<sqlerGetFMPIPC> <sqlerFmpCommon.C:391>
<sqlccGetIPCs> <sqlccipc_srv.C:1185>
<sqlerAllocFmpIPC> <sqlerFmpEngine.C:6593>
<sqlerGetFmpThreadEntry> <sqlerFmpEngine.C:4609>
<sqlerGetFmpFromPool> <sqlerFmpEngine.C:1773>
<sqlerInvokeFencedRoutine> <sqlerFmpEngine.C:818>
--- |
