db2expln hangs because its associated db2fmp crashed due to
"Corrupt pool free tree node". The db2diag.log prints:
2014-10-17-00.41.05.615427+480 I89038A491 LEVEL: Warning
PID : 33816664 TID : 1 PROC : db2fmp
(149139) 0
INSTANCE: istdbs5 NODE : 000 DB : F14G2RC
APPID : 10.48.38.149.39484.141016164020
EDUID : 1 EDUNAME: db2fmp (149139) 0
FUNCTION: DB2 UDB, aps binder service, sqlajbnd_process,
probe:2817
MESSAGE : Error code
DATA #1 : Hexdump, 4 bytes
0x0FFFFFFFFFFFAF20 : FFFF FDD9
....
2014-10-17-00.41.05.629355+480 I89530A926 LEVEL: Warning
PID : 33816664 TID : 1 PROC : db2fmp
(149139) 0
INSTANCE: istdbs5 NODE : 000 DB : F14G2RC
APPID : 10.48.38.149.39484.141016164020
EDUID : 1 EDUNAME: db2fmp (149139) 0
FUNCTION: DB2 UDB, trace services, sqlt_logerr_data (secondary
logging func, probe:0
MESSAGE : sqlaj_error_struct
DATA #1 : Hexdump, 88 bytes
0x0FFFFFFFFFFFAE30 : 4150 534A 0000 0B01 3238 3137 0000 0000
APSJ....2817....
0x0FFFFFFFFFFFAE40 : 0000 0000 FFFF FDD9 7371 6C61 6A50 7265
........sqlajPre
0x0FFFFFFFFFFFAE50 : 7069 6E69 7400 0000 0000 0000 0000 0000
pinit...........
0x0FFFFFFFFFFFAE60 : 0000 0000 0000 0B01 4261 6420 7371 6C61
........Bad sqla
0x0FFFFFFFFFFFAE70 : 6A50 7265 7069 6E69 7400 0000 0000 0000
jPrepinit.......
0x0FFFFFFFFFFFAE80 : 0000 0000 0000 0000
........
2014-10-17-00.41.05.666887+480 E90457A1257 LEVEL: Severe
PID : 33816664 TID : 1 PROC : db2fmp
(149139) 0
INSTANCE: istdbs5 NODE : 000 DB : F14G2RC
APPID : 10.48.38.149.39484.141016164020
EDUID : 1 EDUNAME: db2fmp (149139) 0
FUNCTION: DB2 UDB, SQO Memory Management,
sqloDiagnoseFreeBlockFailure, probe:999
MESSAGE : Memory validation failure, diagnostic file dumped.
DATA #1 : String, 28 bytes
Corrupt pool free tree node.
DATA #2 : File name, 30 bytes
33816664.1.mem_diagnostics.txt
CALLSTCK: (Static functions may not be resolved correctly, as
they are resolved to the nearest symbol)
[0] 0x09000000016A4E58 pdLog + 0xA8
[1] 0x09000000017AF23C
diagnoseMemoryCorruptionAndCrash__13SQLO_MEM_POOLFUlCPCcCb +
0x2A8
[2] 0x09000000017AEEBC
diagnoseMemoryCorruptionAndCrash__13SQLO_MEM_POOLFUlCPCcCb@glue3
7B + 0x7C
[3] 0x090000000294CCF8
.MemTreePut.fdpr.clone.47__13SQLO_MEM_POOLFP8SMemNodeUlP17SqloCh
unkSubgroup + 0xF4
[4] 0x0900000001907F4C sqlofmblkEx + 0x268
[5] 0x0900000001F910F4 sqlerFmpListener + 0x78
[6] 0x0000000100001248 main + 0x928
[7] 0x00000001000002D8 __start + 0x98
[8] 0x0000000000000000 ?unknown + 0x0
[9] 0x0000000000000000 ?unknown + 0x0
2014-10-17-00.41.05.822283+480 I95594A1833 LEVEL: Warning
PID : 33816664 TID : 1 PROC : db2fmp
(149139) 0
INSTANCE: istdbs5 NODE : 000 DB : F14G2RC
APPID : 10.48.38.149.39484.141016164020
EDUID : 1 EDUNAME: db2fmp (149139) 0
FUNCTION: DB2 UDB, SQO Memory Management, sqlogmblkEx,
probe:1000
MESSAGE : ZRC=0x820F0002=-2112946174=SQLO_INV_MEM "Invalid
memory addr"
DIA8561C A invalid memory block was encountered.
DATA #1 : String, 43 bytes
Memory management block allocation failure.
DATA #2 : Codepath, 8 bytes
4:20:28
DATA #3 : Memory pool handle pointer,
PD_TYPE_MEM_POOL_HANDLE_PTR, 8 bytes
0x0000000000000000
DATA #4 : Requested size, PD_TYPE_MEM_REQUESTED_SIZE, 8 bytes
54
DATA #5 : Adjusted block size, PD_TYPE_MEM_ADJUSTED_SIZE, 8
bytes
96
DATA #6 : Options for requested block, PD_TYPE_GET_MEM_OPTIONS,
4 bytes
0x00000000
DATA #7 : Pointer to address that will be set by new allocation,
PD_TYPE_PTR_TO_ADDRESS_OUT, 8 bytes
0x0fffffffffffe468
DATA #8 : File name, PD_TYPE_OSS_MEM_FILE_NAME, 18 bytes
db2explnServer.SQC
DATA #9 : Line of code, PD_TYPE_OSS_MEM_LINE_NUM, 8 bytes
744
DATA #10: Resource binding pointer,
PD_TYPE_RESOURCE_BINDING_PTR, 8 bytes
0x0000000000000000
CALLSTCK: (Static functions may not be resolved correctly, as
they are resolved to the nearest symbol)
[0] 0x09000000016A4E58 pdLog + 0xA8
[1] 0x09000000022567E8 pdLog@glue1EC + 0x254
[2] 0x09000000016E22C4 sqlogmblkEx + 0x764
[3] 0x09000000289ADAE0 explain + 0xEC0
[4] 0x0900000001F9B16C
@76@sqlerRunRoutine__FP13sqleInvokerCBPi + 0x27C
[5] 0x090000000192E9C0 sqlerDyload + 0x13C
[6] 0x090000000191B6BC sqlerFmpListener + 0x170
[7] 0x0000000100001248 main + 0x928
[8] 0x00000001000002D8 __start + 0x98
[9] 0x0000000000000000 ?unknown + 0x0
From 33816664.1.mem_diagnostics.txt, we can see the sqlcode -551
bind error
message returned during bind of db2exdyn.bnd corrupted the free
tree node:
BEGIN MEMORY CORRUPTION DIAGNOSIS CONTEXT DUMP
================================================================
=============
Chunk Subgroup: address: 0x00000001100C8000 alloc pool:
0x00000001100976F0 chunks: 1 blocks: 20 1st addr:
0x00000001100C8068
-- SUSPECT NODE: --
-- BLOCK: code: 0xdb2cafe8 size: 54 file: db2explnServer.SQC
line: 744 seq: 23 CSG: 0xfab00000001100c8 --
00000001100c8f68 : db 2c af e8 00 00 00 36 75 0b 5e ed 02 e8 00
17 .,.....6u.^.....
00000001100c8f78 : fa b0 00 00 00 11 00 c8 00 00 00 01 45 72 72
6f ............Erro
00000001100c8f88 : 72 20 64 75 72 69 6e 67 20 62 69 6e 64 20 6f
66 r during bind of
00000001100c8f98 : 20 64 62 32 65 78 64 79 6e 2e 62 6e 64 20 28
53 db2exdyn.bnd (S
00000001100c8fa8 : 51 4c 43 4f 44 45 20 3d 20 2d 35 35 31 29 2e
20 QLCODE = -551).
00000001100c8fb8 : 4d 65 73 73 61 67 65 73 20 69 6e 20 2f 74 6d
70 Messages in /tmp
-- NODE DETECTED AS CORRUPT --
-- UNKNOWN NODE TYPE --
00000001100c8fc8 : 2f 62 61 61 72 67 62 61 61 62 2e 00 00 00 00
00 /baargbaab......
00000001100c8fd8 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01
00 ................
00000001100c8fe8 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
00000001100c8ff8 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
00000001100c9008 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
00000001100c9018 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
00000001100c9028 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
00000001100c9038 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
00000001100c9048 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
00000001100c9058 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
00000001100c9068 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
00000001100c9078 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
00000001100c9088 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
00000001100c9098 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
00000001100c90a8 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
00000001100c90b8 : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
================================================================
=============
END MEMORY CORRUPTION DIAGNOSIS CONTEXT DUMP |
