This problem effects v91, v95, v97. In partition database
environment (where multiple nodes needs to communicate with each
other via FCM), if one of the nodes run out of the FCM buffer to
communicate back (send reply) to other nodes, it could cause the
receiver nodes to wait forever for the reply, which causes hang.
The following db2diag.log entries will be written when snapshot
monitoring runs out of FCM buffers:
2010-03-25-02.04.24.272731+000 I14931064A484 LEVEL: Error
PID : 648562 TID : 1 PROC : db2agntp 1
INSTANCE: xxxxxx NODE : 001 APPHDL : 0-413
FUNCTION: DB2 UDB, fast comm manager,
sqkfBufferManager::AllocBuffer, probe:55
MESSAGE : Ran out low priority buffers. Total=246125; Num in
freelist=49226; med thres=49225; hi thres=24612
DATA #1 : Hexdump, 4 bytes
0x0FFFFFFFFFFF6184 : 0000 000A
....
2010-03-25-02.04.24.272901+000 I14931549A332 LEVEL: Warning
PID : 648562 TID : 1 PROC : db2agntp 1
INSTANCE: xxxxxx NODE : 001 APPHDL : 0-413
FUNCTION: DB2 UDB, buffer dist serv, sqlkdSendReply, probe:80
RETCODE : ZRC=0x85590027=-2057764825=SQLKF_NO_BUFFER "No FCM
Buffer available"
2010-03-25-02.04.24.273315+000 I14931882A376 LEVEL: Warning
PID : 648562 TID : 1 PROC : db2agntp 1
INSTANCE: xxxxxx NODE : 001 APPHDL : 0-413
FUNCTION: DB2 UDB, buffer dist serv, sqlkdSendReply, probe:90
MESSAGE : Replying to node =
DATA #1 : Hexdump, 2 bytes
0x0FFFFFFFFFFF6570 : 0000
..
2010-03-25-02.04.24.273460+000 I14935899A303 LEVEL: Warning
PID : 648562 TID : 1 PROC :
db2agntp 1
INSTANCE: db2ob8 NODE : 001
APPHDL : 0-413
FUNCTION: DB2 UDB, base sys utilities, sqlePdbRemap, probe:100
MESSAGE : ZRC 85590027 remapped to SQLCODE -6040
2010-03-25-02.04.24.625916+000 I14936203A293 LEVEL: Severe
PID : 648562 TID : 1 PROC : db2agntp 1
INSTANCE: xxxxxx NODE : 001 APPHDL : 0-413
FUNCTION: DB2 UDB, database monitor, sqm_sqlmPdbRequestRouter,
probe:90
RETCODE : ZRC=0xFFFFE868=-6040
Stacktrace on the receiver node will be similar to the
following:
<StackTrace>
-------Frame------
WaitRecvReady
ReceiveBuffer
getNextReply
getNextBuffer
sqlkd_rcv_buffer
sqlkd_rcv_get_next_buffer
sqlkd_rcv_init
sqlkdReceiveReply
sqlmReceiveAndProcessReplies
sqlmSendMonitorRequest
sqlmonssbackend
sqlesrvr
sqleMappingFnServer
sqlerKnownProcedure
sqlerCallDL
sqljs_ddm_excsqlstt
sqljsParseRdbAccessed
sqljsParse |
