|
Problem IC70077
|
Status: Closed |
ACCUMULATION OF DB2FMP (JAVA) WITH MEMORY CONSUMPTION WHEN KEEPFENCED=NO |
| product: |
DB2 FOR LUW / DB2FORLUW / 950 - DB2 |
| Problem description: |
Output of "ps -elf | grep db2fmp" shows a large number of db2fmp
(Java) processes running on your system when KEEPFENCED=NO. The
db2fmp processes are not being reused and appear to be stale.
For those db2fmp processes, the TIME value does not change
between runs of the "ps -elf | grep db2fmp" command. The number
of db2fmp processes may grow until virtual memory is exhausted.
Issue "db2pd -fmp" and check the status flag for the 'stale'
db2fmp processes. The flag value 0x80000003 for the problematic
db2fmp processes means:
0x00000001 the db2fmp has a JVM -> used by Java routine
+ 0x00000002 the db2fmp is threaded -> for execution of
threadsafe routines
+ 0x80000000 FMP will no longer be allowed to create new
threads, either due to an explicit thread limit or reaching an
internal resource limit
=====================================
x80000003
Because your db2fmp is flagged as 0x80000003 you may see
messages like the following in the db2diag.log file indicating
that new threads cannot be created:
2010-05-24-16.56.12.520000-420 I8527H330 LEVEL: Warning
PID : 4040 TID : 3552 PROC :
db2fmp.exe
INSTANCE: DB2 NODE : 000
EDUID : 3552
FUNCTION: DB2 UDB, routine_infrastructure,
sqlerMasterThreadListener,
probe:300
MESSAGE : Java heap is nearly full; no thread created
2010-05-24-16.56.12.520000-420 I8859H575 LEVEL: Warning
PID : 2556 TID : 1564 PROC :
db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : TEST
APPHDL : 0-87 APPID: AC13240C.G77B.E57AE0955EF1
AUTHID : NIC
EDUID : 1564 EDUNAME: db2agent (TEST) 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadReq,
probe:
89
MESSAGE : FMP reported it could not create a new thread
DATA #1 : Hexdump, 4 bytes
0x79E7E830 : C80F 0000 ....
2010-05-24-16.56.12.520000-420 I9436H456 LEVEL: Severe
PID : 2556 TID : 1564 PROC :
db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : TEST
APPHDL : 0-87 APPID: AC13240C.G77B.E57AE0955EF1
AUTHID : NIC
EDUID : 1564 EDUNAME: db2agent (TEST) 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerGetFmpThread,
probe:20
RETCODE : ZRC=0xFFFFFBEE=-1042 |
| Problem Summary: |
****************************************************************
* USERS AFFECTED: *
* ACCUMULATION OF DB2FMP (JAVA) WITH MEMORY CONSUMPTION *
* WHENKEEPFENCED=NO *
****************************************************************
* PROBLEM DESCRIPTION: *
* Output of "ps -elf | grep db2fmp" shows a large number *
* ofdb2fmp(Java) processes running on your system when *
* KEEPFENCED=NO.Thedb2fmp processes are not being reused and *
* appear to bestale.For those db2fmp processes, the TIME value *
* does not changebetween runs of the "ps -elf | grep db2fmp" *
* command. Thenumberof db2fmp processes may grow until *
* virtual memory isexhausted.Issue "db2pd -fmp" and check the *
* status flag for the 'stale'db2fmp processes. The flag value *
* 0x80000003 for theproblematicdb2fmp processes *
* means:0x00000001 the db2fmp has a JVM -> used by Java *
* routine+ 0x00000002 the db2fmp is threaded -> for *
* execution ofthreadsafe routines+ 0x80000000 FMP will no *
* longer be allowed to create newthreads, either due to an *
* explicit thread limit or reachinganinternal resource *
* limit=====================================x80000003Because *
* your db2fmp is flagged as 0x80000003 you may seemessages *
* like the following in the db2diag.log fileindicatingthat new *
* threads cannot be created:2010-05-24-16.56.12.520000-420 *
* I8527H330 LEVEL:WarningPID : 4040 *
* TID : 3552 PROC :db2fmp.exeINSTANCE: DB2 *
* NODE : 000EDUID : 3552FUNCTION: DB2 UDB, *
* routine_infrastructure,sqlerMasterThreadListener,probe:300MESS
* : Java heap is nearly full; no thread *
* created2010-05-24-16.56.12.520000-420 I8859H575 *
* LEVEL:WarningPID : 2556 TID : 1564 *
* PROC :db2syscs.exeINSTANCE: DB2 NODE : 000 *
* DB : TESTAPPHDL : 0-87 *
* APPID:AC13240C.G77B.E57AE0955EF1AUTHID : NICEDUID : 1564 *
* EDUNAME: db2agent (TEST) 0FUNCTION: DB2 UDB, *
* routine_infrastructure,sqlerMasterThreadReq,probe:89MESSAGE *
* : FMP reported it could not create a new threadDATA #1 : *
* Hexdump, 4 bytes0x79E7E830 : C80F 0000 *
* ....2010-05-24-16.56.12.520000-420 I9436H456 *
* LEVEL:SeverePID : 2556 TID : 1564 *
* PROC :db2syscs.exeINSTANCE: DB2 NODE : *
* 000 DB : TESTAPPHDL : 0-87 *
* APPID:AC13240C.G77B.E57AE0955EF1AUTHID : NICEDUID : 1564 *
* EDUNAME: db2agent (TEST) 0FUNCTION: DB2 UDB, *
* routine_infrastructure,sqlerGetFmpThread,probe:20RETCODE : *
* ZRC=0xFFFFFBEE=-1042 *
****************************************************************
* RECOMMENDATION: *
* WORKAROUNDS:1) Review your Java routines and ensure that the *
* code is notheavily dependent on Garbage Collection. (e.g. *
* close off theobjects manually after they are no longer used *
* rather thenrelying on the JVM to clean them up)From the *
* setting perspective2) If you use THREADSAFE Java routines *
* (default), then setKEEPFENCED=YES. In that case we will *
* reuse the same threadandreduce the chance of creating new *
* threads.3) If you declare your routines as NOT THREADSAFE *
* then youcanset KEEPFENCED to either YES or NO. When a *
* routine is in aNOTTHREADSAFE mode, each routine will run in *
* its owndb2fmp/JVM.If KEEPFENCED=NO a new fenced mode process *
* is created anddestroyed for each fenced mode invocation. *
* IfKEEPFENCED=YESthe existing db2fmp will be returned to the *
* pool for reuse. *
**************************************************************** |
| Local Fix: |
WORKAROUNDS:
1) Review your Java routines and ensure that the code is not
heavily dependent on Garbage Collection. (e.g. close off the
objects manually after they are no longer used rather then
relying on the JVM to clean them up)
From the setting perspective
2) If you use THREADSAFE Java routines (default), then set
KEEPFENCED=YES. In that case we will reuse the same thread and
reduce the chance of creating new threads.
3) If you declare your routines as NOT THREADSAFE then you can
set KEEPFENCED to either YES or NO. When a routine is in a NOT
THREADSAFE mode, each routine will run in its own db2fmp/JVM.
If KEEPFENCED=NO a new fenced mode process is created and
destroyed for each fenced mode invocation. If KEEPFENCED=YES
the existing db2fmp will be returned to the pool for reuse. |
| available fix packs: |
DB2 Version 9.5 Fix Pack 7 for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 8 for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 9 for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 10 for Linux, UNIX, and Windows
|
| Solution |
APAR will be fixed in an upcoming v9.5 fixpak release |
| Workaround |
not known / see Local fix |
| BUG-Tracking |
forerunner : APAR is sysrouted TO one or more of the following: IC70081
follow-up : |
| Timestamps |
Date - problem reported :
Date - problem closed :
Date - last modified :
| 21.07.2010
25.10.2010
25.10.2010 |
| Problem solved at the following versions (IBM BugInfos) |
9.5.
|
| Problem solved according to the fixlist(s) of the following version(s) |
| 9.1.0.7
|
 |
| 9.5.0.7
|
|
