When cdr cleanstart is used to reset the replay position and
throw away transactions that have been spooled to disk, it
currently is only removing rows from the trg_send_stxn table.
However, when rows getting written to spool, data is also put in
the trg_send_srep table, and when transactions are removed
normally from spool, this data is also removed. But in the case
of cdr cleanstart the data is not.
Leaving this data behind can cause other ER problems, like
seeing long running CDRFlushQue threads or long running GCReq
threads. These long running threads would have the following
stack traces:
CDRFlushQue:
yield_processor_mvp
mt_lock_wait
mt_lock
rqmGetTxnByStamp
rqmMarkSpoolReplDelete
qrDoFlushQueueForRepl
queuerFlushQueueForRepl
cdrTrampolineThread
th_init_initgls
startup
GCReq:
mt_lock_wait
mt_lock
rqmGetTxnByStamp
rqmMarkSpoolReplDelete
qrFlushQueueForRepl
queuerReplDropped
dac_participate_repl
gsi_participate_repl
clientRequestThread
cdrTrampolineThread
th_init_initgls
startup
The key pieces in the stacks above is the function
rqmMarkSpoolReplDelete, as it's the function which indicates the
trg_send_srep table is being scanned. So in the case of after a
cleanstart there should be nothing in the spool file, so this
should not take time as there should be nothing to do if other
ER onstats (like onstat -g rqm sendq) indicate that there are no
transactions in spool, but if it taking cpu cycles, that would
indicate the presence of the left over, non-purged rows in the
trg_send_srep table.
If the server does have transactions in spool, then taking time
in this function would be normal, as going through the spool
files cleaning things up is required, but when onstat -g rqm
sendq shows no transactions in spool, but there are threads
spending a lot of time in rqmMarkSpoolReplDelete() then that
would indicate the customer had done a cleanstart and now has
these left over rows which is then causing issues. |
