Hi all,
This I feel sure is "pilot error"; I am messing something up in my perl script to create the db. Have a tiny collection of 6 documents 75kb when loaded. It seems to load ok but when I go to use it I am getting Out of Main Memory errors. The JVM is 4gb using -Xms and -Xmx options on the java command to start the basex server (on an 8gb VM).
On reviewing the logs I can see that my client is not doing a CLOSE for the database -- yesterday's log showed 175 OPENs and only 31 closes -- OOPS!! Ok, so I start getting this:
OPEN chaingang Error: Out of Main Memory. 11286.05 ms I can see in top the JVMs ram just being gobbled up rapidly until it runs out and this message appears.
Yes, I know I need to fix my clients but is it possible that even when the clients session closes, somehow the database is being kept open, and eventually we run out of memory? But that doesn;t really fit with the "gobbling ram" I am seeing, unless the former situation triggers the latter.
IF this is the issue, could the basex server be made to nicely close any databases open in a session when that session is closed or the pipe gets broken?
# when trying to recover 2010-08-31.log:22:24:11.781 DROP DB chaingang Error: Database 'chaingang' is currently locked.
Or of a db has become corrupted somehow, could it cause such message? Are there any integrity check commands on the horizon?
I feel a bit nervous about these kinds of memory errors as I operate in a VSphere v4 (VMWARE) environment and my JVM monitoring skills are rudimentary -- the JVM does look like it has a nice big contiguous slab of ram to work with. Clearly something bad/silly I have done is triggering this behaviour: basex just "runs away" with ram.
Any thoughts? Has the api interface changed -- my perl client dates back to late March?
Sorry to be so vague.
Any similarly vague clues most welcome.
On a similar vien, my web interface/query controller is modperl2 under apache, which means the script sits resident in ram. I have it opening/closing a basex session on each request cycle (and am about to go check what I do when it closes!). Perhaps it would be far better to have the basex connection done in the apache initialisation stage (when apache2 launches the modperl2 modules) so the session just sits there until apache restarts. Then each request cycle should just open/close the database? I ask here as it could possibly be related to my grief above. If not my apologies for confusing the matter.
I am using this Sun jdk: /opt/java/jdk1.6.0_18/bin/java. Kind of regretting not using openSolaris as my OS platform rather than Linux. I do so love its trace facilities and its a great Java platform. I'm using 64bit Linux: 2.6.27-17-server #1 SMP Fri Mar 12 02:48:52 UTC 2010 x86_64 GNU/Linux ; and BaseX 6.2.6 [Server]
Best wishes all,
Sandra
Download Eclipse MAT http://www.eclipse.org/mat/ http://www.eclipse.org/mat/and run the JVM with http://stackoverflow.com/questions/542979/using-heapdumponoutofmemoryerror-p...
http://stackoverflow.com/questions/542979/using-heapdumponoutofmemoryerror-parameter-for-heap-dump-for-jbossTake the snapshot and open it in MAT and tell BaseX's developers why it's running out of memory (or put the report MAT creates on your webserver - it's just a bunch of .html files)
- Godmar
On Tue, Aug 31, 2010 at 8:35 PM, Sandra Maria Silcot <ssilcot@unimelb.edu.au
wrote:
Hi all,
This I feel sure is "pilot error"; I am messing something up in my perl script to create the db. Have a tiny collection of 6 documents 75kb when loaded. It seems to load ok but when I go to use it I am getting Out of Main Memory errors. The JVM is 4gb using -Xms and -Xmx options on the java command to start the basex server (on an 8gb VM).
On reviewing the logs I can see that my client is not doing a CLOSE for the database -- yesterday's log showed 175 OPENs and only 31 closes -- OOPS!! Ok, so I start getting this:
OPEN chaingang Error: Out of Main Memory. 11286.05 ms I can see in top the JVMs ram just being gobbled up rapidly until it runs out and this message appears.
Yes, I know I need to fix my clients but is it possible that even when the clients session closes, somehow the database is being kept open, and eventually we run out of memory? But that doesn;t really fit with the "gobbling ram" I am seeing, unless the former situation triggers the latter.
IF this is the issue, could the basex server be made to nicely close any databases open in a session when that session is closed or the pipe gets broken?
# when trying to recover 2010-08-31.log:22:24:11.781 DROP DB chaingang Error: Database 'chaingang' is currently locked.
Or of a db has become corrupted somehow, could it cause such message? Are there any integrity check commands on the horizon?
I feel a bit nervous about these kinds of memory errors as I operate in a VSphere v4 (VMWARE) environment and my JVM monitoring skills are rudimentary -- the JVM does look like it has a nice big contiguous slab of ram to work with. Clearly something bad/silly I have done is triggering this behaviour: basex just "runs away" with ram.
Any thoughts? Has the api interface changed -- my perl client dates back to late March?
Sorry to be so vague.
Any similarly vague clues most welcome.
On a similar vien, my web interface/query controller is modperl2 under apache, which means the script sits resident in ram. I have it opening/closing a basex session on each request cycle (and am about to go check what I do when it closes!). Perhaps it would be far better to have the basex connection done in the apache initialisation stage (when apache2 launches the modperl2 modules) so the session just sits there until apache restarts. Then each request cycle should just open/close the database? I ask here as it could possibly be related to my grief above. If not my apologies for confusing the matter.
I am using this Sun jdk: /opt/java/jdk1.6.0_18/bin/java. Kind of regretting not using openSolaris as my OS platform rather than Linux. I do so love its trace facilities and its a great Java platform. I'm using 64bit Linux: 2.6.27-17-server #1 SMP Fri Mar 12 02:48:52 UTC 2010 x86_64 GNU/Linux ; and BaseX 6.2.6 [Server]
Best wishes all,
Sandra
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Godmar,
I'm not running Jboss -- I running the server as a jar like this:
nohup java -Xmx4200m -cp basex-current.jar org.basex.BaseXServer -p 8888 (basex-current.jar symlinks to -> basex-6.2.6.jar)
Is Jboss required for MAT?
Please note:
* I have gone through my modperl and perl admin clients * I have checked I issue a 'close' for every open and that I close the connection -- several corrections were made where this was NOT happening * I have corrected my clients so I'm not sending empty/null commands (a carriage return or a null was slipping through) * I have closely monitored usage today and basex has been very well behaved - no problems at all * I have cycled through various scripted full creates and all seems well * the JVM is using ram, as I would expect it to: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8823 fas 20 0 4697m 3.7g 8976 S 0 49.3 16:22.60 java
This may be coincidence but it looks very much to me as if my sloppy coding in not "close"(ing) databases was triggering the runaway ram problem.
If I can do the MAT thing without jboss - please let me know, and I will roll back my changes and try to replicate, but I'm feeling fairly sure you could be looking at your server code and perhaps making it a little more defensive against sloppy clients!
I'll continue to monitor. If you like I can send you yesterday's log privately - its too embarassing to post it to the list!!!
Many thanks,
Sandra
Download Eclipse MAT http://www.eclipse.org/mat/ http://www.eclipse.org/mat/and run the JVM with http://stackoverflow.com/questions/542979/using-heapdumponoutofmemoryerror-p...
the snapshot and open it in MAT and tell BaseX's developers why it's running
out of memory (or put the report MAT creates on your webserver - it's
just
a bunch of .html files)
- Godmar
On Tue, Aug 31, 2010 at 8:35 PM, Sandra Maria Silcot <ssilcot@unimelb.edu.au
wrote:
Hi all, This I feel sure is "pilot error"; I am messing something up in my perl
script to create the db. Have a tiny collection of 6 documents 75kb when
loaded. It seems to load ok but when I go to use it I am getting Out of
Main Memory errors. The JVM is 4gb using -Xms and -Xmx options on the java
command to start the basex server (on an 8gb VM). On reviewing the logs I can see that my client is not doing a CLOSE for
the database -- yesterday's log showed 175 OPENs and only 31 closes -- OOPS!! Ok, so I start getting this:
OPEN chaingang Error: Out of Main Memory. 11286.05 ms I can see in top the JVMs ram just being gobbled up rapidly until it
runs
out and this message appears. Yes, I know I need to fix my clients but is it possible that even when
the
clients session closes, somehow the database is being kept open, and
eventually we run out of memory? But that doesn;t really fit with the "gobbling ram" I am seeing, unless the former situation triggers the latter.
IF this is the issue, could the basex server be made to nicely close
any
databases open in a session when that session is closed or the pipe
gets
broken? # when trying to recover 2010-08-31.log:22:24:11.781 DROP DB chaingang Error: Database 'chaingang' is currently
locked.
Or of a db has become corrupted somehow, could it cause such message?
Are
there any integrity check commands on the horizon? I feel a bit nervous about these kinds of memory errors as I operate in
a
VSphere v4 (VMWARE) environment and my JVM monitoring skills are
rudimentary -- the JVM does look like it has a nice big contiguous slab of
ram to work with. Clearly something bad/silly I have done is triggering
this behaviour: basex just "runs away" with ram.
Any thoughts? Has the api interface changed -- my perl client dates
back
to late March? Sorry to be so vague. Any similarly vague clues most welcome. On a similar vien, my web interface/query controller is modperl2 under
apache, which means the script sits resident in ram. I have it
opening/closing a basex session on each request cycle (and am about to
go
check what I do when it closes!). Perhaps it would be far better to
have
the basex connection done in the apache initialisation stage (when
apache2
launches the modperl2 modules) so the session just sits there until
apache
restarts. Then each request cycle should just open/close the database?
I
ask here as it could possibly be related to my grief above. If not my
apologies for confusing the matter.
I am using this Sun jdk: /opt/java/jdk1.6.0_18/bin/java. Kind of
regretting not using openSolaris as my OS platform rather than Linux. I do
so love its trace facilities and its a great Java platform. I'm using
64bit Linux: 2.6.27-17-server #1 SMP Fri Mar 12 02:48:52 UTC 2010 x86_64
GNU/Linux ; and BaseX 6.2.6 [Server] Best wishes all, Sandra _______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Hm… I'm afraid I can't give quick help on that one. Database instances should get correctly closed if connections fail or get lost – but if you (or someone else) manages to create a reproducible example in which that's not the case, that would be great, and should be easily fixed.
Christian
On Wed, Sep 1, 2010 at 8:24 AM, Sandra Maria Silcot ssilcot@unimelb.edu.au wrote:
Godmar,
I'm not running Jboss -- I running the server as a jar like this:
nohup java -Xmx4200m -cp basex-current.jar org.basex.BaseXServer -p 8888 (basex-current.jar symlinks to -> basex-6.2.6.jar)
Is Jboss required for MAT?
Please note:
* I have gone through my modperl and perl admin clients * I have checked I issue a 'close' for every open and that I close the connection -- several corrections were made where this was NOT happening * I have corrected my clients so I'm not sending empty/null commands (a carriage return or a null was slipping through) * I have closely monitored usage today and basex has been very well behaved - no problems at all * I have cycled through various scripted full creates and all seems well
- the JVM is using ram, as I would expect it to:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8823 fas 20 0 4697m 3.7g 8976 S 0 49.3 16:22.60 java
This may be coincidence but it looks very much to me as if my sloppy coding in not "close"(ing) databases was triggering the runaway ram problem.
If I can do the MAT thing without jboss - please let me know, and I will roll back my changes and try to replicate, but I'm feeling fairly sure you could be looking at your server code and perhaps making it a little more defensive against sloppy clients!
I'll continue to monitor. If you like I can send you yesterday's log privately - its too embarassing to post it to the list!!!
Many thanks,
Sandra
Download Eclipse MAT http://www.eclipse.org/mat/ http://www.eclipse.org/mat/and run the JVM with http://stackoverflow.com/questions/542979/using-heapdumponoutofmemoryerror-p...
the snapshot and open it in MAT and tell BaseX's developers why it's running
out of memory (or put the report MAT creates on your webserver - it's
just
a bunch of .html files)
- Godmar
On Tue, Aug 31, 2010 at 8:35 PM, Sandra Maria Silcot <ssilcot@unimelb.edu.au
wrote:
Hi all, This I feel sure is "pilot error"; I am messing something up in my perl
script to create the db. Have a tiny collection of 6 documents 75kb when
loaded. It seems to load ok but when I go to use it I am getting Out of
Main Memory errors. The JVM is 4gb using -Xms and -Xmx options on the java
command to start the basex server (on an 8gb VM). On reviewing the logs I can see that my client is not doing a CLOSE for
the database -- yesterday's log showed 175 OPENs and only 31 closes -- OOPS!! Ok, so I start getting this:
OPEN chaingang Error: Out of Main Memory. 11286.05 ms I can see in top the JVMs ram just being gobbled up rapidly until it
runs
out and this message appears. Yes, I know I need to fix my clients but is it possible that even when
the
clients session closes, somehow the database is being kept open, and
eventually we run out of memory? But that doesn;t really fit with the "gobbling ram" I am seeing, unless the former situation triggers the latter.
IF this is the issue, could the basex server be made to nicely close
any
databases open in a session when that session is closed or the pipe
gets
broken? # when trying to recover 2010-08-31.log:22:24:11.781 DROP DB chaingang Error: Database 'chaingang' is currently
locked.
Or of a db has become corrupted somehow, could it cause such message?
Are
there any integrity check commands on the horizon? I feel a bit nervous about these kinds of memory errors as I operate in
a
VSphere v4 (VMWARE) environment and my JVM monitoring skills are
rudimentary -- the JVM does look like it has a nice big contiguous slab of
ram to work with. Clearly something bad/silly I have done is triggering
this behaviour: basex just "runs away" with ram.
Any thoughts? Has the api interface changed -- my perl client dates
back
to late March? Sorry to be so vague. Any similarly vague clues most welcome. On a similar vien, my web interface/query controller is modperl2 under
apache, which means the script sits resident in ram. I have it
opening/closing a basex session on each request cycle (and am about to
go
check what I do when it closes!). Perhaps it would be far better to
have
the basex connection done in the apache initialisation stage (when
apache2
launches the modperl2 modules) so the session just sits there until
apache
restarts. Then each request cycle should just open/close the database?
I
ask here as it could possibly be related to my grief above. If not my
apologies for confusing the matter.
I am using this Sun jdk: /opt/java/jdk1.6.0_18/bin/java. Kind of
regretting not using openSolaris as my OS platform rather than Linux. I do
so love its trace facilities and its a great Java platform. I'm using
64bit Linux: 2.6.27-17-server #1 SMP Fri Mar 12 02:48:52 UTC 2010 x86_64
GNU/Linux ; and BaseX 6.2.6 [Server] Best wishes all, Sandra _______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
Godmar Back -
Sandra Maria Silcot