Expected behaviour with full disk
Hello, I’m having some problems with BaseX in situations where there it runs out of disk space. I’m delivering XML files to BaseX 8.4 using RestXQ. A request is made and the XML is added to a database depending on some criteria. I receive many hundreds of thousands of requests and everything works well unless I run out of disk space. What I would expect to happen is that BaseX would try to add the document, find it couldn’t due to the disk being full and return a server error (e.g. 500) and the transaction would be aborted leaving the database as it was before the request. Any further requests would all get errors returned. What I’m seeing is that when a request is made that causes the disk to become full BaseX gets part way through adding the document, fails and returns an error but also leaves the database corrupted. Usually the server has to be restarted and then any database that has been affected has to be deleted - it’s rare to be able to save anything from it. The errors vary but it is usually a NullPointer exception and then java.nio.channels.OverlappingFileLockException. for further requests. I’ve tried turning off logging and setting parallel to 1 to see if that made any difference but it doesn’t. Is what I’m seeing unique to my setup or do others see this? Is there anything I can do to make BaseX just stop accepting requests when disk space runs out to protect my databases? Many thanks, James
Hi, On 01. 04. 16 16:48, James Ball wrote:
Is what I’m seeing unique to my setup or do others see this?
I have seen exactly the same behaviour in case of network hard drive loosing connection causing the Kernel to make the filesystem read only. However, I don't really expect BaseX to manage this kind of case contrary to disk full situation. Pablo
Hello James, Did you try to set ADDCACHE ? This way you might run out of disk space during disk caching, before database corruption occurs. Be careful, you may have to serialize the XML in order to use this option with db:add/db:replace (I read you are in the REST-XQ layer). Otherwise, the only way to prevent database corruption might be to check disk size regularly and reject new data by program… http://docs.basex.org/wiki/Options#ADDCACHE http://docs.basex.org/wiki/Database_Module#db:add Best regards, Fabrice Etanchaud Horanet Smart Objects FR-85200 Fontenay le Comte De : basex-talk-bounces@mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] De la part de James Ball Envoyé : vendredi 1 avril 2016 16:48 À : BaseX <basex-talk@mailman.uni-konstanz.de> Objet : [basex-talk] Expected behaviour with full disk Hello, I’m having some problems with BaseX in situations where there it runs out of disk space. I’m delivering XML files to BaseX 8.4 using RestXQ. A request is made and the XML is added to a database depending on some criteria. I receive many hundreds of thousands of requests and everything works well unless I run out of disk space. J What I would expect to happen is that BaseX would try to add the document, find it couldn’t due to the disk being full and return a server error (e.g. 500) and the transaction would be aborted leaving the database as it was before the request. Any further requests would all get errors returned. What I’m seeing is that when a request is made that causes the disk to become full BaseX gets part way through adding the document, fails and returns an error but also leaves the database corrupted. Usually the server has to be restarted and then any database that has been affected has to be deleted - it’s rare to be able to save anything from it. The errors vary but it is usually a NullPointer exception and then java.nio.channels.OverlappingFileLockException. for further requests. I’ve tried turning off logging and setting parallel to 1 to see if that made any difference but it doesn’t. Is what I’m seeing unique to my setup or do others see this? Is there anything I can do to make BaseX just stop accepting requests when disk space runs out to protect my databases? Many thanks, James
Hi James, I’m sorry, there is general solution for that on database level; if a disk is full, basically anything can be happen. However, as you are using RESTXQ, you could check memory via XQuery: declare namespace file = 'java:java.io.File'; let $free := file:getFreeSpace(file:new('/')) return if($free < 100000000) then error((), "Less than 100 MB left on disk") else ( (: continue :) ) This way, you can also decide what is going to happen if the disk runs out of space (raise an error? send an e-mail? etc.). By the way, raising an error will probably be the most elegant solution, as you can catch the error via %rest:error and do some more things [1]. Hope this helps, Christian [1] http://docs.basex.org/wiki/RESTXQ#Error_Handling On Fri, Apr 1, 2016 at 4:48 PM, James Ball <basex-talk@jamesball.co.uk> wrote:
Hello,
I’m having some problems with BaseX in situations where there it runs out of disk space.
I’m delivering XML files to BaseX 8.4 using RestXQ.
A request is made and the XML is added to a database depending on some criteria. I receive many hundreds of thousands of requests and everything works well unless I run out of disk space.
What I would expect to happen is that BaseX would try to add the document, find it couldn’t due to the disk being full and return a server error (e.g. 500) and the transaction would be aborted leaving the database as it was before the request. Any further requests would all get errors returned.
What I’m seeing is that when a request is made that causes the disk to become full BaseX gets part way through adding the document, fails and returns an error but also leaves the database corrupted. Usually the server has to be restarted and then any database that has been affected has to be deleted - it’s rare to be able to save anything from it.
The errors vary but it is usually a NullPointer exception and then java.nio.channels.OverlappingFileLockException. for further requests.
I’ve tried turning off logging and setting parallel to 1 to see if that made any difference but it doesn’t.
Is what I’m seeing unique to my setup or do others see this?
Is there anything I can do to make BaseX just stop accepting requests when disk space runs out to protect my databases?
Many thanks, James
Hi Christian, Thank you for update. I had assumed that BaseX would make a request for more disk space before trying to write to it and if it couldn’t be allocated would abort the transaction. Luckily in my case there is only one function that ever writes to the databases so your suggested solution is ideal. Basically one copy and paste has got it working! Thank you so much. James
On 1 Apr 2016, at 17:23, Christian Grün <christian.gruen@gmail.com> wrote:
Hi James,
I’m sorry, there is general solution for that on database level; if a disk is full, basically anything can be happen.
However, as you are using RESTXQ, you could check memory via XQuery:
declare namespace file = 'java:java.io.File'; let $free := file:getFreeSpace(file:new('/')) return if($free < 100000000) then error((), "Less than 100 MB left on disk") else ( (: continue :) )
This way, you can also decide what is going to happen if the disk runs out of space (raise an error? send an e-mail? etc.). By the way, raising an error will probably be the most elegant solution, as you can catch the error via %rest:error and do some more things [1].
Hope this helps, Christian
[1] http://docs.basex.org/wiki/RESTXQ#Error_Handling
On Fri, Apr 1, 2016 at 4:48 PM, James Ball <basex-talk@jamesball.co.uk> wrote:
Hello,
I’m having some problems with BaseX in situations where there it runs out of disk space.
I’m delivering XML files to BaseX 8.4 using RestXQ.
A request is made and the XML is added to a database depending on some criteria. I receive many hundreds of thousands of requests and everything works well unless I run out of disk space.
What I would expect to happen is that BaseX would try to add the document, find it couldn’t due to the disk being full and return a server error (e.g. 500) and the transaction would be aborted leaving the database as it was before the request. Any further requests would all get errors returned.
What I’m seeing is that when a request is made that causes the disk to become full BaseX gets part way through adding the document, fails and returns an error but also leaves the database corrupted. Usually the server has to be restarted and then any database that has been affected has to be deleted - it’s rare to be able to save anything from it.
The errors vary but it is usually a NullPointer exception and then java.nio.channels.OverlappingFileLockException. for further requests.
I’ve tried turning off logging and setting parallel to 1 to see if that made any difference but it doesn’t.
Is what I’m seeing unique to my setup or do others see this?
Is there anything I can do to make BaseX just stop accepting requests when disk space runs out to protect my databases?
Many thanks, James
Hi James, I'm glad to hear that the XQuery check solves your problems.
I had assumed that BaseX would make a request for more disk space before trying to write to it and if it couldn’t be allocated would abort the transaction.
True, that'd be nice. As the memory consumption per update varies a lot, depending on the chosen database options, it turned out it's pretty hard to guess how much disk space will eventually be required.
Thank you so much.
You're always welcome, Christian
James
On 1 Apr 2016, at 17:23, Christian Grün <christian.gruen@gmail.com>
wrote:
Hi James,
I’m sorry, there is general solution for that on database level; if a disk is full, basically anything can be happen.
However, as you are using RESTXQ, you could check memory via XQuery:
declare namespace file = 'java:java.io.File'; let $free := file:getFreeSpace(file:new('/')) return if($free < 100000000) then error((), "Less than 100 MB left on disk") else ( (: continue :) )
This way, you can also decide what is going to happen if the disk runs out of space (raise an error? send an e-mail? etc.). By the way, raising an error will probably be the most elegant solution, as you can catch the error via %rest:error and do some more things [1].
Hope this helps, Christian
[1] http://docs.basex.org/wiki/RESTXQ#Error_Handling
On Fri, Apr 1, 2016 at 4:48 PM, James Ball <basex-talk@jamesball.co.uk>
wrote:
Hello,
I’m having some problems with BaseX in situations where there it runs out of disk space.
I’m delivering XML files to BaseX 8.4 using RestXQ.
A request is made and the XML is added to a database depending on some criteria. I receive many hundreds of thousands of requests and everything works well unless I run out of disk space.
What I would expect to happen is that BaseX would try to add the document, find it couldn’t due to the disk being full and return a server error (e.g. 500) and the transaction would be aborted leaving the database as it was before the request. Any further requests would all get errors returned.
What I’m seeing is that when a request is made that causes the disk to become full BaseX gets part way through adding the document, fails and returns an error but also leaves the database corrupted. Usually the server has to be restarted and then any database that has been affected has to be deleted - it’s rare to be able to save anything from it.
The errors vary but it is usually a NullPointer exception and then java.nio.channels.OverlappingFileLockException. for further requests.
I’ve tried turning off logging and setting parallel to 1 to see if that made any difference but it doesn’t.
Is what I’m seeing unique to my setup or do others see this?
Is there anything I can do to make BaseX just stop accepting requests when disk space runs out to protect my databases?
Many thanks, James
participants (4)
-
Christian Grün -
Etanchaud Fabrice -
James Ball -
Strasser Pablo