Updates failing due to "opened by another client" on a system running only basexhttp
Howdy -- I recently had an update run via the REST interface fail due to the database being "opened by another client". This would be unsurprising were it taking place on a workstation where a concurrent process such as basexgui might have been running, but the host in question was a server, with its basexhttp instance controlled by system init scripts, running as a dedicated user, and otherwise unlikely to be interfered with by a separate process. There are, however, multiple concurrent REST clients using that same HTTP server. As basexhttp should be the only service directly accessing the underlying database on this host, this surprises me -- my expectation is that the transaction monitor would defer any execution of an update until a full lock could be grabbed, preventing it from interfering with other read or update actions happening within the same process. Is this expectation valid? If not, what do I need to do to be able to apply updates to a database with minimal disruption to readers?
Am Donnerstag, 29. März 2012, 16:23:12 schrieb Charles Duffy:
Howdy --
I recently had an update run via the REST interface fail due to the database being "opened by another client". This would be unsurprising were it taking place on a workstation where a concurrent process such as basexgui might have been running, but the host in question was a server, with its basexhttp instance controlled by system init scripts, running as a dedicated user, and otherwise unlikely to be interfered with by a separate process. There are, however, multiple concurrent REST clients using that same HTTP server.
As basexhttp should be the only service directly accessing the underlying database on this host, this surprises me -- my expectation is that the transaction monitor would defer any execution of an update until a full lock could be grabbed, preventing it from interfering with other read or update actions happening within the same process.
Is this expectation valid? If not, what do I need to do to be able to apply updates to a database with minimal disruption to readers?
hm, actually I think it's a bug: if a running REST request is interrupted, while another one is trying to acquire a write lock on the same database, then the database is left open. I can reproduce it as follows: $ curl -i -X GET "admin:admin@localhost:8984/rest/testdb?query=(1%20to%2010000000000000)" While the first query is running start the following: $ curl -i -X PUT -T "test.xml" "admin:admin@localhost:8984/rest/testdb" The PUT request waits, which is ok, but if the GET request is interrupted, the PUT receives: "Database 'testdb' is currently opened by another client." Subsequent attempts to execute the PUT request fail with the same error. Regards, Dimitar
Hi, Am 30.03.2012 00:03, schrieb Dimitar Popov:
The PUT request waits, which is ok, but if the GET request is interrupted, the PUT receives:
"Database 'testdb' is currently opened by another client."
Subsequent attempts to execute the PUT request fail with the same error.
that seems to be the per-database pin file again that I mistook for the (similar, but distinct) write lock last time. The current approach is simple and effective, but it doesn't work at all if the BaseX instance is terminated before it can delete the pin files. That's actually what we want for the write lock, because terminating the instance while it writes the updates to disk pretty much guarantees curruption, but it's too strict for marking opened databases. My first idea would be to try and lock the main table of the database, `tbl.basex`, via FileChannel.tryLock(). That should be just as effective in preventing concurrent access, with the added benefit of leaving no orphaned files behind when the BaseX instance isn't terminated gracefully. @Christian, could you elaborate on the outcome of the tests you did with file locks? I'll look into it, too. Cheers, Leo P.S.: You can render the database usable again by deleting the pin-* file by hand.
Hi,
My first idea would be to try and lock the main table of the database, `tbl.basex`, via FileChannel.tryLock(). That should be just as effective in preventing concurrent access, with the added benefit of leaving no orphaned files behind when the BaseX instance isn't terminated gracefully.
yes, I think we should rather go for the lock approach. Still, I'd prefer to create a separate file for each process accessing the database; otherwise, we won't know when to unlock the existing files.
P.S.: You can render the database usable again by deleting the pin-* file by hand.
I have just added a section on lock and pin files to our documentation: http://docs.basex.org/wiki/Transaction_Management#Locking As Dimitar indicated, the problem you stumbled upon is a bug (this is a positive side effect of the introduced pin files: we have already found two other bugs related to databases that had not been properly closed). Christian
2012/3/30 Christian Grün <christian.gruen@gmail.com>
Hi,
My first idea would be to try and lock the main table of the database, `tbl.basex`, via FileChannel.tryLock(). That should be just as effective in preventing concurrent access, with the added benefit of leaving no orphaned files behind when the BaseX instance isn't terminated gracefully.
yes, I think we should rather go for the lock approach. Still, I'd prefer to create a separate file for each process accessing the database; otherwise, we won't know when to unlock the existing files.
I'm sorry -- I don't understand this objection. flock()-style shared locks are removed automatically as soon as the file handle is closed, even if the file they're held on still exists, and only after the last one of them is removed can an exclusive lock on the same file be granted. Could you explain what you mean by knowing when to unlock the existing files? (Also, dealing with locks spread across multiple files tends to be, by nature, a bit more prone to race conditions -- often, one has to leverage directory rather than file semantics to do it safely, but even then it's more work across platforms).
Thanks for the teaching lesson on shared locks, which just makes it easier, and obviously makes more sense. Am 30.03.2012 12:38 schrieb "Charles Duffy" <charles@dyfis.net>:
2012/3/30 Christian Grün <christian.gruen@gmail.com>
Hi,
My first idea would be to try and lock the main table of the database, `tbl.basex`, via FileChannel.tryLock(). That should be just as effective in preventing concurrent access, with the added benefit of leaving no orphaned files behind when the BaseX instance isn't terminated gracefully.
yes, I think we should rather go for the lock approach. Still, I'd prefer to create a separate file for each process accessing the database; otherwise, we won't know when to unlock the existing files.
I'm sorry -- I don't understand this objection.
flock()-style shared locks are removed automatically as soon as the file handle is closed, even if the file they're held on still exists, and only after the last one of them is removed can an exclusive lock on the same file be granted. Could you explain what you mean by knowing when to unlock the existing files?
(Also, dealing with locks spread across multiple files tends to be, by nature, a bit more prone to race conditions -- often, one has to leverage directory rather than file semantics to do it safely, but even then it's more work across platforms).
participants (4)
-
Charles Duffy -
Christian Grün -
Dimitar Popov -
Leo Wörteler