File-io:error. File lock by another process (windows-only)
Hi, We have this issue only on Windows system since 9.2 that we know of... could be from 902. It is still happening after upgrade to 923. I am not able to provide a sample to reproduce systematically as it never happens at the same .xqm location. It does however always seem that the 1st error message occurs on a line with a file:write or file;append. I am giving you everything I know, in case that it rings a bell. 1. Start server with basexhttp 2. Run a process that has many functions and many file:write and file:append. 3. This generally works properly. The process ends with success. 4. Run the same process again, there is a good chance that we'll get an error that says something like: [file:io-error] Resource "debug-list-imtes.xml' locked by another process. 5. Run the same process again May get file-oi:error at the same location or a different one. 6. Restarting the server brings us back to 1... but sometimes we need to manually delete the files written with function from the file module before we can go back to 1 and get 1 good run of the process. Note: Although the process runs correctly and seems to finish running (to the same point that releases all locks on Linux). The person on the Windows system also often gets: [db:open] Resource "C:\basex\data\staging-en-us\txt.basex locked by another process and then the related database also remains locked and we may get messages: Unexpected error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 9.2.3 Java: Oracle Corporation, 1.8.0_211 OS: Windows 10, amd64 Stack Trace: java.nio.channels.OverlappingFileLockException at sun.nio.ch.SharedFileLockTable.checkList(Unknown Source) at sun.nio.ch.SharedFileLockTable.add(Unknown Source) at sun.nio.ch.FileChannelImpl.tryLock(Unknown Source) at org.basex.io.random.TableDiskAccess.lock(TableDiskAccess.java:147) at org.basex.io.random.TableDiskAccess.(TableDiskAccess.java:79) at org.basex.data.DiskData.init(DiskData.java:125) at org.basex.data.DiskData.(DiskData.java:84) at org.basex.core.cmd.Open.open(Open.java:99) at org.basex.query.QueryResources.database(QueryResources.java:179) at org.basex.query.func.StandardFunc.checkData(StandardFunc.java:429) at org.basex.query.func.db.DbOpen.value(DbOpen.java:17) at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:45) at org.basex.query.expr.path.IterPath$1.init(IterPath.java:79) at org.basex.query.expr.path.IterPath$1.next(IterPath.java:46) at org.basex.query.expr.path.IterPath$1.next(IterPath.java:1) at org.basex.query.iter.Iter.value(Iter.java:65) at org.basex.query.expr.path.IterPath.nodes(IterPath.java:87) at org.basex.query.expr.path.AxisPath.value(AxisPath.java:39) at org.basex.query.expr.gflwor.Let$LetEval.next(Let.java:173) at org.basex.query.expr.gflwor.GFLWOR.value(GFLWOR.java:80) at org.basex.query.func.StaticFunc.invValue(StaticFunc.java:172) at org.basex.query.func.FuncCall.invoke(FuncCall.java:117) at org.basex.query.func.FuncCall.value(FuncCall.java:56) at org.basex.query.scope.MainModule.value(MainModule.java:135) at org.basex.query.QueryContext.update(QueryContext.java:673) at org.basex.query.QueryContext.iter(QueryContext.java:332) at org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:73) at org.basex.http.web.WebResponse.create(WebResponse.java:56) at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:54) at org.basex.http.BaseXServlet.service(BaseXServlet.java:64) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) at java.lang.Thread.run(Unknown Source)undefined -- France Baril Architecte documentaire / Documentation architect france.baril@architextus.com
Hi France, 2. Run a process that has many functions and many file:write and
file:append.
Do multiple processes write to the same files in parallel, or do all of them write to different files? 4. Run the same process again, there is a good chance that we'll get an
error that says something like: [file:io-error] Resource "debug-list-imtes.xml' locked by another process.
Is this one of the files you are writing to? If not, which operations are accessing this file? Christian
Some of the times the process stops on a file to which we write once in the whole process. Sometimes, it's one where we append content sequentially. let $x := write(...) let $x := append(...) The issue can be on any file, even if the process writes to it only once. If the process writes to a file like the example above, it typically crashes on the 1st write, not the following append. Calls are sequential, not parallel. The error messages are always about a file we write to, except the one about database table. We do db:replace() content in that database, but never touch the system files directly. I hope this helps El vie., 12 de julio de 2019 20:30, Christian Grün < christian.gruen@gmail.com> escribió:
Hi France,
2. Run a process that has many functions and many file:write and
file:append.
Do multiple processes write to the same files in parallel, or do all of them write to different files?
4. Run the same process again, there is a good chance that we'll get an
error that says something like: [file:io-error] Resource "debug-list-imtes.xml' locked by another process.
Is this one of the files you are writing to? If not, which operations are accessing this file?
Christian
Sorry, I doubt we’ll be able to track this down without code to reproduce this, or without a closer look at your system. Maybe it’s better to forward this to our commercial support? If the process writes to a file like the example above, it typically
crashes on the 1st write, not the following append.
So this should be reproducible even without append?
It only happens on the windows system. I tried making a small model... I couldn't reproduce... The fact that calling the same process may create a block at different lines makes it a bit aleatory. I'll keep an eye out for patterns, but I've been looking for around a month now. With luck, maybe someone else has a similar issue that can provide more clues. El vie., 12 de julio de 2019 23:12, Christian Grün < christian.gruen@gmail.com> escribió:
Sorry, I doubt we’ll be able to track this down without code to reproduce this, or without a closer look at your system. Maybe it’s better to forward this to our commercial support?
If the process writes to a file like the example above, it typically
crashes on the 1st write, not the following append.
So this should be reproducible even without append?
And about the write without append... it could happen on the 1st operation, but I noticed that in older code and just removed with write as I didn't know yet that we could have that issue with any write or append,.. but that opreation could have been in a recursive loop. So touched by the process more than once. On Fri, Jul 12, 2019 at 11:45 PM France Baril <france.baril@architextus.com> wrote:
It only happens on the windows system. I tried making a small model... I couldn't reproduce... The fact that calling the same process may create a block at different lines makes it a bit aleatory.
I'll keep an eye out for patterns, but I've been looking for around a month now.
With luck, maybe someone else has a similar issue that can provide more clues.
El vie., 12 de julio de 2019 23:12, Christian Grün < christian.gruen@gmail.com> escribió:
Sorry, I doubt we’ll be able to track this down without code to reproduce this, or without a closer look at your system. Maybe it’s better to forward this to our commercial support?
If the process writes to a file like the example above, it typically
crashes on the 1st write, not the following append.
So this should be reproducible even without append?
-- France Baril Architecte documentaire / Documentation architect france.baril@architextus.com
If all calls are exclusively sequential, such a lock can only appear if a file is not closed after a write operation – and this is actually ensured by the implementation of all functions of the File Module. If the calls are not strictly sequential (e.g. because RESTXQ is used, or because you have more than a single client), you may need to use query locks [1] in order to ensure that your code won’t be executed in parallel. For more hints, we’ll need to look at your code and possibly your setup. [1] http://docs.basex.org/wiki/Transaction_Management#XQuery_Locks France Baril <france.baril@architextus.com> schrieb am Sa., 13. Juli 2019, 00:51:
And about the write without append... it could happen on the 1st operation, but I noticed that in older code and just removed with write as I didn't know yet that we could have that issue with any write or append,.. but that opreation could have been in a recursive loop. So touched by the process more than once.
On Fri, Jul 12, 2019 at 11:45 PM France Baril < france.baril@architextus.com> wrote:
It only happens on the windows system. I tried making a small model... I couldn't reproduce... The fact that calling the same process may create a block at different lines makes it a bit aleatory.
I'll keep an eye out for patterns, but I've been looking for around a month now.
With luck, maybe someone else has a similar issue that can provide more clues.
El vie., 12 de julio de 2019 23:12, Christian Grün < christian.gruen@gmail.com> escribió:
Sorry, I doubt we’ll be able to track this down without code to reproduce this, or without a closer look at your system. Maybe it’s better to forward this to our commercial support?
If the process writes to a file like the example above, it typically
crashes on the 1st write, not the following append.
So this should be reproducible even without append?
-- France Baril Architecte documentaire / Documentation architect france.baril@architextus.com
I'll take a closer look next week. Would that apply if this is tested on a local computer with a single user that we know only calls the process once? We know for sure the process is not launched multiple times. On Fri, Jul 12, 2019 at 11:59 PM Christian Grün <christian.gruen@gmail.com> wrote:
If all calls are exclusively sequential, such a lock can only appear if a file is not closed after a write operation – and this is actually ensured by the implementation of all functions of the File Module.
If the calls are not strictly sequential (e.g. because RESTXQ is used, or because you have more than a single client), you may need to use query locks [1] in order to ensure that your code won’t be executed in parallel.
For more hints, we’ll need to look at your code and possibly your setup.
[1] http://docs.basex.org/wiki/Transaction_Management#XQuery_Locks
France Baril <france.baril@architextus.com> schrieb am Sa., 13. Juli 2019, 00:51:
And about the write without append... it could happen on the 1st operation, but I noticed that in older code and just removed with write as I didn't know yet that we could have that issue with any write or append,.. but that opreation could have been in a recursive loop. So touched by the process more than once.
On Fri, Jul 12, 2019 at 11:45 PM France Baril < france.baril@architextus.com> wrote:
It only happens on the windows system. I tried making a small model... I couldn't reproduce... The fact that calling the same process may create a block at different lines makes it a bit aleatory.
I'll keep an eye out for patterns, but I've been looking for around a month now.
With luck, maybe someone else has a similar issue that can provide more clues.
El vie., 12 de julio de 2019 23:12, Christian Grün < christian.gruen@gmail.com> escribió:
Sorry, I doubt we’ll be able to track this down without code to reproduce this, or without a closer look at your system. Maybe it’s better to forward this to our commercial support?
If the process writes to a file like the example above, it typically
crashes on the 1st write, not the following append.
So this should be reproducible even without append?
-- France Baril Architecte documentaire / Documentation architect france.baril@architextus.com
-- France Baril Architecte documentaire / Documentation architect france.baril@architextus.com
Hi, We think we found the issue. It seems that basexhttp.bat was launching multiple instance of basex. It's the second time we see that on Windows systems. Is it possible that the Windows installation process sets environment variables which creates conflicts if other instances are installed using the .zip files? I was on Mac for years before switching to Linux and have never had this issue. I'm trying to figure out how this happens so I can advise my colleagues on Windows better. Thanks,
Hi France,
It seems that basexhttp.bat was launching multiple instance of basex.
We have multiple Windows instances running with BaseX, but I cannot recollect having had such an issue before (and I’m not sure how this could happen, as the code for starting the server is pretty straightforward). Maybe you had other instances running? Did you really call basexhttp only once? If yes, which flags do you supply as arguments?
It's the second time we see that on Windows systems. Is it possible that the Windows installation process sets environment variables which creates conflicts if other instances are installed using the .zip files?
If you use the Windows installer, the PATH variable will be modified, such that basex will be found on command line. Best, Christian
Thanks... I'll pass this along to the person managing the Windows installations. I did make sure over web sessions that they only started basex and batch processes once. Issue closed. On Mon, Jul 22, 2019 at 12:32 PM Christian Grün <christian.gruen@gmail.com> wrote:
Hi France,
It seems that basexhttp.bat was launching multiple instance of basex.
We have multiple Windows instances running with BaseX, but I cannot recollect having had such an issue before (and I’m not sure how this could happen, as the code for starting the server is pretty straightforward).
Maybe you had other instances running? Did you really call basexhttp only once? If yes, which flags do you supply as arguments?
It's the second time we see that on Windows systems. Is it possible that the Windows installation process sets environment variables which creates conflicts if other instances are installed using the .zip files?
If you use the Windows installer, the PATH variable will be modified, such that basex will be found on command line.
Best, Christian
-- France Baril Architecte documentaire / Documentation architect france.baril@architextus.com
participants (2)
-
Christian Grün -
France Baril