job:execute with unit:test stalls
Hi BaseX team, The new job:execute function in BaseX 12 is incredibly useful. Using job:execute, it's now possible to have a query that fetches data, dynamically computes the names of databases, creates the databases, populates the databases, and then reads from the same databases, all contained in one query or module!!! This can greatly simplify processes that previously required multiple steps. I'm running into trouble when trying to use %unit:test to test code that uses job:execute. Running the test causes a deadlock, even though the code works correctly when run on its own. Here is a small example based on the example in the documentation https://docs.basex.org/main/Job_Functions#job:execute This query produces "true" when run in BaseX GUI. job:execute('db:create("test")'), job:execute('db:put-value("test", 1 to 10, "numbers")'), job:execute('db:get-value("test", "numbers")') = (1 to 10) After adding %unit:test and unit:assert, when this test is run in BaseX GUI it stalls (until I click Stop to interrupt the process). module namespace t = "test"; declare %unit:test function t:test01 () { job:execute('db:create("test")'), job:execute('db:put-value("test", 1 to 10, "numbers")'), unit:assert(job:execute('db:get-value("test", "numbers")') = (1 to 10)) }; Should this work, or am I missing something? The documentation mentions to be careful to avoid deadlocks with updating queries. I've also found that a deadlock can occur when the main query uses fn:doc even if unrelated to job:execute, but this can be solved by wrapping fn:doc in job:execute. I haven't found a way to work around deadlocks that occur with %unit:test. I'm using BaseX version 12.2. Many thanks, Vincent ______________________________________________ Vincent M. Lizzi Head of Information Standards | Taylor & Francis Group E-Mail: vincent.lizzi@taylorandfrancis.com<mailto:vincent.lizzi@taylorandfrancis.com> Web: www.tandfonline.com<http://www.tandfonline.com/> Time zone: US Eastern Taylor & Francis is a trading name of Informa UK Limited, registered in England under no. 1072954 "Everything should be made as simple as possible, but not simpler." [cid:498c7d70-ddef-4485-9ad7-484411b1c0f5]<https://outlook.office.com/bookwithme/user/aa80d42cbb5b46dba06a5ad241d7665b@taylorandfrancis.com?anonymous&ep=owaSlotsEmailSignature> Book time to meet with me<https://outlook.office.com/bookwithme/user/aa80d42cbb5b46dba06a5ad241d7665b@taylorandfrancis.com?anonymous&ep=owaSlotsEmailSignature> Information Classification: General
Hi Vincent, I’m glad to see that job:execute finds its use in the wild. As you have already discovered, care needs to be taken when updates are performed: Code that is executed with this function is started as a separate query job, which will be registered and run once there is no other updating process. When the parent expression is updating, this causes a deadlock. If tests are executed (via the TEST command, via -t on command line, etc), a global write lock is set – which means that an invocation of job:execute inside the test is doomed to stall. One way out could be to analyze all XQUnit functions and set a more fine granular lock; but I assume your tests will not only perform main-memory operations, but read and possibly write data, too? Best, Christian ________________________________ Von: Lizzi, Vincent via BaseX-Talk <basex-talk@mailman.uni-konstanz.de> Gesendet: Donnerstag, 29. Januar 2026 20:55 An: Christian Grün via BaseX-Talk <basex-talk@mailman.uni-konstanz.de> Betreff: [basex-talk] job:execute with unit:test stalls Hi BaseX team, The new job:execute function in BaseX 12 is incredibly useful. Using job:execute, it's now possible to have a query that fetches data, dynamically computes the names of databases, creates the databases, populates the databases, and then reads from the same databases, all contained in one query or module!!! This can greatly simplify processes that previously required multiple steps. I'm running into trouble when trying to use %unit:test to test code that uses job:execute. Running the test causes a deadlock, even though the code works correctly when run on its own. Here is a small example based on the example in the documentation https://docs.basex.org/main/Job_Functions#job:execute This query produces "true" when run in BaseX GUI. job:execute('db:create("test")'), job:execute('db:put-value("test", 1 to 10, "numbers")'), job:execute('db:get-value("test", "numbers")') = (1 to 10) After adding %unit:test and unit:assert, when this test is run in BaseX GUI it stalls (until I click Stop to interrupt the process). module namespace t = "test"; declare %unit:test function t:test01 () { job:execute('db:create("test")'), job:execute('db:put-value("test", 1 to 10, "numbers")'), unit:assert(job:execute('db:get-value("test", "numbers")') = (1 to 10)) }; Should this work, or am I missing something? The documentation mentions to be careful to avoid deadlocks with updating queries. I've also found that a deadlock can occur when the main query uses fn:doc even if unrelated to job:execute, but this can be solved by wrapping fn:doc in job:execute. I haven't found a way to work around deadlocks that occur with %unit:test. I'm using BaseX version 12.2. Many thanks, Vincent ______________________________________________ Vincent M. Lizzi Head of Information Standards | Taylor & Francis Group E-Mail: vincent.lizzi@taylorandfrancis.com<mailto:vincent.lizzi@taylorandfrancis.com> Web: www.tandfonline.com<http://www.tandfonline.com/> Time zone: US Eastern Taylor & Francis is a trading name of Informa UK Limited, registered in England under no. 1072954 "Everything should be made as simple as possible, but not simpler." [cid:498c7d70-ddef-4485-9ad7-484411b1c0f5]<https://outlook.office.com/bookwithme/user/aa80d42cbb5b46dba06a5ad241d7665b@taylorandfrancis.com?anonymous&ep=owaSlotsEmailSignature> Book time to meet with me<https://outlook.office.com/bookwithme/user/aa80d42cbb5b46dba06a5ad241d7665b@taylorandfrancis.com?anonymous&ep=owaSlotsEmailSignature> Information Classification: General
Hi Christian, Your explanation is helpful. The test scenarios that I have do need to read and write data. Currently it is possible to have a function annotated with %unit:before that performs updates to prepare databases for a %unit:test function. For example: module namespace t = "test"; declare %unit:before("t:test02") %updating function t:test01 () { db:create("test") }; declare %unit:before("t:test03") %updating function t:test02 () { db:put-value("test", 1 to 10, "numbers") }; declare %unit:test function t:test03 () { unit:assert(db:get-value("test", "numbers") = (1 to 10)) }; In a similar way, would it be possible to have a %unit:before function run code that uses job:execute, which may perform updates, and then have a %unit:test function verify the results, and avoid a deadlock? I'm not sure if this would be any easier than analyzing XQUnit functions to set a more fine granular lock as you described. I tried this and it resulted in a deadlock: module namespace t = "test"; declare %unit:before("t:test02") function t:test01 () { job:execute('db:create("test")') }; declare %unit:before("t:test03") function t:test02 () { job:execute('db:put-value("test", 1 to 10, "numbers")') }; declare %unit:test function t:test03 () { unit:assert(db:get-value("test", "numbers")) = (1 to 10) }; Perhaps a different approach could be to create tests using BaseX's client/server architecture: have XQUnit tests run in a BaseX client, and have code under test run in BaseX server facilitated by a set of RESTXQ functions. Many thanks, Vincent _____________________________________________ Vincent M. Lizzi Head of Information Standards | Taylor & Francis Group vincent.lizzi@taylorandfrancis.com<mailto:vincent.lizzi@taylorandfrancis.com> Time zone: US Eastern [cid:ccdbe40b-697a-4a46-8940-d2f45e996071]<https://outlook.office.com/bookwithme/user/aa80d42cbb5b46dba06a5ad241d7665b@taylorandfrancis.com?anonymous&ep=owaSlotsEmailSignature> Book time to meet with me<https://outlook.office.com/bookwithme/user/aa80d42cbb5b46dba06a5ad241d7665b@taylorandfrancis.com?anonymous&ep=owaSlotsEmailSignature> ________________________________ From: Christian Grün <cg@basex.org> Sent: Monday, February 2, 2026 10:37 AM To: Christian Grün via BaseX-Talk <basex-talk@mailman.uni-konstanz.de>; Lizzi, Vincent <Vincent.Lizzi@taylorandfrancis.com> Subject: AW: job:execute with unit:test stalls Hi Vincent, I’m glad to see that job:execute finds its use in the wild. As you have already discovered, care needs to be taken when updates are performed: Code that is executed with this function is started as a separate query job, which will be registered and run once there is no other updating process. When the parent expression is updating, this causes a deadlock. If tests are executed (via the TEST command, via -t on command line, etc), a global write lock is set – which means that an invocation of job:execute inside the test is doomed to stall. One way out could be to analyze all XQUnit functions and set a more fine granular lock; but I assume your tests will not only perform main-memory operations, but read and possibly write data, too? Best, Christian ________________________________ Von: Lizzi, Vincent via BaseX-Talk <basex-talk@mailman.uni-konstanz.de> Gesendet: Donnerstag, 29. Januar 2026 20:55 An: Christian Grün via BaseX-Talk <basex-talk@mailman.uni-konstanz.de> Betreff: [basex-talk] job:execute with unit:test stalls Hi BaseX team, The new job:execute function in BaseX 12 is incredibly useful. Using job:execute, it's now possible to have a query that fetches data, dynamically computes the names of databases, creates the databases, populates the databases, and then reads from the same databases, all contained in one query or module!!! This can greatly simplify processes that previously required multiple steps. I'm running into trouble when trying to use %unit:test to test code that uses job:execute. Running the test causes a deadlock, even though the code works correctly when run on its own. Here is a small example based on the example in the documentation https://docs.basex.org/main/Job_Functions#job:execute<https://docs.basex.org/main/Job_Functions#job:execute> This query produces "true" when run in BaseX GUI. job:execute('db:create("test")'), job:execute('db:put-value("test", 1 to 10, "numbers")'), job:execute('db:get-value("test", "numbers")') = (1 to 10) After adding %unit:test and unit:assert, when this test is run in BaseX GUI it stalls (until I click Stop to interrupt the process). module namespace t = "test"; declare %unit:test function t:test01 () { job:execute('db:create("test")'), job:execute('db:put-value("test", 1 to 10, "numbers")'), unit:assert(job:execute('db:get-value("test", "numbers")') = (1 to 10)) }; Should this work, or am I missing something? The documentation mentions to be careful to avoid deadlocks with updating queries. I've also found that a deadlock can occur when the main query uses fn:doc even if unrelated to job:execute, but this can be solved by wrapping fn:doc in job:execute. I haven't found a way to work around deadlocks that occur with %unit:test. I'm using BaseX version 12.2. Many thanks, Vincent ______________________________________________ Vincent M. Lizzi Head of Information Standards | Taylor & Francis Group E-Mail: vincent.lizzi@taylorandfrancis.com<mailto:vincent.lizzi@taylorandfrancis.com> Web: www.tandfonline.com<http://www.tandfonline.com/> Time zone: US Eastern Taylor & Francis is a trading name of Informa UK Limited, registered in England under no. 1072954 "Everything should be made as simple as possible, but not simpler." [cid:498c7d70-ddef-4485-9ad7-484411b1c0f5]<https://outlook.office.com/bookwithme/user/aa80d42cbb5b46dba06a5ad241d7665b@taylorandfrancis.com?anonymous&ep=owaSlotsEmailSignature> Book time to meet with me<https://outlook.office.com/bookwithme/user/aa80d42cbb5b46dba06a5ad241d7665b@taylorandfrancis.com?anonymous&ep=owaSlotsEmailSignature> Information Classification: General Information Classification: General
Hi Vincent, The current behavior may be better to understand by bearing in mind that every execution of a BaseX command results in a new job that is orchestrated by the transaction manager. The TEST command behaves identically: Before it is started, it is ensured that no other job is running in parallel that would conflict with its execution. As we do not know in advance what tests will be performed (without analyzing all tests in more detail), tests will cause a global write lock. If a XQUnit test includes a job:execute call, it will be registered as new job and scheduled by the transaction management to be run after the test execution – which will never happen because everyone is waiting for good. With the current solution, it is a logical consequence that XQUnit tests can also be run with the client/server architecture (but probably in a different way as you may have guessed): A client can initiate the test execution and the server will run the tests once there are no other jobs running. It is simply a consequence of defining test execution via a BaseX command, and I would assume this has rarely, if ever, been done in practice. One possible long-term change could be to define XQuery testing purely as a command-line based operation and ignore transactions completely (after all, there will never be any concurrent operations in a standalone command-line instance). A drawback of the drafted change would be that it would prevent us in future from introducing something like a unit:execute($url) function. But maybe this is over the top anyway. There are good reasons why we have no XQuery function to launch arbitrary BaseX commands: all this increases the danger of circular dependencies in the workflow, such as the one that we are currently discussing. How do you currently start the test execution? Christian ________________________________ Von: Lizzi, Vincent <Vincent.Lizzi@taylorandfrancis.com> Gesendet: Mittwoch, 4. Februar 2026 03:41 An: Christian Grün <cg@basex.org>; Christian Grün via BaseX-Talk <basex-talk@mailman.uni-konstanz.de> Betreff: Re: job:execute with unit:test stalls Hi Christian, Your explanation is helpful. The test scenarios that I have do need to read and write data. Currently it is possible to have a function annotated with %unit:before that performs updates to prepare databases for a %unit:test function. For example: module namespace t = "test"; declare %unit:before("t:test02") %updating function t:test01 () { db:create("test") }; declare %unit:before("t:test03") %updating function t:test02 () { db:put-value("test", 1 to 10, "numbers") }; declare %unit:test function t:test03 () { unit:assert(db:get-value("test", "numbers") = (1 to 10)) }; In a similar way, would it be possible to have a %unit:before function run code that uses job:execute, which may perform updates, and then have a %unit:test function verify the results, and avoid a deadlock? I'm not sure if this would be any easier than analyzing XQUnit functions to set a more fine granular lock as you described. I tried this and it resulted in a deadlock: module namespace t = "test"; declare %unit:before("t:test02") function t:test01 () { job:execute('db:create("test")') }; declare %unit:before("t:test03") function t:test02 () { job:execute('db:put-value("test", 1 to 10, "numbers")') }; declare %unit:test function t:test03 () { unit:assert(db:get-value("test", "numbers")) = (1 to 10) }; Perhaps a different approach could be to create tests using BaseX's client/server architecture: have XQUnit tests run in a BaseX client, and have code under test run in BaseX server facilitated by a set of RESTXQ functions. Many thanks, Vincent
Hi Christian, I typically run XQUnit tests either by clicking the bug icon in the BaseX GUI or by a command line script that essentially runs org.basex.BaseX with -c "TEST path/to/folder". If needed, a command line script can be used to launch BaseX server before running XQUnit tests. In a CI context, I have a CI script that launches BaseX Server and then runs BaseX XQunit tests that use http:send-request to exercise RESTXQ endpoints provided by the BaseX server. Based on your explanation, it sounds like it would cause problems to allow running XQUnit tests and job:execute in the same instance. Having the XQUnit tests and the code under test that uses job:execute run in separate instances could work though to avoid a deadlock. I just tried this example, and it works! module namespace t = "test"; declare variable $t:host := 'localhost'; declare variable $t:port := 1984; declare variable $t:user := 'admin'; declare variable $t:pass := 'password'; declare %unit:test function t:test01 () { let $c := client:connect($t:host, $t:port, $t:user, $t:pass) let $q := ``[ job:execute('db:create("test")'), job:execute('db:put-value("test", 1 to 10, "numbers")'), job:execute('db:get-value("test", "numbers")') ]`` let $result := client:query($c, $q) return unit:assert($result = (1 to 10)) }; I'll try writing my actual test scenarios like this example. Many thanks, Vincent _____________________________________________ Vincent M. Lizzi Head of Information Standards | Taylor & Francis Group vincent.lizzi@taylorandfrancis.com<mailto:vincent.lizzi@taylorandfrancis.com> Time zone: US Eastern [cid:11e706bd-7970-4599-a405-306573d77a77]<https://outlook.office.com/bookwithme/user/aa80d42cbb5b46dba06a5ad241d7665b@taylorandfrancis.com?anonymous&ep=owaSlotsEmailSignature> Book time to meet with me<https://outlook.office.com/bookwithme/user/aa80d42cbb5b46dba06a5ad241d7665b@taylorandfrancis.com?anonymous&ep=owaSlotsEmailSignature> ________________________________ From: Christian Grün <cg@basex.org> Sent: Wednesday, February 4, 2026 3:29 AM To: Lizzi, Vincent <Vincent.Lizzi@taylorandfrancis.com>; Christian Grün via BaseX-Talk <basex-talk@mailman.uni-konstanz.de> Subject: AW: job:execute with unit:test stalls Hi Vincent, The current behavior may be better to understand by bearing in mind that every execution of a BaseX command results in a new job that is orchestrated by the transaction manager. The TEST command behaves identically: Before it is started, it is ensured that no other job is running in parallel that would conflict with its execution. As we do not know in advance what tests will be performed (without analyzing all tests in more detail), tests will cause a global write lock. If a XQUnit test includes a job:execute call, it will be registered as new job and scheduled by the transaction management to be run after the test execution – which will never happen because everyone is waiting for good. With the current solution, it is a logical consequence that XQUnit tests can also be run with the client/server architecture (but probably in a different way as you may have guessed): A client can initiate the test execution and the server will run the tests once there are no other jobs running. It is simply a consequence of defining test execution via a BaseX command, and I would assume this has rarely, if ever, been done in practice. One possible long-term change could be to define XQuery testing purely as a command-line based operation and ignore transactions completely (after all, there will never be any concurrent operations in a standalone command-line instance). A drawback of the drafted change would be that it would prevent us in future from introducing something like a unit:execute($url) function. But maybe this is over the top anyway. There are good reasons why we have no XQuery function to launch arbitrary BaseX commands: all this increases the danger of circular dependencies in the workflow, such as the one that we are currently discussing. How do you currently start the test execution? Christian ________________________________ Von: Lizzi, Vincent <Vincent.Lizzi@taylorandfrancis.com> Gesendet: Mittwoch, 4. Februar 2026 03:41 An: Christian Grün <cg@basex.org>; Christian Grün via BaseX-Talk <basex-talk@mailman.uni-konstanz.de> Betreff: Re: job:execute with unit:test stalls Hi Christian, Your explanation is helpful. The test scenarios that I have do need to read and write data. Currently it is possible to have a function annotated with %unit:before that performs updates to prepare databases for a %unit:test function. For example: module namespace t = "test"; declare %unit:before("t:test02") %updating function t:test01 () { db:create("test") }; declare %unit:before("t:test03") %updating function t:test02 () { db:put-value("test", 1 to 10, "numbers") }; declare %unit:test function t:test03 () { unit:assert(db:get-value("test", "numbers") = (1 to 10)) }; In a similar way, would it be possible to have a %unit:before function run code that uses job:execute, which may perform updates, and then have a %unit:test function verify the results, and avoid a deadlock? I'm not sure if this would be any easier than analyzing XQUnit functions to set a more fine granular lock as you described. I tried this and it resulted in a deadlock: module namespace t = "test"; declare %unit:before("t:test02") function t:test01 () { job:execute('db:create("test")') }; declare %unit:before("t:test03") function t:test02 () { job:execute('db:put-value("test", 1 to 10, "numbers")') }; declare %unit:test function t:test03 () { unit:assert(db:get-value("test", "numbers")) = (1 to 10) }; Perhaps a different approach could be to create tests using BaseX's client/server architecture: have XQUnit tests run in a BaseX client, and have code under test run in BaseX server facilitated by a set of RESTXQ functions. Many thanks, Vincent Information Classification: General
participants (2)
-
Christian Grün -
Lizzi, Vincent