so if you call store:get, store:put or store:write in the first process,
a second process will not wait until the store operations are completed.
In non XQuery contexts a semaphore [1] might be used to ensure that my other threads don't get between a get and put. In the spirit of blurring the XQuery Java boundaries I tried [2]. It seems to work. Is it dangerous?
[1] https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Semaphore.htm...
[2] declare namespace Semphore ="java:java.util.concurrent.Semaphore";
declare function local:config-update($k as xs:string,$v as item(),$sem) { Semphore:acquire($sem), try{ let $u:=store:get("config")=> map:put($k, $v) return store:put("config",$u) }catch * { trace("Errrr",$err:description) } ,Semphore:release($sem) };
let $sem:=Semphore:new(1,true())
let $s1 := store:put("config", map{}) let $s2 := xquery:fork-join( for $i in (1 to 100) return function(){ let $r:=(prof:sleep(10),$i) return local:config-update( string($i),$r,$sem) } )
return count(map:keys(store:get("config")))
On Tue, 28 Jan 2025 at 14:09, Marco Lettere m.lettere@gmail.com wrote:
Ok, thanks for the clarification.
M. On 28/01/25 15:08, Christian Grün wrote:
Sorry Christian, do you mean *not* synchronized?
With »synchronized«, I meant to refer to a lower level: You will not end up with a corrupt key/value store or with I/O conflicts when accessing and updating the store via multiple threads. However, as you have already observed, multiple operations are not executed in a well-defined order, so if you call store:get, store:put or store:write in the first process, a second process will not wait until the store operations are completed.
Oh, wow. Looks great Andy. Thanks for suggestion.
Wonder to know what's Christian's opinion on this.
M.
On 07/02/25 16:45, Andy Bunce wrote:
so if you call store:get, store:put or store:write in the first process, a
second process will not wait until the store operations are completed.
In non XQuery contexts a semaphore [1] might be used to ensure that my other threads don't get between a get and put. In the spirit of blurring the XQuery Java boundaries I tried [2]. It seems to work. Is it dangerous?
[1] https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Semaphore.htm...
[2] declare namespace Semphore ="java:java.util.concurrent.Semaphore";
declare function local:config-update($k as xs:string,$v as item(),$sem) { Semphore:acquire($sem), try{ let $u:=store:get("config")=> map:put($k, $v) return store:put("config",$u) }catch * { trace("Errrr",$err:description) } ,Semphore:release($sem) };
let $sem:=Semphore:new(1,true())
let $s1 := store:put("config", map{}) let $s2 := xquery:fork-join( for $i in (1 to 100) return function(){ let $r:=(prof:sleep(10),$i) return local:config-update( string($i),$r,$sem) } )
return count(map:keys(store:get("config")))
On Tue, 28 Jan 2025 at 14:09, Marco Lettere m.lettere@gmail.com wrote:
Ok, thanks for the clarification. M. On 28/01/25 15:08, Christian Grün wrote:Sorry Christian, do you mean *not* synchronized? With »synchronized«, I meant to refer to a lower level: You will not end up with a corrupt key/value store or with I/O conflicts when accessing and updating the store via multiple threads. However, as you have already observed, multiple operations are not executed in a well-defined order, so if you call store:get, store:put or store:write in the first process, a second process will not wait until the store operations are completed.
One problem with this might be that: When using the fork-join function it is easy to ensure all the threads have a reference to the *same* semaphore. If these were arbitrary BaseX "jobs" it is not clear how this could be done without explicit support from the BaseX runtime. Perhaps it could be done with a new annotation %basex:semaphore ("my-semaphore") that could be applied to functions.
/Andy
On Fri, 7 Feb 2025 at 18:31, Marco Lettere m.lettere@gmail.com wrote:
Oh, wow. Looks great Andy. Thanks for suggestion.
Wonder to know what's Christian's opinion on this.
M. On 07/02/25 16:45, Andy Bunce wrote:
so if you call store:get, store:put or store:write in the first process,
a second process will not wait until the store operations are completed.
In non XQuery contexts a semaphore [1] might be used to ensure that my other threads don't get between a get and put. In the spirit of blurring the XQuery Java boundaries I tried [2]. It seems to work. Is it dangerous?
[1] https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Semaphore.htm...
[2] declare namespace Semphore ="java:java.util.concurrent.Semaphore";
declare function local:config-update($k as xs:string,$v as item(),$sem) { Semphore:acquire($sem), try{ let $u:=store:get("config")=> map:put($k, $v) return store:put("config",$u) }catch * { trace("Errrr",$err:description) } ,Semphore:release($sem) };
let $sem:=Semphore:new(1,true())
let $s1 := store:put("config", map{}) let $s2 := xquery:fork-join( for $i in (1 to 100) return function(){ let $r:=(prof:sleep(10),$i) return local:config-update( string($i),$r,$sem) } )
return count(map:keys(store:get("config")))
On Tue, 28 Jan 2025 at 14:09, Marco Lettere m.lettere@gmail.com wrote:
Ok, thanks for the clarification.
M. On 28/01/25 15:08, Christian Grün wrote:
Sorry Christian, do you mean *not* synchronized?
With »synchronized«, I meant to refer to a lower level: You will not end up with a corrupt key/value store or with I/O conflicts when accessing and updating the store via multiple threads. However, as you have already observed, multiple operations are not executed in a well-defined order, so if you call store:get, store:put or store:write in the first process, a second process will not wait until the store operations are completed.
For us it can be solved with custom annotations or even constant strings because it all comes down to the one workflow engine ...
Il sab 8 feb 2025, 11:55 Andy Bunce bunce.andy@gmail.com ha scritto:
One problem with this might be that: When using the fork-join function it is easy to ensure all the threads have a reference to the *same* semaphore. If these were arbitrary BaseX "jobs" it is not clear how this could be done without explicit support from the BaseX runtime. Perhaps it could be done with a new annotation %basex:semaphore ("my-semaphore") that could be applied to functions.
/Andy
On Fri, 7 Feb 2025 at 18:31, Marco Lettere m.lettere@gmail.com wrote:
Oh, wow. Looks great Andy. Thanks for suggestion.
Wonder to know what's Christian's opinion on this.
M. On 07/02/25 16:45, Andy Bunce wrote:
so if you call store:get, store:put or store:write in the first process,
a second process will not wait until the store operations are completed.
In non XQuery contexts a semaphore [1] might be used to ensure that my other threads don't get between a get and put. In the spirit of blurring the XQuery Java boundaries I tried [2]. It seems to work. Is it dangerous?
[1] https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Semaphore.htm...
[2] declare namespace Semphore ="java:java.util.concurrent.Semaphore";
declare function local:config-update($k as xs:string,$v as item(),$sem) { Semphore:acquire($sem), try{ let $u:=store:get("config")=> map:put($k, $v) return store:put("config",$u) }catch * { trace("Errrr",$err:description) } ,Semphore:release($sem) };
let $sem:=Semphore:new(1,true())
let $s1 := store:put("config", map{}) let $s2 := xquery:fork-join( for $i in (1 to 100) return function(){ let $r:=(prof:sleep(10),$i) return local:config-update( string($i),$r,$sem) } )
return count(map:keys(store:get("config")))
On Tue, 28 Jan 2025 at 14:09, Marco Lettere m.lettere@gmail.com wrote:
Ok, thanks for the clarification.
M. On 28/01/25 15:08, Christian Grün wrote:
Sorry Christian, do you mean *not* synchronized?
With »synchronized«, I meant to refer to a lower level: You will not end up with a corrupt key/value store or with I/O conflicts when accessing and updating the store via multiple threads. However, as you have already observed, multiple operations are not executed in a well-defined order, so if you call store:get, store:put or store:write in the first process, a second process will not wait until the store operations are completed.
Hi there,
The semaphore code looks fine. As Andy noted, things get much more complicated when the code needs to run across multiple concurrent jobs: We might easily run into deadlocks.
Thus, if it turns out that concurrency gets a bigger issue, we would probably embed key/value store updates into our Pending Update List concept, even if it would make it less flexible to use.
Cheers, Christian
On Sat, Feb 8, 2025 at 12:07 PM Marco Lettere m.lettere@gmail.com wrote:
For us it can be solved with custom annotations or even constant strings because it all comes down to the one workflow engine ...
Il sab 8 feb 2025, 11:55 Andy Bunce bunce.andy@gmail.com ha scritto:
One problem with this might be that: When using the fork-join function it is easy to ensure all the threads have a reference to the *same* semaphore. If these were arbitrary BaseX "jobs" it is not clear how this could be done without explicit support from the BaseX runtime. Perhaps it could be done with a new annotation %basex:semaphore ("my-semaphore") that could be applied to functions.
/Andy
On Fri, 7 Feb 2025 at 18:31, Marco Lettere m.lettere@gmail.com wrote:
Oh, wow. Looks great Andy. Thanks for suggestion.
Wonder to know what's Christian's opinion on this.
M. On 07/02/25 16:45, Andy Bunce wrote:
so if you call store:get, store:put or store:write in the first
process, a second process will not wait until the store operations are completed.
In non XQuery contexts a semaphore [1] might be used to ensure that my other threads don't get between a get and put. In the spirit of blurring the XQuery Java boundaries I tried [2]. It seems to work. Is it dangerous?
[1] https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Semaphore.htm...
[2] declare namespace Semphore ="java:java.util.concurrent.Semaphore";
declare function local:config-update($k as xs:string,$v as item(),$sem) { Semphore:acquire($sem), try{ let $u:=store:get("config")=> map:put($k, $v) return store:put("config",$u) }catch * { trace("Errrr",$err:description) } ,Semphore:release($sem) };
let $sem:=Semphore:new(1,true())
let $s1 := store:put("config", map{}) let $s2 := xquery:fork-join( for $i in (1 to 100) return function(){ let $r:=(prof:sleep(10),$i) return local:config-update( string($i),$r,$sem) } )
return count(map:keys(store:get("config")))
On Tue, 28 Jan 2025 at 14:09, Marco Lettere m.lettere@gmail.com wrote:
Ok, thanks for the clarification.
M. On 28/01/25 15:08, Christian Grün wrote:
Sorry Christian, do you mean *not* synchronized?
With »synchronized«, I meant to refer to a lower level: You will not end up with a corrupt key/value store or with I/O conflicts when accessing and updating the store via multiple threads. However, as you have already observed, multiple operations are not executed in a well-defined order, so if you call store:get, store:put or store:write in the first process, a second process will not wait until the store operations are completed.
No please. The store module is perfect as it is for a lot of out-of-band activities. If this can not be implemented in an optional way leave it off. Thanks.
I'll try to go with Andy's suggestion or resolve to using a DB in the critical cases.
Regards,
Marco.
On 22/02/25 11:38, Christian Grün wrote:
Hi there,
The semaphore code looks fine. As Andy noted, things get much more complicated when the code needs to run across multiple concurrent jobs: We might easily run into deadlocks.
Thus, if it turns out that concurrency gets a bigger issue, we would probably embed key/value store updates into our Pending Update List concept, even if it would make it less flexible to use.
Cheers, Christian
On Sat, Feb 8, 2025 at 12:07 PM Marco Lettere m.lettere@gmail.com wrote:
For us it can be solved with custom annotations or even constant strings because it all comes down to the one workflow engine ... Il sab 8 feb 2025, 11:55 Andy Bunce <bunce.andy@gmail.com> ha scritto: One problem with this might be that: When using the fork-join function it is easy to ensure all the threads have a reference to the *same* semaphore. If these were arbitrary BaseX "jobs" it is not clear how this could be done without explicit support from the BaseX runtime. Perhaps it could be done with a new annotation %basex:semaphore ("my-semaphore") that could be applied to functions. /Andy On Fri, 7 Feb 2025 at 18:31, Marco Lettere <m.lettere@gmail.com> wrote: Oh, wow. Looks great Andy. Thanks for suggestion. Wonder to know what's Christian's opinion on this. M. On 07/02/25 16:45, Andy Bunce wrote:>so if you call store:get, store:put or store:write in the first process, a second process will not wait until the store operations are completed. In non XQuery contexts a semaphore [1] might be used to ensure that my other threads don't get between a get and put. In the spirit of blurring the XQuery Java boundaries I tried [2]. It seems to work. Is it dangerous? [1] https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Semaphore.html [2] declare namespace Semphore ="java:java.util.concurrent.Semaphore"; declare function local:config-update($k as xs:string,$v as item(),$sem) { Semphore:acquire($sem), try{ let $u:=store:get("config")=> map:put($k, $v) return store:put("config",$u) }catch * { trace("Errrr",$err:description) } ,Semphore:release($sem) }; let $sem:=Semphore:new(1,true()) let $s1 := store:put("config", map{}) let $s2 := xquery:fork-join( for $i in (1 to 100) return function(){ let $r:=(prof:sleep(10),$i) return local:config-update( string($i),$r,$sem) } ) return count(map:keys(store:get("config"))) On Tue, 28 Jan 2025 at 14:09, Marco Lettere <m.lettere@gmail.com> wrote: Ok, thanks for the clarification. M. On 28/01/25 15:08, Christian Grün wrote:Sorry Christian, do you mean *not* synchronized? With »synchronized«, I meant to refer to a lower level: You will not end up with a corrupt key/value store or with I/O conflicts when accessing and updating the store via multiple threads. However, as you have already observed, multiple operations are not executed in a well-defined order, so if you call store:get, store:put or store:write in the first process, a second process will not wait until the store operations are completed.
basex-talk@mailman.uni-konstanz.de
-
Andy Bunce -
Christian Grün -
Marco Lettere