Running the following in the GUI (8.6.5) ``` let $db:="BEP" return count(collection($db)) ``` The info windows shows $db is inlined
Compiling: - inline $db_0 - pre-evaluate collection("BEP") to document-node() sequence - pre-evaluate count((db:open-pre("BEP",0), ...)) to xs:integer - simplify gflwor Optimized Query: 9193
All good.
Submitting a job ``` let $q:='let $db:="BEP" return count(collection($db))' return jobs:eval($q,(),map{"cache":true()})!(jobs:wait(.),jobs:list-details(.)) ``` returns: <job id="job90" type="ScheduledXQuery" state="cached" user="admin" duration="PT0.007S" reads="(global)" writes="(none)">let $db:="BEP" return count(collection($db))</job>
Note @reads="(global)"
But ``` let $q:='count(collection("BEP"))' return jobs:eval($q,(),map{"cache":true()})!(jobs:wait(.),jobs:list-details(.)) ``` <job id="job92" type="ScheduledXQuery" state="cached" user="admin" duration="PT0.006S" reads="BEP" writes="(none)">count(collection("BEP"))</job>
Note @reads="BEP"
Is this correct and/or of any consequence?
/Andy
Hi Andy,
Note @reads="BEP"
Is this correct and/or of any consequence?
This is correct: If the argument of fn:collection, fn:doc, etc. is directly supplied as string argument, it is possible to statically detect (at parse time) which database needs to be locked. Variables will only be inlined at compile time, i.e., after the locks have already been assigned. As a result, your first query will cause a global lock.
Currently, we only have one compilation step. In an ideal BaseX world, we would have at least 2 steps:
1. Static compilation, in which a query would be typed and rewritten without considering dynamic input (external variables, databases, …). The result could be reused several times, and in your case, the database name would have been inlined.
2. Dynamic compilation, which binds external variables (from clients, RESTXQ calls, etc.), checks databases for index structures, etc.
It might turn out that Step 1 is too expensive in order to be evaluated before locking a query. On the other hand, static compilation needs not be locked, it can always run in parallel with queries that are currently evaluated.
Does this answer your question? Christian
Thanks for the explanation.
Wanting to parameterise the input collections, in fairly simple ways, seems to me a natural and common requirement. but the consequence of a global read lock is all other tasks are unable to access any database. So great care required ;-)
Variables will only be inlined at compile time, i.e., after the locks have
already been assigned.
Consider a function:
bxcode:inline-variables($xqcode-uri) as xs:string @returns xquery code that is the source code from $xqcode-uri but with inlining "constant" variables and operations applied
This could be used as a preprocessor. Is this something that could be created from the bits and pieces in the current BaseX Java code?
/Andy
On 20 September 2017 at 18:27, Christian Grün christian.gruen@gmail.com wrote:
Hi Andy,
Note @reads="BEP"
Is this correct and/or of any consequence?
This is correct: If the argument of fn:collection, fn:doc, etc. is directly supplied as string argument, it is possible to statically detect (at parse time) which database needs to be locked. Variables will only be inlined at compile time, i.e., after the locks have already been assigned. As a result, your first query will cause a global lock.
Currently, we only have one compilation step. In an ideal BaseX world, we would have at least 2 steps:
- Static compilation, in which a query would be typed and rewritten
without considering dynamic input (external variables, databases, …). The result could be reused several times, and in your case, the database name would have been inlined.
- Dynamic compilation, which binds external variables (from clients,
RESTXQ calls, etc.), checks databases for index structures, etc.
It might turn out that Step 1 is too expensive in order to be evaluated before locking a query. On the other hand, static compilation needs not be locked, it can always run in parallel with queries that are currently evaluated.
Does this answer your question? Christian
bxcode:inline-variables($xqcode-uri) as xs:string @returns xquery code that is the source code from $xqcode-uri but with inlining "constant" variables and operations applied
This could be used as a preprocessor. Is this something that could be created from the bits and pieces in the current BaseX Java code?
This reminds me of the discussion with Omar [1]: There is currently no way to generate a correct string representation for compiled code. In the GUI Info View, we output a string representation of the optimized query. In simple cases, this string is a valid and equivalent to the original query string, but it’s fairly easy to generate invalid results.
If it’s an option for you to generate code in XQuery, something like the following could be done:
_ bxcode.xqm _______
module namespace bxcode = 'bxcode'; declare function bxcode:bind-external-variables( $uri as xs:string, $vars as map(xs:string, xs:string) ) as xs:string { let $xqcode := unparsed-text($uri) return fold-left( map:keys($vars), $xqcode, function($string, $name) { let $value := replace($vars($name), ']``', '`{"]``"}`', 'q') return $string => replace('declare variable $' || $name || ' external;', '', 'q') => replace('$' || $name, '``[' || $value || ']``', 'q') } ) };
_ example.xq _______
declare variable $db external; count(collection($db))
_ run-bxcode.xq _______
import module namespace bxcode = 'bxcode' at 'bxcode.xqm'; bxcode:bind-external-variables('query.xq', map { 'db': 'BEP' })
_ output _______
count(collection(``[BEP]``))
Cheers, Christian
[1] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg09863.htm...
The use of String Constructor does not avoid the global read lock, but I can see with some changes and a suitable set of conventions this could be made to work.
Cheers /Andy
<job id="job13" type="ScheduledXQuery" state="cached" user="admin" duration="PT0.007S" reads="(global)" writes="(none)">count( collection(``[BEP]``))</job>
On 20 September 2017 at 23:28, Christian Grün christian.gruen@gmail.com wrote:
bxcode:inline-variables($xqcode-uri) as xs:string @returns xquery code that is the source code from $xqcode-uri but with inlining "constant" variables and operations applied
This could be used as a preprocessor. Is this something that could be created from the bits and pieces in the current BaseX Java code?
This reminds me of the discussion with Omar [1]: There is currently no way to generate a correct string representation for compiled code. In the GUI Info View, we output a string representation of the optimized query. In simple cases, this string is a valid and equivalent to the original query string, but it’s fairly easy to generate invalid results.
If it’s an option for you to generate code in XQuery, something like the following could be done:
_ bxcode.xqm _______
module namespace bxcode = 'bxcode'; declare function bxcode:bind-external-variables( $uri as xs:string, $vars as map(xs:string, xs:string) ) as xs:string { let $xqcode := unparsed-text($uri) return fold-left( map:keys($vars), $xqcode, function($string, $name) { let $value := replace($vars($name), ']``', '`{"]``"}`', 'q') return $string => replace('declare variable $' || $name || ' external;', '', 'q') => replace('$' || $name, '``[' || $value || ']``', 'q') } ) };
_ example.xq _______
declare variable $db external; count(collection($db))
_ run-bxcode.xq _______
import module namespace bxcode = 'bxcode' at 'bxcode.xqm'; bxcode:bind-external-variables('query.xq', map { 'db': 'BEP' })
_ output _______
count(collection(``[BEP]``))
Cheers, Christian
[1] https://www.mail-archive.com/basex-talk@mailman.uni- konstanz.de/msg09863.html
The use of String Constructor does not avoid the global read lock
Good to know, thanks. It now does [1], provided that the constructor consists of a single static strings.
[1] http://files.basex.org/releases/latest/
<job id="job13" type="ScheduledXQuery" state="cached" user="admin" duration="PT0.007S" reads="(global)" writes="(none)">count(collection(``[BEP]``))</job>
On 20 September 2017 at 23:28, Christian Grün christian.gruen@gmail.com wrote:
bxcode:inline-variables($xqcode-uri) as xs:string @returns xquery code that is the source code from $xqcode-uri but with inlining "constant" variables and operations applied
This could be used as a preprocessor. Is this something that could be created from the bits and pieces in the current BaseX Java code?
This reminds me of the discussion with Omar [1]: There is currently no way to generate a correct string representation for compiled code. In the GUI Info View, we output a string representation of the optimized query. In simple cases, this string is a valid and equivalent to the original query string, but it’s fairly easy to generate invalid results.
If it’s an option for you to generate code in XQuery, something like the following could be done:
_ bxcode.xqm _______
module namespace bxcode = 'bxcode'; declare function bxcode:bind-external-variables( $uri as xs:string, $vars as map(xs:string, xs:string) ) as xs:string { let $xqcode := unparsed-text($uri) return fold-left( map:keys($vars), $xqcode, function($string, $name) { let $value := replace($vars($name), ']``', '`{"]``"}`', 'q') return $string => replace('declare variable $' || $name || ' external;', '', 'q') => replace('$' || $name, '``[' || $value || ']``', 'q') } ) };
_ example.xq _______
declare variable $db external; count(collection($db))
_ run-bxcode.xq _______
import module namespace bxcode = 'bxcode' at 'bxcode.xqm'; bxcode:bind-external-variables('query.xq', map { 'db': 'BEP' })
_ output _______
count(collection(``[BEP]``))
Cheers, Christian
[1] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg09863.htm...
Using 9.1.2
let $d:=db:open(``[BEP]``) return db:create(``[BEP-staging]``,$d,$d!base-uri(.))
The GUI query info reports: - Read Locking: (none) - Write Locking: BEP,BEP-staging
I expected BEP-staging to appear only in the Read locking, as it does if I remove the db:create call Is this correct?
/Andy
On Thu, 21 Sep 2017 at 16:26, Christian Grün christian.gruen@gmail.com wrote:
The use of String Constructor does not avoid the global read lock
Good to know, thanks. It now does [1], provided that the constructor consists of a single static strings.
[1] http://files.basex.org/releases/latest/
<job id="job13" type="ScheduledXQuery" state="cached" user="admin" duration="PT0.007S" reads="(global)" writes="(none)">count(collection(``[BEP]``))</job>
On 20 September 2017 at 23:28, Christian Grün <christian.gruen@gmail.com
wrote:
bxcode:inline-variables($xqcode-uri) as xs:string @returns xquery code that is the source code from $xqcode-uri but with inlining "constant" variables and operations applied
This could be used as a preprocessor. Is this something that could be created from the bits and pieces in
the
current BaseX Java code?
This reminds me of the discussion with Omar [1]: There is currently no way to generate a correct string representation for compiled code. In the GUI Info View, we output a string representation of the optimized query. In simple cases, this string is a valid and equivalent to the original query string, but it’s fairly easy to generate invalid results.
If it’s an option for you to generate code in XQuery, something like the following could be done:
_ bxcode.xqm _______
module namespace bxcode = 'bxcode'; declare function bxcode:bind-external-variables( $uri as xs:string, $vars as map(xs:string, xs:string) ) as xs:string { let $xqcode := unparsed-text($uri) return fold-left( map:keys($vars), $xqcode, function($string, $name) { let $value := replace($vars($name), ']``', '`{"]``"}`', 'q') return $string => replace('declare variable $' || $name || ' external;', '',
'q')
=> replace('$' || $name, '``[' || $value || ']``', 'q') } )};
_ example.xq _______
declare variable $db external; count(collection($db))
_ run-bxcode.xq _______
import module namespace bxcode = 'bxcode' at 'bxcode.xqm'; bxcode:bind-external-variables('query.xq', map { 'db': 'BEP' })
_ output _______
count(collection(``[BEP]``))
Cheers, Christian
[1]
https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg09863.htm...
sorry typo there, should be: I expected *BEP *to appear only in the Read locking, as it does if I remove the db:create call
On Fri, 8 Feb 2019 at 16:03, Andy Bunce bunce.andy@gmail.com wrote:
Using 9.1.2
let $d:=db:open(``[BEP]``) return db:create(``[BEP-staging]``,$d,$d!base-uri(.))
The GUI query info reports:
- Read Locking: (none)
- Write Locking: BEP,BEP-staging
I expected BEP-staging to appear only in the Read locking, as it does if I remove the db:create call Is this correct?
/Andy
On Thu, 21 Sep 2017 at 16:26, Christian Grün christian.gruen@gmail.com wrote:
The use of String Constructor does not avoid the global read lock
Good to know, thanks. It now does [1], provided that the constructor consists of a single static strings.
[1] http://files.basex.org/releases/latest/
<job id="job13" type="ScheduledXQuery" state="cached" user="admin" duration="PT0.007S" reads="(global)" writes="(none)">count(collection(``[BEP]``))</job>
On 20 September 2017 at 23:28, Christian Grün <
christian.gruen@gmail.com>
wrote:
bxcode:inline-variables($xqcode-uri) as xs:string @returns xquery code that is the source code from $xqcode-uri but
with
inlining "constant" variables and operations applied
This could be used as a preprocessor. Is this something that could be created from the bits and pieces in
the
current BaseX Java code?
This reminds me of the discussion with Omar [1]: There is currently no way to generate a correct string representation for compiled code. In the GUI Info View, we output a string representation of the optimized query. In simple cases, this string is a valid and equivalent to the original query string, but it’s fairly easy to generate invalid results.
If it’s an option for you to generate code in XQuery, something like the following could be done:
_ bxcode.xqm _______
module namespace bxcode = 'bxcode'; declare function bxcode:bind-external-variables( $uri as xs:string, $vars as map(xs:string, xs:string) ) as xs:string { let $xqcode := unparsed-text($uri) return fold-left( map:keys($vars), $xqcode, function($string, $name) { let $value := replace($vars($name), ']``', '`{"]``"}`', 'q') return $string => replace('declare variable $' || $name || ' external;', '',
'q')
=> replace('$' || $name, '``[' || $value || ']``', 'q') } )};
_ example.xq _______
declare variable $db external; count(collection($db))
_ run-bxcode.xq _______
import module namespace bxcode = 'bxcode' at 'bxcode.xqm'; bxcode:bind-external-variables('query.xq', map { 'db': 'BEP' })
_ output _______
count(collection(``[BEP]``))
Cheers, Christian
[1]
https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg09863.htm...
Hi Andy,
The current behavior is correct indeed – but it might not be what one expects. Currently, we are…
a) collecting all static database references in the query and b) assigning either read or write locks to these databases, depending if the overall query is updating or not.
The reason is that it’s often tricky to determine statically (i.e., while parsing the query and before compiling and optimizing it) which databases will be accessed for read or write operations without analyzing the query in more detail. An arbitrary example:
let $db := db:open('db1') return insert node <new/> into $db/*
We would need to follow the variable reference in order to find out if db1 will be updated. In simple queries such as yours, however, this might be possible; I’ll have some more thoughts on that.
Cheers Christian
On Fri, Feb 8, 2019 at 5:45 PM Andy Bunce bunce.andy@gmail.com wrote:
sorry typo there, should be: I expected BEP to appear only in the Read locking, as it does if I remove the db:create call
On Fri, 8 Feb 2019 at 16:03, Andy Bunce bunce.andy@gmail.com wrote:
Using 9.1.2
let $d:=db:open(``[BEP]``) return db:create(``[BEP-staging]``,$d,$d!base-uri(.))
The GUI query info reports:
- Read Locking: (none)
- Write Locking: BEP,BEP-staging
I expected BEP-staging to appear only in the Read locking, as it does if I remove the db:create call Is this correct?
/Andy
On Thu, 21 Sep 2017 at 16:26, Christian Grün christian.gruen@gmail.com wrote:
The use of String Constructor does not avoid the global read lock
Good to know, thanks. It now does [1], provided that the constructor consists of a single static strings.
[1] http://files.basex.org/releases/latest/
<job id="job13" type="ScheduledXQuery" state="cached" user="admin" duration="PT0.007S" reads="(global)" writes="(none)">count(collection(``[BEP]``))</job>
On 20 September 2017 at 23:28, Christian Grün christian.gruen@gmail.com wrote:
bxcode:inline-variables($xqcode-uri) as xs:string @returns xquery code that is the source code from $xqcode-uri but with inlining "constant" variables and operations applied
This could be used as a preprocessor. Is this something that could be created from the bits and pieces in the current BaseX Java code?
This reminds me of the discussion with Omar [1]: There is currently no way to generate a correct string representation for compiled code. In the GUI Info View, we output a string representation of the optimized query. In simple cases, this string is a valid and equivalent to the original query string, but it’s fairly easy to generate invalid results.
If it’s an option for you to generate code in XQuery, something like the following could be done:
_ bxcode.xqm _______
module namespace bxcode = 'bxcode'; declare function bxcode:bind-external-variables( $uri as xs:string, $vars as map(xs:string, xs:string) ) as xs:string { let $xqcode := unparsed-text($uri) return fold-left( map:keys($vars), $xqcode, function($string, $name) { let $value := replace($vars($name), ']``', '`{"]``"}`', 'q') return $string => replace('declare variable $' || $name || ' external;', '', 'q') => replace('$' || $name, '``[' || $value || ']``', 'q') } ) };
_ example.xq _______
declare variable $db external; count(collection($db))
_ run-bxcode.xq _______
import module namespace bxcode = 'bxcode' at 'bxcode.xqm'; bxcode:bind-external-variables('query.xq', map { 'db': 'BEP' })
_ output _______
count(collection(``[BEP]``))
Cheers, Christian
[1] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg09863.htm...
basex-talk@mailman.uni-konstanz.de
-
Andy Bunce -
Christian Grün