Func def & performance: element()* vs item()*
BaseX is a great tool for analyzing & characterizing large amounts of XML data. I have used it both at work and on personal projects. I hope the following observation is useful. When I define a function that recurs over a sequence of elements in order to build a map of element name counts, I find that when I specify the type of the element sequence as 'element()*', the function runs so slowly that I give up after 5 minutes or so. But when I specify the type as 'item()*', it finishes in 40 seconds or less. Here's an example: -----begin code snippet----- declare namespace local="w00fw00f"; declare function local:count($elems as element()*, $elem_counts as map(*)) as map(*) { let $elem := head($elems), $elem_name := $elem/name(), $elems_new := tail($elems), $elem_name_count := if (map:contains($elem_counts, $elem_name)) then map:get($elem_counts, $elem_name) + 1 else 1, $elem_counts_new := map:put($elem_counts, $elem_name, $elem_name_count) return if (count($elems_new) = 0) then $elem_counts_new else local:count($elems_new, $elem_counts_new) }; let $coll := collection('pure_20190402'), $elems := $coll/result/items/*, $elem_names_map := local:count($elems, map {}) return json:serialize($elem_names_map, map {'format' : 'xquery'}) -----end code snippet----- In the function declaration, changing "$elems as element()*" to "$elems as item()*" makes the difference in performance. Replacing the JSON serialization with a standard XML one does not change the performance. I am running BaseX 9.1.2 under Ubuntu 16.04.6. All the best, Chuck Bearden
Am 11.04.2019 um 00:09 schrieb Chuck Bearden:
BaseX is a great tool for analyzing & characterizing large amounts of XML data. I have used it both at work and on personal projects. I hope the following observation is useful.
When I define a function that recurs over a sequence of elements in order to build a map of element name counts, I find that when I specify the type of the element sequence as 'element()*', the function runs so slowly that I give up after 5 minutes or so. But when I specify the type as 'item()*', it finishes in 40 seconds or less. Here's an example:
-----begin code snippet----- declare namespace local="w00fw00f"; declare function local:count($elems as element()*, $elem_counts as map(*)) as map(*) { let $elem := head($elems), $elem_name := $elem/name(), $elems_new := tail($elems), $elem_name_count := if (map:contains($elem_counts, $elem_name)) then map:get($elem_counts, $elem_name) + 1 else 1, $elem_counts_new := map:put($elem_counts, $elem_name, $elem_name_count) return if (count($elems_new) = 0) then $elem_counts_new else local:count($elems_new, $elem_counts_new) };
let $coll := collection('pure_20190402'), $elems := $coll/result/items/*, $elem_names_map := local:count($elems, map {})
It seems that task to build the map can also be solved with grouping: let $elem_names_map := map:merge( for $item in $coll/result/items/* group by $name := name($item) return map { $name : count($item) } ) Not sure whether that improves performance.
Hi Chuck, Martin already suggested that map construction via map:merge is preferable and faster (my personal experience is that there are just few cases in which map:put is a better choice). Your query was an interesting one, though. In various cases, we drop type information at runtime, as it can be expensive to decorate all newly generated sequences with the correct type. As a result, the type of your function arguments is verified every time the function is called, and this takes additional time. But as it’s always recommendable to declare types, and as this is not the first time that this is chasing me, I had some more thoughts, and I have found a good answer on how to improve generally typing at runtime! You can already be sure that your query will benefit from the upcoming optimizations, i.e., with BaseX 9.2. Due to this, and due to some other minor optimizations that are still in progress, we decided to delay the release until beginning of next week. Cheers Christian On Thu, Apr 11, 2019 at 12:10 AM Chuck Bearden <cfbearden@gmail.com> wrote:
BaseX is a great tool for analyzing & characterizing large amounts of XML data. I have used it both at work and on personal projects. I hope the following observation is useful.
When I define a function that recurs over a sequence of elements in order to build a map of element name counts, I find that when I specify the type of the element sequence as 'element()*', the function runs so slowly that I give up after 5 minutes or so. But when I specify the type as 'item()*', it finishes in 40 seconds or less. Here's an example:
-----begin code snippet----- declare namespace local="w00fw00f"; declare function local:count($elems as element()*, $elem_counts as map(*)) as map(*) { let $elem := head($elems), $elem_name := $elem/name(), $elems_new := tail($elems), $elem_name_count := if (map:contains($elem_counts, $elem_name)) then map:get($elem_counts, $elem_name) + 1 else 1, $elem_counts_new := map:put($elem_counts, $elem_name, $elem_name_count) return if (count($elems_new) = 0) then $elem_counts_new else local:count($elems_new, $elem_counts_new) };
let $coll := collection('pure_20190402'), $elems := $coll/result/items/*, $elem_names_map := local:count($elems, map {}) return json:serialize($elem_names_map, map {'format' : 'xquery'}) -----end code snippet-----
In the function declaration, changing "$elems as element()*" to "$elems as item()*" makes the difference in performance. Replacing the JSON serialization with a standard XML one does not change the performance. I am running BaseX 9.1.2 under Ubuntu 16.04.6.
All the best, Chuck Bearden
Hi Christian, On Thu, 11 Apr 2019 at 13:37, Christian Grün <christian.gruen@gmail.com> wrote:
Hi Chuck,
Martin already suggested that map construction via map:merge is preferable and faster (my personal experience is that there are just few cases in which map:put is a better choice).
Your query was an interesting one, though. In various cases, we drop type information at runtime, as it can be expensive to decorate all newly generated sequences with the correct type. As a result, the type of your function arguments is verified every time the function is called, and this takes additional time.
But as it’s always recommendable to declare types, and as this is not the first time that this is chasing me, I had some more thoughts, and I have found a good answer on how to improve generally typing at runtime! You can already be sure that your query will benefit from the upcoming optimizations, i.e., with BaseX 9.2.
You may be interested in my https://github.com/rhdunn/xquery-intellij-plugin/blob/master/docs/XQuery%20I... document. It is the result of previous investigations in supporting static type analysis in my XQuery plugin. Specifically: 1. 3.2.1 Item Type Union -- computing the best matching union type of two item types. 2. 3.2.2 Sequence Type Union -- computing the union of two sequences for use in disjoint expressions such as the if and else branches of an IfExpr. 3. 3.2.3 Sequence Type Addition -- computing the resulting type that best matches an Expr. The advantage of this is that the type information can be computed at compile time. I was able to get a basic prototype implementation working for some expressions, and have tested the logic for the rules in that document. I haven't worked on this recently, as I have been adding other features to my plugin. Kind regards, Reece Due to this, and due to some other minor optimizations that are still
in progress, we decided to delay the release until beginning of next week.
Cheers Christian
On Thu, Apr 11, 2019 at 12:10 AM Chuck Bearden <cfbearden@gmail.com> wrote:
BaseX is a great tool for analyzing & characterizing large amounts of XML data. I have used it both at work and on personal projects. I hope the following observation is useful.
When I define a function that recurs over a sequence of elements in order to build a map of element name counts, I find that when I specify the type of the element sequence as 'element()*', the function runs so slowly that I give up after 5 minutes or so. But when I specify the type as 'item()*', it finishes in 40 seconds or less. Here's an example:
-----begin code snippet----- declare namespace local="w00fw00f"; declare function local:count($elems as element()*, $elem_counts as
map(*))
as
map(*) {
let $elem := head($elems), $elem_name := $elem/name(), $elems_new := tail($elems), $elem_name_count := if (map:contains($elem_counts, $elem_name)) then map:get($elem_counts, $elem_name) + 1 else 1, $elem_counts_new := map:put($elem_counts, $elem_name,
$elem_name_count)
return if (count($elems_new) = 0) then $elem_counts_new else local:count($elems_new, $elem_counts_new) };
let $coll := collection('pure_20190402'), $elems := $coll/result/items/*, $elem_names_map := local:count($elems, map {}) return json:serialize($elem_names_map, map {'format' : 'xquery'}) -----end code snippet-----
In the function declaration, changing "$elems as element()*" to "$elems as item()*" makes the difference in performance. Replacing the JSON serialization with a standard XML one does not change the performance. I am running BaseX 9.1.2 under Ubuntu 16.04.6.
All the best, Chuck Bearden
You may be interested in my https://github.com/rhdunn/xquery-intellij-plugin/blob/master/docs/XQuery%20I... document. It is the result of previous investigations in supporting static type analysis in my XQuery plugin.
Thanks, I’ll definitely have a look at your document. In BaseX, we don’t have union types; apart from that, our static typing system should be close to complete (if some types in the query plan should indicate otherwise, feedback is welcome). For example, we are iteratively computing the type of built-in higher-order functions at compile time. The static type of the following function is xs:decimal+: fold-left(1 to 123, 4.5, function($tmp, $curr) { ($curr, $tmp) }) And the static type of the next expression is xs:short+, not xs:integer+ as one might guess (the type derived from the actual types of the input arguments, which will never yield an xs:integer result type): fold-left((1 to 20) ! xs:byte(.), (), function($tmp, $curr) { $tmp, if($curr instance of xs:byte) then xs:short($curr) else xs:integer($curr) }) One missing link is that the static type is not always propagated to the result (i.e., to the internal result representation, which can be a plain untyped array of typed items, in particular if the result is generated by untyped iterators). I think I’ve found a good trade-off between performance and better runtime typing. In future, the intersection of the type of the original expression and the resulting type will be assigned to the resulting value instance.
A new snapshot is online! Looking forward to feedback. On Thu, Apr 11, 2019 at 12:10 AM Chuck Bearden <cfbearden@gmail.com> wrote:
BaseX is a great tool for analyzing & characterizing large amounts of XML data. I have used it both at work and on personal projects. I hope the following observation is useful.
When I define a function that recurs over a sequence of elements in order to build a map of element name counts, I find that when I specify the type of the element sequence as 'element()*', the function runs so slowly that I give up after 5 minutes or so. But when I specify the type as 'item()*', it finishes in 40 seconds or less. Here's an example:
-----begin code snippet----- declare namespace local="w00fw00f"; declare function local:count($elems as element()*, $elem_counts as map(*)) as map(*) { let $elem := head($elems), $elem_name := $elem/name(), $elems_new := tail($elems), $elem_name_count := if (map:contains($elem_counts, $elem_name)) then map:get($elem_counts, $elem_name) + 1 else 1, $elem_counts_new := map:put($elem_counts, $elem_name, $elem_name_count) return if (count($elems_new) = 0) then $elem_counts_new else local:count($elems_new, $elem_counts_new) };
let $coll := collection('pure_20190402'), $elems := $coll/result/items/*, $elem_names_map := local:count($elems, map {}) return json:serialize($elem_names_map, map {'format' : 'xquery'}) -----end code snippet-----
In the function declaration, changing "$elems as element()*" to "$elems as item()*" makes the difference in performance. Replacing the JSON serialization with a standard XML one does not change the performance. I am running BaseX 9.1.2 under Ubuntu 16.04.6.
All the best, Chuck Bearden
The snapshot executes both versions of my recursive function (one with 'element()*' and one with 'item()*' for the sequence of elements) equally fast, which is to say in about 4s. I verified that 9.1.2 executes the 'item()*' one in about the same time, but the 'element()*' one drags on for a few minutes before I stop it. Note that I exit & restart BaseX between tests. Thanks Christian! On Fri, Apr 12, 2019 at 1:12 PM Christian Grün <christian.gruen@gmail.com> wrote:
A new snapshot is online! Looking forward to feedback.
On Thu, Apr 11, 2019 at 12:10 AM Chuck Bearden <cfbearden@gmail.com> wrote:
BaseX is a great tool for analyzing & characterizing large amounts of XML data. I have used it both at work and on personal projects. I hope the following observation is useful.
When I define a function that recurs over a sequence of elements in order to build a map of element name counts, I find that when I specify the type of the element sequence as 'element()*', the function runs so slowly that I give up after 5 minutes or so. But when I specify the type as 'item()*', it finishes in 40 seconds or less. Here's an example:
-----begin code snippet----- declare namespace local="w00fw00f"; declare function local:count($elems as element()*, $elem_counts as map(*)) as map(*) { let $elem := head($elems), $elem_name := $elem/name(), $elems_new := tail($elems), $elem_name_count := if (map:contains($elem_counts, $elem_name)) then map:get($elem_counts, $elem_name) + 1 else 1, $elem_counts_new := map:put($elem_counts, $elem_name, $elem_name_count) return if (count($elems_new) = 0) then $elem_counts_new else local:count($elems_new, $elem_counts_new) };
let $coll := collection('pure_20190402'), $elems := $coll/result/items/*, $elem_names_map := local:count($elems, map {}) return json:serialize($elem_names_map, map {'format' : 'xquery'}) -----end code snippet-----
In the function declaration, changing "$elems as element()*" to "$elems as item()*" makes the difference in performance. Replacing the JSON serialization with a standard XML one does not change the performance. I am running BaseX 9.1.2 under Ubuntu 16.04.6.
All the best, Chuck Bearden
participants (4)
-
Christian Grün -
Chuck Bearden -
Martin Honnen -
Reece Dunn