Hello,
I am facing some performances issues, probably due to unexpected query reevaluation. Might it be due to the optimizer ? Here is a code to test
declare function local:test($i){ let $data := fn:for-each(1 to 10000000, function($a){2*$a} ) (: let $data := xquery:eval("fn:for-each(1 to 10000000, function($a){2*$a} )") :) return (1 to $i) ! local:test2($data,.) };
declare function local:test2($data, $dummy) { count($data) };
(1 to 10) ! prof:time(local:test(.))
Output :
683.76 ms 1137.95 ms 1727.68 ms 2151.06 ms 2694.84 ms 3189.57 ms 3725.1 ms 4277.51 ms 4815.74 ms 5417.79 ms
Note that using xquery:eval (toggling the comment inside local:test). seems to perform correctly. However I can't use this trick as a workaround.
(1 to 10) ! prof:time(local:test(.)) Ouput :
2472.71 ms 825.61 ms 3013.14 ms 986.81 ms 2982.97 ms 778.24 ms 1195.33 ms 3280.7 ms 976.96 ms 913.69 ms
Hi Jean-Marc,
interesting one; it seems that $data is inlined and thus evaluated more than once. This doesn't happen if a FLWOR expression is used:
let $data := fn:for-each ... for $n in 1 to $i return local:test2($data, $n)
Should be easy to resolve.
Thanks for your (always concise) examples, Christian
On Mon, Jan 12, 2015 at 9:08 PM, jean-marc Mercier jeanmarc.mercier@gmail.com wrote:
Hello,
I am facing some performances issues, probably due to unexpected query reevaluation. Might it be due to the optimizer ? Here is a code to test
declare function local:test($i){ let $data := fn:for-each(1 to 10000000, function($a){2*$a} ) (: let $data := xquery:eval("fn:for-each(1 to 10000000, function($a){2*$a} )") :) return (1 to $i) ! local:test2($data,.) };
declare function local:test2($data, $dummy) { count($data) };
(1 to 10) ! prof:time(local:test(.))
Output :
683.76 ms 1137.95 ms 1727.68 ms 2151.06 ms 2694.84 ms 3189.57 ms 3725.1 ms 4277.51 ms 4815.74 ms 5417.79 ms
Note that using xquery:eval (toggling the comment inside local:test). seems to perform correctly. However I can't use this trick as a workaround.
(1 to 10) ! prof:time(local:test(.)) Ouput :
2472.71 ms 825.61 ms 3013.14 ms 986.81 ms 2982.97 ms 778.24 ms 1195.33 ms 3280.7 ms 976.96 ms 913.69 ms
Christian,
Hi. Thanks for the FLWOR trick, behaving in constant time. However it seems to generates an unexpected 5X time overhead, and I can't use it straightforwardly as workaround. Can you reproduce it, since this overhead might be due to eclipse ?
declare function local:test(){fn:for-each(1 to 10000000, function($a){2*$a})}; declare function local:test1(){count(local:test())}; declare function local:test2(){let $ data := local:test() return for $n in (1 to 10) return count($data)};
prof:time(local:test1()) ouput : 639.12 ms prof:time(local:test2()) ouput : 2431.67 ms
2015-01-13 1:59 GMT+01:00 Christian Grün christian.gruen@gmail.com:
Hi Jean-Marc,
interesting one; it seems that $data is inlined and thus evaluated more than once. This doesn't happen if a FLWOR expression is used:
let $data := fn:for-each ... for $n in 1 to $i return local:test2($data, $n)
Should be easy to resolve.
Thanks for your (always concise) examples, Christian
On Mon, Jan 12, 2015 at 9:08 PM, jean-marc Mercier jeanmarc.mercier@gmail.com wrote:
Hello,
I am facing some performances issues, probably due to unexpected query reevaluation. Might it be due to the optimizer ? Here is a code to test
declare function local:test($i){ let $data := fn:for-each(1 to 10000000, function($a){2*$a} ) (: let $data := xquery:eval("fn:for-each(1 to 10000000, function($a){2*$a} )") :) return (1 to $i) ! local:test2($data,.) };
declare function local:test2($data, $dummy) { count($data) };
(1 to 10) ! prof:time(local:test(.))
Output :
683.76 ms 1137.95 ms 1727.68 ms 2151.06 ms 2694.84 ms 3189.57 ms 3725.1 ms 4277.51 ms 4815.74 ms 5417.79 ms
Note that using xquery:eval (toggling the comment inside local:test).
seems
to perform correctly. However I can't use this trick as a workaround.
(1 to 10) ! prof:time(local:test(.)) Ouput :
2472.71 ms 825.61 ms 3013.14 ms 986.81 ms 2982.97 ms 778.24 ms 1195.33 ms 3280.7 ms 976.96 ms 913.69 ms
Hi Jean-Marc,
beforehand: The first issue you reported back to us recently [1] has been fixed by our functional guru Leo! A new snapshot is available [2].
Hi. Thanks for the FLWOR trick, behaving in constant time. However it seems to generates an unexpected 5X time overhead
Most probably this is because the results of for-each will be cached, while in the previous case (if no inlining takes place), all items will be iterated.
Hope this helps, Christian
[1] https://github.com/BaseXdb/basex/issues/1052 [2] http://files.basex.org/releases/latest/
On Tue, Jan 13, 2015 at 9:19 AM, jean-marc Mercier jeanmarc.mercier@gmail.com wrote:
Christian,
Hi. Thanks for the FLWOR trick, behaving in constant time. However it seems to generates an unexpected 5X time overhead, and I can't use it straightforwardly as workaround. Can you reproduce it, since this overhead might be due to eclipse ?
declare function local:test(){fn:for-each(1 to 10000000, function($a){2*$a})}; declare function local:test1(){count(local:test())}; declare function local:test2(){let $ data := local:test() return for $n in (1 to 10) return count($data)};
prof:time(local:test1()) ouput : 639.12 ms prof:time(local:test2()) ouput : 2431.67 ms
2015-01-13 1:59 GMT+01:00 Christian Grün christian.gruen@gmail.com:
Hi Jean-Marc,
interesting one; it seems that $data is inlined and thus evaluated more than once. This doesn't happen if a FLWOR expression is used:
let $data := fn:for-each ... for $n in 1 to $i return local:test2($data, $n)
Should be easy to resolve.
Thanks for your (always concise) examples, Christian
On Mon, Jan 12, 2015 at 9:08 PM, jean-marc Mercier jeanmarc.mercier@gmail.com wrote:
Hello,
I am facing some performances issues, probably due to unexpected query reevaluation. Might it be due to the optimizer ? Here is a code to test
declare function local:test($i){ let $data := fn:for-each(1 to 10000000, function($a){2*$a} ) (: let $data := xquery:eval("fn:for-each(1 to 10000000, function($a){2*$a} )") :) return (1 to $i) ! local:test2($data,.) };
declare function local:test2($data, $dummy) { count($data) };
(1 to 10) ! prof:time(local:test(.))
Output :
683.76 ms 1137.95 ms 1727.68 ms 2151.06 ms 2694.84 ms 3189.57 ms 3725.1 ms 4277.51 ms 4815.74 ms 5417.79 ms
Note that using xquery:eval (toggling the comment inside local:test). seems to perform correctly. However I can't use this trick as a workaround.
(1 to 10) ! prof:time(local:test(.)) Ouput :
2472.71 ms 825.61 ms 3013.14 ms 986.81 ms 2982.97 ms 778.24 ms 1195.33 ms 3280.7 ms 976.96 ms 913.69 ms
Glory to our holy Leo for transmitting *brahmavidya* to shishya !!
Turning back to this optimization problem, this might be trickier since execution time does not seem to behave linearly. For instance, consider
declare function local:test1($i){let $data_3 as item() *:= fn:for-each((1 to 1000000*$i), function($a_0) as item()? { (2 * $a_0) }) return for $n in (1 to 2) return fn:count($data_3)}; declare function local:test2($i){count(fn:for-each((1 to 1000000*$i), function($a_0) as item()? { (2 * $a_0) }))};
Then execution time is ( the results are stable while repeating the experience )
prof:time(local:test1(7)) : 631.32 ms prof:time(local:test2(7)) : 451.32 ms prof:time(local:test1(8)) : 2550.17 ms prof:time(local:test2(8)) : 507.32 ms
2015-01-13 22:43 GMT+01:00 Christian Grün christian.gruen@gmail.com:
Hi Jean-Marc,
beforehand: The first issue you reported back to us recently [1] has been fixed by our functional guru Leo! A new snapshot is available [2].
Hi. Thanks for the FLWOR trick, behaving in constant time. However it
seems
to generates an unexpected 5X time overhead
Most probably this is because the results of for-each will be cached, while in the previous case (if no inlining takes place), all items will be iterated.
Hope this helps, Christian
[1] https://github.com/BaseXdb/basex/issues/1052 [2] http://files.basex.org/releases/latest/
On Tue, Jan 13, 2015 at 9:19 AM, jean-marc Mercier jeanmarc.mercier@gmail.com wrote:
Christian,
Hi. Thanks for the FLWOR trick, behaving in constant time. However it
seems
to generates an unexpected 5X time overhead, and I can't use it straightforwardly as workaround. Can you reproduce it, since this
overhead
might be due to eclipse ?
declare function local:test(){fn:for-each(1 to 10000000, function($a){2*$a})}; declare function local:test1(){count(local:test())}; declare function local:test2(){let $ data := local:test() return for $n
in
(1 to 10) return count($data)};
prof:time(local:test1()) ouput : 639.12 ms prof:time(local:test2()) ouput : 2431.67 ms
2015-01-13 1:59 GMT+01:00 Christian Grün christian.gruen@gmail.com:
Hi Jean-Marc,
interesting one; it seems that $data is inlined and thus evaluated more than once. This doesn't happen if a FLWOR expression is used:
let $data := fn:for-each ... for $n in 1 to $i return local:test2($data, $n)
Should be easy to resolve.
Thanks for your (always concise) examples, Christian
On Mon, Jan 12, 2015 at 9:08 PM, jean-marc Mercier jeanmarc.mercier@gmail.com wrote:
Hello,
I am facing some performances issues, probably due to unexpected query reevaluation. Might it be due to the optimizer ? Here is a code to
test
declare function local:test($i){ let $data := fn:for-each(1 to 10000000, function($a){2*$a} ) (: let $data := xquery:eval("fn:for-each(1 to 10000000, function($a){2*$a} )") :) return (1 to $i) ! local:test2($data,.) };
declare function local:test2($data, $dummy) { count($data) };
(1 to 10) ! prof:time(local:test(.))
Output :
683.76 ms 1137.95 ms 1727.68 ms 2151.06 ms 2694.84 ms 3189.57 ms 3725.1 ms 4277.51 ms 4815.74 ms 5417.79 ms
Note that using xquery:eval (toggling the comment inside local:test). seems to perform correctly. However I can't use this trick as a workaround.
(1 to 10) ! prof:time(local:test(.)) Ouput :
2472.71 ms 825.61 ms 3013.14 ms 986.81 ms 2982.97 ms 778.24 ms 1195.33 ms 3280.7 ms 976.96 ms 913.69 ms
Christian,
Note that executing within BaseX gui leads to different results, but seems also to behave non linearly : for instance for $i in (15 to 20) return prof:time(local:test1($i)) returns random results : for instance Evaluating: - 1368.54 ms - 1314.6 ms - 5151.12 ms - 1719.01 ms - 6023.38 ms - 1996.37 ms
Might it be garbage collector stuff ?
2015-01-14 8:18 GMT+01:00 jean-marc Mercier jeanmarc.mercier@gmail.com:
Glory to our holy Leo for transmitting *brahmavidya* to shishya !!
Turning back to this optimization problem, this might be trickier since execution time does not seem to behave linearly. For instance, consider
declare function local:test1($i){let $data_3 as item() *:= fn:for-each((1 to 1000000*$i), function($a_0) as item()? { (2 * $a_0) }) return for $n in (1 to 2) return fn:count($data_3)}; declare function local:test2($i){count(fn:for-each((1 to 1000000*$i), function($a_0) as item()? { (2 * $a_0) }))};
Then execution time is ( the results are stable while repeating the experience )
prof:time(local:test1(7)) : 631.32 ms prof:time(local:test2(7)) : 451.32 ms prof:time(local:test1(8)) : 2550.17 ms prof:time(local:test2(8)) : 507.32 ms
2015-01-13 22:43 GMT+01:00 Christian Grün christian.gruen@gmail.com:
Hi Jean-Marc,
beforehand: The first issue you reported back to us recently [1] has been fixed by our functional guru Leo! A new snapshot is available [2].
Hi. Thanks for the FLWOR trick, behaving in constant time. However it
seems
to generates an unexpected 5X time overhead
Most probably this is because the results of for-each will be cached, while in the previous case (if no inlining takes place), all items will be iterated.
Hope this helps, Christian
[1] https://github.com/BaseXdb/basex/issues/1052 [2] http://files.basex.org/releases/latest/
On Tue, Jan 13, 2015 at 9:19 AM, jean-marc Mercier jeanmarc.mercier@gmail.com wrote:
Christian,
Hi. Thanks for the FLWOR trick, behaving in constant time. However it
seems
to generates an unexpected 5X time overhead, and I can't use it straightforwardly as workaround. Can you reproduce it, since this
overhead
might be due to eclipse ?
declare function local:test(){fn:for-each(1 to 10000000, function($a){2*$a})}; declare function local:test1(){count(local:test())}; declare function local:test2(){let $ data := local:test() return for $n
in
(1 to 10) return count($data)};
prof:time(local:test1()) ouput : 639.12 ms prof:time(local:test2()) ouput : 2431.67 ms
2015-01-13 1:59 GMT+01:00 Christian Grün christian.gruen@gmail.com:
Hi Jean-Marc,
interesting one; it seems that $data is inlined and thus evaluated more than once. This doesn't happen if a FLWOR expression is used:
let $data := fn:for-each ... for $n in 1 to $i return local:test2($data, $n)
Should be easy to resolve.
Thanks for your (always concise) examples, Christian
On Mon, Jan 12, 2015 at 9:08 PM, jean-marc Mercier jeanmarc.mercier@gmail.com wrote:
Hello,
I am facing some performances issues, probably due to unexpected
query
reevaluation. Might it be due to the optimizer ? Here is a code to
test
declare function local:test($i){ let $data := fn:for-each(1 to 10000000, function($a){2*$a} ) (: let $data := xquery:eval("fn:for-each(1 to 10000000, function($a){2*$a} )") :) return (1 to $i) ! local:test2($data,.) };
declare function local:test2($data, $dummy) { count($data) };
(1 to 10) ! prof:time(local:test(.))
Output :
683.76 ms 1137.95 ms 1727.68 ms 2151.06 ms 2694.84 ms 3189.57 ms 3725.1 ms 4277.51 ms 4815.74 ms 5417.79 ms
Note that using xquery:eval (toggling the comment inside local:test). seems to perform correctly. However I can't use this trick as a workaround.
(1 to 10) ! prof:time(local:test(.)) Ouput :
2472.71 ms 825.61 ms 3013.14 ms 986.81 ms 2982.97 ms 778.24 ms 1195.33 ms 3280.7 ms 976.96 ms 913.69 ms
Hi Jean-Marc,
Might it be garbage collector stuff ?
That's exactly what it is.
2015-01-14 8:18 GMT+01:00 jean-marc Mercier jeanmarc.mercier@gmail.com:
Glory to our holy Leo for transmitting brahmavidya to shishya !!
Turning back to this optimization problem, this might be trickier since
execution time does not seem to behave linearly. For instance, consider
declare function local:test1($i){let $data_3 as item() *:=
fn:for-each((1 to 1000000*$i), function($a_0) as item()? { (2 * $a_0) }) return for $n in (1 to 2) return fn:count($data_3)};
declare function local:test2($i){count(fn:for-each((1 to 1000000*$i),
function($a_0) as item()? { (2 * $a_0) }))};
Then execution time is ( the results are stable while repeating the
experience )
prof:time(local:test1(7)) : 631.32 ms prof:time(local:test2(7)) :
451.32 ms
prof:time(local:test1(8)) : 2550.17 ms prof:time(local:test2(8)) :
507.32 ms
2015-01-13 22:43 GMT+01:00 Christian Grün christian.gruen@gmail.com:
Hi Jean-Marc,
beforehand: The first issue you reported back to us recently [1] has been fixed by our functional guru Leo! A new snapshot is available [2].
Hi. Thanks for the FLWOR trick, behaving in constant time. However it
seems
to generates an unexpected 5X time overhead
Most probably this is because the results of for-each will be cached, while in the previous case (if no inlining takes place), all items will be iterated.
Hope this helps, Christian
[1] https://github.com/BaseXdb/basex/issues/1052 [2] http://files.basex.org/releases/latest/
On Tue, Jan 13, 2015 at 9:19 AM, jean-marc Mercier jeanmarc.mercier@gmail.com wrote:
Christian,
Hi. Thanks for the FLWOR trick, behaving in constant time. However it
seems
to generates an unexpected 5X time overhead, and I can't use it straightforwardly as workaround. Can you reproduce it, since this
overhead
might be due to eclipse ?
declare function local:test(){fn:for-each(1 to 10000000, function($a){2*$a})}; declare function local:test1(){count(local:test())}; declare function local:test2(){let $ data := local:test() return for
$n in
(1 to 10) return count($data)};
prof:time(local:test1()) ouput : 639.12 ms prof:time(local:test2()) ouput : 2431.67 ms
2015-01-13 1:59 GMT+01:00 Christian Grün christian.gruen@gmail.com:
Hi Jean-Marc,
interesting one; it seems that $data is inlined and thus evaluated more than once. This doesn't happen if a FLWOR expression is used:
let $data := fn:for-each ... for $n in 1 to $i return local:test2($data, $n)
Should be easy to resolve.
Thanks for your (always concise) examples, Christian
On Mon, Jan 12, 2015 at 9:08 PM, jean-marc Mercier jeanmarc.mercier@gmail.com wrote:
Hello,
I am facing some performances issues, probably due to unexpected
query
reevaluation. Might it be due to the optimizer ? Here is a code to
test
declare function local:test($i){ let $data := fn:for-each(1 to 10000000, function($a){2*$a} ) (: let $data := xquery:eval("fn:for-each(1 to 10000000, function($a){2*$a} )") :) return (1 to $i) ! local:test2($data,.) };
declare function local:test2($data, $dummy) { count($data) };
(1 to 10) ! prof:time(local:test(.))
Output :
683.76 ms 1137.95 ms 1727.68 ms 2151.06 ms 2694.84 ms 3189.57 ms 3725.1 ms 4277.51 ms 4815.74 ms 5417.79 ms
Note that using xquery:eval (toggling the comment inside
local:test).
seems to perform correctly. However I can't use this trick as a
workaround.
(1 to 10) ! prof:time(local:test(.)) Ouput :
2472.71 ms 825.61 ms 3013.14 ms 986.81 ms 2982.97 ms 778.24 ms 1195.33 ms 3280.7 ms 976.96 ms 913.69 ms
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
jean-marc Mercier