Cache compiled expressions for use with QueryProcessor
Hi all, We have lots of non dynamic (i.e. do not change at run-time) queries/expressions that are always parsed/compiled using a new QueryProcessor instance. How to cache compiled expressions for use with QueryProcessor? Or can we just cache QueryProcessor instances? If yes, are QueryProcessor instances thread-safe? Thanks!
Hi Erdal, parsing and compilation phase is pretty fast in BaseX. Before you think about precompiling your queries, it would be interesting to know if parsing and compilation is a real bottleneck.. Have you already done some profiling? The following example of running a (very simple) query 100'000 times demonstrates that there may be no need to cache queries that are to be executed multiple times: basex -V -r100000 "1+2" Parsing: 0.04 ms (avg) Compiling: 0.01 ms (avg) Evaluating: 0.0 ms (avg) Printing: 0.03 ms (avg) Total Time: 0.08 ms (avg) In a complex query with lots of modules being imported, however, the time for parsing and compiling a query may increase. For those cases, we indeed have some plans to precompile queries in the background. I can’t give you a timeframe yet. Christian ___________________________ 2013/10/22 Erdal Karaca <erdal.karaca.de@gmail.com>:
Hi all, We have lots of non dynamic (i.e. do not change at run-time) queries/expressions that are always parsed/compiled using a new QueryProcessor instance.
How to cache compiled expressions for use with QueryProcessor? Or can we just cache QueryProcessor instances? If yes, are QueryProcessor instances thread-safe?
Thanks!
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Hi Christian, In this case, BaseX is embedded in a Java application (Eclipse RCP) and provides "live editing" of parts of big XML files (managed by BaseX). Just imagine we have an O(n^2) algorithm which visits 100x100 "data items" while executing 10 different (and "simple") expressions (in the inner loop). That would be 100x100x10 = 100'000 parsing/compiling operations taking in sum 5000 ms if 1 parsing/compiling operation takes 0.05 ms. If the expressions would be cacheable, then, maybe it would just take 10 x (0.05 ms + (lookup of compiled expression)) Of course, O(n^2) is bad, but that is another story. The idea is to be able to cache compiled expressions (like parsing/compiling regex in java). Thanks! 2013/10/22 Christian Grün <christian.gruen@gmail.com>
Hi Erdal,
parsing and compilation phase is pretty fast in BaseX. Before you think about precompiling your queries, it would be interesting to know if parsing and compilation is a real bottleneck.. Have you already done some profiling?
The following example of running a (very simple) query 100'000 times demonstrates that there may be no need to cache queries that are to be executed multiple times:
basex -V -r100000 "1+2"
Parsing: 0.04 ms (avg) Compiling: 0.01 ms (avg) Evaluating: 0.0 ms (avg) Printing: 0.03 ms (avg) Total Time: 0.08 ms (avg)
In a complex query with lots of modules being imported, however, the time for parsing and compiling a query may increase. For those cases, we indeed have some plans to precompile queries in the background. I can’t give you a timeframe yet.
Christian ___________________________
2013/10/22 Erdal Karaca <erdal.karaca.de@gmail.com>:
Hi all, We have lots of non dynamic (i.e. do not change at run-time) queries/expressions that are always parsed/compiled using a new QueryProcessor instance.
How to cache compiled expressions for use with QueryProcessor? Or can we just cache QueryProcessor instances? If yes, are QueryProcessor instances thread-safe?
Thanks!
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Hi Erdal, in this case, you’ll either need to wait, or think about sponsoring the feature to speed up its implementation… ;) Best, Christian ___________________________
Hi Christian, In this case, BaseX is embedded in a Java application (Eclipse RCP) and provides "live editing" of parts of big XML files (managed by BaseX). Just imagine we have an O(n^2) algorithm which visits 100x100 "data items" while executing 10 different (and "simple") expressions (in the inner loop). That would be 100x100x10 = 100'000 parsing/compiling operations taking in sum 5000 ms if 1 parsing/compiling operation takes 0.05 ms. If the expressions would be cacheable, then, maybe it would just take 10 x (0.05 ms + (lookup of compiled expression)) Of course, O(n^2) is bad, but that is another story.
The idea is to be able to cache compiled expressions (like parsing/compiling regex in java).
Thanks!
2013/10/22 Christian Grün <christian.gruen@gmail.com>
Hi Erdal,
parsing and compilation phase is pretty fast in BaseX. Before you think about precompiling your queries, it would be interesting to know if parsing and compilation is a real bottleneck.. Have you already done some profiling?
The following example of running a (very simple) query 100'000 times demonstrates that there may be no need to cache queries that are to be executed multiple times:
basex -V -r100000 "1+2"
Parsing: 0.04 ms (avg) Compiling: 0.01 ms (avg) Evaluating: 0.0 ms (avg) Printing: 0.03 ms (avg) Total Time: 0.08 ms (avg)
In a complex query with lots of modules being imported, however, the time for parsing and compiling a query may increase. For those cases, we indeed have some plans to precompile queries in the background. I can’t give you a timeframe yet.
Christian ___________________________
2013/10/22 Erdal Karaca <erdal.karaca.de@gmail.com>:
Hi all, We have lots of non dynamic (i.e. do not change at run-time) queries/expressions that are always parsed/compiled using a new QueryProcessor instance.
How to cache compiled expressions for use with QueryProcessor? Or can we just cache QueryProcessor instances? If yes, are QueryProcessor instances thread-safe?
Thanks!
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Hehe :-) 2013/10/22 Christian Grün <christian.gruen@gmail.com>
Hi Erdal,
in this case, you’ll either need to wait, or think about sponsoring the feature to speed up its implementation… ;)
Best, Christian ___________________________
Hi Christian, In this case, BaseX is embedded in a Java application (Eclipse RCP) and provides "live editing" of parts of big XML files (managed by BaseX). Just imagine we have an O(n^2) algorithm which visits 100x100 "data items" while executing 10 different (and "simple") expressions (in the inner loop). That would be 100x100x10 = 100'000 parsing/compiling operations taking in sum 5000 ms if 1 parsing/compiling operation takes 0.05 ms. If the expressions would be cacheable, then, maybe it would just take 10 x (0.05 ms + (lookup of compiled expression)) Of course, O(n^2) is bad, but that is another story.
The idea is to be able to cache compiled expressions (like parsing/compiling regex in java).
Thanks!
2013/10/22 Christian Grün <christian.gruen@gmail.com>
Hi Erdal,
parsing and compilation phase is pretty fast in BaseX. Before you think about precompiling your queries, it would be interesting to know if parsing and compilation is a real bottleneck.. Have you already done some profiling?
The following example of running a (very simple) query 100'000 times demonstrates that there may be no need to cache queries that are to be executed multiple times:
basex -V -r100000 "1+2"
Parsing: 0.04 ms (avg) Compiling: 0.01 ms (avg) Evaluating: 0.0 ms (avg) Printing: 0.03 ms (avg) Total Time: 0.08 ms (avg)
In a complex query with lots of modules being imported, however, the time for parsing and compiling a query may increase. For those cases, we indeed have some plans to precompile queries in the background. I can’t give you a timeframe yet.
Christian ___________________________
2013/10/22 Erdal Karaca <erdal.karaca.de@gmail.com>:
Hi all, We have lots of non dynamic (i.e. do not change at run-time) queries/expressions that are always parsed/compiled using a new QueryProcessor instance.
How to cache compiled expressions for use with QueryProcessor? Or can we just cache QueryProcessor instances? If yes, are QueryProcessor instances thread-safe?
Thanks!
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
participants (2)
-
Christian Grün -
Erdal Karaca