Fwd: Re: How to process very long node sequences
Wrong mail account... -------- Weitergeleitete Nachricht -------- Betreff: Re: [basex-talk] How to process very long node sequences Datum: Wed, 18 Mar 2015 19:26:00 +0100 Von: Leo Wörteler <lw@basex.org> An: Hans-Juergen Rennau <hrennau@yahoo.de> Kopie (CC): basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de> Dear Hans-Jürgen, Am 18.03.2015 um 15:23 schrieb Hans-Juergen Rennau:
I want to persuade the processor to visit, process and forget the nodes one after the other, rather than to attempt loading all nodes into memory before proceeding to process them.
BaseX already uses iterative processing as default mode. It only falls back to caching if it has to, e.g. for sorting, reversing or duplicate elimination. Calls to user-defined functions are currently also blocking, but we are currently investigating if that can be changed.
Schematically:
declare function f:processNode($node as node()) as empty-sequence() {...};
for $node in doc('huge-doc')/a/b/c return f:processNode($node)
The example you gave will always be evaluated without caching the node sequence. The `for` clause requests the result of its argument iteratively, and the XPath expression you used only contains child steps, which can be evaluated in document order without duplicates.
Well, in some cases it works, in others in doesn't. Is there any safe way how to enforce sequential visit-process-forget processing?
When the XPath expression becomes more complex, it is not as easy to predict if it uses caching internally. BaseX tries quite hard to detect paths that do not need it, the algorithm can be seen in [1]. If you see a `CachedPath` in the Info View of the GUI, you can try to reformulate the query.
I call this problem general because it is a critical aspect of dealing with huge documents.
XQuery does not have a dedicated *streaming mode* like that of XSLT 3.0 [2] (yet), but it would definitely be possible to check if some part of a query (e.g. marked by a pragma) is evaluated without caching. It would however be quite some work. Hope that helps, Leo [1] https://github.com/BaseXdb/basex/blob/0b828a8/basex-core/src/main/java/org/b... [2] http://www.w3.org/TR/xslt-30/#dt-guaranteed-streamable
Hi Hans-Jürgen, As I understand the first part of answer, you tell that when there is a user defined function Basex do cache, but after when you analyze the example, you tell that in this cxase there is no cache else there is a f:processNode function in the return. Do I you don't understand? Marc Le 18/03/2015 19:38, Leonard Wörteler a écrit :
Wrong mail account...
-------- Weitergeleitete Nachricht -------- Betreff: Re: [basex-talk] How to process very long node sequences Datum: Wed, 18 Mar 2015 19:26:00 +0100 Von: Leo Wörteler <lw@basex.org> An: Hans-Juergen Rennau <hrennau@yahoo.de> Kopie (CC): basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de>
Dear Hans-Jürgen,
Am 18.03.2015 um 15:23 schrieb Hans-Juergen Rennau:
I want to persuade the processor to visit, process and forget the nodes one after the other, rather than to attempt loading all nodes into memory before proceeding to process them.
BaseX already uses iterative processing as default mode. It only falls back to caching if it has to, e.g. for sorting, reversing or duplicate elimination. Calls to user-defined functions are currently also blocking, but we are currently investigating if that can be changed.
Schematically:
declare function f:processNode($node as node()) as empty-sequence() {...};
for $node in doc('huge-doc')/a/b/c return f:processNode($node)
The example you gave will always be evaluated without caching the node sequence. The `for` clause requests the result of its argument iteratively, and the XPath expression you used only contains child steps, which can be evaluated in document order without duplicates.
Well, in some cases it works, in others in doesn't. Is there any safe way how to enforce sequential visit-process-forget processing?
When the XPath expression becomes more complex, it is not as easy to predict if it uses caching internally. BaseX tries quite hard to detect paths that do not need it, the algorithm can be seen in [1]. If you see a `CachedPath` in the Info View of the GUI, you can try to reformulate the query.
I call this problem general because it is a critical aspect of dealing with huge documents.
XQuery does not have a dedicated *streaming mode* like that of XSLT 3.0 [2] (yet), but it would definitely be possible to check if some part of a query (e.g. marked by a pragma) is evaluated without caching. It would however be quite some work.
Hope that helps, Leo
[1] https://github.com/BaseXdb/basex/blob/0b828a8/basex-core/src/main/java/org/b...
Hi Marc, Am 18.03.2015 um 21:08 schrieb Marc:
As I understand the first part of answer, you tell that when there is a user defined function Basex do cache, but after when you analyze the example, you tell that in this cxase there is no cache else there is a f:processNode function in the return.
Do I you don't understand?
well, kind of. When evaluating a call to a user-defined function (that was not inlined during compilation), both the arguments and the result of the call are fully evaluated and cached in memory. This is still the case in Hans-Jürgen's example. As I understand his question however, he is mostly concerned about the (potentially very big) sequence of nodes that the `for` loop iterates over. I just pointed out that that sequence will not normally be cached by BaseX. Hope that clears it up, chears, Leo
Thank you very much, Leo, and thanks for your interest, Marc. An observation I made, and which turned out to be crucial to my concrete problem: caching: for $node at $pos in doc('otds-fti')/foo/bar return ... not caching: for $node at $pos in doc('otds-fti')/foo/bar/document{.} return ... So the final step /document{.} turns the for loop into caching. Interestingly, the problem can be worked around very simply: not caching: for $node at $pos in doc('otds-fti')/foo/bar let $node := document{.} return ... I am very glad that the problem could be solved. Your pointing out the importance of the expression details "and the XPath expression you used only contains child steps, which can be evaluated in document order without duplicates." helped me to make the revealing experiment. Thanks again -Hans-Jürgen PS: Interestingly, the following variant is also not caching: f:evaluate($expr, $context) { let $ctx := map{'': $context} return xquery:eval($expr $ctx) }; let $context := doc('otds-fti') for $node at $pos in f:evaluate('/foo/bar', $context) let $node := document{.} return ... Leonard Wörteler <leonard.woerteler@uni-konstanz.de> schrieb am 23:12 Mittwoch, 18.März 2015: Hi Marc, Am 18.03.2015 um 21:08 schrieb Marc:
As I understand the first part of answer, you tell that when there is a user defined function Basex do cache, but after when you analyze the example, you tell that in this cxase there is no cache else there is a f:processNode function in the return.
Do I you don't understand?
well, kind of. When evaluating a call to a user-defined function (that was not inlined during compilation), both the arguments and the result of the call are fully evaluated and cached in memory. This is still the case in Hans-Jürgen's example. As I understand his question however, he is mostly concerned about the (potentially very big) sequence of nodes that the `for` loop iterates over. I just pointed out that that sequence will not normally be cached by BaseX. Hope that clears it up, chears, Leo
Hi Hans-Jürgen,
for $node at $pos in doc('otds-fti')/foo/bar/document{.} return ...
So the final step /document{.}
turns the for loop into caching.
A minor thing I'd like to add: * The semantics of the path expression is that all results need to be returned in "document order". * Document order is well-defined within a single document. However, it is implementation-defined when working across multiple documents. * As a result, the following query... let $doc1 := document { 'a' } let $doc2 := document { 'a' } return ($doc1, $doc2)/self::node() …may either return doc1 or doc2 as first result. The nodes won't be reordered with the simple map operator: for $node at $pos in doc('otds-fti')/foo/bar ! document{.} Cheers, Christian
participants (4)
-
Christian Grün -
Hans-Juergen Rennau -
Leonard Wörteler -
Marc