February 2022 - BaseX-Talk - mailman.uni-konstanz.de

'Flatten' a collection
by Ben Engbers 14 Feb '22

14 Feb '22

Hi, I have a collection of 740 XML-documents which I want to flatten. The files all have the same structure: <handeling id="h_1"> <datum date="d_1"/> <spreekbeurt><spreker>spreker_1</spreker></spreekbeurt> <spreekbeurt><spreker>spreker_3</spreker></spreekbeurt> </handeling> <handeling id="h_2"> <datum date="d_2"/> <spreekbeurt> <spreker>spreker_2</spreker></spreekbeurt><spreekbeurt> <spreker>spreker_1</spreker><spreekbeurt> <spreker>spreker_4</spreker></spreekbeurt> </handeling> <handeling id="h_3"> <datum date="d_3"/> <spreekbeurt><spreker>spreker_2</spreker></spreekbeurt> <spreekbeurt><spreker>spreker_3</spreker></spreekbeurt> <spreekbeurt><spreker>spreker_2</spreker></spreekbeurt> <spreekbeurt><spreker>spreker_1</spreker></spreekbeurt> </handeling> The following query gives this result: import module namespace functx = "http://www.functx.com"; let $Blogs := collection("Blog") let $Turns := collection("Blog") for $Blog in collection("Blog"), $Turn in collection("Blog") where $Turn//datum/@date = $Blog//datum/@date order by $Blog//datum/@date count $Count let $Id := $Blog/handeling/@id let $Datum := $Blog//datum/@date let $Speaker := $Turn//spreker/text() return($Id, $Datum, $Speaker, $Count) id="h_1" date="d_1" spreker_1 spreker_3 1 id="h_2" date="d_2" spreker_2 spreker_1 spreker_4 2 id="h_3" date="d_3" spreker_2 spreker_3 spreker_2 spreker_1 3 But what I eventually need is this (for clarity shown as a table): 1, id="h_1", date="d_1", 1, spreker_1 1, id="h_1", date="d_1", 2, spreker_3 2, id="h_2", date="d_2", 1, spreker_2 2, id="h_2", date="d_2", 2, spreker_1 2, id="h_2", date="d_2", 3, spreker_4 3, id="h_3", date="d_3", 1, spreker_2 3, id="h_3", date="d_3", 2, spreker_3 3, id="h_3", date="d_3", 3, spreker_2 3, id="h_3", date="d_3", 4, spreker_1 The first counter indicates the position in $Blog. and the second counter indicates the position in $Turn I expected that the following query would return what I was looking for: for $Blog in collection("Blog") order by $Blog//datum/@date let $Id := $Blog/handeling/@id let $Datum := $Blog//datum/@date count $countOuter return ( for $Turn in collection("Blog") where $Turn//datum/@date = $Blog//datum/@date let $Speaker := $Turn//spreker/text() return ($countOuter, $Id, $Datum, $Speaker)) Instead it returns 1, id="h_1", date="d_1", 1, spreker_1, spreker_3 2, id="h_2", date="d_2", 1, spreker_2, spreker_1, spreker_4 3, id="h_3", date="d_3", 1, spreker_2, spreker_3, spreker_2, spreker_1 I have 2 questions: 1: Is it possible to use separate counters for the inner and the outer loop? (How should I define the $countInner?) 2: How can I formulate the query for getting the correct output? Ben Engbers

4 3

combining queries from mutiple sources
by Eric Levy 14 Feb '22

14 Feb '22

I am completely new to BaseX, and trying to conceive of a way to use it to process a compound query that combines an external query with a static one. The core idea is a very simple application, expressed as an XQuery file, that results in the following sequence of actions: 1. Extraction of data from a database, by some query. 2. Modification of the results of (1), by some query. 3. Persistence of the results from (2), back into the database from which the data was originally extracted. The modifications given in (2) would be expressed external to the application, supplied as input per invocation, in a separate file or in a command-line parameter. Thus, the overall operation would entail static components, the fixed parts of the application, represented by (1) and (3), and dynamic components, passed by the invocation, represented by (2). I realize such a processing pipeline may be achieved through queries supplied in a full application accessing the database by one of the language-specific APIs, but I am trying to consider an approach that requires use only of XQuery statements, without involving a separate programming language or formal build process. Does the design of BaseX accommodate an application having the structure as described?

4 4

Feature request - file listing
by Hans-Juergen Rennau 12 Feb '22

12 Feb '22

Dear BaseX people, I have a feature request: a function which does the same as file:list, but returns the full paths. (In the explanation, I call it "file:list2") Rationale: (a) It is the full paths what we need in order to parse the files (doc(), json:doc(), csv:doc(), html:doc()). (b) The combination of file listing and parsing in a single expression is of matchless elegance and expressivenessJust think about how we can extend this into a single list+parse+navigate+report expression, like: file:list2($folder, true(), $fname) ! doc(.) ! //returnCode[. ne 0]/concat(base-uri(.), ': ', .) (Wow! Is there any language or tool competing with this compactness and readability?) Currently, if I don't overlook something, I have to do this: file:list($dir, false(), $fname) ! concat($dir, '/', .) ! doc(.) => count() I emphasize the psychological aspect. Every time I have the pleasure to give a workshop about XQuery, I show off how elegant it is to parse a whole file system tree into a forest of nodes in a single expression. And I am always embarrassed about this clumsy fumbling montage of a path. It's as if Mozart had fallen asleep while writing his xquery sonata, knocked over the inkwell and now there is this splash smeared out on the score. You would be my heroes if you can spare us that. Kind regards,Hans-Jürgen

6 9

my application's performance statistics
by Rob Stapper 11 Feb '22

11 Feb '22

Hi Christian, I did some performance statistics on my application and something caught my eye, which I hope you can shed some light on. I found that when running the application the distribution of the life-cycle fases: Parsing, Compiling, Evaluating and Printing, is: Parsing : 95%, Compiling : 4%, Evaluating : 0.4% and Printing : 0.03% . My development-paradigm is maintainability and not as much as performance optimization. So I expect is a lot of code-optimization during parsing and therefore a relatively high amount of time consumed for parsing. This is confirmed by the relatively low amount of time for the Compiling- and Evaluating-time. Code-optimization actually is a strong point of the parser. However I had expected/hoped that this code-optimization only occurs at the first run and that the optimized code is kept for following runs. This does not seem to be the case. First of all: am I correct in my analysis? Secondly, can you give me any advise on improving the parsing-time, not being giving up on my development-paradigm, based on these stats? Some more stats: The info-view gives for the compiling -fase a total of 2124 lines of which: - Inline …… 923 lines - Remove type check …. 710 lines - Simplify FLWOR …. 218 lines - Rewrite … 150 lines - Unroll …. 12 lines - Merge: concat ….. 11 lines - Open database … 3 lines - Swap operant … 1 lines - Simplify catch * …. 4 lines - Thnx in advance for your reply. Rob Stapper Sent from Mail for Windows -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

2 1

file:descendants - returns relative paths
by Hans-Juergen Rennau 11 Feb '22

11 Feb '22

Dear BaseX people, functions file:children() file:descendants() should return the full paths of the resources found, but they do so only if the argument was in turn a full path. A bug? Example: file:children('.') =>PS C:\projects\xquery-introduction\tips> basex "file:children('.')" .\foo.xq .\bar.docx Kind regards,Hans-Jürgen

2 1

How to extract value from fn:analyze-string
by Ben Engbers 11 Feb '22

11 Feb '22

Hi, This query produces the following result: let $debates := collection("Parliament") for $debate-item in $debates let $item-file := $debate-item/officiele-publicatie//meta/@content let $debate-id := fn:analyze-string( $debate-item/officiele-publicatie//meta/@content, "(\d{8}-\d*)-(\d*)") return ($debate-id) => <analyze-string-result xmlns="http://www.w3.org/2005/xpath-functions"> <non-match>https://zoek.officielebekendmakingen.nl/h-tk-</non-match> <match> <group nr="1">20202021-102</group>-<group nr="2">1</group> </match> <non-match>/metadata.xml</non-match> </analyze-string-result> ... I am trying to extract the values from group 1 and 2 but this query returns 0 results: let $debates := collection("Parliament") for $debate-item in $debates let $item-file := $debate-item/officiele-publicatie//meta/@content let $debate-id := fn:analyze-string( $debate-item/officiele-publicatie//meta/@content, "(\d{8}-\d*)-(\d*)") let $debate-nr := $debate-id//match/group[@nr="1"]/text() let $item-nr := $debate-id//match/group[@nr="2"]/text() return ($debate-nr, $item-nr) My guess is that analyze-string inserts new elements in the query and that that is the reason why this does not work. How can I extract debate-nr and item-nr from $debate-id? Ben Engbers

4 4

BaseX Doing Lots of Work on Startup: How to Diagnose?
by Eliot Kimber 10 Feb '22

10 Feb '22

I now have my server loaded with lots of data and I’m seeing that when the server starts it is taking 100% of a processor for 10s of minutes or more. I expect some initial startup cost of course as BaseX prepares whatever it needs to satisfy queries but this feels like something is not right. I don’t see anything obvious in the logs but I suspect I’m not looking in the right place or don’t have appropriate logging options turned on. Any tips on what I can look for to see what might be causing this startup issue? Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

3 7

Controlling order of attributes with updates
by Jonathan Robie 10 Feb '22

10 Feb '22

Yes, I know that the order of attributes is insignificant. Except to some human beings who complain to me about the order of elements. Is there any way for me to control the order of attributes in BaseX when I insert new attributes? Jonathan

3 5

BaseX and 'view'?
by Ben Engbers 09 Feb '22

09 Feb '22

Hi, I am writing a blog for R-bloggers with the aim of raising awareness of my RbaseX package. The latest version is super fast! Loading and saving 740 xml documents from the R environment took only 34 seconds! Before I can use those documents I will probably have to convert them using xsl. I was wondering if BaseX, like Oracle, also has a 'view' over the original data? I can then define multiple views that are better adapted to the intended use. ben

2 1

Managing/Debugging Server Load and Performance
by Eliot Kimber 08 Feb '22

08 Feb '22

I’m getting my validation dashboard server almost ready for use and will be making available to real users. I want to make sure that I’ve done everything I can to ensure maximum performance and stability of the server and I’m realizing I’m not entirely sure how to do that. This is running on a Centos virtual machine with 16GB of ram and 1TB of available disk space and 4 cores—not the beefiest but it’s what IT gave us (and we’re not sure how to ask for more on short notice). Here’s what I’ve done so far: * Using the JRE provided with Oxygen, allocated with 4GB (we are also using this server to run Oxygen via scripting and it needs 8GB to handle our insanely huge DITA maps) * Set parallel to 4 (to match the number of cores, but just guessing that this is a useful setting based on the docs) I have not modified any index settings but I’m also not doing anything other than querying on element types and attribute values, so from the docs it looks like the default indexes should be what I need. I’m seeing some apparent occasional slowness on pages that should not be slow (don’t reflect long-running queries or huge data volumes) but I’m not really sure how to diagnose it or even verify that I’ve succeeded in giving BaseX all the resources it needs. The same webapp performs well on my development mac, which makes me think the issue is one of resource allocation, so probably my user error setting up the server. Any guidance on how to ensure best server performance would be most helpful. Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

4 12