BaseX-Talk December 2019

basex-talk@mailman.uni-konstanz.de

22 participants
27 discussions

Start a nNew thread

Re: [basex-talk] HTTPServer + gzip compression
by Stefan Koch 23 Dec '19

23 Dec '19

1 0

R package
by Ben Engbers 23 Dec '19

23 Dec '19

Hi, Some years ago, I have written a R-client for BaseX. That client used a R-file that could be sourced into your R-code (https://github.com/BaseXdb/basex/tree/master/basex-api/src/main/r/RbaseXCli…). Since not all functions from the server protocol were implemented, I decided to rewrite the client and build a package. This package has now been accepted by Cran (https://cran.r-project.org/web/packages/RBaseX/index.html). I am still working on some improvements but that will take some time. In the meantime, enjoy it. Ben Engbers

1 0

Improper use or potential bug
by France Baril 23 Dec '19

23 Dec '19

This .xq works the first time when returning db:create. However, once the DB are created, if I run it again, files are note replaced. Instead, I get java.lang.ArrayIndexOutOfBoundsException. I can send the DB in private if necessary for debugging. The result is not exactly an index, it's just flattening a few levels of indirection to make future queries faster. I was testing this on 9.2.3. Also a side question, is there a way to rewrite this query so not all DBs are locked? I get lock: global. I would like to lock only DBs that start with 'index-prompt-' I tried hard coding the prefix in the db:create and db:replace fonctions but that didn't help. Since I will have this running on a loop (cannot catch updated on src DB), I don't want to lock the whole system each time this query is running. It does take a good 12 seconds to run. (: USAGE: for $prompt in db:attribute('index-prompt-en-us', 'setup_as_new', 'name')/.. return $prompt => prof:time() :) let $langs := for $lang in db:list()[string-length(.) = 5][substring(., 3, 1) = '-'] return $lang return for $lang in $langs let $index-db := 'index-prompt-' || $lang let $prompt-file-name := 'prompts-' || $lang || '.xml' let $index := <index>{ for $string-group in db:open('global-content')//*[name()='string-group'] let $prompt-this-lang := $string-group/child::*[@xml:lang=$lang][normalize-space(.)!=''] let $prompt-base := if (exists($prompt-this-lang)) then $prompt-this-lang else $string-group/child::*[@xml:lang='en-us'] let $p2 := db:attribute('global-content', $string-group/data(@key), 'key')/..[name()='prompt'][ancestor::*[last()]/name()='prompt-refs'][@package=$string-group/data(@package)][@pkg-version=$string-group/data(@pkg-version)] let $prompts := ( for $prompt-match in $p2 return copy $copy := $prompt-base modify ( for $attr in $prompt-match/@*[name()!='package' and name()!='pkg-version' and name()!='key'] return insert node $attr into $copy ) return $copy ) return $prompts }</index> return if (db:exists($index-db)) then db:replace($index-db, $prompt-file-name, $index) else db:create($index-db, $index, $prompt-file-name , map { 'updindex': true() }) -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 2

UPDINDEX on custom index structures not working... where is my missing link?
by France Baril 21 Dec '19

21 Dec '19

Hi, I'm trying to build something that is similar to custom indices but that actually returns resolved nodes (nodes created by manipulating and merging nodes from the original DB). 1- I created a script that creates new index DBs. UPDINDEX and AUTOOPTIMIZED are set on both the 'index' DB and the DB from which the new DBs were created 2- I can create a query on any of the new 'index' DBs to extract the nodes I want. 3- After I change the content in the source DB, the response shows that the nodes in the 'index' DBs are not updated. I get the old result. I feel that I am missing a link between the source and the index DBs to make the update mechanism work. Since I do not want to get the original node, but one that I create, I did not use db:node-id or db:open-id shown in the documentation example. Is that where the link is created? CODE THAT CREATES THE INDEX DBs let $langs := for $lang in ('en-us', 'es-es') (:db:list()[string-length(.) = 5][substring(., 3, 1) = '-']:) return $lang return for $lang in $langs let $index := <index>{ for $string-group in db:open('global-content')//*[name()='string-group'] let $prompt-this-lang := $string-group/child::*[@xml:lang=$lang][normalize-space(.)!=''] let $prompt-base := if (exists($prompt-this-lang)) then $prompt-this-lang else $string-group/child::*[@xml:lang='en-us'] let $prompts := for $prompt-match in db:open('global-content')/*[name()='prompt-refs']/*[name()='prompt'][@package=$string-group/data(@package)][@pkg-version=$string-group/data(@pkg-version)][@key=$string-group/data(@key)] return copy $copy := $prompt-base modify ( for $attr in $prompt-match/@*[name()!='package' and name()!='pkg-version' and name()!='key'] return insert node $attr into $copy ) return $copy return $prompts }</index> return db:create('index-prompt-' || $lang, $index, 'prompts-' || $lang || '.xml', map { 'updindex': true(), 'autooptimize': true() }) CODE THAT QUERIES THE 'INDEX' DBS (# db:enforceindex #) { for $prompt in db:attribute('index-prompt-en-us', 'setup_as_new', 'name')/.. return $prompt => prof:time() } -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 7

BaseX 9.3.1: Bug Fixes, Performance Tweaks
by Christian Grün 20 Dec '19

20 Dec '19

Hi all, A new BaseX maintenance release is available. It contains minor bug fixes and performance tweaks: http://basex.org Have fun with the new release, have a good time, Christian BaseX Team

1 0

How many QueryModule instances can be created?
by Johannes Echterhoff 18 Dec '19

18 Dec '19

Hello, We are struggling with a situation where we have built a custom Java QueryModule (as described in http://docs.basex.org/wiki/Java_Bindings) - let's call it M1, and that module is imported by two other XQuery modules (let's call them M2 and M3). When executing an XQuery that imports M1 and the other XQuery modules M2 and M3 within the BaseX GUI, multiple different instances of M1 are created. That is an issue, because we initialise the module M1 with some information (a custom geometry index) before calling any other function of M1. This initialization shall only occur once, because it is quite time-consuming. Also, the information from this initialization procedure must be available whenever the (non-initialization) functions of M1 are called - whether in the XQuery or the other modules M2 and M3. The situation is as follows: * testquery.xq imports the custom module M1 * testquery.xq also imports the XQuery modules M2 and M3, which are stored in other XQuery files; M2 and M3 both import M1 * the custom module is initialized in testquery.xq - not in M2 and M3 * non-initialization functions of M1 are called in testquery.xq, as well as by functions declared in M2 and M3 - but only after initialization functions of M1 have been called in testquery.xq We get exceptions when functions of M1 are called by functions declared in M2 and M3, apparently because multiple instances of M1 have been created by BaseX (tested with the BaseX GUI), and only one of them has been initialized (in testquery.xq). After this lengthy introduction of the issue at hand, here are my questions: * Is it expected behavior that multiple instances of a Java QueryModule (M1, in my scenario) may be created and used during an execution of a query scenario like the one described above - or in general? Or should there only ever be a single such instance, regardless of how many times the module is imported? o Note: The functions used to initialize M1 are non-deterministic and declared as such. Not entirely sure if that makes a difference regarding how many times M1 would be created. * I have the same questions for the case that the Java QueryModule M1 was packaged in a XAR (as described in http://docs.basex.org/wiki/Repository#EXPath_Packaging). Would such a packaging approach actually make any difference? Apologies that I do not have a small, self-contained example project to demonstrate this. I hope that I explained the issue with sufficient detail and clarity. If not, just let me know. Best regards, Johannes P.S.: If you have suggestions for a better approach of handling such a scenario, where a QueryModule must be initialized before it can be used, and there shall only be a single instance of this module within the execution of an XQuery, let me know.

2 8

Re: [basex-talk] Huge No of XML files.
by Liam R. E. Quin 18 Dec '19

18 Dec '19

On Wed, 2019-12-18 at 11:10 +0530, Sreenivasulu Yadavalli wrote: > > > What exactly do you mean by moving collections around?. > > A: moving the collections in the same system. So, you use the Linux "mv" command to do this? Or what? What exactly do you mean by collections? I for one would find it easier if you would stop talking in riddles, as my telepathy skills are weak. > And every day we have to > update the existing collection with call data. So finding the > collection is > taking more time How do you look for the collection? Isn't it a separate BaseX database? > > Are you taking a database with 100 million documents and renaming > 50,000 of them? > > What operations exactly are slow? > > A: finding the existing collection. find / -name collection.db ? This is a little frustrating in that you are asking for people's help but not explaining the problem. Are you saying that fn:collection() is slow in BaseX? What arguments are you passing it exactly? What is the size, in gigabytes, of the database, on disk? How many documents are in it? Can you give step-by-step EXACT AND PRECISSE instructions so someone else could reproduce the problem you have having? Complete and exact instructions, with sample files if needed, so they can reproduce the problem on their own computer? A database with 80,000 files is easy to "find" here, and opens quickly, in a small fraction of a second. It doesn't take hours. Is something else running on your computer that makes it slow?? Note: please remember to copy the list in your replies, as the BaseX people are far more knowledgeable about BaseX than i am :) My goal as an analyst is to get you to explain the problem you are having clearly enough that you can get an answer :) Liam -- Liam Quin, https://www.delightfulcomputing.com/ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org

1 0

Text index
by Giuseppe G. A. Celano 12 Dec '19

12 Dec '19

Hi, I am wondering whether it is possible to use indexes in a script even if a database is not created explicitly/previously. I see that sometimes, when comparing different XML texts accessed via doc(), text indexes are automatically created/used, but it is not clear to me whether/how I can specify/force the use of them also in the script. Thanks. Best, Giuseppe

2 1

Xslt:processor() and xslt:version() incorrect with Saxon 9.1B
by Majewski, Steven Dennis (sdm7g) 12 Dec '19

12 Dec '19

I have noticed that BaseX xslt:processor() and xslt:version() returns incorrect info when used with Saxon 9.1B It appears to identify it as a Saxon processor, and assumes it’s Saxon HE and version 3.0 . This is different from what the XSLT system-property function returns. ( I’m using Saxon9B because I have stylesheets based on that version that use external java functions. I’m going to be updating these to use BaseX XQuery functions to pass params into the stylesheet instead of those external functions internal to the stylesheets, and I was hoping I could use those xslt:processor() function to keep track of which Saxon library was active as I was migrating the stylesheets. Setting CLASSPATH on the command line works OK for that use case, but I’m working on RESTXQ web apps, so I need to keep track of config when web app was started. ) Xq/processor.xq: let $XS := <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0" > <xsl:output method="text" /> <xsl:template match="/" > <xsl:value-of select="system-property( 'xsl:vendor' )" />, <xsl:value-of select="system-property( 'xsl:vendor-url' )" /> XSLT Version=<xsl:value-of select="system-property( 'xsl:version' )" /> </xsl:template> </xsl:stylesheet> return (xslt:transform-text( <null/>, $XS ), concat('(', xslt:processor(), ', Version:', xslt:version(), ')') ) $ basex Xq/processor.xq Apache Software Foundation (Xalan XSLTC), http://xml.apache.org/xalan-j XSLT Version=1.0 (Java, Version:1.0) $ CLASSPATH=$SAXON9HE basex Xq/processor.xq Saxonica, http://www.saxonica.com/ XSLT Version=3.0 (Saxon HE, Version:3.0) $ CLASSPATH=$SAXON91B basex Xq/processor.xq SAXON 9.1.0.8 from Saxonica, http://www.saxonica.com/ XSLT Version=2.0 (Saxon HE, Version:3.0) I think it would be better to make XQuery xslt functions return the same values as XSLT system-property function. However, I would understand why you might not want to make an incompatible and slower change to current behavior, especially for a rather rare use case. If you keep the current behavior, I suggest a documentation note about how the function determines that value and that it may not be accurate in all circumstances. I can obviously write my own function as above to get correct values. ( I know I mostly only write to this list when I have a problem or find an issue, however this *really* is a very solid implementation. Thank You! ) — Steve Majewski

2 1

running in parallel
by celano＠informatik.uni-leipzig.de 09 Dec '19

09 Dec '19

Hi, I am trying to run two BaseX scripts in parallel using: xquery:fork-join( ( function() {xquery:eval(xs:anyURI('extract_from_ocr1.xq'))} , function (){xquery:eval(xs:anyURI('extract_from_ocr2.xq'))} ) ) As far as I can understand (read below), the scripts are kind of run in parallel, but still the time benefit of that does not seem much in comparison with running in sequence (~25s vs ~28s). The files contain the same function, which reads files from a directory, performs some calculation, and saves the result in a file (the two scripts work on different directories). I infer that the previous script is run in parallel because the files for the results are created at the same time. I tried to do the same with GNU parallel, and in that case the files are actually run in parallel. Do we know why the execution time is not (more or less) halved in BaseX? Thanks. Ciao, Giuseppe

5 5

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk December 2019