May 2022 - BaseX-Talk - mailman.uni-konstanz.de

BaseX 9.6: The Summer Edition
by Christian Grün 28 Nov '24

28 Nov '24

Dear all, We provide you with a new and fresh version of BaseX, our open source XML framework, database system and XQuery 3.1 processor: https://basex.org/ Apart from our main focus (query rewritings and optimizations), we have added the following enhancements: XQUERY: MODULES, FEATURES - Archive Module, archive:write: stream large archives to file - SQL Module: support for more SQL types - Full-Text Module, ft:thesaurus: perform Thesaurus queries - Fulltext, fuzzy search: specify Levenshtein limit - UNROLLLIMIT option: control limit for unrolling loops XQUERY: JAVA BINDINGS - Java objects of unknown type are wrapped into function items - results of constructor calls are returned as function items - the standard package "java.lang." has become optional - array arguments can be specified with the middle dot notation - conversion can be controlled with the WRAPJAVA option - better support for XQuery arrays and maps WEB APPLICATIONS - RESTXQ: Server-Timing HTTP headers are attached to the response For a more comprehensive list of added and updated features, look into our documentation (docs.basex.org) and check out the GitHub issues (github.com/BaseXdb/basex/issues) Have fun, Your BaseX Team

11 43

recursively used variables
by Rob Stapper 12 Aug '24

12 Aug '24

Hi, The code[1] below and send as attachment generates a error message: “Static variable depends on itself: $Q{http://www.w3.org/2005/xquery-local-functions}test”. I use these variables to refer to my private functions in my modules so I can easyly refer to them in a inheritance situation. It’s not a big problem for me but I was wondering if the error-triggering is justified or that it should work. [1]=========================================== declare variable $local:test := local:test#1 ; declare %private function local:test( $i) { if ( $i > 0) then $local:test( $i - 1) } ; $local:test( 10) =========================================== Kind regards, Rob Stapper Sent from Mail for Windows 10 -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

4 7

xslt:transform-report result
by Andy Bunce 11 Mar '24

11 Mar '24

Hi, Using BaseX 9.7.1 and saxon9he-9.9.1.jar The documentation suggests the ?result from xslt:transform-report should be *a* document-node where possible [1] This seems not quite to be the case when there are processing instructions or comments at the top level. In these cases a sequence of document-nodes is returned. /Andy [1] https://docs.basex.org/wiki/XSLT_Module#xslt:transform-report let $xslt:=<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0"> <xsl:mode on-no-match="shallow-copy"/> </xsl:stylesheet> let $xml:=document{ <?something type="compact"?>, <foo/>} return xslt:transform-report($xml,$xslt) Returns map { "messages": (), "result": (<?something type="compact"?>, <foo/>) }

3 7

First steps with client/server on Linux - bulk import; error message in dba
by Florian Schmitt 18 Jul '22

18 Jul '22

Hi all, i'm currently evaluating BaseX running on a headless Linux Server. It took me quite a while to get it running (the docs seem to be focused on Windows environments)... Is there a way to contribute to the documentation? First question: After importing some of documents nicely organized in collections, i've found that the REST client will display all documents on the same "level" instead of showing just the first level of collections. This isn't nice if the db contains a great number of documents.... When accessing the same db using WebDAV (Oxygen), one can nicelly navigate using the collection structure. With REST, adding the collection names to the URL will restrict the output to the db content of those collections. But without such a restriction, it would be great if the REST web interface would just return the first level of collections without any XML resource entries. Second question: What's the "best practice" for bulk imports while retaining the directory structure of the imported files using collections? I've found that Oxygen "translates" subdirectories into collections when using the "import directories" feature. Is there somehing similar using the command line client? And another issue: The "Logs" page of the DBA webapp shows the following error warning: "Unexpected error: Improper use? Potential bug? Your feedback is welcome: Contact..." Is this a critical error? How to debug this? So far, i'found the wrapper-basex log, but it doesn't contain anything useful. Since i'm using the Yajsw Wrapper, there's no debug output or debug option (?). Yours sincerely. Florian

4 8

Feature Preview: Caching Module
by Christian Grün 17 Jun '22

17 Jun '22

Hi there, BaseX 10 will come with a new Caching Module [1], which will allow you to store XQuery values (atomic items, nodes, sequences, maps, arrays, anything except function items) in a main-memory cache. Caches are persistent: Its contents will be written to disk at shutdown time and retrieved from disk in a new or restarted BaseX instance when accessed for the first time. The cache size is only limited by the available RAM. We started off with an XML format for representing caches in files, but we switched to a binary format to speed up the processing of large caches. It’s also possible to put large XML documents in the cache, but the classic database representation will give you better results in most cases. A first snapshot is available [2]. In addition, we’ll soon introduce new database functions, which will enable you to store XQuery values (…including maps) in a database. Have fun, Christian [1] https://docs.basex.org/wiki/Caching_Module [2] https://files.basex.org/releases/latest-10/

5 9

Encoding hassle ....
by Marco Lettere 13 Jun '22

13 Jun '22

Dear all, after wrapping our heads around this for hours today, we don't know how to get rid of this inconsistency. Thus I ask for help ... SSCE: BaseX 9.6.4 [Standalone] Try 'help' to get more information. > xquery file:write-text("a1.txt", "°" || out:nl()) (: Same with codepoints-to-string(176) instead of "°" :) Query executed in 183.94 ms. > xquery file:read-text("a1.txt") ° Query executed in 1.49 ms. > xquery file:write-text("a2.txt", file:read-text("a1.txt")) Query executed in 3.4 ms. > xquery file:read-text("a2.txt") [file:io-error] Decoding error: xb0 Testing the files with linux command-line tool "file", this is the output: > file a1.txt a1.txt: Unicode text, UTF-8 text > file a2.txt a2.txt: ISO-8859 text Reading the file after "copying" it seems to change the encoding. How is this supposed to be handled? Regards, Marco.

3 5

Lock avoidance using inspect:functions?
by Andy Bunce 13 Jun '22

13 Jun '22

Following Omar's message on lock management [1] I thought I would share a technique that I have tried. It makes use of the function inspect:functions [2]. The code below shows two versions of an eval-update function. After running each version the info.result view reports: - local:eval-update => a global write lock. - local:eval-update2 => no locks at all! I am pretty sure this is a bug/oversight in inspect:functions. I suspect running it risks data integrity, so it is not a good idea. However, I do make use of a non-updating version of this technique, as a way to do string evaluation without triggering a global read lock. If I know that no lockable resources are referenced then it seems safe to me? Note: inspect:functions is similar to the unimplemented standard XQuery 3.1 function fn:load-xquery-module[3] /Andy [1] https://mailman.uni-konstanz.de/pipermail/basex-talk/2022-May/017207.html [2] https://docs.basex.org/wiki/Inspection_Module#inspect:functions [3] https://www.w3.org/TR/xpath-functions-31/#func-load-xquery-module (:~ evaluate XQuery string, string is enriched with external variable definitions for convenience :) declare %updating function local:eval-update($exp as xs:string,$binds as map(*)){ let $decl:= map:keys($binds)!``[declare variable $`{ . }` external;]``=>string-join("") return xquery:eval-update($decl || $exp, $binds) }; declare %updating function local:eval-update2($exp as xs:string,$binds as map(*)){ let $decl:= map:keys($binds)!``[let $`{ . }` :=$binds("`{ . }`") ]``=>string-join("") let $mod:=("module namespace _='abuse'; declare %updating function _:foo($binds){" , $decl, "return " , $exp , "};")=>string-join("") let $p:=file:create-temp-file("APB", ".xqm") let$_:=file:write-text($p,$mod) return updating inspect:functions($p)($binds) }; (: note: the bind functionality is not used in this example) let $binds:=map{"p1":42} let $exp:="db:replace('factbook', 'abuse.xml', '<a/>')" return local:eval-update2($exp,$binds)

2 2

Cannot file:write when path has chinese characters...
by France Baril 10 Jun '22

10 Jun '22

[file:invalid-path] Invalid file path: 'Temp/pdfs/ e5-轻奢版-80-global-en-us-dh9.fo <http://xn--e5--80-global-en-us-dh9-3195b4226a2w0g.fo> Any chance this limit can be overcome? -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

4 8

Constructing and querying map structures in a dynamic way
by Markus Elfring 27 May '22

27 May '22

Hello, I would like to construct a map structure based on a dynamic selection by the means of a FLWOR expression. https://www.w3.org/TR/2017/REC-xquery-31-20170321/#id-maps The keys and associated values should be presented as a CSV-like file by a subsequent data processing step. Can it become easier to find existing XQuery script examples for such an use case? Regards, Markus

6 12

Executing optimized XQuery in RestXQ without having to deal with global lock situations
by Omar Siam 25 May '22

25 May '22

What I like about BaseX is that it is very good at optimizing self-contained queries about the size a user can read and understand [1] [2] and that it has a DB locking system for transaction management [3] that is robust and easy to understand. What I don’t like so much about BaseX is that these two mechanisms don’t work very well with complex code that is split into various modules. I use modules for code that may be shared among projects or just as a means of grouping common concerns in one module. That I don’t like this behaviour does not mean I know (or have any hope) that this can be solved in a better way without at least make unpleasant sacrifices elsewhere. It is just the setting I have to deal with. When BaseX cannot determine anymore which DBs are used in a query and which are not, it falls back to assuming there are no indexes, so automatic optimization in this regard is stopped, and it assumes that just all DBs known to BaseX are used in that query so it acquires a global lock. [4] When doing only reading queries this is not much of a problem. Using indexes in queries can be forced with functions or with the db:enforceindex pragma [5]. Problems start showing when trying to implement a CRUD RestXQ application. Create, update and delete can be implemented using the XQuery update standard but of course now this will get slow and cumbersome when for many read operations it cannot be determined which DBs they use and so a global read lock is held. That of course means that no global write lock can be acquired until all read operations are finished on all DBs known to a BaseX instance. This is especially problematic when one instance of BaseX with a RestXQ application is used to serve data from independent databases. Say one instance of BaseX has a RestXQ API that servers a lot of different dictionaries for different natural languages. This is my use case. Although the content of dictionary entries is different, the parts in the TEI/XML I try to manipulate, that are created, read, updated or deleted, are the same. So, a common API should handle many independent dictionaries, edited by many users, using one instance of BaseX. Also, when working with my biggest XML database of several GB I ran into problems when reindexing after an update. Reindexing all those GB of data takes too long and makes small updates in there impossible. Why not multiple instances of BaseX? Well because for better or worse BaseX runs in a JVM and even after I tried to minimize the memory footprint of an idle BaseX it is still a little less than 300 MB and we run a lot of services here on shared servers so RAM usage matters. Also, RAM usage is a part of the costs when using commercial cloud services. But of course, not running BaseX at all if not used is best if you pay per minute. And also: as recently discussed on the list: BaseX as any Java program gets optimized while running by the JVM and then those optimizations as well as caching will benefit all the data hosted in one instance but would be less efficient with multiple instances I assume. So how do I achieve four goals: * Keep the XQuery short and concise because that is what the optimizer can handle best? * Keep the code separated into Modules that deal with one particular aspect? * Use RestXQ and not another technique to actually implement the RESTful API? * All this while being able to split GB of XML data into portions that can be reindexed in a reasonable amount of time? The two thing that help here a lot is eval functions like xquery:eval [6] and String Constructors [7]. Say, I want to run a query but on different collections (databases). I can do this by having a list of collections and executing the actual query in a for loop with the concrete collection as a variable. If I just write the XQuery code down like this the problem is that the optimizer would need to evaluate the query to find out which databases to lock and what indexes can be used. BaseX is not built to do this (yet). It does not mock run the query. So, it decides that a global lock needs to be used. Depending on the use of XQuery Update either a global write lock or a global read lock is acquired. Easy to understand but does not help with performance here. If I want to make the situation worse for the optimizer I can use xquery:eval. That of course makes the XQuery code totally opaque to the optimizer. A global lock is guaranteed. Still another eval function is a solution here. There is the jobs module jobs:eval [8]. If I break up my code into jobs only these jobs hold locks for as long as they run. This can be a much shorter period of time than what it takes to run a whole RestXQ request. It is also possible to find a place that needs to be changed in a number of databases and then only write lock one of them to change something. So, if my data is stored in not one but several database files I can make them look like one big XML for API purposes, but still have small enough independent parts that can be indexed separately so updates with reindexing are relatively fast. If I have a search I want to perform on parts of databases that are in principle independent, like dictionary entries in a large dictionary, I can do this in parallel on each database. I tried to implement this idea with jobs:eval and it actually worked very well. Only the interface of the function was cumbersome to use the way I wanted to make use of this functionality. So, I wrote a wrapper around jobs:eval and jobs:wait that makes it easy to generate small self-contained XQuery code [9] using String Constructors [10] and some other functions used for querying the structure of the data stored in BaseX like listing and filtering databases by name [11]. Another other goal for this util:eval(s) function was to make it still easy to see errors [12]. A typical use is something like: run a filter query in all databases [13] that are found using a database name filter in some settings database [14] and use a string for comparison from a request URL parameter [15]. Find an entry out of a few million and replace it with an updated version. Of course, with reindexing [16]. What were some (unexpected) problems? Because now jobs and especially write jobs lock databases while the RestXQ code is running the RestXQ code itself cannot hold any read or write lock. That is possible in BaseX but some functions force a global lock. For example db:list. I think there are good reasons why you want to have a global lock and therefore atomicity during a query when you ask for the list of databases. Of course, my code happens to need to list databases quite often. And my code should not hold a global lock here after getting the list. My list of databases may change during a RestXQ call but I don’t care yet about that situation. I think it does not matter to me. There is also a now simple solution: Outsource db:list to its own job [17]. I also remember there was a problem with an automatic conversion of RestXQ parameters creating a randomly named lock. But it was no problem to do the conversion explicitly in XQuery code and so have the RestXQ code not hold any lock again. Now there already was a question on the mailing list about BaseX behaviour in a multithreaded environment. I don’t use that BaseX.jar in such a way with my own Java code but jobs are (Java) threads. And the interesting thing here is now that with a lot of threads (say 700) that don’t lock each other, a bottleneck shows in the way BaseX handles file access. At least the Java profiler showed me this as a primary source of wasted time [18]. If I get it correctly then file access is as usual done in 4KB portions which are read into a buffer and smaller parts are accessed from there. This way is by far the most efficient way to do this on any current operating system and file system. But now this buffer’s handling needs protection from the buffer being manipulated in different threads. All I found in the JDK for this is a performance nightmare was the Jave nio streams systems [19], which tries to guarantee quite a few threads related consistencies [20] and seems really slow. This seems to be a well-known fact documented numerous times on the internet [21]. I also tried with one of the tests BaseX contains [22] and an attempt to use FileChannel instead of the current RandomAccessFile base implementation and found the documented behaviour: Java nio file classes are no replacement for the current implementation when it comes to performance. Looking at other databases I saw they implement something OS dependent but it is hard to compare [23]. [1] https://docs.basex.org/wiki/Indexes [2] https://docs.basex.org/wiki/XQuery_Optimizations [3] https://docs.basex.org/wiki/Transaction_Management [4] https://docs.basex.org/wiki/Transaction_Management#Limitations [5] https://docs.basex.org/wiki/Indexes#Enforce_Rewritings [6] https://docs.basex.org/wiki/XQuery_Module#xquery:eval [7] https://www.w3.org/TR/2017/REC-xquery-31-20170321/#id-string-constructors [8] https://docs.basex.org/wiki/Jobs_Module#jobs:eval [9] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/util.xqm#L… [10] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/data/acces… [11] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/data/profi… [12] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/util.xqm#L… [13] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/data/acces… [14] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/data/acces… [15] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/entries.xq… [16] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/data/acces… [17] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/dicts.xqm#… [18] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/b… and other read methods there [19] https://blogs.oracle.com/javamagazine/post/java-nio-nio2-buffers-channels-a… [20] https://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html The view of a file provided by an instance of this class is guaranteed to be consistent with other views of the same file provided by other instances in the same program. [21] https://www.mathematik.uni-marburg.de/~alexmaurer/files/NioVsIo.pdf as an example. May be more recent evaluations of Java 11 or 17 nio or nio.2 performance is better? [22] https://github.com/BaseXdb/basex/blob/master/basex-core/src/test/java/org/b… [23] https://github.com/neo4j/neo4j/blob/4.4/community/native/src/main/java/org/… Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at | www.oeaw.ac.at/acdh

2 2