February 2022 - BaseX-Talk - mailman.uni-konstanz.de

BaseX 9.6: The Summer Edition
by Christian Grün 28 Nov '24

28 Nov '24

Dear all, We provide you with a new and fresh version of BaseX, our open source XML framework, database system and XQuery 3.1 processor: https://basex.org/ Apart from our main focus (query rewritings and optimizations), we have added the following enhancements: XQUERY: MODULES, FEATURES - Archive Module, archive:write: stream large archives to file - SQL Module: support for more SQL types - Full-Text Module, ft:thesaurus: perform Thesaurus queries - Fulltext, fuzzy search: specify Levenshtein limit - UNROLLLIMIT option: control limit for unrolling loops XQUERY: JAVA BINDINGS - Java objects of unknown type are wrapped into function items - results of constructor calls are returned as function items - the standard package "java.lang." has become optional - array arguments can be specified with the middle dot notation - conversion can be controlled with the WRAPJAVA option - better support for XQuery arrays and maps WEB APPLICATIONS - RESTXQ: Server-Timing HTTP headers are attached to the response For a more comprehensive list of added and updated features, look into our documentation (docs.basex.org) and check out the GitHub issues (github.com/BaseXdb/basex/issues) Have fun, Your BaseX Team

11 43

recursively used variables
by Rob Stapper 12 Aug '24

12 Aug '24

Hi, The code[1] below and send as attachment generates a error message: “Static variable depends on itself: $Q{http://www.w3.org/2005/xquery-local-functions}test”. I use these variables to refer to my private functions in my modules so I can easyly refer to them in a inheritance situation. It’s not a big problem for me but I was wondering if the error-triggering is justified or that it should work. [1]=========================================== declare variable $local:test := local:test#1 ; declare %private function local:test( $i) { if ( $i > 0) then $local:test( $i - 1) } ; $local:test( 10) =========================================== Kind regards, Rob Stapper Sent from Mail for Windows 10 -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

4 7

User group meeting in Prague?
by Imsieke, Gerrit, le-tex 10 Apr '22

10 Apr '22

Fellow BaseX Users! You might have heard that XML Prague 2022 will take place in June. (Unless the then-prevalent Greek letter makes it impossible even in that time of year, of course.) I asked Christian whether the BaseX team will organize a user group meeting after it had not happened for years now. Christian didn’t seem to be very fond of organizing such a meeting. I asked him whether he would be available to present new features, the roadmap, and for a Q&A session if the users themselves organized such a meeting. He agreed, and therefore I hereby ask the list members whether anyone will join me in organizing this. The plan looks as follows: We will apply for one or two 90-minute slots via the CFP process (https://www.xmlprague.cz/cfp/) We don’t need to have a fixed schedule yet by Dec. 20 (end of CFP date as currently announced – it will be extended anyway). Christian was so kind as to create a new repo, user-group, on Github. We will use one or more of its Wiki pages [1] in order to plan the event. The page will eventually evolve into an agenda if you agree. Looking forward to meeting many of you in Prague in June. And another organizing volunteer (or other volunteers), please come forward. Maybe we can also deal with it in the Wiki [2]. Gerrit [1] https://github.com/BaseXdb/user-group/wiki/2022-06-XML-Prague [2] https://github.com/BaseXdb/user-group/wiki/Members

3 3

Results of some experiments for improving full-text search speeds
by Tamara Marnell 05 Mar '22

05 Mar '22

Good morning--I hope everyone in Europe is safe and well. This week I've been working on improving the speed of full-text searches in our website, and I wanted to share what I've found to be helpful and not so helpful. Our project: We run a website of finding aids from 47 archival repositories in the western US. Archivists submit their finding aids as EADs through a CMS that indexes each document for full-text searching in BaseX, with other custom indexes for facets like subject, material type, etc. https://archiveswest.orbiscascade.org/ Our issue: We need to perform full-text searches that match documents with all of the terms in the query. The BaseX full-text index and ft:search() function work at the text node level. So we need to use the mode "any word," then group by a unique identifier in the root and run an ft:contains() with the mode "all words," like this: for $db_id in tokenize($d, '\|') for $result in ft:search($db_id, $terms, map{'mode':'any word','fuzzy':$f}) let $ead := $result/ancestor::ead let $ark := normalize-space($ead/eadheader/eadid/@identifier) (: Other node values fetched here :) group by $ark where ft:contains(string-join($result, ' '), $terms, map{'mode':'all words','fuzzy':$f}) (: Custom ranking calculations based on other node values performed here :) return <ark>{$ark}</ark> Most users of the site are interested in their repositories only, and they use only a few words to find known collections, so most of the time results are delivered in less than 5 seconds. If users want to search for multiple terms across all 47 repositories, though, the search was taking 30-40 seconds. What I DIDN'T find helpful: Modifying the XQuery 1. Performing a separate search for each term and mapping to fetch with db:open-id() later, instead of searching for all terms at once and grouping. This was slightly slower: for $db_id in tokenize($d, '\|') let $results := fold-left( (: Get the ft:search() results as a map of node IDs and results :) ) for $node_id in map:keys($results) let $ead := db:open-id($db_id, $node_id) (: Ranking etc. :) 2. For multi-term queries, creating a temporary table to search instead of the complete full-text indexes of all databases. a. Create a database like temp123456 with a full-text index. b. For the first term, loop through all databases add the documents to temp123456 with db:add($temp_id, db:path($ead)) and optimize. c. For subsequent terms, map the results of a full-text search temp123456 instead of looping through all 47 databases, and then loop through all EADs in the temp database and remove the ones that don't match with... if (not(exists($temp_results(db:node-id($ead))))) then db:delete($temp_id, db:path($ead)) d. For the final term, return the usual ranked <ark /> values, then drop the database. This didn't work because the optimization of the temporary databases took a long time--25-30 seconds for a term with 1K document results, according to the BaseX logs. It would work in theory only if the existing full-text index could be parceled out for subsequent terms, instead of being freshly rebuilt. What I DID find helpful: Changing the server We use Amazon Web Services, so I experimented with putting the project on different instance types: General, Memory-Optimized, and Compute-Optimized. The contenders: General t2.medium: 2 vCPU, 4 GB memory General t3.large: 2 vCPU, 8 GB memory Compute-optimized c6g.xlarge: 4 vCPU, 8 GB memory Memory-optimized r6g.large: 2 vCPU, 16 GB memory At the time of testing we were on t2.medium, which is the most affordable and the smallest I thought would be sufficient. For one-term queries, the difference in speed was not that much across the instance types. For longer queries, the number of virtual CPUs and memory made a big difference. Query: native american tribes in oregon t2.medium: 55 seconds t3.large: 30 seconds r6g.large: 22 seconds c6g.xlarge: 21 seconds Memory: - Increasing from 4 GB on the t2.medium to 8 GB on the t3.large and c6g.xlarge nearly doubled the speed of my queries, even for queries of 2 words like "women's march" (20 seconds on t2.medium, 8-10 seconds on the others). - Increasing from 8 GB to 16 GB on a memory-optimized instance didn't affect the speed. The java process used only 10% of the available memory, versus 50-60% on other instance types, but the speed wasn't any faster, even increasing the maximum heap size in the basexhttp startup script. This could be because heap size isn't the right property to adjust. CPU: - Increasing from 2 vCPU to 4 vCPU on a compute-optimized instance improved speeds only for extreme queries of 4 words or more. papers t3.large: 6 seconds c6g.xlarge: 7 seconds family papers t3.large: 18 seconds c6g.xlarge: 18 seconds adams family papers t3.large: 15 seconds c6g.xlarge: 14 seconds adams family papers utah t3.large: 25 seconds c6g.xlarge: 21 seconds For most queries our users are performing, which are short and restricted to one repository, it's a negligible difference that didn't justify the increase in cost for a compute-optimized instance ($40 per month vs. $60 per month). -Tamara -- Tamara Marnell Program Manager, Systems Orbis Cascade Alliance (orbiscascade.org <https://www.orbiscascade.org/>) Pronouns: she/her/hers

3 7

Re: [basex-talk] feature request: opening database at arbitrary file path
by Eric Levy 03 Mar '22

03 Mar '22

On Tue, 2022-02-22 at 23:10 +0100, Christian Grün wrote: > > The other contributor's comments were deeply rooted in particular > > assumptions that stood in stark contrast to the scenario I had > > described. > > It always depends on the way you interpret others’ comments. But > we're getting circular again (there must be a reason for it ;). > > > To me, "embedded" and "standalone" carry the same > > implication in the context of a database > > I see. With standalone, we want to indicate that the application > stands “on its own” and needs no external accessors. In contrast, > something that's embedded is fully dependant on other applications. > > In other words, if we use BaseX embedded, we import it as Java > library. If we use it as standalone processor, we access it > externally, similar to other installed applications and along with > all their idiosyncrasies. > > Would Java be an option for you? I understand the distinction. It seems the issues I originally raised would be the same in both cases. The difference is whether the application would be built as a collection of scripts versus a full Java application. I had planned to begin with Bash scripts calling the command-line database application, and then migrate to a more integrated application model (e.g. Java application using a library) only if required by the evolving needs. Frankly, I would hope to avoid the latter for the foreseeable future, as both the overall application complexity and maintenance resources are rather limited. However, as I explained, the overriding concern is whether the databases may be maintained inside of projects' folders without any dependency on resources elsewhere on the file system.

3 11

CREATEFILTER Setting Not Being Applied
by Eliot Kimber 02 Mar '22

02 Mar '22

In my basex/.basex file I have this local option: # Local Options CREATEFILTER=*.xml,*.dita,*.ditamap I have three BaseX instances running. And in the DBA panel for an instance not running on the default port (i.e., on port 10894) I see this in the Local Options column: CREATEFILTER *.xml,*.dita,*.ditamap SKIPCORRUPT ✓ However, if I do this from the Query console of that same server: db:create(‘mydatabase’, ‘/path/to/docs’) The resulting database only reflects .xml files, not .dita or .ditamap files. I’m wondering if this is a bug in how CREATEFILTER is being applied or my misunderstanding of the implications for CREATEFILTER? This is running 9.6.4 on Centos. If I specify createfilter directly on the db:create() call then of course it works as expected. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

3 3

Identify Unparseable XML Files in File System
by Eliot Kimber 26 Feb '22

26 Feb '22

My validation server loads all the XML documents it can from the file system. I have “skipcorrupt” set to true so that non-well-formed documents don’t fail the database creation or update attempt. However, as part of my validation services, I need to be able to report those documents that are not well formed and therefore didn’t make it into the database. I’m wondering what the easiest/most efficient way to do that would be within BaseX? I’m working with on the order of 36K files. The files as stored in BaseX have the same path and filename as the files on the file system relative to the directory I import from, so the correlation between files and docs in BaseX is direct and simple. One easy solution would be to simply get the disjoint of the list of files on the file system and the docs in the database and then attempt to load each file to verify that is in fact unparseable and not just not-yet-imported. But maybe there’s a more direct way that I’ve overlooked? Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 2

Integration with LDAP for Authentication?
by Eliot Kimber 26 Feb '22

26 Feb '22

For my validation application (now christened “Project Mirabel” after the Encanto character Mirabel Madrigal), I want to add user validation and authentication via connection to our internal LDAP server I think the solution is to build a custom version of BaseX server that replaces the build-in base User implementation with one that uses JNDI to talk to the LDAP server for user validation and authentication. Looking at the Java code I’m not quite sure what the best way to go about this is but I think it’s to provide a library that provides my custom LDAP-using User classes and then replace the built-in user module implementation with one that uses my User classes rather than the base User classes. Am I on the right track? I didn’t see anything that looked like a way to do a run-time configuration of the classes used for authentication. I kind of expected something like a UserValidator interface that is then used by the UserCheck implementation, which would allow for different UserValidator implementations to be set at runtime but the code is much simpler than that, which is obviously sufficient for the majority of BaseX uses. Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

1 0

Multithreading
by Philip Oliver 26 Feb '22

26 Feb '22

There are some very old messages regarding the subject of multithreading in the BaseX server but nothing recent that I can find. With the proliferation of CPUs with large core counts, it seems logical to revisit the topic so that a BaseX server can make efficient use of that resource. Has this been considered lately? BaseX is a fantastic system but I'm afraid that it receives less attention than it should because competitors, particularly Elasticsearch/OpenSearch, are architected to take advantage of multithreading and clustering. In my case, the vast majority of the time, the engine is used on a static database, for full text searching. Even a constrained case of permitted multithreading on a (temporarily) read-only database would be very useful.

1 0

Develop a module, HOWTO
by Ben Engbers 25 Feb '22

25 Feb '22

Hi, I know that it is possible to create a module with functions (I have even done that once), but I can't find the documentation anymore on how to do that. Could please someone provide the URL to this information? Thanks, Ben Engbers

3 3