BaseX-Talk July 2020

basex-talk@mailman.uni-konstanz.de

22 participants
26 discussions

Fwd: [xsl] [OT] New Slack workspace for XML discussions
by Majewski, Steven Dennis (sdm7g) 25 Jul '20

25 Jul '20

FYI: > Begin forwarded message: > > From: "Adam Retter adam.retter(a)googlemail.com" <xsl-list-service(a)lists.mulberrytech.com> > Subject: [xsl] [OT] New Slack workspace for XML discussions > Date: April 16, 2020 at 11:59:25 AM EDT > To: xsl-list(a)lists.mulberrytech.com > Reply-To: <xsl-list(a)lists.mulberrytech.com> > > Hopefully the moderators won't mind me posting this? I think it has > value for readers of this list... > > Just a quick heads up in-case anyone is interested, there is now a new > vendor neutral Slack Workspace for discussing all things XML related - > https://www.xml.com/news/2020-04-slack-workspace-for-the-xml-community/ > > Cheers Adam. > -- > Adam Retter > > skype: adam.retter > tweet: adamretter > http://www.adamretter.org.uk > --~---------------------------------------------------------------- > XSL-List info and archive: http://www.mulberrytech.com/xsl/xsl-list > EasyUnsubscribe: http://lists.mulberrytech.com/unsub/xsl-list/504751 > or by email: xsl-list-unsub(a)lists.mulberrytech.com > --~--

4 3

Transaction Support
by Reto Peter 17 Jul '20

17 Jul '20

I am evaluating BaseX for my XML project. I need transaction support like - Start transaction - Run queries (read, write, update) - Commit or rollback transaction When I see the documentation, it lists Transaction Manager. But when I look at the details, I cannot find anything like that. Anyone can explain me how is the support, or is there an add-on or planned something? Best regards Reto, Frauenfeld, Schweiz

2 3

Possible bug in database size (bytes) after database optimize from BaseX Database Administration
by Geoff Alexander 16 Jul '20

16 Jul '20

I've found that when I perform a database optimize on an unoptimized database that while the record count decreases as expected, the database size (bytes column) stay the same. Is this a bug in reporting the database size (less severe problem) or a bug in the database not reducing on optimize (more severe problem)? I've encountered this with both BaseX 8.6.7 and 9.3.3 running on Windows 10. Thanks, Geoff Alexander, Ph.D. Software Engineer, Corporate Tools Development IBM Corporation Charlotte, NC

2 8

Database file path
by Vladimir Churyukin 16 Jul '20

16 Jul '20

Hello, We have a data transformation pipeline that works with XML files of different sizes, sometimes big (up to several gigabytes). We are using BaseX to do the transformations. For smaller files we use the MAINMEM option, because the whole database can fit in memory. But for some files we can't do that, but all databases we create are simply disposable, and we'd like to control where we put them, and destroy them after the processing. Is there a special option or any other way to specify where particular database's files will reside? How to specify that when we call CreateDB command? thank you, Vladimir

2 4

Corrupt database after update
by Johannes Bauer 16 Jul '20

16 Jul '20

Hi, we have a rather strange and hard to track problem with corrupted databases. Our setup is: * Docker container with a Tomcat that hosts BaseX with some custom RESTXQ services * BaseX 9.2.4 * Java 14 * Docker runs on a Linux VM Workflow * Create database with RESTXQ service call * Import JSON document with RESTXQ service call o call this multiple times. After some Import calls, the import fails and the database is corrupt from this point on. We first thought that it has something to do with the content of the document. But we found no pattern. Sometimes it works, but sometime it does not. There is no concurrency involved. There are no other clients that read or write to the database. We also tried to deactivate the UPDINDEX setting. But it had no effect and we could reproduce the error with and without the automatic index update. The logs in case of errors look like this: 06:58:00.683 172.18.0.2:33728 admin REQUEST [PUT] /c42-core/api/v1/restxq/user/documents/c42-index/metadata%40document06:58:00.700 172.18.0.2:33728 admin 500 Unexpected error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 9.2.4 Java: Oracle Corporation, 14.0.1 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: Index 4 out of bounds for length 1 at org.basex.io.random.TableDiskAccess.fpre(TableDiskAccess.java:507) at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:467) at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:156) at org.basex.data.Data.kind(Data.java:304) at org.basex.query.up.DataUpdates.prepare(DataUpdates.java:133) at org.basex.query.up.ContextModifier.prepare(ContextModifier.java:90) at org.basex.query.up.Updates.prepare(Updates.java:168) at org.basex.query.QueryContext.update(QueryContext.java:678) at org.basex.query.QueryContext.iter(QueryContext.java:332) at org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:73) at org.basex.http.web.WebResponse.create(WebResponse.java:63) at org.basex.http... 16.59 ms Our service does not much. It just calls db:replace(). declare variable $documents:IMPORT_OPTS := map {'chop': fn:false(), 'stripns': fn:false(), 'intparse': fn:true()};declare%rest:PUT("{$xml}")%rest:consumes("application/xml")%rest:produces("application/json")%rest:path("/user/documents/{$databaseId}/{$documentId}")%updatingfunction documents:create($databaseId as xs:string, $documentId as xs:string, $xml as document-node()){if (db:exists($databaseId)) then (update:output(response:empty(204, ())),db:replace($databaseId, documents:decode($documentId), $xml, $documents:IMPORT_OPTS))else (update:output(response:json(errors:error('C42UDO002', map {'databaseId': $databaseId}), 404)), ())}; I've attached the example input document. One addition: We could not reproduce this error running the Docker container on a Windows host. Any feedback or hints to solve this are greatly appreciated. Best regards Johannes

1 0

Maven release
by Vladimir Churyukin 15 Jul '20

15 Jul '20

Hello, Are there any regular maven artifact releases? The last one I see on mvnrepository.com is https://mvnrepository.com/artifact/org.basex/basex/9.2.4. Is it possible to release all new versions there? Thank you, Vladimir

2 1

BaseX 9.4: The Summer Edition
by Christian Grün 14 Jul '20

14 Jul '20

We are glad to give you Version 9.4 of BaseX, our XML framework, database system and XQuery 3.1 processor: https://basex.org/ We have focused on rewriting and optimizing complex XQuery code and speeding up your RESTXQ apps: WEB APPLICATIONS - DBA: support for millions of log entries - LOGTRACE option: write trace output to logs or STDERR - rest:init: Full or partial initialization of RESTXQ cache - basexhttp: register job services GUI - result view: show number of results and result size - Shift F4-F7: toggle search options (case, regeex, ..) - Ctrl-Shift-B: jump to matching bracket XQUERY LOCKING - single lock option for reading and updating queries - Java bindings: annotation added for updating functions XQUERY FUNCTIONS - convenience functions: json:doc, csv:doc, html:doc - element names: convert:encode-key, convert:decode-key - jobs:eval: option added for writing log entries - session module: only create new sessions if required XQUERY PERFORMANCE - boolean comparisons: flatten nested expressions - boolean expressions: merge conjunctions - comparisons: simplify operands with if expression - database functions: always open at compile time - databases: propagate to more expressions at compile time - filters: inline context for single items - filters: rewrite to simple maps - FLWOR: inline where clauses into let clauses - FLWOR: merge last with return clause, rewrite to simple map - FLWOR: rewrite "return if ..." to "where ... return" - fn:for-each, fn:filter: rewrite to FLWORs and filters - if expression, EBV tests: simplify boolean tests - lists, sets, logical expressions: flatten expressions - lists: rewrite to union expressions - logical expressions: apply more boolean algebra rules - paths: remove redundant predicates - predicates with name functions: rewrite to name tests - predicates: merge expressions, discard redundant tests - predicates: remove of superfluous and redundant tests - set expressions: merge operands - simple map: inline cast expressions - simple maps: faster evaluation - simple maps: merge operands - simple maps: rewrite to paths, optimize for index access - switch: rewrite to if expression - types: skip redundant checks, promotions and conversions For a more comprehensive list of added and updated features, check out our documentation (docs.basex.org) and check out the GitHub issues (github.com/BaseXdb/basex/issues). Have fun, Your BaseX Team

3 2

Joining large files
by Giuseppe G. A. Celano 13 Jul '20

13 Jul '20

Hi, I am trying to perform a join operation between two large XML files (~490 MB and ~40 MB), which are the result of the automatic conversion of old sql dumps into XML files. I created two databases for the files. The query I wrote to join them is correct because it works when I limit the join to just a few items, but it never ends if I apply it to all items: here is the xquery: https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master… <https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master…> here is the first file: https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master… <https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master…> here is the second file: https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master… <https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master…> I have also tried to use the database module functions, but without success. Am I missing anything here? Thanks. Ciao, Giuseppe

3 5

Change static directory for html pages
by Ben Pracht 13 Jul '20

13 Jul '20

Hi, I'd like to serve static content in the same server as the one handling the restxq. However, I noticed it expects the content to be in some level under the static directory. Then, the URL also needs the static directory. Is it possible to change both the directory the static files live and the URL? That way I can create a site without requiring the end user to append static to the URL to get to index.html. Ultimately, I want to create static pages, restxq services, schemas and a database, but have them isolated from the BaseX distribution. I've seen some flags to specify some directory locations, but not the static one. I'm using Ant so I can take the source distribution, copy that to a target directory, and in another step, copy my source files. I realize Ant is old, but I have no idea how to do this in Maven. By the way, I absolutely love how easy it is to do a site yet it actually seems quite powerful. Kind Regards, Ben

2 3

Re: [basex-talk] Joining large files
by Christian Grün 12 Jul '20

12 Jul '20

---------- Forwarded message ---------- From: Christian Grün <christian.gruen(a)gmail.com> Date: Sun, 12 Jul 2020 15:25:21 +0200 Subject: Re: [basex-talk] Joining large files To: "Giuseppe G. A. Celano" <celano(a)informatik.uni-leipzig.de> Hi Giuseppe, The optimizer chooses a suboptimal optimization strategy for your query: The first predicate is rewritten for index access, and the second one is sequentially processed. The search key '3' is static, so it’s possible for the compiler to estimate the evaluation time. The search key for the second key isn’t, that’s why the first predicate is given preference. I used two pragmas to speed up your query: • With db:enforceindex, the optimizer is instructed to rewrite the second predicate for index access. • With db:copynode, memory usage is reduced, and 500 MB should suffice to evaluate the query (even with the GUI, which caches the results in main memory) See [1] for more details. Your query is an interesting one; I’ll have more thoughts on how to improve the heuristics of our query optimizer. Hope this helps, Christian [1] https://docs.basex.org/wiki/XQuery_Extensions#Database_Pragmas declare variable $t:= db:open('hib_parses'); declare variable $u := db:open('hib_lemmas'); for $j in $t//row for $nn in $u//row [field[@name = 'lemma_lang_id']/text() = '3'] [(# db:enforceindex #) { field[@name = 'lemma_id']/text() = $j/field[@name='lemma_id']/text() }] return (# db:copynode false #) { element wf { <f>{ $j/* }</f>, <l>{ $nn/* }</l> } } On 7/11/20, Giuseppe G. A. Celano <celano(a)informatik.uni-leipzig.de> wrote: > Hi, > > I am trying to perform a join operation between two large XML files (~490 MB > and ~40 MB), which are the result of the automatic conversion of old sql > dumps into XML files. I created two databases for the files. The query I > wrote to join them is correct because it works when I limit the join to just > a few items, but it never ends if I apply it to all items: > > here is the xquery: > https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master… > <https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master…> > here is the first file: > https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master… > <https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master…> > here is the second file: > https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master… > <https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master…> > > I have also tried to use the database module functions, but without success. > Am I missing anything here? Thanks. > > Ciao, > Giuseppe

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk July 2020