BaseX-Talk

basex-talk@mailman.uni-konstanz.de

9 participants
5256 discussions

Possible bug in database size (bytes) after database optimize from BaseX Database Administration
by Geoff Alexander 16 Jul '20

16 Jul '20

I've found that when I perform a database optimize on an unoptimized database that while the record count decreases as expected, the database size (bytes column) stay the same. Is this a bug in reporting the database size (less severe problem) or a bug in the database not reducing on optimize (more severe problem)? I've encountered this with both BaseX 8.6.7 and 9.3.3 running on Windows 10. Thanks, Geoff Alexander, Ph.D. Software Engineer, Corporate Tools Development IBM Corporation Charlotte, NC

2 8

Database file path
by Vladimir Churyukin 16 Jul '20

16 Jul '20

Hello, We have a data transformation pipeline that works with XML files of different sizes, sometimes big (up to several gigabytes). We are using BaseX to do the transformations. For smaller files we use the MAINMEM option, because the whole database can fit in memory. But for some files we can't do that, but all databases we create are simply disposable, and we'd like to control where we put them, and destroy them after the processing. Is there a special option or any other way to specify where particular database's files will reside? How to specify that when we call CreateDB command? thank you, Vladimir

2 4

Corrupt database after update
by Johannes Bauer 16 Jul '20

16 Jul '20

Hi, we have a rather strange and hard to track problem with corrupted databases. Our setup is: * Docker container with a Tomcat that hosts BaseX with some custom RESTXQ services * BaseX 9.2.4 * Java 14 * Docker runs on a Linux VM Workflow * Create database with RESTXQ service call * Import JSON document with RESTXQ service call o call this multiple times. After some Import calls, the import fails and the database is corrupt from this point on. We first thought that it has something to do with the content of the document. But we found no pattern. Sometimes it works, but sometime it does not. There is no concurrency involved. There are no other clients that read or write to the database. We also tried to deactivate the UPDINDEX setting. But it had no effect and we could reproduce the error with and without the automatic index update. The logs in case of errors look like this: 06:58:00.683 172.18.0.2:33728 admin REQUEST [PUT] /c42-core/api/v1/restxq/user/documents/c42-index/metadata%40document06:58:00.700 172.18.0.2:33728 admin 500 Unexpected error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 9.2.4 Java: Oracle Corporation, 14.0.1 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: Index 4 out of bounds for length 1 at org.basex.io.random.TableDiskAccess.fpre(TableDiskAccess.java:507) at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:467) at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:156) at org.basex.data.Data.kind(Data.java:304) at org.basex.query.up.DataUpdates.prepare(DataUpdates.java:133) at org.basex.query.up.ContextModifier.prepare(ContextModifier.java:90) at org.basex.query.up.Updates.prepare(Updates.java:168) at org.basex.query.QueryContext.update(QueryContext.java:678) at org.basex.query.QueryContext.iter(QueryContext.java:332) at org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:73) at org.basex.http.web.WebResponse.create(WebResponse.java:63) at org.basex.http... 16.59 ms Our service does not much. It just calls db:replace(). declare variable $documents:IMPORT_OPTS := map {'chop': fn:false(), 'stripns': fn:false(), 'intparse': fn:true()};declare%rest:PUT("{$xml}")%rest:consumes("application/xml")%rest:produces("application/json")%rest:path("/user/documents/{$databaseId}/{$documentId}")%updatingfunction documents:create($databaseId as xs:string, $documentId as xs:string, $xml as document-node()){if (db:exists($databaseId)) then (update:output(response:empty(204, ())),db:replace($databaseId, documents:decode($documentId), $xml, $documents:IMPORT_OPTS))else (update:output(response:json(errors:error('C42UDO002', map {'databaseId': $databaseId}), 404)), ())}; I've attached the example input document. One addition: We could not reproduce this error running the Docker container on a Windows host. Any feedback or hints to solve this are greatly appreciated. Best regards Johannes

1 0

Maven release
by Vladimir Churyukin 15 Jul '20

15 Jul '20

Hello, Are there any regular maven artifact releases? The last one I see on mvnrepository.com is https://mvnrepository.com/artifact/org.basex/basex/9.2.4. Is it possible to release all new versions there? Thank you, Vladimir

2 1

BaseX 9.4: The Summer Edition
by Christian Grün 14 Jul '20

14 Jul '20

We are glad to give you Version 9.4 of BaseX, our XML framework, database system and XQuery 3.1 processor: https://basex.org/ We have focused on rewriting and optimizing complex XQuery code and speeding up your RESTXQ apps: WEB APPLICATIONS - DBA: support for millions of log entries - LOGTRACE option: write trace output to logs or STDERR - rest:init: Full or partial initialization of RESTXQ cache - basexhttp: register job services GUI - result view: show number of results and result size - Shift F4-F7: toggle search options (case, regeex, ..) - Ctrl-Shift-B: jump to matching bracket XQUERY LOCKING - single lock option for reading and updating queries - Java bindings: annotation added for updating functions XQUERY FUNCTIONS - convenience functions: json:doc, csv:doc, html:doc - element names: convert:encode-key, convert:decode-key - jobs:eval: option added for writing log entries - session module: only create new sessions if required XQUERY PERFORMANCE - boolean comparisons: flatten nested expressions - boolean expressions: merge conjunctions - comparisons: simplify operands with if expression - database functions: always open at compile time - databases: propagate to more expressions at compile time - filters: inline context for single items - filters: rewrite to simple maps - FLWOR: inline where clauses into let clauses - FLWOR: merge last with return clause, rewrite to simple map - FLWOR: rewrite "return if ..." to "where ... return" - fn:for-each, fn:filter: rewrite to FLWORs and filters - if expression, EBV tests: simplify boolean tests - lists, sets, logical expressions: flatten expressions - lists: rewrite to union expressions - logical expressions: apply more boolean algebra rules - paths: remove redundant predicates - predicates with name functions: rewrite to name tests - predicates: merge expressions, discard redundant tests - predicates: remove of superfluous and redundant tests - set expressions: merge operands - simple map: inline cast expressions - simple maps: faster evaluation - simple maps: merge operands - simple maps: rewrite to paths, optimize for index access - switch: rewrite to if expression - types: skip redundant checks, promotions and conversions For a more comprehensive list of added and updated features, check out our documentation (docs.basex.org) and check out the GitHub issues (github.com/BaseXdb/basex/issues). Have fun, Your BaseX Team

3 2

Joining large files
by Giuseppe G. A. Celano 13 Jul '20

13 Jul '20

Hi, I am trying to perform a join operation between two large XML files (~490 MB and ~40 MB), which are the result of the automatic conversion of old sql dumps into XML files. I created two databases for the files. The query I wrote to join them is correct because it works when I limit the join to just a few items, but it never ends if I apply it to all items: here is the xquery: https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master… <https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master…> here is the first file: https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master… <https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master…> here is the second file: https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master… <https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master…> I have also tried to use the database module functions, but without success. Am I missing anything here? Thanks. Ciao, Giuseppe

3 5

Change static directory for html pages
by Ben Pracht 13 Jul '20

13 Jul '20

Hi, I'd like to serve static content in the same server as the one handling the restxq. However, I noticed it expects the content to be in some level under the static directory. Then, the URL also needs the static directory. Is it possible to change both the directory the static files live and the URL? That way I can create a site without requiring the end user to append static to the URL to get to index.html. Ultimately, I want to create static pages, restxq services, schemas and a database, but have them isolated from the BaseX distribution. I've seen some flags to specify some directory locations, but not the static one. I'm using Ant so I can take the source distribution, copy that to a target directory, and in another step, copy my source files. I realize Ant is old, but I have no idea how to do this in Maven. By the way, I absolutely love how easy it is to do a site yet it actually seems quite powerful. Kind Regards, Ben

2 3

Re: [basex-talk] Joining large files
by Christian Grün 12 Jul '20

12 Jul '20

---------- Forwarded message ---------- From: Christian Grün <christian.gruen(a)gmail.com> Date: Sun, 12 Jul 2020 15:25:21 +0200 Subject: Re: [basex-talk] Joining large files To: "Giuseppe G. A. Celano" <celano(a)informatik.uni-leipzig.de> Hi Giuseppe, The optimizer chooses a suboptimal optimization strategy for your query: The first predicate is rewritten for index access, and the second one is sequentially processed. The search key '3' is static, so it’s possible for the compiler to estimate the evaluation time. The search key for the second key isn’t, that’s why the first predicate is given preference. I used two pragmas to speed up your query: • With db:enforceindex, the optimizer is instructed to rewrite the second predicate for index access. • With db:copynode, memory usage is reduced, and 500 MB should suffice to evaluate the query (even with the GUI, which caches the results in main memory) See [1] for more details. Your query is an interesting one; I’ll have more thoughts on how to improve the heuristics of our query optimizer. Hope this helps, Christian [1] https://docs.basex.org/wiki/XQuery_Extensions#Database_Pragmas declare variable $t:= db:open('hib_parses'); declare variable $u := db:open('hib_lemmas'); for $j in $t//row for $nn in $u//row [field[@name = 'lemma_lang_id']/text() = '3'] [(# db:enforceindex #) { field[@name = 'lemma_id']/text() = $j/field[@name='lemma_id']/text() }] return (# db:copynode false #) { element wf { <f>{ $j/* }</f>, <l>{ $nn/* }</l> } } On 7/11/20, Giuseppe G. A. Celano <celano(a)informatik.uni-leipzig.de> wrote: > Hi, > > I am trying to perform a join operation between two large XML files (~490 MB > and ~40 MB), which are the result of the automatic conversion of old sql > dumps into XML files. I created two databases for the files. The query I > wrote to join them is correct because it works when I limit the join to just > a few items, but it never ends if I apply it to all items: > > here is the xquery: > https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master… > <https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master…> > here is the first file: > https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master… > <https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master…> > here is the second file: > https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master… > <https://git.informatik.uni-leipzig.de/celano/perseus_morpheus/-/blob/master…> > > I have also tried to use the database module functions, but without success. > Am I missing anything here? Thanks. > > Ciao, > Giuseppe

1 0

xslt:transform function not working with XML Catalog
by Lizzi, Vincent 10 Jul '20

10 Jul '20

I've encountered a problem using xslt:transform in to transform some old XML that contains a DTD DOCTYPE system literal pointing to a non-working URI and also uses ENTITYREF attributes to refer to image files. I have the XML Catalog configured correctly using CATFILE. The XSLT is using unparsed-entity-uri() to convert the ENTITYREF into a file name for an href attribute. I've tested the XSLT and the XML Catalog in oXygen and in XSpec, so I know they work. I've tried a few alternatives to get this to work in BaseX and have not arrived at a working solution. The BaseX setup uses BaseX 9.3.3, XML Resolver 1.2, and Saxon HE 9.9.1-7. Passing the XML directly to xslt:transform does not use the XML Catalog for parsing the XM. I've tried making the first parameter of xslt:transform a path to a XML file, an xs:anyURI pointing to the XML file, or string containing the XML. Each time the XML fails to parse due to the non-working URI in the system literal. When I change the system literal to a URI that works the transformation works and the unparsed-entity-uri() function produces the expected file name for the href attribute. I also tried parsing the XML using doc() or fetch:xml() first and then using the parsed XML as the first parameter of xslt:transform(). The doc() or fetch:xml does use the XML Catalog to parse the XML. However, unparsed-entity-uri() function produces an empty string because the DTD information is no longer available. Hoping to be able to spot the problem I looked through the BaseX code. XsltTransform is using CatalogWrapper to set a URIResolver, but is not setting an EntityResolver. SAXWrapper is using CatalogWrapper to set an EntityResolver before parsing XML. It looks like this should work, but it's not. The next alternative is to pre-process the XML using a regular expression to replace the non-working URI in the DOCTYPE system literal with a working URI prior to xslt:transform(). This works, but it seems like this is just working around a problem. Has anyone else encountered this problem and found a better solution? Thanks, Vincent ______________________________________________ Vincent M. Lizzi Head of Information Standards | Taylor & Francis Group 530 Walnut St., Suite 850, Philadelphia, PA 19106 E-Mail: vincent.lizzi(a)taylorandfrancis.com Phone: 215-606-4221 Web: www.tandfonline.com Taylor & Francis is a trading name of Informa UK Limited, registered in England under no. 1072954 "Everything should be made as simple as possible, but not simpler." Information Classification: General

3 12

unknown type error returned for xs:anySimpleType and xs:anyType
by Yasir B 10 Jul '20

10 Jul '20

Hello, I ran the following query using BaseX 9.3.3 command-line client: 'hello' instance of xs:anyAtomicType the result of the query is `true`. But if I run this query: 'hello' instance of xs:anySimpleType an error "[XPST0051] Unknown type: xs:anySimpleType" is returned Similarly, for the query: 'hello' instance of xs:anyType an error "[XPST0051] Unknown type: xs:anyType" is returned Should xs:anySimpleType and xs:anyType be recognized? Thank you, Yasir

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk