BaseX-Talk

basex-talk@mailman.uni-konstanz.de

9 participants
5256 discussions

Merge/combine databases in memory constrained environment
by Matthew Dziuban 27 Jul '21

27 Jul '21

Hi all, I have two databases in BaseX, source_db and target_db, and would like to merge them by matching on the id attribute of each element and upserting the element with a `replace` or an `insert` depending on whether the element was found in the `target_db`. `source_db` has about 100,000 elements, and `target_db` has about 1,000,000 elements. The databases look like this:  <root> <element id="1"> <element id="2"> </root>  <root> <element id="1"> </root> And my query to merge the two looks like this: for $e in (db:open("source_db")/root/element) return ( if (exists(db:open("target_db")/root/element[@id = data($e/@id)])) then replace node db:open("target_db")/root/element[@id = data($e/@id)] with $e else insert node $e into db:open("target_db")/root ) When running the query, however, I keep getting memory constraint errors. Using a POST request to BaseX's REST interface I get "Out of Main Memory" and using the BaseX java client ( https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/o…) I get "java.io.IOException: GC overhead limit exceeded". Ideally I would like to just process one element from source_db at a time to avoid memory issues, but it seems like my query isn't doing this. I've tried using the `db:copynode false` pragma but it did not make a difference. Is there any way to accomplish this? Thanks in advance! Matt

3 7

Converting a Java Set to a BaseX sequence
by Reece Dunn 23 Jul '21

23 Jul '21

Hi, I'm making use of the Java bindings in BaseX, with some of the functions returning List<String> and Set<String> types. For List<String> I can adapt that to a sequence using: declare namespace List = "java:java.util.List"; declare function util:list-to-sequence($list) { for $n in 0 to List:size($list) - 1 return List:get($list, $n cast as xs:int) }; however, I'm not sure how to do the equivalent for Set<String> (or more generally, any Iterator<T>) without converting it to a list or array first, as Set only has size() and iterator() methods. Has anyone done this before? The best I can come up with is the following, which relies on the size of the set and the number of next calls in the iterator to be the same (where it should be checking hasNext): declare namespace Set = "java:java.util.Set"; declare namespace Iterator = "java:java.util.Iterator"; declare function util:set-to-sequence($set) { let $iterator := Set:iterator($set) for $n in 0 to Set:size($set) - 1 return Iterator:next($iterator) }; More generally, it would be helpful for BaseX to have adapters for Java arrays, Lists, Sets, Maps, and Iterables/Iterators to XQuery (XDM) types and functions to construct them in XQuery (like my util:list-to-sequence function above). Kind regards, Reece

2 12

Dropping namespaces on import, serialization
by Jonathan Robie 23 Jul '21

23 Jul '21

I am playing with datasets in which namespaces are unhelpful because they make output harder to read. For production use, though, I want the namespaces to be there. Ideally, I would like to tell BaseX to drop namespaces while serializing. Sometimes. Ideally, I would like to be able to set this as an option in a .bxs file. Is that possible? Thanks! Jonathan

3 2

Different result sets using positional predicates in path expression and where clause
by Kristian Kankainen 23 Jul '21

23 Jul '21

Hello, I noticed I get different result sets with the following queries and I can't understand how they differ from each other. 1) Correct behaviour, returns 3 records let $file := db:list("Snomed")[matches(.,"MRCM")][3] return for $record in (db:open("Snomed", $file)/csv/record) [position() = (1 to 3)] return $record 2) Unexpected behaviour, returns 21 records let $file := db:list("Snomed")[matches(.,"MRCM")][3] return for $record in (db:open("Snomed", $file)/csv/record) where position() = (1 to 3) return $record I have highlighted the one line that differs between them, e.g the first uses the positional predicate in a path expression and the second uses it in a where clause. Best regards, Kristian Kankainen

3 2

Autocomplete with RESTXQ
by Imsieke, Gerrit, le-tex 21 Jul '21

21 Jul '21

Hi List, Can anyone recommend a lightweight vanilla Javascript autocomplete library that can easily be used together with BaseX RESTXQ? Maybe even a readily cloneable/modifiable example? I don’t have a preference for a server response format. The RESTXQ service may be configured to return XML, JSON, or HTML No CORS restrictions need to be considered, the query host is the same that delivers the HTML pages. Gerrit

4 7

Solidarity and closeness
by Marco Lettere 18 Jul '21

18 Jul '21

We at Nubisware would like to express all our solidarity to all the members of the list who are currently affected by the flood disaster in Central EU. In particular, our closeness with Christian, Michael and the German BaseX team who has been backing our daily work for years. Looking forward to sustain whatever initiative will be organized to help out. Stay strong. Marco & the Nubisware team.

2 1

Preserving whitespace for XQuery Updates
by Jonathan Robie 16 Jul '21

16 Jul '21

I am doing some transformations of datasets, then submitting pull requests to upstream sources on GitHub. For instance, today I am inserting some attributes, but I may be restructuring in various ways or enhancing data in various ways. To make upstreams happy, I need to be disciplined about not changing whitespace. What do I have to do? Is it sufficient to preserve whitespace when importing, do an XQuery update, and export, or can that change whitespace beyond what the update operations explicitly say? Thanks! Jonathan

2 10

BaseX 9.5.2 simple query has slow Compilation time
by arcogabbo 13 Jul '21

13 Jul '21

Hi, I admit this is my first experience with BaseX and XQuery language, but this seems weird and i can't find anything on docs or issues related to this. I created a random dataset of calls and i wanted to query this dataset using XQuery, however it turns out compile time takes too long for a relatively simple query. The dataset can be found here <https://gist.github.com/arcogabbo/8042d08b1feb5c860f86ff46d4333e84> . The query is: doc("basex/etc/dataset25k.xml")/users/user[number= "605 6230540" and calls/call/location/@city = "Sandy Springs"]/fullName Timing: - Parsing: 0.61 ms - Compiling: 382.43 ms - Evaluating: 2.1 ms - Printing: 0.14 ms - Total Time: 385.28 ms Compiling: - rewrite fn:doc(uri) to document-node() item: doc("basex/etc/dataset25k.xml") -> document { "file:///home/arcogabbo/Scrivania/basex/etc/dataset25k.xml" } - rewrite to predicate: ((number = "605 6230540") and (calls/call/location/@city = "Sandy Springs")) - apply attribute index for "Sandy Springs" Even if i try a simple query on the example dataset of the application(factbook.xml) the compilation time takes too long, for example: Query: doc("basex/etc/factbook.xml")/mondial/country[@name="Norway"]/province/city Timing: - Parsing: 0.34 ms - Compiling: 82.29 ms - Evaluating: 1.05 ms - Printing: 1.49 ms - Total Time: 85.17 ms Compiling: - rewrite fn:doc(uri) to document-node() item: doc("basex/etc/factbook.xml") -> document { "file:///home/arcogabbo/Scrivania/basex/etc/factbook.xml" } - apply attribute index for "Norway" I also tried BaseX version 9.5 and on another computer, yet compilation times are approximately the same. Any help is appreciated.

2 1

Can the full text tokenizer be instructed not to tokenize at hyphens?
by Imsieke, Gerrit, le-tex 13 Jul '21

13 Jul '21

A customer wants to include composite terms such as 'third-generation' as single tokens so that they may be offered in a completion list. I don’t think this is configurable, or is it? Gerrit

2 2

Fuzzy full-text query
by Tim Thompson 06 Jul '21

06 Jul '21

Hello, I am trying to match a local dataset against entries in a full-text index. I ran into the following case and was confused about the fuzzy querying performance: <x>Baxter, Richard, 1615-1691</x>[text() contains text "Baxter, Richard, 615-1691" using fuzzy] The BaseX documentation says that the default fuzzy search will preserve a minimum of 1 error. However, the query above returns no result. If I add the "errors" parameter with a value of 1, it does work: <x>Baxter, Richard, 1615-1691</x>[text() contains text "Baxter, Richard, 615-1691" using fuzzy 1 errors] Is this expected behavior? Thanks in advance, Tim -- Tim A. Thompson Librarian for Applied Metadata Research Yale University Library

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk