BaseX-Talk

basex-talk@mailman.uni-konstanz.de

13 participants
5244 discussions

Copying databases between servers: pth.basex?
by James Ball 12 Dec '20

12 Dec '20

Hello, I’ve been debugging a strange issue on a system and I can’t quite understand what’s causing it. I create some databases on my local laptop (macOS). They provide master data for a larger system so I then copy them to a cloud server (Linode/Amazon). I recently switched updated some queries to use db:open with a path restriction to make them run faster. Everything worked on the laptop but nothing was returned on the server for the same paths. If I optimise(all) the database on the server it then works. After the optimisation, on the server there is a file called pth.basex and a file called idp.basex. On the Mac there is NO file in the database folder called pth.basex or idp.basex. I’m not sure what pth.basex or idp.basex are as they’re not referenced on Storage Layout[1] This is copying the database files directly. I have had mixed results using backup zip files - sometimes it works and sometimes it fails. Has anyone else observed this behaviour? What are the steps I should take to ensure reliable movement of databases? What are these mysterious files? I’m happy to update the Wiki with any new information . Many thanks, James [1] https://docs.basex.org/wiki/Storage_Layout <https://docs.basex.org/wiki/Storage_Layout>

2 3

Performance: first and second xQuery execution / local PC and server
by BIRKNER Michael 30 Nov '20

30 Nov '20

Hello BaseX-Community, I have a question concerning performance ... once again ... sorry ;-) First thing I noticed is that when I run a query that involves quite a lot of data (more information below), the first execution of the query takes much more time as the second execution. In my example, the first execution on a certain machine (more information below) took 4,61 hours, the second execution of the very same query took 23,8 minutes. I suppose that there is some caching mechanism in place? Is that true? Second thing is that the same xQuery mentioned above has an extremly different performance on two different machines. I am testing the xQuery on my local PC where it runs quite fast (4,7 minutes on first execution; 4,5 minutes on second execution). On a (virtual) server (where the xQuery should run in the end) it takes much longer (as mentioned above: 4,61 hours on first execution; 23,8 minutes on second execution). Here are some more details: Local PC (fast execution): - OS: Ubuntu Desktop 18.04.5 LTS - RAM: 16GB LPDDR3, 2133 MT/s - Processor: Intel Core i7-8565U 1.80GHz, 8 CPUs - Java version: OpenJDK 11.0.9.1 - BaseX version: 9.4.4 (from the zip archive), using the command line client with Xmx6g Virtual Server, VMware (slow execution): - OS: Ubuntu Server 20.04.1 LTS - RAM: 16GB (no other details available) - Processor: Intel Xeon Gold 6126 2.60GHz, 4 CPUs - Java version: OpenJDK 11.0.9.1 - BaseX version: 9.4.4 (from the zip archive), using the command line client with Xmx6g The data for testing and the xQuery that is executed are exactly the same, also the BaseX version (9.4.4) and the assigned memory (6 GB). The data that I use are freely available library data. They can be downloaded here if that is for interest: 1. https://data.dnb.de/GND/authorities-geografikum_dnbmarc_20201013.mrc.xml.gz 2. https://data.dnb.de/GND/authorities-koerperschaft_dnbmarc_20201013.mrc.xml.… 3. https://data.dnb.de/GND/authorities-kongress_dnbmarc_20201013.mrc.xml.gz 4. https://data.dnb.de/GND/authorities-person_dnbmarc_20201013.mrc.xml.gz 5. https://data.dnb.de/GND/authorities-sachbegriff_dnbmarc_20201013.mrc.xml.gz 6. https://data.dnb.de/GND/authorities-werk_dnbmarc_20201013.mrc.xml.gz So I have 6 data sets from the downloaded data. For each of these data sets I create a separate BaseX database (with options INTPARSE=true and STRIPNS=true ... other options are not touched, so that they have their default values). The databases are named "geografikum", "koerperschaft", "kongress", "person", "sachbegriff" and "werk". I have to mention that the data set "person" is quite big (~20 GB). The others are between 1 GB and 6 GB in size. The databases are not altered before the execution, so all indexes (ATTRINDEX, TEXTINDEX) should be up to date. The xQuery (see below) that I execute does this: 1. Get ID numbers from two other databases ("db1" and "db2" in the xQuery below). I get 144560 ID numbers in total. 2. Iterate over the 6 databases created from the data sets mentioned above and get all records that have one of the 144560 ID numbers that were retrieved before. 3. Count the retreived records from the 6 databases. The result is 134388 records. As mentioned: The execution (especially the first run) on the virtual server is very slow compared to the execution on the local PC. The final goal is not to count the retrieved records, but to create a new database from them. I just encountered the performance issues when I did some testing with "count()". But creating the new database has the same performance issues. So now I basically have 3 questions: 1. Why is the first execution of the xQuery so slow and the second one faster, especially on the server? 2. Why is the execution on the server much slower than on the local PC? 3. I there any way to speed that up? Would it e. g. make sense to split up the very big dataset "person" into multiple databases? Could that big dataset be the bottleneck? Finally, here is the xQuery that is executed: (: Get IDs from 2 other databases :) let $otherDBs := ('db1', 'db2') let $idsInOtherDBs := (# db:enforceindex #) { for $otherDB in $otherDBs return distinct-values(db:open($otherDB)/collection/record /datafield[@tag=('689', '100', '110', '111', '700', '710', '711')] /subfield[@code='0']) } (: Get records from 6 datasets by ID from other databases :) let $searchDBs := ('geografikum', 'koerperschaft', 'kongress', 'person', 'sachbegriff', 'werk') let $result := (# db:enforceindex #) { for $searchDB in $searchDBs return (db:text($searchDB, $idsInOtherDBs) /parent::element(subfield))[(@attribute(code)='a')] /parent::element(datafield)[(@attribute(tag)='035')] /parent::element(record) union (db:text($searchDB, $idsInOtherDBs) /parent::element(controlfield))[(@attribute(tag)='001')] /parent::element(record) } (: Count the records :) return count($result) As always: Thank you very much for any suggestions! Michael

2 2

Future of the BaseX crypto library
by Joe Wicentowski 28 Nov '20

28 Nov '20

Dear fellow BaseX users, For anyone interested in the future of the EXPath Crypto spec and lib, I posted an important update to the EXPath community group with questions for users of the spec and implementations thereof. Please feel free to chime in: https://github.com/expath/expath-cg/issues/132. Thanks very much to Christian, who already has. I'd welcome any other users of the crypto library to register their ideas. Thank you, Joe

1 0

Re: [basex-talk] Re: Sequence comparison
by Martin Honnen 27 Nov '20

27 Nov '20

4 3

Sequence comparison
by Giuseppe G. A. Celano 26 Nov '20

26 Nov '20

Hi, I have to compare two sequences and find common items, irrespective of their positions: for $a in (1,2,3,5,3) for $u in (1,3,2,3) where $a = $u group by $o := $a return <c>{$a}</c> This returns <c>1</c> <c>2</c> <c>3 3 3 3</c> I would like 3 to be repeated only twice (i.e., each item in the first sequence should pair only with one in the second sentence): is there an XQuery "trick" for that in the FLOWR expression?

3 4

Improper use/potential bug error
by Giuseppe G. A. Celano 25 Nov '20

25 Nov '20

Hi, I got an "Improper use? Potential bug?” error (see below) with a (complex) query which actually works when applied to many files (but not to all). This is the query (temporary links): https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/scripts… <https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/scripts…> applied to all files here: https://git.informatik.uni-leipzig.de/celano/latinnlp/-/tree/master/texts/p… Essentially, I have some words associated to an XPath expression + offsets (see, for example, [1]), which (words) I can retrieve by using substring($xpath, $start, $long) applied to [2]. My goal is to try to identify the text nodes the words are children of, because I want to check whether they are contained in some specific elements. Ciao, Giuseppe [1] https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/texts/p… <https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/texts/p…> [2] https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/texts/p… <https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/texts/p…> --- Error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de <mailto:basex-talk@mailman.uni-konstanz.de> Version: BaseX 9.3.2 Java: Ubuntu, 11.0.9.1 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: Maximum array size reached. at org.basex.util.Array.newSize(Array.java:299) at org.basex.util.Array.newSize(Array.java:288) at org.basex.util.TokenBuilder.add(TokenBuilder.java:267) at org.basex.util.TokenBuilder.add(TokenBuilder.java:252) at org.basex.query.QueryInfo.toString(QueryInfo.java:142) at org.basex.query.QueryContext.info <http://org.basex.query.querycontext.info/>(QueryContext.java:474) at org.basex.query.QueryProcessor.info <http://org.basex.query.queryprocessor.info/>(QueryProcessor.java:272) at org.basex.core.cmd.AQuery.extError(AQuery.java:212) at org.basex.core.cmd.AQuery.query(AQuery.java:130) at org.basex.core.cmd.XQuery.run(XQuery.java:22) at org.basex.core.Command.run(Command.java:257) at org.basex.core.Command.execute(Command.java:93) at org.basex.gui.GUI.exec(GUI.java:416) at org.basex.gui.GUI.lambda$execute$4(GUI.java:359) at java.base/java.lang.Thread.run(Thread.java:834)

2 3

RESTXQ and posting files
by celano＠informatik.uni-leipzig.de 25 Nov '20

25 Nov '20

Hi, I have a script which takes a path to an xml or txt file as an input and outputs its content (as xml or txt) after a few modifications have been applied. I would like to make the script available as a RESTXQ application, but it is not clear to me how I can specify, in a RESTXQ function, that an xml or txt file should be accepted as an input. I have been able to post by sending the content of an entire file as a string, but I guess there is a specific way to handle files. Ciao, Giuseppe

3 2

where is the "lib" folder?
by Nicholas 23 Nov '20

23 Nov '20

Is there even a lib directory? https://unix.stackexchange.com/a/621089/101935 My understanding is that, for example, a JDBC driver can be "added" to BaseX itself. Perhaps this is incorrect. thanks, Nick

2 2

A first RESTXQ function
by Nicholas 21 Nov '20

21 Nov '20

For this example: module namespace page = 'http://basex.org/examples/web-page'; declare %rest:path("hello/{$who}") %rest:GET function page:hello($who) { <response> <title>Hello { $who }!</title> </response> }; https://docs.basex.org/wiki/RESTXQ where/how is this function stored or saved? thanks, Nick

2 2

testing of updating functions in basex
by dudzikp＠yahoo.com 18 Nov '20

18 Nov '20

Dear all, I have not found a way to make basex unit module to test updating functions. I have a function site:add-movies which results in several db:add instructions. I try to achieve something like: declare %unit:test %updating function test-site:download-movies() { let $result := site:add-movies( $test-site:site-db, test-site:fake-fetch-file#1, 7 ) return unit:assert-equals(1, 1) (: here I am going to check if db contains expected elements :) }; But let command is not valid for updating expression. Basex docs mention something about update:output and update:cache which may be of some use here but no exhaustive example is given for. Anyone had similar problem? BR

2 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk