- BaseX-Talk - mailman.uni-konstanz.de

set parser csv header
by Nicholas 15 Dec '20

15 Dec '20

How do I set the CSV parser to use "header" true? so that when the record is imported it has the header information as below? nicholas@mordor:~/flwor/csv$ nicholas@mordor:~/flwor/csv$ basex BaseX 9.0.1 [Standalone] Try 'help' to get more information. > > create database addressbook Database 'addressbook' created in 240.02 ms. > > open addressbook Database 'addressbook' was opened in 0.06 ms. > > set parser csv PARSER: csv > > add addressbook.csv Resource(s) added in 23.74 ms. > > xquery . <csv> <record> <entry>Name</entry> <entry>First Name</entry> <entry>Address</entry> <entry>City</entry> </record> <record> <entry>Huber</entry> <entry>Sepp</entry> <entry>Hauptstraße 13</entry> <entry>93547 Hintertupfing</entry> </record> </csv> Query executed in 219.65 ms. > > exit Enjoy life. nicholas@mordor:~/flwor/csv$ nicholas@mordor:~/flwor/csv$ basex addressbook.xq <csv> <record> <Name>Huber</Name> <First_Name>Sepp</First_Name> <Address>Hauptstraße 13</Address> <City>93547 Hintertupfing</City> </record> </csv>nicholas@mordor:~/flwor/csv$ nicholas@mordor:~/flwor/csv$ nicholas@mordor:~/flwor/csv$ cat addressbook.csv Name,First Name,Address,City Huber,Sepp,Hauptstraße 13,93547 Hintertupfing nicholas@mordor:~/flwor/csv$ nicholas@mordor:~/flwor/csv$ cat addressbook.xq xquery version "3.1"; let $text := file:read-text('addressbook.csv') return csv:parse($text, map { 'header': true() }) nicholas@mordor:~/flwor/csv$ see also: https://stackoverflow.com/q/65275618/4531180 thanks, Nick

2 1

java bindings - Data types
by Andre Bovy 15 Dec '20

15 Dec '20

Dear BasexTalk, for different (good or bad) reasons, I would like to use the java HTTPConnection instead of the http:client module. But, I am not able to find the correct syntax for these statements. I have tried the middle dot character to force the correct type, without success I receive always the message : [XPTY0004] java.net.URL instance expected as first argument, xs:anyURI found. Any idea? Thank you in advance declare namespace HttpURLConnection ="java.net.HttpURLConnection"; declare namespace URL ="java.net.URL"; let $urlString := "https:www.google.com" let $url := URL:new($urlString) let $httpConn := URL:openConnection($url) return $urlString

2 1

Slow BaseX cold performance
by Peter Villadsen 14 Dec '20

14 Dec '20

All, I am facing some issues with BaseX (9.3.1) performance when running cold, i.e. immediately after the server is started. The problem I am seeing is that the performance of moderately complex queries is terrible for a some time after the database is opened, but then improves dramatically after a few query evaluations have taken place. For instance, a certain query takes around 7 minutes to when the server has just been started, but then performance improves to consistently to take around 18 seconds. This happens for most queries, it is not a characteristic of any particular query, as far as I can tell. This is annoying, of course, and users are complaining that the system is not responsive. Here are some observations: 1. The first time the query is submitted the java process does not consume any CPU cycles and no footprint is added to the 10GB that is allocated when the database is opened (it takes around 1 minute to open the database. You can see the database info below). I would have expected a surge of activity as soon as the query was submitted, as all the 16 cores contribute to evaluating the result. 2. If I use the BaseX GUI (with an allowed 26GB but consuming around 10GB in this case) I never get any delays - The query performance is consistently around 18 secs, immediately after opening the database. 3. Both the HTTP server or TCP server exhibit this anomaly. 4. After the initial sluggishness, when the database is running warm, the performance stays satisfactory, at around 18 seconds. I have never seen it return to the bad situation. 5. The first time the query is run, the footprint rises about 1GB to around 11GB. It does not increase with subsequent query evaluations. 6. I think that the system remains warm even if I close down the java server process and start it again. Is this an accepted behavior or an anomaly? Why is the behavior different if the database is queried in the stand-alone situation (the BaseX UI client)? Is there some setting that I need to provide to the Java process to ameliorate the cold performance? Any indexes that I have to put in place? Any suggestions are welcome. Here are some stats describing the hardware setup: The machine hosting both the server and the client is a powerful 16 core box running a AMD EPYC 7452 Processor with 64GB RAM running in the cloud; since they both the client and the server are running on the same box, there should be negligible network latency in this scenario. The machine (running windows server 2016 datacenter) is doing nothing else at the time of these tests: The Idle process consumes 99% of the CPU time and almost no disk or network access takes place. The machine does not have hardware disks - They are hosted on another box, I believe. The database information: <database> <databaseproperties> <name>generated</name> <size>16 GB</size> <nodes>705422480</nodes> <documents>136673</documents> <binaries>0</binaries> <timestamp>2020-09-17T22:24:15.037Z</timestamp> <uptodate>false</uptodate> </databaseproperties> <resourceproperties> <inputpath>C:/Users/pvillads/Desktop/generated/</inputpath> <inputsize>16 GB</inputsize> <inputdate>2020-09-17T22:24:14.943Z</inputdate> </resourceproperties> <indexes> <textindex>false</textindex> <attrindex>false</attrindex> <tokenindex>false</tokenindex> <ftindex>false</ftindex> <textinclude/> <attrinclude/> <tokeninclude/> <ftinclude/> <language>English</language> <stemming>false</stemming> <casesens>false</casesens> <diacritics>false</diacritics> <stopwords/> <updindex>false</updindex> <autooptimize>false</autooptimize> <maxcats>100</maxcats> <maxlen>96</maxlen> <splitsize>0</splitsize> </indexes> </database> $ java -version java version "1.8.0_271" Java(TM) SE Runtime Environment (build 1.8.0_271-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode) Thanks, Best Regards Peter Villadsen Principal Architect Microsoft Business Applications Group

2 1

Copying databases between servers: pth.basex?
by James Ball 13 Dec '20

13 Dec '20

Hello, I’ve been debugging a strange issue on a system and I can’t quite understand what’s causing it. I create some databases on my local laptop (macOS). They provide master data for a larger system so I then copy them to a cloud server (Linode/Amazon). I recently switched updated some queries to use db:open with a path restriction to make them run faster. Everything worked on the laptop but nothing was returned on the server for the same paths. If I optimise(all) the database on the server it then works. After the optimisation, on the server there is a file called pth.basex and a file called idp.basex. On the Mac there is NO file in the database folder called pth.basex or idp.basex. I’m not sure what pth.basex or idp.basex are as they’re not referenced on Storage Layout[1] This is copying the database files directly. I have had mixed results using backup zip files - sometimes it works and sometimes it fails. Has anyone else observed this behaviour? What are the steps I should take to ensure reliable movement of databases? What are these mysterious files? I’m happy to update the Wiki with any new information . Many thanks, James [1] https://docs.basex.org/wiki/Storage_Layout <https://docs.basex.org/wiki/Storage_Layout>

2 3

Performance: first and second xQuery execution / local PC and server
by BIRKNER Michael 30 Nov '20

30 Nov '20

Hello BaseX-Community, I have a question concerning performance ... once again ... sorry ;-) First thing I noticed is that when I run a query that involves quite a lot of data (more information below), the first execution of the query takes much more time as the second execution. In my example, the first execution on a certain machine (more information below) took 4,61 hours, the second execution of the very same query took 23,8 minutes. I suppose that there is some caching mechanism in place? Is that true? Second thing is that the same xQuery mentioned above has an extremly different performance on two different machines. I am testing the xQuery on my local PC where it runs quite fast (4,7 minutes on first execution; 4,5 minutes on second execution). On a (virtual) server (where the xQuery should run in the end) it takes much longer (as mentioned above: 4,61 hours on first execution; 23,8 minutes on second execution). Here are some more details: Local PC (fast execution): - OS: Ubuntu Desktop 18.04.5 LTS - RAM: 16GB LPDDR3, 2133 MT/s - Processor: Intel Core i7-8565U 1.80GHz, 8 CPUs - Java version: OpenJDK 11.0.9.1 - BaseX version: 9.4.4 (from the zip archive), using the command line client with Xmx6g Virtual Server, VMware (slow execution): - OS: Ubuntu Server 20.04.1 LTS - RAM: 16GB (no other details available) - Processor: Intel Xeon Gold 6126 2.60GHz, 4 CPUs - Java version: OpenJDK 11.0.9.1 - BaseX version: 9.4.4 (from the zip archive), using the command line client with Xmx6g The data for testing and the xQuery that is executed are exactly the same, also the BaseX version (9.4.4) and the assigned memory (6 GB). The data that I use are freely available library data. They can be downloaded here if that is for interest: 1. https://data.dnb.de/GND/authorities-geografikum_dnbmarc_20201013.mrc.xml.gz 2. https://data.dnb.de/GND/authorities-koerperschaft_dnbmarc_20201013.mrc.xml.… 3. https://data.dnb.de/GND/authorities-kongress_dnbmarc_20201013.mrc.xml.gz 4. https://data.dnb.de/GND/authorities-person_dnbmarc_20201013.mrc.xml.gz 5. https://data.dnb.de/GND/authorities-sachbegriff_dnbmarc_20201013.mrc.xml.gz 6. https://data.dnb.de/GND/authorities-werk_dnbmarc_20201013.mrc.xml.gz So I have 6 data sets from the downloaded data. For each of these data sets I create a separate BaseX database (with options INTPARSE=true and STRIPNS=true ... other options are not touched, so that they have their default values). The databases are named "geografikum", "koerperschaft", "kongress", "person", "sachbegriff" and "werk". I have to mention that the data set "person" is quite big (~20 GB). The others are between 1 GB and 6 GB in size. The databases are not altered before the execution, so all indexes (ATTRINDEX, TEXTINDEX) should be up to date. The xQuery (see below) that I execute does this: 1. Get ID numbers from two other databases ("db1" and "db2" in the xQuery below). I get 144560 ID numbers in total. 2. Iterate over the 6 databases created from the data sets mentioned above and get all records that have one of the 144560 ID numbers that were retrieved before. 3. Count the retreived records from the 6 databases. The result is 134388 records. As mentioned: The execution (especially the first run) on the virtual server is very slow compared to the execution on the local PC. The final goal is not to count the retrieved records, but to create a new database from them. I just encountered the performance issues when I did some testing with "count()". But creating the new database has the same performance issues. So now I basically have 3 questions: 1. Why is the first execution of the xQuery so slow and the second one faster, especially on the server? 2. Why is the execution on the server much slower than on the local PC? 3. I there any way to speed that up? Would it e. g. make sense to split up the very big dataset "person" into multiple databases? Could that big dataset be the bottleneck? Finally, here is the xQuery that is executed: (: Get IDs from 2 other databases :) let $otherDBs := ('db1', 'db2') let $idsInOtherDBs := (# db:enforceindex #) { for $otherDB in $otherDBs return distinct-values(db:open($otherDB)/collection/record /datafield[@tag=('689', '100', '110', '111', '700', '710', '711')] /subfield[@code='0']) } (: Get records from 6 datasets by ID from other databases :) let $searchDBs := ('geografikum', 'koerperschaft', 'kongress', 'person', 'sachbegriff', 'werk') let $result := (# db:enforceindex #) { for $searchDB in $searchDBs return (db:text($searchDB, $idsInOtherDBs) /parent::element(subfield))[(@attribute(code)='a')] /parent::element(datafield)[(@attribute(tag)='035')] /parent::element(record) union (db:text($searchDB, $idsInOtherDBs) /parent::element(controlfield))[(@attribute(tag)='001')] /parent::element(record) } (: Count the records :) return count($result) As always: Thank you very much for any suggestions! Michael

2 2

Future of the BaseX crypto library
by Joe Wicentowski 28 Nov '20

28 Nov '20

Dear fellow BaseX users, For anyone interested in the future of the EXPath Crypto spec and lib, I posted an important update to the EXPath community group with questions for users of the spec and implementations thereof. Please feel free to chime in: https://github.com/expath/expath-cg/issues/132. Thanks very much to Christian, who already has. I'd welcome any other users of the crypto library to register their ideas. Thank you, Joe

1 0

Re: [basex-talk] Re: Sequence comparison
by Martin Honnen 27 Nov '20

27 Nov '20

4 3

Sequence comparison
by Giuseppe G. A. Celano 26 Nov '20

26 Nov '20

Hi, I have to compare two sequences and find common items, irrespective of their positions: for $a in (1,2,3,5,3) for $u in (1,3,2,3) where $a = $u group by $o := $a return <c>{$a}</c> This returns <c>1</c> <c>2</c> <c>3 3 3 3</c> I would like 3 to be repeated only twice (i.e., each item in the first sequence should pair only with one in the second sentence): is there an XQuery "trick" for that in the FLOWR expression?

3 4

Improper use/potential bug error
by Giuseppe G. A. Celano 25 Nov '20

25 Nov '20

Hi, I got an "Improper use? Potential bug?” error (see below) with a (complex) query which actually works when applied to many files (but not to all). This is the query (temporary links): https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/scripts… <https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/scripts…> applied to all files here: https://git.informatik.uni-leipzig.de/celano/latinnlp/-/tree/master/texts/p… Essentially, I have some words associated to an XPath expression + offsets (see, for example, [1]), which (words) I can retrieve by using substring($xpath, $start, $long) applied to [2]. My goal is to try to identify the text nodes the words are children of, because I want to check whether they are contained in some specific elements. Ciao, Giuseppe [1] https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/texts/p… <https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/texts/p…> [2] https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/texts/p… <https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/texts/p…> --- Error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de <mailto:basex-talk@mailman.uni-konstanz.de> Version: BaseX 9.3.2 Java: Ubuntu, 11.0.9.1 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: Maximum array size reached. at org.basex.util.Array.newSize(Array.java:299) at org.basex.util.Array.newSize(Array.java:288) at org.basex.util.TokenBuilder.add(TokenBuilder.java:267) at org.basex.util.TokenBuilder.add(TokenBuilder.java:252) at org.basex.query.QueryInfo.toString(QueryInfo.java:142) at org.basex.query.QueryContext.info <http://org.basex.query.querycontext.info/>(QueryContext.java:474) at org.basex.query.QueryProcessor.info <http://org.basex.query.queryprocessor.info/>(QueryProcessor.java:272) at org.basex.core.cmd.AQuery.extError(AQuery.java:212) at org.basex.core.cmd.AQuery.query(AQuery.java:130) at org.basex.core.cmd.XQuery.run(XQuery.java:22) at org.basex.core.Command.run(Command.java:257) at org.basex.core.Command.execute(Command.java:93) at org.basex.gui.GUI.exec(GUI.java:416) at org.basex.gui.GUI.lambda$execute$4(GUI.java:359) at java.base/java.lang.Thread.run(Thread.java:834)

2 3

RESTXQ and posting files
by celano＠informatik.uni-leipzig.de 25 Nov '20

25 Nov '20

Hi, I have a script which takes a path to an xml or txt file as an input and outputs its content (as xml or txt) after a few modifications have been applied. I would like to make the script available as a RESTXQ application, but it is not clear to me how I can specify, in a RESTXQ function, that an xml or txt file should be accepted as an input. I have been able to post by sending the content of an entire file as a string, but I guess there is a specific way to handle files. Ciao, Giuseppe

3 2