- BaseX-Talk - mailman.uni-konstanz.de

Great work! Thank you basex folks
by Liedtke, Alexander 23 Mar '25

23 Mar '25

Thanks for the great work and the new version update! I think everybody using basex should consider using the (pretty good hidden) donate button at the bottom of the basex.org homepage, to at least spend the devs a good weekend with coffee or beer… 😉

2 2

csv:parse header option not working in version 11.7
by Lizzi, Vincent 20 Mar '25

20 Mar '25

Hi BaseX team, Has the "header" option on csv:parse changed between BaseX version 11.6 and 11.7? Here is an example to illustrate what looks like a problem or change. Given a tab separated text file such as this: aaaa bbbb cccc a1 b1 c1 a2 b2 c2 This query uses the "header": "yes" option on csv:parse which should map column names to element names. "tsv-test.tsv" => file:resolve-path() => file:read-text() => csv:parse(map{'separator': 'tab', 'header': 'yes'}) BaseX version 11.6 produces the expected parse: <csv> <record> <aaaa>a1</aaaa> <bbbb>b1</bbbb> <cccc>c1</cccc> </record> <record> <aaaa>a2</aaaa> <bbbb>b2</bbbb> <cccc>c2</cccc> </record> </csv> BaseX version 11.7 seems to be ignoring the header option and produces this output: <csv> <record> <entry>aaaa</entry> <entry>bbbb</entry> <entry>cccc</entry> </record> <record> <entry>a1</entry> <entry>b1</entry> <entry>c1</entry> </record> <record> <entry>a2</entry> <entry>b2</entry> <entry>c2</entry> </record> </csv> If this is an expected change, is there any equivalent to the option "header": "yes" ? Best regards, Vincent ______________________________________________ Vincent M. Lizzi Head of Information Standards | Taylor & Francis Group E-Mail: vincent.lizzi(a)taylorandfrancis.com<mailto:vincent.lizzi@taylorandfrancis.com> Web: www.tandfonline.com<http://www.tandfonline.com/> Time zone: US Eastern Taylor & Francis is a trading name of Informa UK Limited, registered in England under no. 1072954 "Everything should be made as simple as possible, but not simpler." Information Classification: General

3 4

HTTP server performance seems very slow...
by Peter Villadsen 15 Mar '25

15 Mar '25

All, I have been using BaseX for a while, connecting to the TCP endpoint. I know the performance I typically get, and it is impressive! However, now I wanted to use the HTTP endpoint, and it seems the performance is at least 2 orders of magnitude worse! Here is the query that I am POSTing to http://localhost:8984/rest/RainFnd_6.0.10.0 <query xmlns=http://basex.org/rest> <text>/Class[@Package='ApplicationPlatform']/@Name</text> </query> This simple query will generate around 1500 results from the 13GB database (RainFnd_6.0.10.0<http://localhost:8984/rest/RainFnd_6.0.10.0>). It takes just over 7 seconds to do this. If I do this in the BaseX GUI that is self contained, it takes around 20ms. However, it seems that the time spent executing the query against the database is negligible. Please consider this query: <query xmlns=http://basex.org/rest> <text>1 + 2</text> </query> In which there there is obviously no database access. It takes almost the same amount of time as the query that accesses the database. 7 seconds to calculate 1 + 2 is too long. If I post the 1 + 2 query to the endpoint without specifying the database on the URL: http://localhost:8984/rest it takes around 7 milliseconds, close to what I expected, certainly within expectations for the time spent sending the query over the wire and serializing etc. This leads me to believe that a lot of the time (>7 seconds) may be spent opening the database each time a POST is done? Is there a way to tweak the HTTP server to "remember" the connection with the current database for a little while? This may be against the REST principles, of course. The database is guaranteed to be read-only in my case. The problem is that this makes the HTTP server inappropriate for interactive applications. I can still use the TCP server, where I get the results I need, but using the HTTP would be simpler, and have less overhead in terms of code needed to communicate with the server. Please let me know if there is a way to accomplish acceptable performance with the HTTP server. Best Regards Peter Villadsen Principal Technical Program Manager Microsoft Business Applications Group

3 7

Protecting against XML vulnerabilities
by Nico Verwer (Rakensi) 15 Mar '25

15 Mar '25

I am trying to protect my BaseX application from XML vulnerabilities, like the ones described in [https://gist.github.com/mgeeky/4f726d3b374f0a34267d4f19c9004870] and [https://learn.microsoft.com/en-us/archive/msdn-magazine/2009/november/xml-d…] My application runs as `basexhttp` inside a docker container, and I set the options in web.xml: <context-param> <param-name>org.basex.dtd</param-name> <param-value>false</param-value> </context-param> <context-param> <param-name>org.basex.xinclude</param-name> <param-value>false</param-value> </context-param> I have not found other options, for example to let the parser limit expansion of internal entities. Is there a way to set parser properties like `jdk.xml.entityExpansionLimit` in BaseX?

4 6

Incremental updates: Best practices?
by Florian Schmitt 14 Mar '25

14 Mar '25

Hi all, I’m searching for a way to apply “incremental updates” to a database. By “incremental updates”, I refer to the following situation: 1. Starting point is a database containing several xml resources, e.g. /coll1/res1.xml /coll1/res2.xml /coll2/res3.xml. 2. The update consists of a file structure like /coll1/res2.xml /coll2/res4.xml /coll3/res5.xml - so I want to update the existing resource /coll1/res2.xml and add some additional resources /coll2.res4.xml, /coll3/res5.xml. The desired result is a structure like this: /coll1/res1.xml (untouched) /coll1/res2.xml (updated) /coll2/res3.xml (untouched) /coll2/res4.xml (added) /coll3/res5.xml (added) Using the command line client and the “ADD” command, I’m able to apply the update content in a single step, but (as the documentation predicted) /coll1/res2.xml now appears twice. This isn’t the desired result - /coll1.res2.xml should be overwritten by the content of / coll1.res2.xml from the update. Using the command line client and the “PUT” command replaces / coll1.res2.xml and adds the additional resource. But it deletes / coll1/res1.xml and /coll2/res3.xml - again, not the desired result, since resources missing in the update should simply be kept untouched. In my real use case, the initial dataset contains about 18,000 resources, updates may contain hundreds of updated or new resources, so separating ADD and PUT operations manually isn't possible. A simply but ugly solution would be to keep the initial set of resources in the file system, "apply" the update there and import the complete set of files as new Database. Another solution would be to write a dedicated xquery that checks for each single resource to apply a PUT or ADD operation. Is there a better way for such a use case? Florian

3 2

Upgrading BaseX
by Csaba Fekete 02 Mar '25

02 Mar '25

Hi all, What is the best way to upgrade BaseX on a production server? My current version is 11.1 and I'd like to upgrade to the latest. My gut feeling is that all it takes to overwrite the following (and leave everything else): *bin/etc/lib/BaseX.jar* ... and then restart the http server. Any thoughts? Thanks

4 4

Re: Performance tuning
by Christian Grün 02 Mar '25

02 Mar '25

Hi Csaba (cc to the list), You already found the reason for the excessive memory consumption by yourself. If you don’t need to rely on the pathological element names, you can replace them with a simple <i/> element, e.g. as follows: let $file := file:base-dir() || 'SPANYOLORSZÁG.xml' return file:write-text-lines( 'normalized-' || $file, file:read-text-lines($file) ! replace(., '<(/?)i\d+>', '<$1i>') ) This will also reduce memory consumption quite a lot. It would be a considerable effort to change the limit for element and attribute names, and it would also increase the database size for ordinary XML input, which is it’s improbable we will touch this. However, a relatively easy exercise would be to output error messages once the limits are exceeded, and not at the very end. Best, Christian

2 1

Copy document via REST API
by Liedtke, Alexander 02 Mar '25

02 Mar '25

Hello all, I am trying to copy (duplicate) an existing document via REST API, but can not get it working. I tried via POST as commands. Can somebody please provide me a working example ? My use case is simply duplicate an existing document in a databse, as a backup file. From the documentation: <add (path='...')>[input]</add>, I translated it to xml command similar: <commands> <command>OPEN db</command> <command>ADD copy.xml source.xml</command> </commands> with diffferent variations, like adding path argument etc. However, in the end I got nothing working. I also tried it via xquery, but I would prefer xml like above. But in the end I am happy with any working example. Two more questions in that context. What is the recommended header Content-Type xml or text/plain? Does it make sense rto use the http://basex_rest_url/?command url or just the REST URL? I am pretty sure, there is an easy solution for copying documents, so any help on this is appreciated. Thank you in advance Alexander Liedtke

2 3

Re: Performance tuning
by Christian Grün 27 Feb '25

27 Feb '25

Just some quick feedback: Does it work if you specify the input along with CREATE DB? basex -c"CREATE DB taurus SPANYOLORSZÁG.xml" You can also specify a directory as input. Thanks, Christian Csaba Fekete <feketecsaba(a)gmail.com> schrieb am Do., 27. Feb. 2025, 17:36: > Hi Christian > Sorry, I thought I was sending this to the mailing list. Thanks for > answering anyway! > Now I'm trying with a smaller dataset and I am adding the documents one by > one. I also upgraded BaseX to the latest version. > The largest document is 1151M in size and it can't be imported, even if I > use attrindex and textindex. > The file is actually publicly available: > http://taurusreisen.hu/partner/v2/SPANYOLORSZAG.zip > Here is my command and the output: > /opt/basex/bin/basex -Oattrindex=true -Otextindex=true -v -V -c"OPEN > taurus; ADD ./SPANYOLORSZÁG.xml" > Database 'taurus' was opened in 18.21 ms. > Out of Main Memory. > I am thinking of solving the problem by splitting the file to several > chunks, which will be CPU-demanding but could make it work. > Any ideas are welcome. > Thank you again, and a million thanks for BaseX! It is a fantastic tool. > Regards, > Csaba > > On Thu, 27 Feb 2025 at 15:52, Christian Grün <christian.gruen(a)gmail.com> > wrote: > >> Hi Csaba, >> >> It’s difficult to give a general advice; XML documents are just too >> different. In principle, a few GB or even MB can be sufficient to create >> databases for very large collections (10 GB and more), but sometimes >> namespaces are a showstopper. See [1] for some statistics. >> >> What’s the total size of your XML documents? Can you create the database >> if you enable the text and attribute index? >> >> Best, >> Christian >> >> [1] https://docs.basex.org/main/Statistics >> >> >> >> >> On Tue, Feb 25, 2025 at 2:10 PM Csaba Fekete <feketecsaba(a)gmail.com> >> wrote: >> >>> Hi >>> I have a web server that runs Basex 11.1. The server is a VPS with 18G >>> of RAM. >>> I have a directory of documents in various sizes, ranging from a few >>> kilobytes up to 2G. >>> I am trying to import these documents with the command >>> CREATE DB mydb /path/to/docs >>> With the default jvm max heap size (2GB) I get the error: Out of main >>> memory >>> If I raise the max heap size to 4GB, I get the same error. >>> If I raise it to 8GB, the system becomes unresponsive. >>> How can I determine how much system memory I need to be able to carry >>> out this task? >>> Thanks >>> >>

2 2

Performance tuning
by Csaba Fekete 27 Feb '25

27 Feb '25

Hi I have a web server that runs Basex 11.1. The server is a VPS with 18G of RAM. I have a directory of documents in various sizes, ranging from a few kilobytes up to 2G. I am trying to import these documents with the command CREATE DB mydb /path/to/docs With the default jvm max heap size (2GB) I get the error: Out of main memory If I raise the max heap size to 4GB, I get the same error. If I raise it to 8GB, the system becomes unresponsive. How can I determine how much system memory I need to be able to carry out this task? Thanks

2 1