April 2022 - BaseX-Talk - mailman.uni-konstanz.de

User group meeting in Prague?
by Imsieke, Gerrit, le-tex 10 Apr '22

10 Apr '22

Fellow BaseX Users! You might have heard that XML Prague 2022 will take place in June. (Unless the then-prevalent Greek letter makes it impossible even in that time of year, of course.) I asked Christian whether the BaseX team will organize a user group meeting after it had not happened for years now. Christian didn’t seem to be very fond of organizing such a meeting. I asked him whether he would be available to present new features, the roadmap, and for a Q&A session if the users themselves organized such a meeting. He agreed, and therefore I hereby ask the list members whether anyone will join me in organizing this. The plan looks as follows: We will apply for one or two 90-minute slots via the CFP process (https://www.xmlprague.cz/cfp/) We don’t need to have a fixed schedule yet by Dec. 20 (end of CFP date as currently announced – it will be extended anyway). Christian was so kind as to create a new repo, user-group, on Github. We will use one or more of its Wiki pages [1] in order to plan the event. The page will eventually evolve into an agenda if you agree. Looking forward to meeting many of you in Prague in June. And another organizing volunteer (or other volunteers), please come forward. Maybe we can also deal with it in the Wiki [2]. Gerrit [1] https://github.com/BaseXdb/user-group/wiki/2022-06-XML-Prague [2] https://github.com/BaseXdb/user-group/wiki/Members

3 3

Checking the display of line breaks in the info view
by Markus Elfring 10 Apr '22

10 Apr '22

Hello, I tried the following XQuery script out with the software “BaseX 9.7”. let $t1 := ``[line 1 line 2]``, $t2 := ``[line 3 line 4]`` return ('"' || $t1 || '"', "---", '"' || $t2 || '"') I observed the following display then in the info view. “… Optimized Query: ("""line 1
line 2""", "---", """line 3
line 4""") Query: let $t1 := ``[line 1 line 2]``, $t2 := ``[line 3 line 4]`` return ('"' || $t1 || '"', "---", '"' || $t2 || '"') …” May I expect that original line breaks will be preserved in the shown query display? Regards, Markus

2 4

html document retrieval runs out of main memory
by Graydon Saunders 08 Apr '22

08 Apr '22

Hello -- I'm using the basexgui to run (minus some identifying actual values defined previously in the query) (: for each path, retrieve the document :) for $remote in $paths let $name as xs:string := file:name($remote) let $target as xs:string := file:resolve-path($name,$targetBase) let $fetched := http:send-request(<http:request method='get' override-media-type='application/octet-stream' username='{$id}' password='{$pass}' />, $remote)[2] let $use as item() := try { html:parse($fetched) } catch * { $fetched } return if ($use instance of document-node()) then file:write($target,$use) else file:write-binary($target,$use) It works, in that I get exactly 100 documents retrieved. (There are unfortunately 140+ documents in the list.) However, the query fails with an "out of main memory" error when using a recent 10.0 beta or 9.7 with Xmx set to 2g. Setting Xmx to 16g with 9.7 produces the same "out of memory" error in the same length of time (about 5 minutes). java -version says 20:27 test % java -version openjdk version "11.0.14.1" 2022-02-08 OpenJDK Runtime Environment 18.9 (build 11.0.14.1+1) OpenJDK 64-Bit Server VM 18.9 (build 11.0.14.1+1, mixed mode, sharing) It's entirely possible I'm going about fetching files off a web server the wrong way; it's possible there's something there that's rather large, but I doubt it's that large. What should I be doing instead? Thanks! Graydon

2 4

I would like to share some code I use for easier RestXQ development
by Omar Siam 08 Apr '22

08 Apr '22

Hi list, I am using BaseX here at my institution a lot as an XML database that provides data using RESTful APIs [1] or serves HTML, JS and CSS with RestXQ which in turn uses HTTP requests to fetch more data [2]. Our main use of BaseX databases is about TEI/XML encoded dictionaries of various sizes. That means we mainly want to search for, get and change small tei:entry parts in larger TEI/XML documents. BaseXwas amazingly stable (compared to some other open source XML database existing today) over the last few years and has a very solid set of built in functions that almost always are sufficient to get a job done. I also like BaseX because I am pretty sure I always could understand why things did not work and what I can do about it even without doing any Java programming (although I saw some weirdness or the other over the last few years). I created or ported some XQuery modules [3][4][6][7] and a containerization environment [5] that make my life easier and I would like to share them here and maybe have a discussion about my implementations and if others can make use of them and how (for example can these [6][7] be expath packages, I see some obstacles in the way RestXQ annotations work). I will try to give an introduction to each of the modules in some separate mails. Finally, I now write to the list as I have a performance problem with a CRUD API I created [1] for the task mentioned above when using it with a dataset that is about 7GB looking at the BaseX databases that make it up. This API uses many of the modules and techniques I came up with so I thought it might be helpful to first talk about those parts. I hope that they may be useful to others as well. I tried to get creative at finding a way to optimize querying the data without having many long lasting global lock situations and having BaseX using indexes as much as possible (this started before the db:enforceindex pragma was introduced and still works for me as expected without it) while still writing the RESTful API in BaseX' implementation of RestXQ. That is why I created [7]. It heavily uses (abuses?) BaseX' jobs module. It allows me to query in smaller BaseX databases in parallel and present them as if they were one big XML DB, which vastly improves performance on update and reindex, to a point. Still there is a file based lock (or is it even class based?) [8], I think the JVM profiling tells me, that severely limits the number of (read) operations that can be done over the API without the user having to wait so long they think the operation failed. This is a multi threading problem as I see it. Or maybe I overlooked something that would solve my problems without all the creative stuff I tried? That probably will be obvious when I have explained the current implentation in more detail which I intend to do in the next few days. [1] https://vle-curation.acdh.oeaw.ac.at/openapi/, https://github.com/acdh-oeaw/vleserver_basex [2] https://vicav.acdh.oeaw.ac.at/, https://github.com/acdh-oeaw/vicav-app [3] https://github.com/acdh-oeaw/openapi4restxq [4] https://github.com/acdh-oeaw/api-problem4restxq [5] https://github.com/simar0at/heroku-buildpack-basex [6] https://github.com/acdh-oeaw/vicav-app/blob/master/http.xqm, https://github.com/acdh-oeaw/openapi4restxq/blob/master_basex/swagger-ui.xqm [7] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/util.xqm [8] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/b… and other read mehods there Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at |www.oeaw.ac.at/acdh

2 1

Support for format “XQuery” by CSV parser configuration?
by Markus Elfring 08 Apr '22

08 Apr '22

Hello, I would like to create another database from a CSV file with the software “BaseX 9.7”. It seems that I can not choose the format option “XQuery” on the tab “Parsing” of the dialogue “Create Database”. https://docs.basex.org/wiki/CSV_Module#Conversion Will any further configuration parameters become relevant for the CSV data import? Regards, Markus

2 1

Reversal of Hebrew?
by Patrick Durusau 08 Apr '22

08 Apr '22

I have attached two images, one that shows an unpointed Hebrew word entered in find, and the result as presented in the result window. Notice the "b" and "r" read right to left in Find, but read left to right in the result. If you switch the character positions in the result it would be correct. I'm not sure that searching without pointing but returning pointed results is a feature, but it would be nice to turn that on or off. Including cantillation marks (which are a separate category of mark from vowels). That is: 1) match exact text as entered 2) match text if only consonants match (for both vowels and cantillation) 3) match text if consonants and vowels match but not cantillation 4) match text if some consonants, some vowels, or some cantillation matches (yes, I know about regexes but I can either ask scholars to type in their "find" conditions or I can try to teach them regexes. Which do you think I'm going to pick? If that's a problem we can always create a XQuery library/pod whatever to keep users from having to type the filters in manually. Thanks! Hope everyone is having a great week! Patrick -- Patrick Durusau patrick(a)durusau.net Technical Advisory Board, OASIS (TAB) Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300 Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps) Another Word For It (blog): http://tm.durusau.net Homepage: http://www.durusau.net Twitter: patrickDurusau

2 1

Enclosing strings by double quotes for CSV data output
by Markus Elfring 08 Apr '22

08 Apr '22

Hello, I constructed a small XQuery script which contains the following specifications. declare option output:method "csv"; declare option output:csv "header=yes, separator=|, quotes=yes"; … The query execution succeeded (in principle) with the software “BaseX 9.7”. I observed data processing results where displayed columns contained space characters. I would expect that such data should be handled as text strings then. Thus I would expect also that these data should be enclosed by double quotes (according to the safe handling of CSV files). But this did not happen for my software test. Will any additional settings become relevant for this use case? Regards, Markus

5 23

An automated workflow for creating tested and sustainable REST API/RestXQ containers that use BaseX
by Omar Siam 07 Apr '22

07 Apr '22

For the last few years, I tried to integrate BaseX into a CI/CD workflow (the one used by gitlab [1]). My understanding of CI/CD explicitly includes automated tests so I can be as sure as possible I don’t break anything when doing further development. I pondered with using BaseX’ built in unit test framework [2], which I think is very good in many situations, but I was not really satisfied when it comes to checking RestXQ endpoints. The way it needs to be launched [3] is a bit awkward to me as a test for RestXQ and I could not come up with an easy way to automate this. When serving HTML based apps that interact with BaseX as a data store there is also no well-established way to do end-to-end testing, that is to for example programmatically click a button and capture the result, but they do exist for Node.js [4]. Unfortunately my projects almost never allow for more than a few tests due to budget and time constraints. So, I want to make them count. My experience is that such end-to-end tests are what can test the most code in JS and RestXQ with the least amount of test code. Admittedly, if you get an error, you go hunting for the real problem. Similarly, I use Node.js to do kind of an outside integration test for RESTful APIs. I have to admit I have a personal aversion against using many programming languages throughout my day. I try to use two or maximum three on any given day. But there are also other reasons: * For one JS and TypeScript are very likely consumers of a RESTful API. It is hard to escape JS or TypeScript today. * Other people are probably much better at their preferred programming languages but I hope almost anyone has a passive knowledge of JS. If this is true, then tests written in JS are good examples and candidates for translation into another languages test framework if necessary. The (now old) gitlab CI/CD workflow uses herokuish [5] and Heroku buildpacks [6] (but not the actual commercial Heroku service) to achieve two things: * Bundle an application written in some programming language in a container without the author of the program needing to have any knowledge of containers, Dockerfiles etc. The author just needs to know the dependency tools of their programming language well * Having a well-defined way to start up tests and proceed with deployment if the tests succeed or stopping right there. The second point is something that the currently used process for building containers in gitlab, Cloud Native Buildpacks [7], still cannot do because they are only in the planning phase of a container interface for launching tests. As I wanted to bundle everything needed to run and test an application running in the RestXQ environment provided by BaseX for the GitLab CI/CD process, I created a Heroku buildpack for BaseX based on the one for Node.js [8]. As I extended the Node.js buildpack everything is still controlled from package.json in principle. The “engines” definitions are amended to also include a version of BaseX and a version of Saxon to use [9]. Due to inheriting from the Node.js buildpack, JS dependencies for a HTML page that acts as the user interface for some RestXQ endpoints can be easily specified, a “build” script can build some sources written to use the Vue.js, React or angular frameworks. Or maybe only the dependencies of a more elaborate test suite need to be fetched like mocha, chai etc. [10] Also, there is a shell script “deployment/initial.sh” [11] is run so XML data can be fetched and loaded into BaseX at the time the container is built. For example, I use it to pull data from another git repository and execute a BaseX bxs script [12] to load the data, generate indices etc. A “test” script then needs to be defined to run the RESTful API endpoint tests [13] or the end-to-end test [14] automatically. Using this approach, it is possible to also launch any external process for testing so for example BaseX Unit module based tests could be launched additionally or instead of something truly Node.js based. We have decided a while ago that we want to have private git repositories with all their CI/CD at gitlab.com. We provide the compute resources to run all builds and checks using a Kubernetes cluster we own. Back then github just didn’t provide any private repositories for free but we wanted to be present there and so we have all the public projects at github.com. The good thing about public repositories on github.com is the Actions workflows there which run in a VM that provides just about any conceivable programming languages build environment plus container building tools and more. The bad thing is: there is no immediately obvious recipe of how to make good use of that VM on github.com. But if there are container building tools we can run pretty much a copy of the gitlab CI/CD workflow that people over there came up with and that is documented in their AutoDevOps workflow definitions. So I cannot really show you the gitlab CI/CD workflow I made the buildpack for in action, you have to take my word for it or try it yourself on some gitlab instance. Reenabling the old herokuish workflow for build (AUTO_DEVOPS_BUILD_IMAGE_CNB_ENABLED=false) and referencing my BaseX buildpack (BUILDPACK_URL=https://github.com/simar0at/heroku-buildpack-basex) will integrate BaseX into the AutoDevOps workflow. But you can also see it running in some of our public projects on github.com. There just a pretty stereotypical .github/workflows YAML file is needed and you can use the github.com Actions just fine [15]. That also has the advantage that one can utilize the infrastructure on github.com instead of providing some compute resources. The package might be still a little rough around the edges but I use it now for a few projects of mine (and my institution). I hope someone finds this useful. There are probably a lot of things that need to be fine tuned and I would be happy to work together to get this package better with time. [1] https://docs.gitlab.com/ee/topics/autodevops/ [2] https://docs.basex.org/wiki/Unit_Module [3] https://docs.basex.org/wiki/Unit_Module#Example [4] for one of the solutions existing for Node.js check out https://www.cypress.io/ . That is what I used recently. [5] https://github.com/gliderlabs/herokuish [6] https://devcenter.heroku.com/articles/buildpacks [7] https://buildpacks.io/ [8] https://github.com/simar0at/heroku-buildpack-basex [9] https://github.com/acdh-oeaw/zuludict-app/blob/master/package.json#L6-L12 [10] https://github.com/acdh-oeaw/vleserver_basex/blob/main/package.json#L19-L25 [11] https://github.com/acdh-oeaw/zuludict-app/tree/master/deployment [12] https://github.com/acdh-oeaw/zuludict-app/blob/master/deployment/deploy-zul… [13] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/package.json#L6… [14] https://github.com/acdh-oeaw/vicav-app/tree/master/cypress/integration [15] https://github.com/acdh-oeaw/vleserver_basex/blob/main/.github/workflows/bu… Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at | www.oeaw.ac.at/acdh

1 0

Incompatibility with Tomcat 10
by Marc Balston 06 Apr '22

06 Apr '22

Hello, I've been successfully running BaseX on Amazon AWS using Tomcat 8.5, but I've been looking to upgrade my server to Tomcat 10.0.2.However, I've run into an issue: BaseX does not startup and from the logs the cause appears to be a missing class: java.lang.NoClassDefFoundError: javax/servlet/http/HttpSessionListener I believe this is because there was a change between Tomcat 9.x and Tomcat 10 which led to a renaming of the javax.* package to jakata.* (related to a change in the servlet API version) see here for a discussion: https://stackoverflow.com/questions/66711660/tomcat-10-x-throws-java-lang-n… Is my understanding correct and is this a known issue?If it is a genuine incompatibility, it would be useful to mention this in the documentation (Web Application - BaseX Documentation) ThanksMarc Marc Balston

2 5

Support for performing queries on query results
by Markus Elfring 05 Apr '22

05 Apr '22

Hello, XQuery scripts can be used also to compute values from selected XML data structures. Query results can be exported then into other file formats. But I would like to reuse query results for further queries directly. The SQL standard is providing named views for this purpose. How will the support evolve for such data processing requirements with XML tools? Regards, Markus

6 18