December 2024 - BaseX-Talk - mailman.uni-konstanz.de

How best to manage link data collection and storage?
by Eliot Kimber 10 Dec '24

10 Dec '24

Over the last couple of years I’ve developed the Mirabel system that provides DITA link management and query features over large volumes of content (the ServiceNow product documentation source). In particular it knows what links to what and enables viewing the content with all link information available. For a given version of the product docs we have about 60K DITA topics and 100 root maps that organize those topics into publications. The primary job of Mirabel is to capture all the hyperlink details as defined by the DITA source and enable queries about the element-to-element and document-to-document relationships established by those links. My implementation approach for loading the link knowledge uses a multi-step process: 1. Load the entire source content into a database 2. Create a “key space” database that reflects the DITA key-to-resource mappings defined by each root DITA map. The key spaces are XQuery maps that map from key names to resources identified by their database node IDs (essentially, each use of a topic from a map has an associated unique key by which that use of the topic can be referenced). The key spaces are a prerequisite for resolving cross references to keys from one topic to other topics in the context of some root map (the same root map or a different one). 3. Create a “link record keeping” database that contains the “where used” index for the content. The where-used index maps element node IDs to a record of every reference to that node (cross references, content references, topic references from maps). The where-used index is the core data used to know where a given map or topic is used, which is used to answer questions like “what publications use this topic?” or “is this topic used at all?”. The where-used table is constructed as an XQuery map that is then turned into XML for storage (I implemented this before BaseX added direct storage of maps but given the size I think it still makes sense to store it as XML, but I could be wrong). * Process all map-to-map and map-to-topic references and create the initial map entries, one for each map and topic. * For topics referenced from maps, process all topic-to-topic references and update the records for each target topic to reflect the references to it. The map context of a given topic determines the targets of key references from that topic, so it is necessary to process the topics in the context of the root maps that use them (in DITA, root maps determine the key-to-resource bindings to which key references resolve). * For topics not referenced from any maps, add entries for them to the where-used table and process any topic-to-topic references (key references cannot be resolved but direct URI references can be). Convert the XQuery map to a single XML document and store in the link record keeping database. The resulting database takes about 150MB of storage. This third step can take two-to-three hours: 60K topics times 0.2 seconds for each topic is 3.3 hours. 0.1 seconds is about as fast as the link processing can go based on my testing. This is all done using temporary databases so as not to disturb the working databases used by the running Mirabel web application. The work is performed by a BaseX “worker” server, not the main server that serves the web site. I essentially have one BaseX http server for each core on my server and allocate work to them based on load, so queries coming from the web app will not be allocated to a worker currently doing a content update process. Once all the new link data is loaded, the temporary databases are swapped into production by renaming the production databases, renaming the temp databases to their production names, then dropping the old databases. (Saying this just now I’m realizing that I don’t know how to pause or wait for active queries against the in-production databases to finish so I can swap the databases.) Because all the index entries use node IDs, the content database and record-keeping databases have to be put into production at the same time, otherwise the content node IDs will be out of sync with the indexed record IDs. I’m working on the assumption that renaming databases is essentially instantaneous and so I can use that to swap the temp databases into production reliably. I use my job orchestration module (https://github.com/ekimbernow/basex-orchestration) to manage the sequence of operations, where each job calls the next job in the sequence once it has finished. This process works reliably for smaller volumes of content—for example, a content set with only a couple of thousand topics and four or five root maps. But at full scale I’m consistently seeing that the link record keeping database, which only has two large XML documents in it, never completes optimization: The database page shows the database with two things in it, but when you open the database’s page, they do not show up and the job that performs the optimization never completes, leaving the database in a locked state. This means the new where-used index can’t be put into production. I feel like I’m going about this the wrong way to make best use of BaseX and avoid this problem with very large databases but I don’t see any obvious alternative approaches. But it feels like I’m missing something fundamental or making a silly error that I can’t see. So my question: How would you solve this problem? In particular, how would you go about constructing the where-used index in a way that works best with BaseX? Or maybe the question is “should I be updating the in-production database with the new data and doing the swapping into the production within the database itself?” (i.e., by renaming the where used index document rather than the database itself.) I am currently using 11.6 and can move to 12 once it is released. Thanks, Eliot _____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368 servicenow servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | X<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Instagram<https://www.instagram.com/servicenow>

1 0

How to unlock a locked database?
by Eliot Kimber 10 Dec '24

10 Dec '24

BaseX 11.6 on Redhat Linux. I have long-running update processes that are failing at some point, leaving databases in a locked state that is not cleared by restarting the server. Short of manually deleting the database from the file system, how can I clear these locks or determine what process has the lock? I do not have FAIRLOCK turned on. This is in the context of my Mirabel system, which loads content into one database and then creates secondary link data databases, which requires lots of data processing and may take several hours to complete. Thanks, Eliot _____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368 servicenow servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | X<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Instagram<https://www.instagram.com/servicenow>

1 0

BaseX GUI dark theme
by ykhabins＠bellsouth.net 10 Dec '24

10 Dec '24

Hello, I am using BaseX 11.6 on Windows OS 10 and 11 machines. I followed the following post https://basex-talk.mailman.uni-konstanz.narkive.com/w8VctrkD/editor-backgrou nd-color And downloaded latest JTattoo-1.6.13.jar, and copied it to the c:\Program Files (x86)\BaseX\lib\custom directory. All new themes show up in the Visualization tab dropdown upon BaseX GUI restart. If I select Noire or HiFi themes, i.e. dark themes, BaseX GUI changes menus, toolbars, dropdowns, except Editor, Result, Info, etc. panes background. Please advise. Regards, Yitzhak Khabinsky

2 1

Content is not allowed in prolog.
by Ben Engbers 07 Dec '24

07 Dec '24

Hi, I have been working very hard on my BasexCpp client project for the last few months. The most effort has been to convert the code that worked flawlessly in Eclipse to a library that meets all the requirements (see also https://medium.com/@pananton/modern-cmake-tutorial-for-c-library-developers…, Modern CMake tutorial for C++ library developers). During that work, there turned out to be imperfections anyway. Fixing a memory-leak took the most effort. I now have 2 versions of the code. The first version is based on CMake and builds a shared library that has to be installed. The second version does not use CMake. This version has to be compiled manually. Both versions use the same code for the library. As soon as I fix the last,following problem, I will publish both new versions. The last problem is in the test program. That program contains the following line: Session.Add(“Add_XML_2”, “/home/bengbers/Git/Private/Raw/libBasexCpp/src/extdata/Articles.xml”); This line calls the "Add command" from the server protocol. The above line is executed without errors. However, the problem is that when I change the path to "./Raw/libBasexCpp/src/extdata/Articles.xml" I get the following error message: "Content is not allowed in prolog.". And since the same testprogram is used to test the different versions, the path has to be relative. My question is how I can use a relative path in the Add-command. -- Ben Engbers Grietjeshof 77 6721 VH Bennekom +31 6 23634840

3 2

Limit on Parallelism
by celano＠informatik.uni-leipzig.de 07 Dec '24

07 Dec '24

Is there a way to specify how many parallel processes Basex 11 is allowed to spawn (I am not using the database)? In combination with GNU parallel, it becomes difficult to have control on parallelism. Best, Giuseppe

4 5

org/basex/BaseX has been compiled by a more recent version of the Java Runtime (class file version 55.0)
by Patrick Durusau 06 Dec '24

06 Dec '24

Greetings, Trying to run BaseX 11.6 on Windows 11 with Java: java -version java version "1.8.0_431" Java(TM) SE Runtime Environment (build 1.8.0_431-b10) Java HotSpot(TM) Client VM (build 25.431-b10, mixed mode, sharing) Error message: Exception in thread "main" java.lang.UnsupportedClassVersionError: org/basex/BaseX has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0 Apologies but I'm still on Windows but hope to return to Linux soon, but not today. Is this a known problem? Thanks! Patrick

3 4

RestXQ OAuth2 permission checks
by Jörn Willhöft 05 Dec '24

05 Dec '24

Hi all, I recently implemented a small OAuth2 / OpenID permission check library in XQuery, complete with a small example page and a docker compose setup, including an identity provider (FusionAuth in this case). I saw this popped up once or twice on the mailing list, but I could not make much use of the provided code, so I hope this might be of interest to anybody. The example project is available on Github: https://github.com/willhoeft-it/basex-oauth2 I would be glad to hear about any opinions or get some feedback. While working on this, I realized that BaseX' permission attributes unfortunately lack a small feature that would be quite useful: The "perm" parameter to the checking function provides the method and the path, but unfortunately not any URL parameters. In my case, I would like to remember the URL (including parameters) the user called before I redirect him to the authorization server to login, so I can afterward redirect him to the page he actually wanted. But in a more general context, I can imagine it would also be useful to do permission checks on the parameters, too. Or maybe I missed something? Kind regards Jörn Willhöft

3 5