December 2024 - BaseX-Talk - mailman.uni-konstanz.de

Deeper discussion of BaseX client/server and web app implementation?
by Eliot Kimber 27 Jan '25

27 Jan '25

I’m rethinking the implementation of my Mirabel system and trying to understand how best to manage concurrency for the web app (satisfying multiple read requests from web clients) while also managing updates to databases in a way that avoids blocking of reads or delaying of updates. I’ve done a read through of the current documentation at https://docs.basex.org/ and also reviewed what I could find online and such. In the documentation I find a number of references to the “client/server” architecture but I’m not finding any particularly deep discussion of it, either in the docs or by searching on i.e., “basex client server”. When I started my Mirabel project I understood that the way to get concurrency was to use multiple BaseX HTTP instances, which can make concurrent read requests on a single set of databases. But now I can’t find the source that led me to that conclusion—I know the product docs have been reworked significantly since then, so maybe something got lost in the update? Based on my latest reading, it seems clear that in order for BaseX to best manage concurrency of reads and writes that the requests need to be within a single server instance running in a single JVM. If I’m understanding it correctly, for FAIRLOCK to enable interleaving of read and update operations it has to be operating in a single JVM, which means that having different JVMs reading from a database being updated by yet another JVM is going to be problematic because the only lock mechanism in that case is the global lock, Given that, it’s not clear how you implement a multi-user web application that needs to not block while waiting for longer-running queries to complete but still have a single server instance to satisfy the queries. So I feel like I’m missing something. In my current solution I run multiple basexhttp servers where the first server is the web app server. It uses a REST handler that inspects the load on each server and sends requests to the lowest-load server. This works but in my reading I’m starting to think that this level of complexity is not required (or maybe, more accurately, surely others would have needed the same mechanism?). In my web app I’ve implemented a general asynchronous query mechanism where I can serve web pages with elements that then trigger async requests back to the server to fetch elements (for example, I cache HTML renderings of DITA tables that are then fetched asynchronously to populate report tables that include the rendered tables). I don’t see a way to avoid this level of optimization and it’s not too complicated. This seems to work well and there are more opportunities for caching things. I’m also continuing to think about Tamara’s suggestion to use polling or web sockets to perform long-running queries asynchronously, store the results in a results database, then trigger fetching in the client (in response to an earlier question of mine about how to avoid keeping HTTP request sessions open for a long time). So my question: What is the general architecture for a web application that needs to support large numbers of concurrent users making large numbers of requests and manage updates to databases? Thanks, Eliot _____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368 servicenow servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | X<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Instagram<https://www.instagram.com/servicenow>

5 8

Re: [basex-talk] XSD validation: cache invalidation stopped working
by ykhabins＠bellsouth.net 30 Dec '24

30 Dec '24

Hello, >> Note that the GUI runs in an independent JVM; it does not influence HTTP calls. The explanation was very helpful. I ended up with a CURL command to invalidate XSD that is cached behind HTTP listener calls (different JVM). curl -i -u <username>:<password> -X POST -H "Content-Type: application/xml" -d "<query xmlns='http://basex.org/rest'><text>validate:xsd-init()</text></query>" "http://<host>:8080/rest" It emits the following: HTTP/1.1 200 OK Date: Mon, 30 Dec 2024 14:44:03 GMT Content-Type: application/xml; charset=UTF-8 Content-Length: 0 Server: Jetty(11.0.24) And everything is working as expected. Thanks a lot. Regards, Yitzhak Khabinsky

1 0

Re: [basex-talk] XSD validation: cache invalidation stopped working
by ykhabins＠bellsouth.net 30 Dec '24

30 Dec '24

Here is a full repro. Initial setup, four files: 1) XML: e:\Temp\XSD_CACHE\input.xml 2) XSD: e:\Temp\XSD_CACHE\state.xsd 3) XQuery: c:\Program Files (x86)\BaseX\webapp\XSD11_Validation_RECEIPT_2.xq 4) c# file making BaseX REST HTTP call XML file: e:\Temp\XSD_CACHE\input.xml ====================================== <?xml version="1.0"?> <root> <state>FL</state> <state>TX</state> </root> XSD file: e:\Temp\XSD_CACHE\state.xsd ========================================= <?xml version="1.0" encoding="UTF-8"?> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" version="1.0.0"> <xs:element name="root"> <xs:complexType> <xs:sequence> <xs:element maxOccurs="unbounded" ref="state"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="state" type="state_type"/> <xs:simpleType name="state_type"> <xs:restriction base="xs:string"> <xs:length value="2"/> <xs:enumeration value="FL"/>  </xs:restriction> </xs:simpleType> </xs:schema> XQuery file: XSD11_Validation_RECEIPT_2.xq ========================================== declare variable $xml as xs:string external; declare variable $xsd as xs:string external; try { let $timezone := timezone-from-dateTime(current-dateTime()) let $before_datetime := adjust-dateTime-to-timezone(convert:integer-to-dateTime(prof:current-ms()), xs:dayTimeDuration($timezone)) (: current-dateTime() :) let $result := validate:xsd-report($xml, $xsd, map { 'http://apache.org/xml/features/validation/cta-full-xpath-checking': true(), 'cache': true() }) let $after_datetime := adjust-dateTime-to-timezone(convert:integer-to-dateTime(prof:current-ms()), xs:dayTimeDuration($timezone)) (: current-dateTime() :) return <root><metadata> <result>{data($result/status)}</result> <errors>{count($result/message[@level="Error"])}</errors> <warnings>{count($result/message[@level="Warning"])}</warnings> <startTime>{$before_datetime}</startTime> <endTime>{$after_datetime}</endTime> <xmlFile>{$xml}</xmlFile> <xsdFile>{$xsd}</xsdFile> <xsdProcessor>BaseX {data(db:system()//version)}, Java {proc:property('java.vendor')} v.{proc:property('java.runtime.version')}, {validate:xsd-processor()} 2.12.2</xsdProcessor> <xsdVersion>{data(doc($xsd)/*:schema/@version)}</xsdVersion> </metadata> <messages>{ $result/message transform with { delete node @url } }</messages> </root> } catch * { <error> <errorCode>{$err:code}</errorCode> <errorDescription>{$err:description}</errorDescription> </error> } c# REST HTTP call: ================== void Main() { const string HOST = "HostName"; const int PORT = 8080; const string USERNAME = "username"; const string PASSWORD = "password"; System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls; const string XQUERY = "XSD11_Validation_RECEIPT_2.xq"; string xmlFile = @"e:\Temp\XSD_CACHE\input.xml"; string xsdFile = @"e:\Temp\XSD_CACHE\state.xsd"; string REQUESTURL = $"http://{HOST}:{PORT}/rest?run={XQUERY}&$xml={xmlFile}&$xsd={xsdFile}"; try { using (HttpClientHandler handler = new HttpClientHandler() {UseDefaultCredentials = true}) // for mdpd proxy //{UseDefaultCredentials = true, UseProxy = true, DefaultProxyCredentials = CredentialCache.DefaultNetworkCredentials}) using (HttpClient client = new HttpClient(handler)) { // Encode username and password pair with a base64 implementation. String encoded = Convert.ToBase64String(Encoding.UTF8.GetBytes(USERNAME + ":" + PASSWORD)); client.DefaultRequestHeaders.Add("Authorization", "Basic " + encoded); client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/xml")); var response = client.GetAsync(REQUESTURL).Result; System.Net.Http.HttpContent content = response.Content; var contentString = content.ReadAsStringAsync().Result; // get the actual content string if (response.StatusCode == HttpStatusCode.OK) // 200 //IsSuccessStatusCode == True { XDocument xdoc2 = XDocument.Parse(contentString); Console.WriteLine(xdoc2); string fileRECEIPT = $@"{Path.Combine(Path.GetDirectoryName(xmlFile), Path.GetFileNameWithoutExtension(xmlFile))}_RECEIPT_{DateTime.Now.ToString(" yyyy-MM-ddTHH-mm-ss")}.xml"; xdoc2.Save(fileRECEIPT, SaveOptions.None); } else { string fileErrorResponse = $@"{Path.Combine(Path.GetDirectoryName(xmlFile), Path.GetFileNameWithoutExtension(xmlFile))}_ErrorResponse_{DateTime.Now.ToSt ring("yyyy-MM-ddTHH-mm-ss")}.xml"; File.WriteAllText(fileErrorResponse, contentString); } } } catch (Exception ex) { ex.Message.Dump(); } } In BaseX GUI: SET BINDINGS xml=e:\Temp\XSD_CACHE\input.xml,xsd=e:\Temp\XSD_CACHE\state.xsd Start BaseX HTTP listener via launching c:\Program Files (x86)\BaseX\bin\basexhttp.bat First execution emits the following correct result in both BaseX GUI as well as c# REST HTTP: ============================================================================ ================= <root> <metadata> <result>invalid</result> <errors>2</errors> <warnings>0</warnings> <startTime>2024-12-29T21:00:55.659-05:00</startTime> <endTime>2024-12-29T21:00:55.729-05:00</endTime> <xmlFile>e:\Temp\XSD_CACHE\input.xml</xmlFile> <xsdFile>e:\Temp\XSD_CACHE\state.xsd</xsdFile> <xsdProcessor>BaseX 11.6, Java Eclipse Adoptium v.22.0.2+9, Xerces 2.12.2</xsdProcessor> <xsdVersion>1.0.0</xsdVersion> </metadata> <messages> <message level="Error" line="4" column="19">cvc-enumeration-valid: Value 'TX' is not facet-valid with respect to enumeration '[FL]'. It must be a value from the enumeration.</message> <message level="Error" line="4" column="19">cvc-type.3.1.3: The value 'TX' of element 'state' is not valid.</message> </messages> </root> Modified XSD file: xsd=e:\Temp\XSD_CACHE\state.xsd 1) attribute version="1.0.1" 2) uncommented <xs:enumeration value="TX"/> to make XML file valid ========================================================== <?xml version="1.0" encoding="UTF-8"?> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" version="1.0.1"> <xs:element name="root"> <xs:complexType> <xs:sequence> <xs:element maxOccurs="unbounded" ref="state"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="state" type="state_type"/> <xs:simpleType name="state_type"> <xs:restriction base="xs:string"> <xs:length value="2"/> <xs:enumeration value="FL"/> <xs:enumeration value="TX"/> </xs:restriction> </xs:simpleType> </xs:schema> Execute in BaseX GUI: validate:xsd-init() Re-run c# REST HTTP CALL emits incorrect result. It shows correct xsdVersion: 1.0.1, but result is invalid with the same error ============================================================================ = <root> <metadata> <result>invalid</result> <errors>2</errors> <warnings>0</warnings> <startTime>2024-12-29T21:20:07.457-05:00</startTime> <endTime>2024-12-29T21:20:07.459-05:00</endTime> <xmlFile>e:\Temp\XSD_CACHE\input.xml</xmlFile> <xsdFile>e:\Temp\XSD_CACHE\state.xsd</xsdFile> <xsdProcessor>BaseX 11.6, Java Eclipse Adoptium v.22.0.2+9, Xerces 2.12.2</xsdProcessor> <xsdVersion>1.0.1</xsdVersion> </metadata> <messages> <message level="Error" line="4" column="19">cvc-enumeration-valid: Value 'TX' is not facet-valid with respect to enumeration '[FL]'. It must be a value from the enumeration.</message> <message level="Error" line="4" column="19">cvc-type.3.1.3: The value 'TX' of element 'state' is not valid.</message> </messages> </root> Though executing XSD11_Validation_RECEIPT_2.xq directly in the BaseX GUI emits the correct result: <root> <metadata> <result>valid</result> <errors>0</errors> <warnings>0</warnings> <startTime>2024-12-29T21:23:51.738-05:00</startTime> <endTime>2024-12-29T21:23:51.739-05:00</endTime> <xmlFile>e:\Temp\XSD_CACHE\input.xml</xmlFile> <xsdFile>e:\Temp\XSD_CACHE\state.xsd</xsdFile> <xsdProcessor>BaseX 11.6, Java Eclipse Adoptium v.22.0.2+9, Xerces 2.12.2</xsdProcessor> <xsdVersion>1.0.1</xsdVersion> </metadata> <messages/> </root> Just restart of the BaseX HTTP listener resolves the issue with the BaseX REST HTTP call. Regards, Yitzhak Khabinsky

2 1

XSD validation: cache invalidation stopped working
by ykhabins＠bellsouth.net 29 Dec '24

29 Dec '24

Hello, My environment has BaseX 11.6 installed. The XQuery is called in c# via REST HTTP call: string REQUESTURL = $"http://{HOST}:{PORT}/rest?run={XQUERY}&$xml={xmlFile}&$xsd={xsdFile}"; XQuery is using XSD validation with caching, along the following: let $result := validate:xsd-report($xml, $xsd, map { 'http://apache.org/xml/features/validation/cta-full-xpath-checking': true(), 'cache': true() }) It seems that the cache invalidation call stopped working ?! It is not picking up the latest XSD. I launch it from the BaseX GUI as follows: validate:xsd-init() Just restart of the HTTP listener picks up the latest/modified XSD. Please advise. Regards, Yitzhak Khabinsky

2 1

serialization order of map in fiddle
by Martin Honnen 29 Dec '24

29 Dec '24

After some recent announcement of ordered maps in the BaseX fiddle I played around with some JSON serialization, to my surprise the result is different whether I use the serialize function directly on a single map <https://fiddle.basex.org/?share=%28%27query%21%27%2803.*2%2C%5Cn%28.30*24%2…>: {'foo' : 'bar', 'data' : array { 1 to 5 }} => serialize({'method' : 'json', 'indent' : true() }), {'data' : array { 1 to 5 }, 'foo' : 'bar'} => serialize({'method' : 'json', 'indent' : true() }) gives { "foo": "bar", "data": [ 1, 2, 3, 4, 5 ] } { "data": [ 1, 2, 3, 4, 5 ], "foo": "bar" } or in a for .. return <https://fiddle.basex.org/?share=%28%27query%21%27for+%24map+in+%7B%280.-%29…>: for $map in ({'foo' : 'bar', 'data' : array { 1 to 5 }}, {'data' : array { 1 to 5 }, 'foo' : 'bar'}) return serialize($map, {'method' : 'json', 'indent' : true() }) gives { "foo": "bar", "data": [ 1, 2, 3, 4, 5 ] } { "foo": "bar", "data": [ 1, 2, 3, 4, 5 ] } Should the JSON serialization reflect the map as created or is that still random/arbitrary order for the properties?

2 1

Help with recursion
by Chuck Bearden 27 Dec '24

27 Dec '24

I'm trying to ingest a sequence of 6293 strings, each 8 characters long, and create a map in which each key is an integer from 1 to 63, and each value is a subsequence of the 6293 strings, from ($key+1) * 100) to that starting value + 99. I wrote a function to do this recursively (so I thought), but it runs and runs (I've given it over 5 minutes), and so I can see I borked something. I wrote a function to output the starting and ending sequence indices for each execution of the function, but it (as the only change to the first function) quickly causes a stack overflow: *Error:* Stack Overflow: Try tail recursion? Since the recursive call was the last expression in the function, I thought I was doing tail recursion. I'm running 11.6.0. The code with explanatory comments follows. I'd be grateful for any corrections and insights. declare function local:write_results($elems as element()*, $count as xs:integer) { let $dir_path := '/home/cfbearden/projects/mesh_for_pure/20241220/data/' let $count_as_str := xs:string($count) let $content_out := <PubmedArticleSet>{$elems}</PubmedArticleSet> let $path := $dir_path || 'pmed2022_' || $count_as_str || '.xml' let $params := { 'method' : 'xml', 'indent' : 'yes' } return file:write($path, $elems, $params) }; declare function local:by_hundreds($pmids as xs:string*, $pmid_map as map(*)) as map(*) { let $key_count := count(map:keys($pmid_map)) return (: base case: each value except the last should have 100 items in it; the last should have <= 100; in this case we return the map goal is to build a map with 63 keys, with each value a sequence of 100 or 93 (in the last case) strings :) if ((($key_count + 1) * 100) >= count($pmids)) then $pmid_map else (: starting index for subsequence() :) let $start := 1 + ($key_count * 100) (: ending index for subsequence() :) let $end := $start + 99 (: value of the map key for this 100 items :) let $key := $key_count + 1 (: attempt to log all recurrent executions to files let $foo := local:write_results(<res><start>{$start}</start><end>{$end}</end></res>, $key_count) :) (: create a map with $key as key and the subsequence from n to n + 99 of $pmids and merge it with input accumulator map to make a new map :) let $pmid_map_new := map:merge($pmid_map, map { $key : subsequence($pmids, $start, $end) }) (: recur; the number of keys in $pmid_map_new will enable the next execution to determine which items from $pmids to use :) return local:by_hundreds($pmids, $pmid_map_new) }; (: The data format is simple: <DATA_RECORDS> <DATA_RECORD> <pub_uuid>0008980a-f1f3-40bf-8c13-cb0b79dfa81e</pub_uuid> <pmid>35732127</pmid> </DATA_RECORD> ... </DATA_RECORDS> :) let $pure_2022_db := collection('pure_2022') (: yields a sequence of 6293 strings, all 8 characters long :) let $pmids := ( for $drec in $pure_2022_db/DATA_RECORDS/DATA_RECORD let $pmid := $drec/pmid/text() return $pmid ) let $result := local:by_hundreds($pmids, {}) return $result All the best, Chuck Bearden

2 4

Upgrading BaseX by several major versions
by Rémy Delanaux 18 Dec '24

18 Dec '24

Hello, I manage a project I've worked on from 2020 to 2022 and have recently come back to (no one has really worked on since then). The project currently runs BaseX 9.4, and we obviously would like to catch up on versions of BaseX. But I'm wondering what is the best way to process here. I don't remember upgrading major versions of BaseX during my time on project (we changed servers so for BaseX 9 we did a fresh install and backup restore) What is the correct way to do this? Should I straight up install the latest version (probably BaseX at the time of writing) and find all blocking migration changes, or gradually install major version after major version (10 then 11 then 12 when available) and ook for smaller migration processes? I wasn't able to find migration guides except for upgrading from 9 to 10 (although there are individual changelogs for each individual module). Thanks! Best, Remy Delanaux

6 10

parse-json() function reorder keys in the resulting map{}
by Oleksandr Shpak 16 Dec '24

16 Dec '24

Hi team, Found one "issue" with key ordering in the resulting map. Need to clarify, is it normal behavior or a bug. json:parse('{"b":1, "a":3}', { 'format': 'xquery' }) returns {"a":3.0e0,"b":1.0e0} but json:parse('{"b":1, "a":3}') <json type="object"><b type="number">1</b><a type="number">3</a></json> As you see, the standard function returns keys not in the order as it was set in the original JSON string. I didn't find any requirement in xQuery spec that the resulting map entries should be sorted by the key name. Is it a bug or a feature? If a feature I would be really happy to see one additional param to preserve key ordering. Thanks! -- s0rr0w

3 7

BaseX freeze
by Jerome Chauveau 16 Dec '24

16 Dec '24

hi, my basex server (11.5) stops responding when too many clients request my rxq application. I've published a little demo to reproduce the bug here : [ https://git.unicaen.fr/chauveau/basex-freeze-app | https://git.unicaen.fr/chauveau/basex-freeze-app ] I don't know if the problem comes from BaseX, Jetty or the Saxon XSLT processor. Thanks for your help Jérôme

2 3

Runtime failure in DBA optimize database panel
by Eliot Kimber 16 Dec '24

16 Dec '24

Using 11.6, the DBA app’s “optimize” panel gives this runtime error: ``` Stopped at /data1/basex/webapp/dba/databases/db-optimize.xqm, 104/10: [basex:restxq] Value of "name" must be xs:string, supplied: (). ``` Must be a simple code bug but I couldn’t find the source in github. Where is the source for the DBA app? Cheers, E. _____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368 servicenow servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | X<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Instagram<https://www.instagram.com/servicenow>

2 1