BaseX-Talk

basex-talk@mailman.uni-konstanz.de

7 participants
5254 discussions

Re: [basex-talk] XSD validation: cache invalidation stopped working
by ykhabins＠bellsouth.net 30 Dec '24

30 Dec '24

Here is a full repro. Initial setup, four files: 1) XML: e:\Temp\XSD_CACHE\input.xml 2) XSD: e:\Temp\XSD_CACHE\state.xsd 3) XQuery: c:\Program Files (x86)\BaseX\webapp\XSD11_Validation_RECEIPT_2.xq 4) c# file making BaseX REST HTTP call XML file: e:\Temp\XSD_CACHE\input.xml ====================================== <?xml version="1.0"?> <root> <state>FL</state> <state>TX</state> </root> XSD file: e:\Temp\XSD_CACHE\state.xsd ========================================= <?xml version="1.0" encoding="UTF-8"?> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" version="1.0.0"> <xs:element name="root"> <xs:complexType> <xs:sequence> <xs:element maxOccurs="unbounded" ref="state"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="state" type="state_type"/> <xs:simpleType name="state_type"> <xs:restriction base="xs:string"> <xs:length value="2"/> <xs:enumeration value="FL"/>  </xs:restriction> </xs:simpleType> </xs:schema> XQuery file: XSD11_Validation_RECEIPT_2.xq ========================================== declare variable $xml as xs:string external; declare variable $xsd as xs:string external; try { let $timezone := timezone-from-dateTime(current-dateTime()) let $before_datetime := adjust-dateTime-to-timezone(convert:integer-to-dateTime(prof:current-ms()), xs:dayTimeDuration($timezone)) (: current-dateTime() :) let $result := validate:xsd-report($xml, $xsd, map { 'http://apache.org/xml/features/validation/cta-full-xpath-checking': true(), 'cache': true() }) let $after_datetime := adjust-dateTime-to-timezone(convert:integer-to-dateTime(prof:current-ms()), xs:dayTimeDuration($timezone)) (: current-dateTime() :) return <root><metadata> <result>{data($result/status)}</result> <errors>{count($result/message[@level="Error"])}</errors> <warnings>{count($result/message[@level="Warning"])}</warnings> <startTime>{$before_datetime}</startTime> <endTime>{$after_datetime}</endTime> <xmlFile>{$xml}</xmlFile> <xsdFile>{$xsd}</xsdFile> <xsdProcessor>BaseX {data(db:system()//version)}, Java {proc:property('java.vendor')} v.{proc:property('java.runtime.version')}, {validate:xsd-processor()} 2.12.2</xsdProcessor> <xsdVersion>{data(doc($xsd)/*:schema/@version)}</xsdVersion> </metadata> <messages>{ $result/message transform with { delete node @url } }</messages> </root> } catch * { <error> <errorCode>{$err:code}</errorCode> <errorDescription>{$err:description}</errorDescription> </error> } c# REST HTTP call: ================== void Main() { const string HOST = "HostName"; const int PORT = 8080; const string USERNAME = "username"; const string PASSWORD = "password"; System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls; const string XQUERY = "XSD11_Validation_RECEIPT_2.xq"; string xmlFile = @"e:\Temp\XSD_CACHE\input.xml"; string xsdFile = @"e:\Temp\XSD_CACHE\state.xsd"; string REQUESTURL = $"http://{HOST}:{PORT}/rest?run={XQUERY}&$xml={xmlFile}&$xsd={xsdFile}"; try { using (HttpClientHandler handler = new HttpClientHandler() {UseDefaultCredentials = true}) // for mdpd proxy //{UseDefaultCredentials = true, UseProxy = true, DefaultProxyCredentials = CredentialCache.DefaultNetworkCredentials}) using (HttpClient client = new HttpClient(handler)) { // Encode username and password pair with a base64 implementation. String encoded = Convert.ToBase64String(Encoding.UTF8.GetBytes(USERNAME + ":" + PASSWORD)); client.DefaultRequestHeaders.Add("Authorization", "Basic " + encoded); client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/xml")); var response = client.GetAsync(REQUESTURL).Result; System.Net.Http.HttpContent content = response.Content; var contentString = content.ReadAsStringAsync().Result; // get the actual content string if (response.StatusCode == HttpStatusCode.OK) // 200 //IsSuccessStatusCode == True { XDocument xdoc2 = XDocument.Parse(contentString); Console.WriteLine(xdoc2); string fileRECEIPT = $@"{Path.Combine(Path.GetDirectoryName(xmlFile), Path.GetFileNameWithoutExtension(xmlFile))}_RECEIPT_{DateTime.Now.ToString(" yyyy-MM-ddTHH-mm-ss")}.xml"; xdoc2.Save(fileRECEIPT, SaveOptions.None); } else { string fileErrorResponse = $@"{Path.Combine(Path.GetDirectoryName(xmlFile), Path.GetFileNameWithoutExtension(xmlFile))}_ErrorResponse_{DateTime.Now.ToSt ring("yyyy-MM-ddTHH-mm-ss")}.xml"; File.WriteAllText(fileErrorResponse, contentString); } } } catch (Exception ex) { ex.Message.Dump(); } } In BaseX GUI: SET BINDINGS xml=e:\Temp\XSD_CACHE\input.xml,xsd=e:\Temp\XSD_CACHE\state.xsd Start BaseX HTTP listener via launching c:\Program Files (x86)\BaseX\bin\basexhttp.bat First execution emits the following correct result in both BaseX GUI as well as c# REST HTTP: ============================================================================ ================= <root> <metadata> <result>invalid</result> <errors>2</errors> <warnings>0</warnings> <startTime>2024-12-29T21:00:55.659-05:00</startTime> <endTime>2024-12-29T21:00:55.729-05:00</endTime> <xmlFile>e:\Temp\XSD_CACHE\input.xml</xmlFile> <xsdFile>e:\Temp\XSD_CACHE\state.xsd</xsdFile> <xsdProcessor>BaseX 11.6, Java Eclipse Adoptium v.22.0.2+9, Xerces 2.12.2</xsdProcessor> <xsdVersion>1.0.0</xsdVersion> </metadata> <messages> <message level="Error" line="4" column="19">cvc-enumeration-valid: Value 'TX' is not facet-valid with respect to enumeration '[FL]'. It must be a value from the enumeration.</message> <message level="Error" line="4" column="19">cvc-type.3.1.3: The value 'TX' of element 'state' is not valid.</message> </messages> </root> Modified XSD file: xsd=e:\Temp\XSD_CACHE\state.xsd 1) attribute version="1.0.1" 2) uncommented <xs:enumeration value="TX"/> to make XML file valid ========================================================== <?xml version="1.0" encoding="UTF-8"?> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" version="1.0.1"> <xs:element name="root"> <xs:complexType> <xs:sequence> <xs:element maxOccurs="unbounded" ref="state"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="state" type="state_type"/> <xs:simpleType name="state_type"> <xs:restriction base="xs:string"> <xs:length value="2"/> <xs:enumeration value="FL"/> <xs:enumeration value="TX"/> </xs:restriction> </xs:simpleType> </xs:schema> Execute in BaseX GUI: validate:xsd-init() Re-run c# REST HTTP CALL emits incorrect result. It shows correct xsdVersion: 1.0.1, but result is invalid with the same error ============================================================================ = <root> <metadata> <result>invalid</result> <errors>2</errors> <warnings>0</warnings> <startTime>2024-12-29T21:20:07.457-05:00</startTime> <endTime>2024-12-29T21:20:07.459-05:00</endTime> <xmlFile>e:\Temp\XSD_CACHE\input.xml</xmlFile> <xsdFile>e:\Temp\XSD_CACHE\state.xsd</xsdFile> <xsdProcessor>BaseX 11.6, Java Eclipse Adoptium v.22.0.2+9, Xerces 2.12.2</xsdProcessor> <xsdVersion>1.0.1</xsdVersion> </metadata> <messages> <message level="Error" line="4" column="19">cvc-enumeration-valid: Value 'TX' is not facet-valid with respect to enumeration '[FL]'. It must be a value from the enumeration.</message> <message level="Error" line="4" column="19">cvc-type.3.1.3: The value 'TX' of element 'state' is not valid.</message> </messages> </root> Though executing XSD11_Validation_RECEIPT_2.xq directly in the BaseX GUI emits the correct result: <root> <metadata> <result>valid</result> <errors>0</errors> <warnings>0</warnings> <startTime>2024-12-29T21:23:51.738-05:00</startTime> <endTime>2024-12-29T21:23:51.739-05:00</endTime> <xmlFile>e:\Temp\XSD_CACHE\input.xml</xmlFile> <xsdFile>e:\Temp\XSD_CACHE\state.xsd</xsdFile> <xsdProcessor>BaseX 11.6, Java Eclipse Adoptium v.22.0.2+9, Xerces 2.12.2</xsdProcessor> <xsdVersion>1.0.1</xsdVersion> </metadata> <messages/> </root> Just restart of the BaseX HTTP listener resolves the issue with the BaseX REST HTTP call. Regards, Yitzhak Khabinsky

2 1

XSD validation: cache invalidation stopped working
by ykhabins＠bellsouth.net 29 Dec '24

29 Dec '24

Hello, My environment has BaseX 11.6 installed. The XQuery is called in c# via REST HTTP call: string REQUESTURL = $"http://{HOST}:{PORT}/rest?run={XQUERY}&$xml={xmlFile}&$xsd={xsdFile}"; XQuery is using XSD validation with caching, along the following: let $result := validate:xsd-report($xml, $xsd, map { 'http://apache.org/xml/features/validation/cta-full-xpath-checking': true(), 'cache': true() }) It seems that the cache invalidation call stopped working ?! It is not picking up the latest XSD. I launch it from the BaseX GUI as follows: validate:xsd-init() Just restart of the HTTP listener picks up the latest/modified XSD. Please advise. Regards, Yitzhak Khabinsky

2 1

serialization order of map in fiddle
by Martin Honnen 29 Dec '24

29 Dec '24

After some recent announcement of ordered maps in the BaseX fiddle I played around with some JSON serialization, to my surprise the result is different whether I use the serialize function directly on a single map <https://fiddle.basex.org/?share=%28%27query%21%27%2803.*2%2C%5Cn%28.30*24%2…>: {'foo' : 'bar', 'data' : array { 1 to 5 }} => serialize({'method' : 'json', 'indent' : true() }), {'data' : array { 1 to 5 }, 'foo' : 'bar'} => serialize({'method' : 'json', 'indent' : true() }) gives { "foo": "bar", "data": [ 1, 2, 3, 4, 5 ] } { "data": [ 1, 2, 3, 4, 5 ], "foo": "bar" } or in a for .. return <https://fiddle.basex.org/?share=%28%27query%21%27for+%24map+in+%7B%280.-%29…>: for $map in ({'foo' : 'bar', 'data' : array { 1 to 5 }}, {'data' : array { 1 to 5 }, 'foo' : 'bar'}) return serialize($map, {'method' : 'json', 'indent' : true() }) gives { "foo": "bar", "data": [ 1, 2, 3, 4, 5 ] } { "foo": "bar", "data": [ 1, 2, 3, 4, 5 ] } Should the JSON serialization reflect the map as created or is that still random/arbitrary order for the properties?

2 1

Help with recursion
by Chuck Bearden 27 Dec '24

27 Dec '24

I'm trying to ingest a sequence of 6293 strings, each 8 characters long, and create a map in which each key is an integer from 1 to 63, and each value is a subsequence of the 6293 strings, from ($key+1) * 100) to that starting value + 99. I wrote a function to do this recursively (so I thought), but it runs and runs (I've given it over 5 minutes), and so I can see I borked something. I wrote a function to output the starting and ending sequence indices for each execution of the function, but it (as the only change to the first function) quickly causes a stack overflow: *Error:* Stack Overflow: Try tail recursion? Since the recursive call was the last expression in the function, I thought I was doing tail recursion. I'm running 11.6.0. The code with explanatory comments follows. I'd be grateful for any corrections and insights. declare function local:write_results($elems as element()*, $count as xs:integer) { let $dir_path := '/home/cfbearden/projects/mesh_for_pure/20241220/data/' let $count_as_str := xs:string($count) let $content_out := <PubmedArticleSet>{$elems}</PubmedArticleSet> let $path := $dir_path || 'pmed2022_' || $count_as_str || '.xml' let $params := { 'method' : 'xml', 'indent' : 'yes' } return file:write($path, $elems, $params) }; declare function local:by_hundreds($pmids as xs:string*, $pmid_map as map(*)) as map(*) { let $key_count := count(map:keys($pmid_map)) return (: base case: each value except the last should have 100 items in it; the last should have <= 100; in this case we return the map goal is to build a map with 63 keys, with each value a sequence of 100 or 93 (in the last case) strings :) if ((($key_count + 1) * 100) >= count($pmids)) then $pmid_map else (: starting index for subsequence() :) let $start := 1 + ($key_count * 100) (: ending index for subsequence() :) let $end := $start + 99 (: value of the map key for this 100 items :) let $key := $key_count + 1 (: attempt to log all recurrent executions to files let $foo := local:write_results(<res><start>{$start}</start><end>{$end}</end></res>, $key_count) :) (: create a map with $key as key and the subsequence from n to n + 99 of $pmids and merge it with input accumulator map to make a new map :) let $pmid_map_new := map:merge($pmid_map, map { $key : subsequence($pmids, $start, $end) }) (: recur; the number of keys in $pmid_map_new will enable the next execution to determine which items from $pmids to use :) return local:by_hundreds($pmids, $pmid_map_new) }; (: The data format is simple: <DATA_RECORDS> <DATA_RECORD> <pub_uuid>0008980a-f1f3-40bf-8c13-cb0b79dfa81e</pub_uuid> <pmid>35732127</pmid> </DATA_RECORD> ... </DATA_RECORDS> :) let $pure_2022_db := collection('pure_2022') (: yields a sequence of 6293 strings, all 8 characters long :) let $pmids := ( for $drec in $pure_2022_db/DATA_RECORDS/DATA_RECORD let $pmid := $drec/pmid/text() return $pmid ) let $result := local:by_hundreds($pmids, {}) return $result All the best, Chuck Bearden

2 4

Upgrading BaseX by several major versions
by Rémy Delanaux 18 Dec '24

18 Dec '24

Hello, I manage a project I've worked on from 2020 to 2022 and have recently come back to (no one has really worked on since then). The project currently runs BaseX 9.4, and we obviously would like to catch up on versions of BaseX. But I'm wondering what is the best way to process here. I don't remember upgrading major versions of BaseX during my time on project (we changed servers so for BaseX 9 we did a fresh install and backup restore) What is the correct way to do this? Should I straight up install the latest version (probably BaseX at the time of writing) and find all blocking migration changes, or gradually install major version after major version (10 then 11 then 12 when available) and ook for smaller migration processes? I wasn't able to find migration guides except for upgrading from 9 to 10 (although there are individual changelogs for each individual module). Thanks! Best, Remy Delanaux

6 10

parse-json() function reorder keys in the resulting map{}
by Oleksandr Shpak 16 Dec '24

16 Dec '24

Hi team, Found one "issue" with key ordering in the resulting map. Need to clarify, is it normal behavior or a bug. json:parse('{"b":1, "a":3}', { 'format': 'xquery' }) returns {"a":3.0e0,"b":1.0e0} but json:parse('{"b":1, "a":3}') <json type="object"><b type="number">1</b><a type="number">3</a></json> As you see, the standard function returns keys not in the order as it was set in the original JSON string. I didn't find any requirement in xQuery spec that the resulting map entries should be sorted by the key name. Is it a bug or a feature? If a feature I would be really happy to see one additional param to preserve key ordering. Thanks! -- s0rr0w

3 7

BaseX freeze
by Jerome Chauveau 16 Dec '24

16 Dec '24

hi, my basex server (11.5) stops responding when too many clients request my rxq application. I've published a little demo to reproduce the bug here : [ https://git.unicaen.fr/chauveau/basex-freeze-app | https://git.unicaen.fr/chauveau/basex-freeze-app ] I don't know if the problem comes from BaseX, Jetty or the Saxon XSLT processor. Thanks for your help Jérôme

2 3

Runtime failure in DBA optimize database panel
by Eliot Kimber 16 Dec '24

16 Dec '24

Using 11.6, the DBA app’s “optimize” panel gives this runtime error: ``` Stopped at /data1/basex/webapp/dba/databases/db-optimize.xqm, 104/10: [basex:restxq] Value of "name" must be xs:string, supplied: (). ``` Must be a simple code bug but I couldn’t find the source in github. Where is the source for the DBA app? Cheers, E. _____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368 servicenow servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | X<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Instagram<https://www.instagram.com/servicenow>

2 1

How best to manage link data collection and storage?
by Eliot Kimber 10 Dec '24

10 Dec '24

Over the last couple of years I’ve developed the Mirabel system that provides DITA link management and query features over large volumes of content (the ServiceNow product documentation source). In particular it knows what links to what and enables viewing the content with all link information available. For a given version of the product docs we have about 60K DITA topics and 100 root maps that organize those topics into publications. The primary job of Mirabel is to capture all the hyperlink details as defined by the DITA source and enable queries about the element-to-element and document-to-document relationships established by those links. My implementation approach for loading the link knowledge uses a multi-step process: 1. Load the entire source content into a database 2. Create a “key space” database that reflects the DITA key-to-resource mappings defined by each root DITA map. The key spaces are XQuery maps that map from key names to resources identified by their database node IDs (essentially, each use of a topic from a map has an associated unique key by which that use of the topic can be referenced). The key spaces are a prerequisite for resolving cross references to keys from one topic to other topics in the context of some root map (the same root map or a different one). 3. Create a “link record keeping” database that contains the “where used” index for the content. The where-used index maps element node IDs to a record of every reference to that node (cross references, content references, topic references from maps). The where-used index is the core data used to know where a given map or topic is used, which is used to answer questions like “what publications use this topic?” or “is this topic used at all?”. The where-used table is constructed as an XQuery map that is then turned into XML for storage (I implemented this before BaseX added direct storage of maps but given the size I think it still makes sense to store it as XML, but I could be wrong). * Process all map-to-map and map-to-topic references and create the initial map entries, one for each map and topic. * For topics referenced from maps, process all topic-to-topic references and update the records for each target topic to reflect the references to it. The map context of a given topic determines the targets of key references from that topic, so it is necessary to process the topics in the context of the root maps that use them (in DITA, root maps determine the key-to-resource bindings to which key references resolve). * For topics not referenced from any maps, add entries for them to the where-used table and process any topic-to-topic references (key references cannot be resolved but direct URI references can be). Convert the XQuery map to a single XML document and store in the link record keeping database. The resulting database takes about 150MB of storage. This third step can take two-to-three hours: 60K topics times 0.2 seconds for each topic is 3.3 hours. 0.1 seconds is about as fast as the link processing can go based on my testing. This is all done using temporary databases so as not to disturb the working databases used by the running Mirabel web application. The work is performed by a BaseX “worker” server, not the main server that serves the web site. I essentially have one BaseX http server for each core on my server and allocate work to them based on load, so queries coming from the web app will not be allocated to a worker currently doing a content update process. Once all the new link data is loaded, the temporary databases are swapped into production by renaming the production databases, renaming the temp databases to their production names, then dropping the old databases. (Saying this just now I’m realizing that I don’t know how to pause or wait for active queries against the in-production databases to finish so I can swap the databases.) Because all the index entries use node IDs, the content database and record-keeping databases have to be put into production at the same time, otherwise the content node IDs will be out of sync with the indexed record IDs. I’m working on the assumption that renaming databases is essentially instantaneous and so I can use that to swap the temp databases into production reliably. I use my job orchestration module (https://github.com/ekimbernow/basex-orchestration) to manage the sequence of operations, where each job calls the next job in the sequence once it has finished. This process works reliably for smaller volumes of content—for example, a content set with only a couple of thousand topics and four or five root maps. But at full scale I’m consistently seeing that the link record keeping database, which only has two large XML documents in it, never completes optimization: The database page shows the database with two things in it, but when you open the database’s page, they do not show up and the job that performs the optimization never completes, leaving the database in a locked state. This means the new where-used index can’t be put into production. I feel like I’m going about this the wrong way to make best use of BaseX and avoid this problem with very large databases but I don’t see any obvious alternative approaches. But it feels like I’m missing something fundamental or making a silly error that I can’t see. So my question: How would you solve this problem? In particular, how would you go about constructing the where-used index in a way that works best with BaseX? Or maybe the question is “should I be updating the in-production database with the new data and doing the swapping into the production within the database itself?” (i.e., by renaming the where used index document rather than the database itself.) I am currently using 11.6 and can move to 12 once it is released. Thanks, Eliot _____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368 servicenow servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | X<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Instagram<https://www.instagram.com/servicenow>

1 0

How to unlock a locked database?
by Eliot Kimber 10 Dec '24

10 Dec '24

BaseX 11.6 on Redhat Linux. I have long-running update processes that are failing at some point, leaving databases in a locked state that is not cleared by restarting the server. Short of manually deleting the database from the file system, how can I clear these locks or determine what process has the lock? I do not have FAIRLOCK turned on. This is in the context of my Mirabel system, which loads content into one database and then creates secondary link data databases, which requires lots of data processing and may take several hours to complete. Thanks, Eliot _____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368 servicenow servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | X<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Instagram<https://www.instagram.com/servicenow>

1 0

← Newer
1
...
9
10
11
12
13
14
15
...
526
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk