June 2023 - BaseX-Talk - mailman.uni-konstanz.de

Values stored to a database vs using the new store
by Eliot Kimber 28 Jun '23

28 Jun '23

I’m coming up to speed on BaseX 10 and trying to understand the practical difference between using the built-in store (or a custom store) and simply saving values into a database. As far as I can tell from the docs the basic feature is the same: given e.g., a map with stuff in it, I can store the map to a database with the new db:put-value() function or to the build-in store (or a custom store) using store:put() and in both cases can get the map back. Given that the store is automatically persistent, it seems like the obvious choice if you’re using maps as a result caching technique, so I’m wondering what the reasons would be for not using it and using a database instead. I think the answer includes: * Databases provide locking semantics that the store does not—the store docs are clear that the store is non-deterministic and may have side effects. So if you need to ensure locking consistency, you want to use a database. * Custom stores require that you manage any required persistence yourself, while databases are inherently persistent. Have I missed any other important considerations? Without taking the time to do the measuring, I’m guessing that there is negligible performance difference between using the store and a database—I’d be very surprised if the underlying implementation was not based on the same code. In our Project Mirabel system I need to do caching of large query results, so being able to quickly and conveniently store maps is ideal and I’m very excited to take advantage of this new feature. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 1

Variable value
by Giuseppe G. A. Celano 19 Jun '23

19 Jun '23

Hi, I have stambled upon an unexpected issue concerning the value of a variable (in BaseX 10.6), which seems to be affected by an independent function. I have a function like page:open-url($an1) which takes a url as a parameter and opens a document. The url usually consists of an invariable part + a final number that identifies different documents. If I only pass the final number of the url, the function, as expected, returns an error (because it tries to resolve a relative path pointing to no document). However, if I change the script into let $f := page:documents($an1, $an2) return page:open-url($an1) the function page:open-url($an1) works properly (i.e., the presence of the independent let clause seems to affect the value of the variable $an1): I would not expect this in a functional programming language like XQuery. I have posted a gist here [1], where there are more details (look at the very end of the script) Ciao, Giuseppe —— [1] https://gist.github.com/gcelano/31ef1880ac8439398c7f6de1de6d78d3

2 1

Minor GUI irritation
by Andy Bunce 19 Jun '23

19 Jun '23

Hi, In the GUI when viewing an open database with many namespaces via database -> properties->information. The properties window is excessively tall, often larger than the screen height. There seems to be extra blank space below the "Resource Properties" section and perhaps the namespaces section that could have a scroll bar if required. db:create("nstest",(1 to 60)!``[<foo xmlns="ns`{.}`" />]``,(1 to 60)!(. || ".xml")) Creates a database that shows the behavior on my screen. Seen with BaseXGUI 10.6 on windows, but I think I also saw it with v9. Regards /Andy [image: image.png]

2 2

Timeframe for supporting fn:load-xquery-module()?
by Andrew Sales 08 Jun '23

08 Jun '23

Hello, The docs say: "The three functions fn:transform, fn:load-xquery-module and fn:collation-key may be added in a future version of BaseX as their implementation might require the use of additional external libraries." Is there a timeframe for this? I'm interested in particular in the load-xquery-module() to enable an implementation of Schematron written in pure XQuery[1] to be portable between engines[2]. Many thanks, Andrew [1] https://github.com/AndrewSales/XQS [2] https://github.com/AndrewSales/XQS/issues/8

4 4

Possible to optimize XPath lookup by string match on element content?
by Eliot Kimber 07 Jun '23

07 Jun '23

In my application I load numerous Oxygen validation report reports, consisting of many <incident> elements, where each incident is associated with a specific input document, captured in a <systemID> element: <incident xmlns=http://www.oxygenxml.com/ns/report> <engine>oXygen</engine> <severity>error</severity> <description>Cannot find definition for key "image.install-store-app-menu-path". Key Scopes:[[bundle-itsm], [reuse], [reuse]]. Keys are gathered from: bundle-itsm-it-service-management.ditamap.</description> <systemID>/Users/eliot.kimber/git-basex/utah/doc/source/reuse/activation/install-store-apps-steps.dita</systemID> <profile>suite-prod</profile> <type>Key reference</type> <location> <start> <line>33</line> <column>26</column> </start> <end> <line>33</line> <column>61</column> </end> <length>35</length> </location> </incident> I need correlate these incidents to the docs as stored in the database, where the match is just on the filename, not any part of the path (although I could match on the part of the path starting with “doc/source”). In my testing, with about 27K of these incident elements, it takes about 60 seconds to build a set of maps where the keys are the filenames and the values are the sequences of <incident> elements that match a given filename, i.e.: let $incidentsByDoc as map(*) := map:merge( let $docNames as xs:string* := $incidents/report:systemID ! string(.) ! relpath:getName(.) => distinct-values() return for $docName in $docNames return map{ $docName : $incidents[report:systemID/text() contains text { '/' || $docName}] } ) I think the only optimization is to save the resulting map as an XML file or (with BaseX 10) just save the map for later use. But I’m curious if there’s some other XPath-level optimization that would make this lookup faster? I’m already using the text index via “contains text”, although I suspect that it’s not really offering an advantage over a simple ends-with(.) check. Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

1 1

Trying to apply Query Update 3.0 update/transform operations
by Eliot Kimber 03 Jun '23

03 Jun '23

I found my error: I was passing in a variable from outside the transform/update call, rather than using the context node. This is the correct invocation: return ( $inputMap transform with { local:resolveMaprefs(.) } ,() ) I also successfully applied the transform recursively, so I think I have, despite my best efforts, figured out how do this map resolution as efficiently as possible. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

1 1

Trying to apply Query Update 3.0 update/transform operations
by Eliot Kimber 03 Jun '23

03 Jun '23

In the context of my attempt to implement DITA map resolution in BaseX, I’m experimenting with using the new XQuery update 3 in-memory updates, which seem to be just what I need. However, my test script fails with: Stopped at /Users/eliot.kimber/git/mirabel/src/main/xquery/resolve-map-test.xqy, 32/9: [XUDY0014] Node was not created by transform expression: <mapref.../>. What subtlety of XQuery update am I missing? The code works as expected through the start if the THEN clause when $refTarget exists (meaning the mapref was resolved to its target document and we’re trying to construct the new nodes. The script is: (: Start of XQuery 😊 import module namespace dutils=http://servicenow.com/xquery/module/now-dita-utils; import module namespace relpath=http://servicenow.com/xquery/module/now-relpath-utils; declare %updating function local:resolveMaprefs($context) { let $maprefs as element()* := $context//mapref[@href][string(@scope) ne 'peer'] let $debug := prof:dump('local:resolveMaprefs(): have ' || count($maprefs) || ' maprefs') return for $mapref in $maprefs let $refTarget as element()? := dutils:resolveReference($mapref/@href) let $debug := (prof:dump('refTarget:'), prof:dump($refTarget)) return if (exists($refTarget)) then let $debug := prof:dump('Got a reference target, attempting to replace original href with it...') let $newNodes as element() := <topicgroup class="+ map/topicref mapgroup-d/topicgroup " base="submap {$refTarget/@base} {$mapref/@base}" xml:base="{relpath:base-uri($refTarget)}" >{ (: $mapref/@* except ($mapref/@href, $mapref/@format, $mapref/@base), $refTarget/@* except ($refTarget/@href, $refTarget/@*:DITAArchVersion, $refTarget/@domains, $refTarget/@base) :) } <topicmeta class="- map/topicmeta "> <navtitle class="- topic/navtitle ">{ $refTarget/*[dutils:class(., 'topic/title')]/node() }</navtitle> </topicmeta> {$refTarget/* except ($refTarget/title, $refTarget/topicmeta)} </topicgroup> return replace node $mapref with $newNodes else () }; let $mapurl as xs:string := 'bundle-devops-enterprise-dev-ops.ditamap' let $inputMap as document-node() := db:open('utah', $mapurl) let $maprefs as element()* := $inputMap//mapref[@href][string(@scope) ne 'peer'] return ( $maprefs, $inputMap update { local:resolveMaprefs(db:open('utah', $mapurl)) } ) (: End of XQuery :) A typical input map would be: <map xml:lang="en-US"> <title>Utah DevOps</title> <mapref href="now-keys-common.ditamap"/> <mapref href="devops-change-velocity.ditamap"/> <mapref href="devops-config.ditamap"/> </map> _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

1 0

In XSLT run by Saxon against doc in a database, how to resolve URI references?
by Eliot Kimber 03 Jun '23

03 Jun '23

I’m trying to run an XSLT transform that works from the file system to work when run from BaseX, where the input is a document in a database. The transform needs to resolve URI references from the input source to their targets in the same database. It looks like using base-uri() in XSLT returns a file:/ URL: ["+ [DEBUG] resolveTopicRef(): base-uri($context)="file:///utah/bundle-devops-enterprise-dev-ops.ditamap""] ["+ [DEBUG] df:resolveTopicRef(): context is a topicref."] ["+ [DEBUG] df:resolveTopicRef(): href=""now-keys-common.ditamap"""] ["+ [DEBUG] df:resolveTopicRef(): topicUri is a string, trying to resolve..."] ["+ [DEBUG] $context = ", <mapref href="now-keys-common.ditamap"/>] ["+ [DEBUG] baseuri(context) = file:///utah/bundle-devops-enterprise-dev-ops.ditamap"] ["+ [DEBUG] df:resolveTopicRef(): document at uri ""file:/utah/now-keys-common.ditamap"" is not available, returning empty list"] (This is the output of xslt:transform-report()?messages, which is very nice BTW) So my question is: Is it possible, short of using the BaseX REST API, to have a Saxon-run XSLT access docs from a database? I can easily set the @xml:base attribute on the input document before passing it to Saxon if there was some base URI that would then resolve from the database. I didn’t see anything relevant in the documentation for the XSLT module. In this case the transform is doing DITA map resolution, creating a single result document from a tree of DITA maps by resolving <mapref> elements to their referenced maps and replacing the reference with the referenced element. I haven’t figured out an efficient way to do this directly in XQuery, so I thought I’d apply my existing XSLT transform, but this seems to be a blocker. I’m sure it would work to use the BaseX REST API to access the docs from the XSLT, but that of course requires having a running server and HTTP handler. I was hoping this would just work. Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twase\itter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

1 0