BaseX-Talk

basex-talk@mailman.uni-konstanz.de

9 participants
5241 discussions

Help dealing with slow API responses
by Bridger Dyson-Smith 24 Nov '21

24 Nov '21

Hi all - I have the sneaking suspicion that the answer to this plea for help will be something like, "Use RESTXQ!!" or something similar, but let me describe the problem: I'm pulling data back from an OAI-PMH endpoint that is slow; i.e. response times are ~1/minute. Embarrassingly, I think I've spent several hours trying to figure out if my requests were buggy before I realized that the API endpoint was just *slow* (at least compared to others that I use regularly). I typically use the IntelliJ plugin or the BaseX GUI for the vast majority of my XQuery work, but these requests effectively build up a sequence of responses (as files) and then serialize them all to disk when the requests have finished. When the requests are answered quickly, there's prompt feedback (my script finishes quickly, I can see serialized files, etc), but when the requests are answered slowly, I'm left waiting (and then thinking, "Oh no - I've mistyped the endpoint URL." or something similar). Initially I think I have two requests for guidance: 1. Is there a better way, using the BaseX GUI (or the command line), to get feedback on a querying process like this? Something... asynchronous, or something clever with builtin functions in the `jobs` or `xquery` modules? 2. If this can be addressed relatively directly with RESTXQ, I'm game to (finally) get more comfortable with it, but does anyone have any examples, applications, scripts, etc they would be willing to share? I want to say that several examples and ideas have been shared before here on the list, but I'm having a terrible time finding them. I've attached a simple SSCCE, where the basic idea is: query an API for some data, and get a response like so: <example> <stuff>...</stuff> <resumptionToken>abc123</resumptionToken> </example> then take the text() of the resumptionToken, and resubmit a request to the API, which would return: <example> <stuff>...</stuff> <resumptionToken>def456</resumptionToken> </example> The full responses are written to the temp directory on your system (file:temp-dir()), with a date-stamp name. The endpoint in my attached example has a fairly small response from what I've selected, but the idea would be: how let me (or another user) know that the query is active and running, not hung up or failing? I understand that given the functional nature of XQuery these sorts of things can be a bit more complicated, so I would appreciate any thoughts, opinions, links, etc. Many thanks for your time and trouble. Best, Bridger PS the endpoint in the example doesn't necessarily exhibit the same slow behavior that is the basis for my woes -- for some list readers it may be very fast indeed -- but I felt like it might be apropos to use this particular endpoint for an illustration.

2 3

Module imports are not transitive?
by Andy Bunce 24 Nov '21

24 Nov '21

Hi, "Module imports are not transitive—that is, importing a module provides access only to function and variable declarations contained directly in the imported module. For example, if module A imports module B, and module B imports module C, module A does not have access to the functions and variables declared in module C." https://www.w3.org/TR/xquery-30/#id-module-import I have noticed before that BaseX does not appear to implement this see [1] My case was a conflict in a module that was imported internally within an EXpath Package and the same namespace used as a module in an application. I have just seen another example today where the current behavior was regarded as a "good" feature. I don't remember ever seeing any discussion on this. Is this something that might be implemented at some point? /Andy [1] https://gist.github.com/apb2006/9702104456e5ee2c48f4ff4f9eba5c1b#file-main-…

2 1

Question about ft:normalize
by Tim Thompson 24 Nov '21

24 Nov '21

Hello, I have a question about the BaseX ft:normalize function. What kind of Unicode normalization is performed by this function, and how might it be implemented using standard XPath functions? Thanks in advance! Tim -- Tim A. Thompson (*he, him*) Librarian for Applied Metadata Research Yale University Library www.linkedin.com/in/timathompson timothy.thompson(a)yale.edu

2 5

random items not sorted
by Tom De Herdt 24 Nov '21

24 Nov '21

Hi, While doing some tests, I noticed an unexpected query result in Basex 9.6.3 (on Windows 10). This query: let $ids := for $i in 1 to 10 return random:uuid() for $id in $ids order by $id return $id ... returns 10 UUIDs, but they are /not/ sorted. Information from the Info pane in the GUI: Compiling: - rewrite range expression to range sequence: (1 to 10) - rewrite simple map to util:replicate(items,count[,multiple]): (1 to 10) ! random:uuid() -> util:replicate(random:uuid(), 10, true()) - inline for $i_1 in (1 to 10) - simplify FLWOR expression: util:replicate(random:uuid(), 10, true()) - inline let $ids_0 := util:replicate(random:uuid(), 10, true()) - rewrite fn:sort(items[,collation[,function]]) to util:replicate(items,count[,multiple]): sort(util:replicate(random:uuid(), 10, true())) -> util:replicate(random:uuid(), 10, true()) - inline for $id_2 in util:replicate(random:uuid(), 10, true()) - simplify FLWOR expression: util:replicate(random:uuid(), 10, true()) Optimized Query: util:replicate(random:uuid(), 10, true()) In Basex 9.3.2 (on another Windows machine) items are sorted alphabetically, as I would expect them to be – but I may be overlooking something. Any ideas? Thanks, Tom De Herdt

2 1

how to redirect http error 404 to root
by Günter Dunz-Wolff 16 Nov '21

16 Nov '21

Hi all, how can I handle it, that in my RESTXQ-App every request, where the response is an 404-error, will be redirected to the index-page / Thanks in advance Guenter

2 2

Returning multi-part MIME in RESTXQ? Query parameters?
by Jonathan Robie 15 Nov '21

15 Nov '21

I'm having fun with RESTXQ, this is an easy and efficient way to create REST interfaces once you learn how. Two questions: 1. How do I return a forest of elements? Multi-part MIME types? If so, how do I specify that? What do I need to add to this particular declaration? (:~ Look up all entries in the Abbott-Smith lexicon that contain an English word in the gloss. :) declare %rest:path("/api/abbott-smith/gloss/{$g}") %rest:GET %output:method('xml') function as:gloss($g) { $as:abbott-smith/*:entry[.//*:gloss[contains-token(lower-case(.), lower-case($g))] ] }; 2. Can I use query strings to specify parameters in RESTXQ? What if I want URLs like this: /api/dictionary/?lemma=cheese Thanks! Jonathan

2 1

Re: [basex-talk] how to redirect http error 404 to root
by Adam Law 14 Nov '21

14 Nov '21

For flexibility you may wish to consider using a golang proxy/edge router. Also useful for https / acme. There is a solution using traefik here: https://imandrea.me/blog/traefik-custom-404/ This seems to use traefik.http.routers.error-router. Adam

1 0

Re: [basex-talk] Performance issue on copy-modify-return when too many items are being replaced
by France Baril 12 Nov '21

12 Nov '21

Forwarding to the mailing list in order to share knowledge. On Fri, Nov 12, 2021 at 1:41 PM BaseX Support <support(a)basex.org> wrote: > Hi France, > > I’d need to get my hands on your code to tell you exactly where it’s > best used, but I can give you some more details on the XQuery > specification: > > When creating new nodes in XQuery via node constructors [1], copies of > all enclosed nodes will be created, and the copied nodes get new node > identities. As a result, the following query yields false: > > let $a := <a/> > let $b := <b>{ $a }</b> > return $b/a is $a > > This step can be very expensive and memory consuming. If the option is > enabled, child nodes will only be linked to the new parent nodes, and > the upper query returns true. > > As the option changes the semantics of XQuery, it should preferably be > used in Pragmas. > > Best, > Christian > > PS: Mails to our mailing list are preferred; this way, other users > might benefit from the replies as well. > > [1] https://www.w3.org/TR/xquery-31/#id-constructors > > > > On Fri, Nov 12, 2021 at 2:13 PM France Baril > <france.baril(a)architextus.com> wrote: > > > > Can you give me more information about how copynode changes the behavior > of the xquery and where it is best used. > > > > I see in the example that the pragma is on db:open. My process is: > > > > 1. Read a document A from DB called lang that has references to other > documents in the same DB lang (where lang is a 4 letter code for a locale). > > 2. Merge all the references into document A to create an aggregate. > > 3. Send the aggregate through multiple functions (that use > copy-modify-return) that each resolve a type of reference (most references > grab referenced content from a DB called global, but others grab it from > the lang DB). These references do not grad entire documents, but smaller > snippets within XML documents. > > 4. Save the result in a DB called staging-lang (where lang is a 4 letter > code for a locale). > > > > So should the pragma apply when reading the 1st document (1), when > reading the documents we aggregate into the 1st document (2), when grabbing > the snippets (3) and/or when saving the end result in the staging DB (4)? > Or maybe for all db:open() and db:attribute()/.. functions in this process? > > > > > > > > > > > > > > > > On Fri, Nov 12, 2021 at 12:16 PM BaseX Support <support(a)basex.org> > wrote: > >> > >> One more suggestion: > >> > >> If node construction turns out to consume too much memory, it sometimes > helps to disable the COPYNODE option: > >> > >> https://docs.basex.org/wiki/XQuery_Extensions#Database_Pragmas > >> > >> > >> > >> France Baril <france.baril(a)architextus.com> schrieb am Fr., 12. Nov. > 2021, 13:09: > >>> > >>> Hi, > >>> > >>> Thanks for your answer. > >>> > >>> I tried rebuilding the document instead of using copies, I have > >>> implemented 3/4 of the functions that resolve references and I'm > >>> already at double the time I had before. So I will set that one aside > >>> as an unsuccessful alternative. If memory serves me correctly we might > >>> have moved from a transform that rebuilds the document to a > >>> copy-modify-return approach to improve performance over a year ago. > >>> > >>> I will try grouping the references of the same names in the example > >>> above to limit the number of queries to the DB. If that still doesn't > >>> help, I will see if I can send you a good example without having to > >>> send too many of our. > >>> > >>> We have a short term solution where we removed some references in > >>> references, which reduces substantially the number of items to resolve > >>> (80% improvement), but it does impact the user experience, so we are > >>> still looking into code-based solutions as opposed to (or to use in > >>> conjunction with) content-based solutions. > >>> > >>> On Fri, Nov 5, 2021 at 5:22 PM BaseX Support <support(a)basex.org> > wrote: > >>> > > >>> > Hi France, > >>> > > >>> > Do you have some sample data that allows us to test your code? > >>> > > >>> > If documents are pretty large, it’s sometimes faster to rebuild a > >>> > document with node constructors instead of performing updates on it. > >>> > > >>> > Best, > >>> > Christian > >>> > ____________________________________ > >>> > > >>> > > We have a query that looks like this: > >>> > > > >>> > > declare function content-refs:resolve-prompt-refs-new($node as > node(), > >>> > > $lang as xs:string) as node()*{ > >>> > > let $result := > >>> > > copy $copy := $node > >>> > > modify( > >>> > > let $entries := > >>> > > $copy/descendant-or-self::*[@name-ref][name()='prompt-ref' or > >>> > > name()='gui-ctrl-ref' > >>> > > or name()='feature-ref' or name()='app-ref' (: or > >>> > > name()='screen-ref':)] > >>> > > > >>> > > let $entries-hd := > >>> > > > $copy/descendant-or-self::*[@id='T1700243243']/descendant-or-self::*[@name-ref][name()='prompt-ref' > >>> > > or name()='gui-ctrl-ref' > >>> > > or name()='feature-ref' or name()='app-ref' (: or > >>> > > name()='screen-ref':)] > >>> > > > >>> > > let $trace := trace('Prompts count: ' || count($entries)) > >>> > > let $trace := trace('Prompts in Hardware diagram: ' || > >>> > > count($entries-hd)) > >>> > > > >>> > > for $entry in $entries > >>> > > (:let $trace := trace('start processing entry'):) > >>> > > let $name := $entry/data(@name-ref) > >>> > > let $trace := > >>> > > if (exists($entry/ancestor::*[@id = 'T1700243243'])) > >>> > > then trace( $name , ' Promptref
') > >>> > > else () > >>> > > let $prompts-from-index := db:attribute('index-prompt-' > || > >>> > > $lang, $name, 'name')/.. (:=> prof:time('index prompt attr: '):) > >>> > > (:let $prompts-from-index := db:open('index-prompt-' || > >>> > > $lang)//*[@name = $name] => prof:time('index prompt open: '):) > >>> > > let $prompts := > >>> > > for $prompt in $prompts-from-index > >>> > > let $original-elem-name := $entry/self::*/name() > >>> > > let $new-elem-name := > >>> > > switch ($original-elem-name) > >>> > > case 'prompt-ref' return $original-elem-name > >>> > > default return > substring-before($original-elem-name, '-ref') > >>> > > return > >>> > > copy $prompt-renamed := $prompt > >>> > > modify( > >>> > > rename node $prompt-renamed as $new-elem-name > >>> > > ) > >>> > > return $prompt-renamed (:=> prof:time('index prompt > new > >>> > > elem-name: '):) > >>> > > let $new-node := > >>> > > if (count($prompts) = 0) > >>> > > then > >>> > > <filter-group error="{concat("No target found in > for: ", > >>> > > $entry/name(), '/@name-ref=', > $entry/@name-ref)}"/> > >>> > > else <filter-group-inline>{ > >>> > > $prompts > >>> > > }</filter-group-inline> > >>> > > let $trace := ('Ready to replace old entry with > new-node') > >>> > > return replace node $entry with $new-node (:=> > >>> > > prof:time('index prompt new node: '):) > >>> > > > >>> > > ) > >>> > > return $copy (:=> prof:time('index prompt return copy: '):) > >>> > > return $result > >>> > > > >>> > > }; > >>> > > > >>> > > As you can see, we are using prof:time to see how quickly items are > >>> > > resolved. Querying to the db for each item goes fairly quickly (2 > >>> > > seconds). However that last 'return $copy' line, after all the > >>> > > replacements are processed takes between 11 and 25 minutes > depending > >>> > > on the system. Memory usage is low, but the CPU usage goes to the > >>> > > roof. > >>> > > > >>> > > We are updating a little over 110 000 items in this operation, so > it > >>> > > is a big operation on a file of about 89000 indented lines. We are > >>> > > wondering if there is a way we could improve the performance. > Before > >>> > > this operation occurs, we are processing the file multiple times to > >>> > > replace other items with very similar functions > (copy-modify.return), > >>> > > they all go fairly quickly so it does seem that the culprit is the > >>> > > number of items being replaced. > >>> > > > >>> > > > >>> > > -- > >>> > > France Baril > >>> > > Architecte documentaire / Documentation architect > >>> > > france.baril(a)architextus.com > >>> > >>> > >>> > >>> -- > >>> France Baril > >>> Architecte documentaire / Documentation architect > >>> france.baril(a)architextus.com > > > > > > > > -- > > France Baril > > Architecte documentaire / Documentation architect > > france.baril(a)architextus.com > -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

1 0

Wrong MIME type ...
by Jonathan Robie 11 Nov '21

11 Nov '21

Using RESTXQ in an application for the first time. I expected this function to have the MIME type application/json: declare %rest:path("/api/abbott-smith/index") %rest:GET %rest:produces("application/json") function as:index() { array { $as:abbott-smith/*:entry ! @lemma ! string() } ! serialize(., map {'method' : 'json'}) }; But both POSTMAN and WADL give me application/xml as the mime type: <wadl:resource path="/api/abbott-smith/index"> <wadl:method name="GET"> <wadl:request/> <wadl:response> <wadl:representation mediaType="application/xml"/> </wadl:response> </wadl:method> </wadl:resource> What am I doing wrong? Jonathan

3 4

Access to "https://raw.githubusercontent.com/BaseXdb/basex/master/basex-api/src/test/resources/first.xml" blocked?
by Ben Engbers 11 Nov '21

11 Nov '21

Hi, Allthough I have never had any feedback on my R-client to BaseX, I have steadily been working on a new version (even while being retired, I still like to program :-). At present all tests are passed, except one test on adding content to a database. On my Linux-machine 'url exists("https://raw.githubusercontent.com/BaseXdb/basex/master/basex-api/src/test/r…")' returns TRUE and 'first.xml' is added to a database. When executed on a Windows machine, the same test returns FALSE. I have tested other URL's - starting with https or http - and they are all accepted. Any clue why "https://raw.githubusercontent.com/BaseXdb/basex/master/basex-api/src/test/r…" is blocked? Ben

2 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk