BaseX-Talk

basex-talk@mailman.uni-konstanz.de

9 participants
5256 discussions

Returning multi-part MIME in RESTXQ? Query parameters?
by Jonathan Robie 15 Nov '21

15 Nov '21

I'm having fun with RESTXQ, this is an easy and efficient way to create REST interfaces once you learn how. Two questions: 1. How do I return a forest of elements? Multi-part MIME types? If so, how do I specify that? What do I need to add to this particular declaration? (:~ Look up all entries in the Abbott-Smith lexicon that contain an English word in the gloss. :) declare %rest:path("/api/abbott-smith/gloss/{$g}") %rest:GET %output:method('xml') function as:gloss($g) { $as:abbott-smith/*:entry[.//*:gloss[contains-token(lower-case(.), lower-case($g))] ] }; 2. Can I use query strings to specify parameters in RESTXQ? What if I want URLs like this: /api/dictionary/?lemma=cheese Thanks! Jonathan

2 1

Re: [basex-talk] how to redirect http error 404 to root
by Adam Law 14 Nov '21

14 Nov '21

For flexibility you may wish to consider using a golang proxy/edge router. Also useful for https / acme. There is a solution using traefik here: https://imandrea.me/blog/traefik-custom-404/ This seems to use traefik.http.routers.error-router. Adam

1 0

Re: [basex-talk] Performance issue on copy-modify-return when too many items are being replaced
by France Baril 12 Nov '21

12 Nov '21

Forwarding to the mailing list in order to share knowledge. On Fri, Nov 12, 2021 at 1:41 PM BaseX Support <support(a)basex.org> wrote: > Hi France, > > I’d need to get my hands on your code to tell you exactly where it’s > best used, but I can give you some more details on the XQuery > specification: > > When creating new nodes in XQuery via node constructors [1], copies of > all enclosed nodes will be created, and the copied nodes get new node > identities. As a result, the following query yields false: > > let $a := <a/> > let $b := <b>{ $a }</b> > return $b/a is $a > > This step can be very expensive and memory consuming. If the option is > enabled, child nodes will only be linked to the new parent nodes, and > the upper query returns true. > > As the option changes the semantics of XQuery, it should preferably be > used in Pragmas. > > Best, > Christian > > PS: Mails to our mailing list are preferred; this way, other users > might benefit from the replies as well. > > [1] https://www.w3.org/TR/xquery-31/#id-constructors > > > > On Fri, Nov 12, 2021 at 2:13 PM France Baril > <france.baril(a)architextus.com> wrote: > > > > Can you give me more information about how copynode changes the behavior > of the xquery and where it is best used. > > > > I see in the example that the pragma is on db:open. My process is: > > > > 1. Read a document A from DB called lang that has references to other > documents in the same DB lang (where lang is a 4 letter code for a locale). > > 2. Merge all the references into document A to create an aggregate. > > 3. Send the aggregate through multiple functions (that use > copy-modify-return) that each resolve a type of reference (most references > grab referenced content from a DB called global, but others grab it from > the lang DB). These references do not grad entire documents, but smaller > snippets within XML documents. > > 4. Save the result in a DB called staging-lang (where lang is a 4 letter > code for a locale). > > > > So should the pragma apply when reading the 1st document (1), when > reading the documents we aggregate into the 1st document (2), when grabbing > the snippets (3) and/or when saving the end result in the staging DB (4)? > Or maybe for all db:open() and db:attribute()/.. functions in this process? > > > > > > > > > > > > > > > > On Fri, Nov 12, 2021 at 12:16 PM BaseX Support <support(a)basex.org> > wrote: > >> > >> One more suggestion: > >> > >> If node construction turns out to consume too much memory, it sometimes > helps to disable the COPYNODE option: > >> > >> https://docs.basex.org/wiki/XQuery_Extensions#Database_Pragmas > >> > >> > >> > >> France Baril <france.baril(a)architextus.com> schrieb am Fr., 12. Nov. > 2021, 13:09: > >>> > >>> Hi, > >>> > >>> Thanks for your answer. > >>> > >>> I tried rebuilding the document instead of using copies, I have > >>> implemented 3/4 of the functions that resolve references and I'm > >>> already at double the time I had before. So I will set that one aside > >>> as an unsuccessful alternative. If memory serves me correctly we might > >>> have moved from a transform that rebuilds the document to a > >>> copy-modify-return approach to improve performance over a year ago. > >>> > >>> I will try grouping the references of the same names in the example > >>> above to limit the number of queries to the DB. If that still doesn't > >>> help, I will see if I can send you a good example without having to > >>> send too many of our. > >>> > >>> We have a short term solution where we removed some references in > >>> references, which reduces substantially the number of items to resolve > >>> (80% improvement), but it does impact the user experience, so we are > >>> still looking into code-based solutions as opposed to (or to use in > >>> conjunction with) content-based solutions. > >>> > >>> On Fri, Nov 5, 2021 at 5:22 PM BaseX Support <support(a)basex.org> > wrote: > >>> > > >>> > Hi France, > >>> > > >>> > Do you have some sample data that allows us to test your code? > >>> > > >>> > If documents are pretty large, it’s sometimes faster to rebuild a > >>> > document with node constructors instead of performing updates on it. > >>> > > >>> > Best, > >>> > Christian > >>> > ____________________________________ > >>> > > >>> > > We have a query that looks like this: > >>> > > > >>> > > declare function content-refs:resolve-prompt-refs-new($node as > node(), > >>> > > $lang as xs:string) as node()*{ > >>> > > let $result := > >>> > > copy $copy := $node > >>> > > modify( > >>> > > let $entries := > >>> > > $copy/descendant-or-self::*[@name-ref][name()='prompt-ref' or > >>> > > name()='gui-ctrl-ref' > >>> > > or name()='feature-ref' or name()='app-ref' (: or > >>> > > name()='screen-ref':)] > >>> > > > >>> > > let $entries-hd := > >>> > > > $copy/descendant-or-self::*[@id='T1700243243']/descendant-or-self::*[@name-ref][name()='prompt-ref' > >>> > > or name()='gui-ctrl-ref' > >>> > > or name()='feature-ref' or name()='app-ref' (: or > >>> > > name()='screen-ref':)] > >>> > > > >>> > > let $trace := trace('Prompts count: ' || count($entries)) > >>> > > let $trace := trace('Prompts in Hardware diagram: ' || > >>> > > count($entries-hd)) > >>> > > > >>> > > for $entry in $entries > >>> > > (:let $trace := trace('start processing entry'):) > >>> > > let $name := $entry/data(@name-ref) > >>> > > let $trace := > >>> > > if (exists($entry/ancestor::*[@id = 'T1700243243'])) > >>> > > then trace( $name , ' Promptref
') > >>> > > else () > >>> > > let $prompts-from-index := db:attribute('index-prompt-' > || > >>> > > $lang, $name, 'name')/.. (:=> prof:time('index prompt attr: '):) > >>> > > (:let $prompts-from-index := db:open('index-prompt-' || > >>> > > $lang)//*[@name = $name] => prof:time('index prompt open: '):) > >>> > > let $prompts := > >>> > > for $prompt in $prompts-from-index > >>> > > let $original-elem-name := $entry/self::*/name() > >>> > > let $new-elem-name := > >>> > > switch ($original-elem-name) > >>> > > case 'prompt-ref' return $original-elem-name > >>> > > default return > substring-before($original-elem-name, '-ref') > >>> > > return > >>> > > copy $prompt-renamed := $prompt > >>> > > modify( > >>> > > rename node $prompt-renamed as $new-elem-name > >>> > > ) > >>> > > return $prompt-renamed (:=> prof:time('index prompt > new > >>> > > elem-name: '):) > >>> > > let $new-node := > >>> > > if (count($prompts) = 0) > >>> > > then > >>> > > <filter-group error="{concat("No target found in > for: ", > >>> > > $entry/name(), '/@name-ref=', > $entry/@name-ref)}"/> > >>> > > else <filter-group-inline>{ > >>> > > $prompts > >>> > > }</filter-group-inline> > >>> > > let $trace := ('Ready to replace old entry with > new-node') > >>> > > return replace node $entry with $new-node (:=> > >>> > > prof:time('index prompt new node: '):) > >>> > > > >>> > > ) > >>> > > return $copy (:=> prof:time('index prompt return copy: '):) > >>> > > return $result > >>> > > > >>> > > }; > >>> > > > >>> > > As you can see, we are using prof:time to see how quickly items are > >>> > > resolved. Querying to the db for each item goes fairly quickly (2 > >>> > > seconds). However that last 'return $copy' line, after all the > >>> > > replacements are processed takes between 11 and 25 minutes > depending > >>> > > on the system. Memory usage is low, but the CPU usage goes to the > >>> > > roof. > >>> > > > >>> > > We are updating a little over 110 000 items in this operation, so > it > >>> > > is a big operation on a file of about 89000 indented lines. We are > >>> > > wondering if there is a way we could improve the performance. > Before > >>> > > this operation occurs, we are processing the file multiple times to > >>> > > replace other items with very similar functions > (copy-modify.return), > >>> > > they all go fairly quickly so it does seem that the culprit is the > >>> > > number of items being replaced. > >>> > > > >>> > > > >>> > > -- > >>> > > France Baril > >>> > > Architecte documentaire / Documentation architect > >>> > > france.baril(a)architextus.com > >>> > >>> > >>> > >>> -- > >>> France Baril > >>> Architecte documentaire / Documentation architect > >>> france.baril(a)architextus.com > > > > > > > > -- > > France Baril > > Architecte documentaire / Documentation architect > > france.baril(a)architextus.com > -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

1 0

Wrong MIME type ...
by Jonathan Robie 11 Nov '21

11 Nov '21

Using RESTXQ in an application for the first time. I expected this function to have the MIME type application/json: declare %rest:path("/api/abbott-smith/index") %rest:GET %rest:produces("application/json") function as:index() { array { $as:abbott-smith/*:entry ! @lemma ! string() } ! serialize(., map {'method' : 'json'}) }; But both POSTMAN and WADL give me application/xml as the mime type: <wadl:resource path="/api/abbott-smith/index"> <wadl:method name="GET"> <wadl:request/> <wadl:response> <wadl:representation mediaType="application/xml"/> </wadl:response> </wadl:method> </wadl:resource> What am I doing wrong? Jonathan

3 4

Access to "https://raw.githubusercontent.com/BaseXdb/basex/master/basex-api/src/test/resources/first.xml" blocked?
by Ben Engbers 11 Nov '21

11 Nov '21

Hi, Allthough I have never had any feedback on my R-client to BaseX, I have steadily been working on a new version (even while being retired, I still like to program :-). At present all tests are passed, except one test on adding content to a database. On my Linux-machine 'url exists("https://raw.githubusercontent.com/BaseXdb/basex/master/basex-api/src/test/r…")' returns TRUE and 'first.xml' is added to a database. When executed on a Windows machine, the same test returns FALSE. I have tested other URL's - starting with https or http - and they are all accepted. Any clue why "https://raw.githubusercontent.com/BaseXdb/basex/master/basex-api/src/test/r…" is blocked? Ben

2 2

Re: [basex-talk] Access to "https://raw.githubusercontent.com/BaseXdb/basex/master/basex-api/src/test/resources/first.xml" blocked?
by Imsieke, Gerrit, le-tex 11 Nov '21

11 Nov '21

Maybe related to the HTTP header field x-content-type-options: nosniff https://docs.microsoft.com/en-us/previous-versions/windows/internet-explore… What is the tool/library you are using on Windows? Is it an R HTTP client that is interfacing some Windows DLL? Maybe they put this rejection in the DLL. Maybe you can find another proxying server for raw Github files? https://stackoverflow.com/questions/40728554/resource-blocked-due-to-mime-t… Gerrit On 11.11.2021 16:37, Ben Engbers wrote: > They are both rejected. I also tried with "https://www.cnn.com" and > "https://nos.nl". The first is accepted, the second is rejected. > > In firefox all URL's are accepted. > > Ben > > Op 11-11-2021 om 16:24 schreef Imsieke, Gerrit, le-tex: >> Hi Ben, >> >> What about other resources there, like: >> >> https://raw.githubusercontent.com/BaseXdb/basex/master/basex-api/src/test/r… >> >> https://raw.githubusercontent.com/BaseXdb/basex/master/basex-api/src/test/r… >> >> >> Do they pass on Windows? >> >> Gerrit >> >> >> On 11.11.2021 16:14, Ben Engbers wrote: >>> Hi, >>> >>> Allthough I have never had any feedback on my R-client to BaseX, I >>> have steadily been working on a new version (even while being >>> retired, I still like to program :-). >>> At present all tests are passed, except one test on adding content to >>> a database. >>> >>> On my Linux-machine 'url >>> exists("https://raw.githubusercontent.com/BaseXdb/basex/master/basex-api/src/test/r…")' >>> returns TRUE and 'first.xml' is added to a database. >>> >>> When executed on a Windows machine, the same test returns FALSE. >>> I have tested other URL's - starting with https or http - and they >>> are all accepted. >>> >>> Any clue why >>> "https://raw.githubusercontent.com/BaseXdb/basex/master/basex-api/src/test/r…" >>> is blocked? >>> >>> Ben >>

2 1

items of type map(*) cannot be serialized
by Jonathan Robie 10 Nov '21

10 Nov '21

I want to serialize JSON. Consider the following function: declare function local:index() { array{ $levinsohn/levinsohn/feature/header ! map {./name : ./description ! string() } } }; When I run that, I get output that looks like this: [map { "Ambiguous": "Marked but ambiguous constituent order." }, map { "annotations": "Inline annotations." } !!! SNIP !!!] That won't work as output in a REST API, so I want to serialize that without the map keyword. I tried to use fn:serialize(): declare function local:index() { array{ $levinsohn/levinsohn/feature/header ! map {./name : ./description ! string() } } ! serialize(.) }; That gave me this error: items of type map(*) cannot be serialized So ... what's the right way to do this? Jonathan

2 2

Running dba as a web app .... what am I doing wrong?
by Jonathan Robie 09 Nov '21

09 Nov '21

I am confused about what is required for running a web app. Here's what I have tried so far: 1. Running basexhttp from ./Downloads/basex If I download the .zip file from https://basex.org/download/ and run basexhttp from the /Downloads/basex subdirectory, I can see the "BaseX HTTP Services" page at localhost:8984, but when I look at localhost:8984/rest, I see no databases: <rest:databases resources="0"/> 2. Running basexhttp from a random directory If I run basexhttp from a random or empty directory, localhost:8984 returns this message: Service not found. But localhost:8984/rest lists all of my databases. 3. Running basexhttp using a copy of ./Downloads/basex/webapp I created a ./basex directory in a temporary directory and did this: cp -r /Users/xxx/Downloads/basex/webapp . I then tried running basexhttp from the new ./basex directory I created in the temporary directory. The results were the same as running from a random directory. Running from within the webapp directory does not work in either case. What am I doing wrong? Thanks! Jonathan

1 0

Passing options to saxon through xslt:transform
by Graydon Saunders 09 Nov '21

09 Nov '21

Hello -- Somewhat to my surprise I've got the large XSLT 3.0 transform to load, or at least not complain about includes. (Load the input as a db; load the stylesheet from the file system. Be extra-careful about constructing URIs.) Which raises the prospect that I'll actually be able to run this transform from BaseX, but I cannot figure out how to set a value for the initial template. This query: let $source := <dummy></dummy> let $template := <xsl:stylesheet version="3.0" xmlns:xsl=' http://www.w3.org/1999/XSL/Transform'> <xsl:template name="xsl:initial-template"> <thingy>Hello, World!</thingy> </xsl:template> <xsl:template match="/"> <fail>matched on root</fail> </xsl:template> </xsl:stylesheet> return xslt:transform($source,$template,map{'initial-template': '{ http://www.w3.org/1999/XSL/Transform}initial-template'}) returns <fail>matched on root</fail> Which isn't surprising; setting initial-template is a Saxon option, not a parameter. If I try return xslt:transform($source,$template,map{},map{'initial-template': '{ http://www.w3.org/1999/XSL/Transform}initial-template'}) I get [XPTY0004] Unknown option 'initial-template'. (same with 'it'.) Is there a way to set Saxon options with xslt:transform? Thanks! Graydon

4 3

Content-Encoding: gzip on incoming RESTXQ POST
by James Ball 08 Nov '21

08 Nov '21

Hello, I have a RESTXQ function that receives an XML file via HTTP POST. This works perfectly. The client would now like to compress the data with gzip and set the Content-Encoding: gzip header. I set GZIP = true in my .basex file based on the documentation [1] But these gzipped post requests fail - my function is called but content seems to arrive at my function still compressed. (Responses from the server become compressed dynamically as expected - and that works great). Is there something else I need to do to make it work for requests? Many thanks, James [1] https://docs.basex.org/wiki/Options#GZIP

3 7

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk