June 2021 - BaseX-Talk - mailman.uni-konstanz.de

getting request's headers inside a Web Application
by dudzikp＠yahoo.com 16 Jun '21

16 Jun '21

Dear all, I do not see an option to obtain the headers from the HTTP request from inside webapp function. Just like in the example: declare %rest:GET %rest:path('tests-html') ???%rest:headers('User-Agent')??? %output:method('html') function page-main:start-empty( ???$user-agent as xs:string??? ) as element(html) { ... } Is this possible? BR

3 3

http:sent-request with username, password and auth-method results in improper use msg
by Rob Stapper 15 Jun '21

15 Jun '21

Hi Cristian, When I add the username, password,auth-method to a http-request [1] I get an improper use message from BaseX[2]. The message suggest a bug. I hope this helps solving it. Kind regards, Rob [1] http:send-request( <http:request method='post' username='admin' password='admin' auth-method='Basic' > <http:body media-type='text/plain'/> </http:request> , 'http://LAPTOP-F17D4BIM:8984/test' , "test" ) [2] Error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 9.5.3 beta Java: Oracle Corporation, 1.8.0_271 OS: Windows 10, amd64 Stack Trace: java.lang.IllegalStateException: Already connected at sun.net.www.protocol.http.HttpURLConnection.setRequestProperty(Unknown Source) at org.basex.util.http.HttpClient.setContentType(HttpClient.java:258) at org.basex.util.http.HttpClient.sendRequest(HttpClient.java:82) at org.basex.query.func.http.HttpSendRequest.value(HttpSendRequest.java:34) at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:46) at org.basex.query.scope.MainModule.iter(MainModule.java:97) at org.basex.query.QueryContext.iter(QueryContext.java:326) at org.basex.query.QueryContext.cache(QueryContext.java:607) at org.basex.query.QueryProcessor.cache(QueryProcessor.java:113) at org.basex.core.cmd.AQuery.query(AQuery.java:99) at org.basex.core.cmd.XQuery.run(XQuery.java:22) at org.basex.core.Command.run(Command.java:257) at org.basex.core.Command.execute(Command.java:93) at org.basex.gui.GUI.exec(GUI.java:416) at org.basex.gui.GUI.lambda$4(GUI.java:359) at java.lang.Thread.run(Unknown Source) Sent from Mail for Windows 10 -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

2 2

differencing the string value of documents
by Graydon Saunders 12 Jun '21

12 Jun '21

Hello -- So I have XML documents A and B. For this purpose, the markup (which is known to be different) is not relevant; the thing I need to do is prove that all of the text in the string value of A is present in B in the same order in which it exists in A. Document B can have additional text and still be correct, so it is not sufficient to check if string(A) = string(B). (Ideally, B's extra text can be extracted and then compared against the sources of extra text in the transformation between A and B so as to confirm that it's at least all plausible new text.) I've been poking at this for a while and am getting that "reinventing the wheel by gnawing on anvils" feeling. Does a facility to do this exist in BaseX? I've had a look at the documentation and I observe some mention of diff.xqm but that appears to be a node-based diff. thanks! Graydon

5 9

Permissions
by SW-Service 12 Jun '21

12 Jun '21

Hello, should perm force authentication and only the user "admin" would be authorised? Best regards Frank module namespace page = 'http://basex.org/test'; declare %rest:path("basex-rest/test/check-rest-perm") %perm:allow("admin") %rest:GET %output:method("xml") function page:test-check-rest-perm() as element(response) { let $response := element response { 'Response' } return $response }; <response>Response</response>

2 1

HTTP Client DELETE with body
by Marco Lettere 12 Jun '21

12 Jun '21

Dear all, I have to call an API which model an HTTP DELETE operation based on the body of the request. It looks like currently http:send-request is not allowing body to be present ("Body not expected for method DELETE."). I feel that this is bit too restrictive interpretation of the spec which states (as far as I recall) that there is no defined semantic for the operation but not that it is forbidden. Is there any possibility to relax this behaviour and allow for bodies in DELETE calls? [1] Thank you very much. Marco. [1] https://github.com/BaseXdb/basex/blob/382c8097afb1ec5bac1e9511a2287686c33f6…

3 6

Re: [basex-talk] Error with full-text and fuzzy query
by Christian Grün 07 Jun '21

07 Jun '21

cc to the list (2)… Christian Grün <christian.gruen(a)gmail.com> schrieb am Mo., 7. Juni 2021, 15:47: > …and another snapshot for you: modifier letters [1] will now be > regarded as diacritical characters, and queries such as… > > tokenize('faraʼid faraid Birkan')[. contains text 'faraʼid' using fuzzy] > > …will only regard the first two terms as similar. As a consequence, > fuzzy queries for terms with modifiers should get a lot faster. > > You’ll need to recreate your full-text index to take full advantage of the > fix. > > [1] https://de.wikipedia.org/wiki/Unicodeblock_Spacing_Modifier_Letters > > > > On Mon, Jun 7, 2021 at 2:00 PM Christian Grün <christian.gruen(a)gmail.com> > wrote: > > > > > Out of curiosity, is there a way to access those individual term > similarity statistics via XQuery? > > > > I assume there is no straightforward way, but you can check the first > > results of a full-text query: > > > > ft:search('your-db', 'faraʼid', map { "fuzzy": true() })[position() < > 10] > > > > And I noticed that due to the modifier letter (ʼ), many non-similar > > results are returned as well. I’ve raised another issue to track this > > down [1]. > > > > [1] https://github.com/BaseXdb/basex/issues/2015 >

1 0

Re: [basex-talk] Error with full-text and fuzzy query
by Christian Grün 07 Jun '21

07 Jun '21

cc to the list… Christian Grün <christian.gruen(a)gmail.com> schrieb am Mo., 7. Juni 2021, 13:50: > Thanks, Tim, that was helpful. It turned out that the index contains > an unusually high number of similar tokens for one search terms in > your query: > > faraʼid : 13709 similar terms > muhammad : 131 similar terms > siraj : 50 similar terms > ... > > A new snapshot with a bug fix is available, in which similar terms are > collected non-recursively [1]. I’d have wished that your search was > evaluated faster (it takes around 6 seconds on machine) but at least > it succeeds. I’ll have some more thoughts. > > Cheers, > Christian > > [1] https://files.basex.org/releases/latest/ > > > > On Mon, Jun 7, 2021 at 1:17 PM Tim Thompson <timathom(a)gmail.com> wrote: > > > > Thank you, Christian. If it's still helpful, here is a link to download > the document I am using as my full-text index. > > > > All best, > > Tim > > > > > > -- > > Tim A. Thompson > > Metadata Librarian > > Yale University Library > > > > On Mon, Jun 7, 2021 at 6:19 AM Christian Grün <christian.gruen(a)gmail.com> > wrote: > >> > >> I think I managed to build up a bug scenario that should resemble > >> yours [1]: If thousands of similar terms are found, the results cannot > >> be recursively joined anymore. > >> > >> [1] https://github.com/BaseXdb/basex/issues/2014 > >> > >> > >> > >> On Mon, Jun 7, 2021 at 10:28 AM Christian Grün > >> <christian.gruen(a)gmail.com> wrote: > >> > > >> > Hi Tim, > >> > > >> > I’m still trying to get this reproduced. > >> > > >> > The error is caused by excessive recursive functions call. Out of > >> > interest, could you try to increase the size of the Java Stack Trace > >> > and see if the error persists? This can be achieved by assigning a > >> > large value to the JVM via the Xss flag [1]. You could e.g. add > >> > -Xss64m to the BASEX_JVM variable in the BaseX start scripts [2]. > >> > > >> > Internal notes… If yes, we might be confronted with an infinite loop. > >> > If no, we should try to rewrite a recursive index lookup to an > >> > iterative one. > >> > > >> > If your data is not confidential, feel free to provide me with a > download link. > >> > > >> > Cheers, > >> > Christian > >> > > >> > [1] > https://stackoverflow.com/questions/3700459/how-to-increase-the-java-stack-… > >> > [2] https://docs.basex.org/wiki/Start_Scripts > >> > > >> > > >> > > >> > On Fri, Jun 4, 2021 at 11:03 PM Tim Thompson <timathom(a)gmail.com> > wrote: > >> > > > >> > > Thanks, Christian. The rest of the stack trace was just many lines > of "at > org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74)" until > the end: > >> > > > >> > > [qtp1546693040-47] WARN > org.eclipse.jetty.server.handler.ErrorHandler - Error page too large: 500 > java.lang.StackOverflowError Request(POST // > 10.5.157.229:10214/dba/query-eval)@7dcadc62 > >> > > [qtp1546693040-47] INFO > org.eclipse.jetty.server.handler.ErrorHandler - Disabling showsStacks for > ErrorPageErrorHandler@53032c30{STARTED} > >> > > > >> > > Best regards, > >> > > Tim > >> > > > >> > > > >> > > -- > >> > > Tim A. Thompson > >> > > Metadata Librarian > >> > > Yale University Library > >> > > > >> > > On Fri, Jun 4, 2021 at 1:30 PM Christian Grün < > christian.gruen(a)gmail.com> wrote: > >> > >> > >> > >> Hi Tim, > >> > >> > >> > >> > [qtp1546693040-47] WARN org.eclipse.jetty.server.HttpChannel - > /dba/query-eval > >> > >> > java.lang.StackOverflowError > >> > >> > at > org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:73) > >> > >> > at > org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74) > >> > >> > >> > >> Could you share some more lines of the stack trace with us? > >> > >> > >> > >> Thanks in advance > >> > >> Christian >

1 0

Error with full-text and fuzzy query
by Tim Thompson 07 Jun '21

07 Jun '21

Hello, I have a full-text index defined on a database with 21,936,670 entries. I'm hitting a stack overflow error when I try to query the following (normalized) string with a fuzzy query: "sajawandi, siraj al-din muhammad ibn muhammad, active 12th century. faraʼid al-sirajiyah" The index entries look like this: <d a="http://id.loc.gov/authorities/names/n2008037847"> <n>Sajāwandī, Sirāj al-Dīn Muḥammad ibn Muḥammad, active 12th century. Farāʼiḍ al-Sirājīyah</n> <d>12th cent.</d> </d> If I create a new DB ("ftindex-test") with only that one entry and try the following lookup, it works: ft:search( "ftindex-test", "sajawandi, siraj al-din muhammad ibn muhammad, active 12th century. faraʼid al-sirajiyah" , map {"mode": "phrase", "content": "entire", "fuzzy": true()} )/.. However, if I try the same query against the full index (with ~22 million entries), it fails with a stack overflow error: [qtp1546693040-47] WARN org.eclipse.jetty.server.HttpChannel - /dba/query-eval java.lang.StackOverflowError at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:73) at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74) ... If I remove the "fuzzy" parameter, the query does work against the full index. Is this a bug, a known limitation here, or something that I'm missing? Thanks in advance, Tim -- Tim A. Thompson Metadata Librarian Yale University Library

2 4

speaking effectively to the optimizer
by Graydon Saunders 05 Jun '21

05 Jun '21

hello -- So I want to subtract one list of sentences from another list of sentences. (As a way of checking "what did processing do to this document?"; we know it added some words, and if we added words to the sentence it was at the start of the sentence, but none of the original words should be changed or missing or reordered relative to the document.) I have tested that I have the sequences of strings which are the list of sentences before I use them to call: declare function local:subtractSentences($before as xs:string*,$after as xs:string*) as xs:string* { let $thisBefore as xs:string? := head($before) let $thisAfter as xs:string? := head($after) return if (not(exists($thisBefore)) or not(exists($thisAfter))) then $after (: we're done because we're out of strings, but if we have any after left we want it :) else if (ends-with($thisAfter,$thisBefore)) then (replace($thisAfter,$thisBefore,''),local:subtractSentences(tail($before),tail($after))) else ($thisAfter,local:subtractSentences($before,tail($after))) }; In ~150 ms, I get Error: Stack Overflow: Try tail recursion? so I have written the function wrong. What's the more useful pattern for this case? Thanks! Graydon

4 3

thesaurus URILiteral vs node
by Liam R. E. Quin 01 Jun '21

01 Jun '21

What would be involved in extending BaseX's full text to allow a thesaurus to be supplied as a node instead of a URI? I want to construct one on the fly, and using file:write() seems excessive. Liam -- Liam Quin, https://www.delightfulcomputing.com/ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org

2 6