August 2023 - BaseX-Talk - mailman.uni-konstanz.de

BaseX 9.6: The Summer Edition
by Christian Grün 28 Nov '24

28 Nov '24

Dear all, We provide you with a new and fresh version of BaseX, our open source XML framework, database system and XQuery 3.1 processor: https://basex.org/ Apart from our main focus (query rewritings and optimizations), we have added the following enhancements: XQUERY: MODULES, FEATURES - Archive Module, archive:write: stream large archives to file - SQL Module: support for more SQL types - Full-Text Module, ft:thesaurus: perform Thesaurus queries - Fulltext, fuzzy search: specify Levenshtein limit - UNROLLLIMIT option: control limit for unrolling loops XQUERY: JAVA BINDINGS - Java objects of unknown type are wrapped into function items - results of constructor calls are returned as function items - the standard package "java.lang." has become optional - array arguments can be specified with the middle dot notation - conversion can be controlled with the WRAPJAVA option - better support for XQuery arrays and maps WEB APPLICATIONS - RESTXQ: Server-Timing HTTP headers are attached to the response For a more comprehensive list of added and updated features, look into our documentation (docs.basex.org) and check out the GitHub issues (github.com/BaseXdb/basex/issues) Have fun, Your BaseX Team

11 43

recursively used variables
by Rob Stapper 12 Aug '24

12 Aug '24

Hi, The code[1] below and send as attachment generates a error message: “Static variable depends on itself: $Q{http://www.w3.org/2005/xquery-local-functions}test”. I use these variables to refer to my private functions in my modules so I can easyly refer to them in a inheritance situation. It’s not a big problem for me but I was wondering if the error-triggering is justified or that it should work. [1]=========================================== declare variable $local:test := local:test#1 ; declare %private function local:test( $i) { if ( $i > 0) then $local:test( $i - 1) } ; $local:test( 10) =========================================== Kind regards, Rob Stapper Sent from Mail for Windows 10 -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

4 7

xslt:transform-report result
by Andy Bunce 11 Mar '24

11 Mar '24

Hi, Using BaseX 9.7.1 and saxon9he-9.9.1.jar The documentation suggests the ?result from xslt:transform-report should be *a* document-node where possible [1] This seems not quite to be the case when there are processing instructions or comments at the top level. In these cases a sequence of document-nodes is returned. /Andy [1] https://docs.basex.org/wiki/XSLT_Module#xslt:transform-report let $xslt:=<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0"> <xsl:mode on-no-match="shallow-copy"/> </xsl:stylesheet> let $xml:=document{ <?something type="compact"?>, <foo/>} return xslt:transform-report($xml,$xslt) Returns map { "messages": (), "result": (<?something type="compact"?>, <foo/>) }

3 7

Support non-admin user access to database admin interface (DBA)
by Matthew Dziuban 30 Aug '23

30 Aug '23

Hi all, While the subject might sound contradictory, I'm curious what you think about opening up the DBA code to allow non-admin users to access it and perform actions for which they have permissions? I currently maintain and run a fork of the DBA web app at work to make this possible, but I'd love to have the behavior built into BaseX if possible. You can view the changes I've made against BaseX 10.7 here: https://github.com/mblink/basex-webapp/compare/upstream-webapp...webapp-10.7 If you're open to this, I'd be happy to open a pull request with my changes! Thanks, Matt

2 5

copy/modify/return and the base URI
by Hans-Juergen Rennau 28 Aug '23

28 Aug '23

Dear BaseX team, I seem to remember that in earlier versions of BaseX a copy/modify/return did not change the base URI of the processed document. However, now it is lost, or more precisely, replaced with the static base URI of the processing XQuery module. Am I mistaken? At any rate it would be very desirable for copy/modify/return to preserve the base URI of the processed document or fragment. Kind regards,Hans-Jürgen

2 5

XQuery-Java-Binding: Nullpointer "nsMap" is null (on returning a w3c.dom.Node)
by Andreas Hengsbach | nexoma 28 Aug '23

28 Aug '23

Hi all, in a Java module I create a "org.w3c.dom.Node" (with java.xml.parsers.DocumentBuilder) and return it to XQuery/BaseX. Since 10.7 I get this exception: # schnipp Exception in thread "main" java.lang.NullPointerException: Cannot invoke "org.basex.util.hash.TokenMap.put(byte[], byte[])" because "nsMap" is null at org.basex.query.value.node.FElem.build(FElem.java:99) at org.basex.query.value.type.NodeType$3.cast(NodeType.java:60) at org.basex.query.value.type.NodeType$3.cast(NodeType.java:55) at org.basex.query.func.java.JavaCall.toValue(JavaCall.java:262) at org.basex.query.func.java.DynJavaFunc.eval(DynJavaFunc.java:90) at org.basex.query.func.java.JavaCall.value(JavaCall.java:81) at org.basex.query.expr.List.value(List.java:230) at org.basex.query.expr.gflwor.GFLWOR.value(GFLWOR.java:92) at org.basex.query.expr.gflwor.Let$LetEval.next(Let.java:146) at org.basex.query.expr.gflwor.GFLWOR.value(GFLWOR.java:92) at org.basex.query.expr.Try.value(Try.java:93) at org.basex.query.expr.List.value(List.java:226) at org.basex.query.expr.gflwor.GFLWOR.value(GFLWOR.java:92) at org.basex.query.expr.List.value(List.java:230) at org.basex.query.expr.If.value(If.java:158) at org.basex.query.expr.List.value(List.java:230) at org.basex.query.expr.Try.value(Try.java:93) at org.basex.query.expr.gflwor.GFLWOR.value(GFLWOR.java:92) at org.basex.query.expr.If.value(If.java:158) at org.basex.query.expr.List.value(List.java:226) at org.basex.query.expr.Try.value(Try.java:93) at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:46) at org.basex.query.scope.MainModule.iter(MainModule.java:48) at org.basex.query.QueryContext.lambda$iter$4(QueryContext.java:357) at org.basex.query.QueryContext.run(QueryContext.java:766) at org.basex.query.QueryContext.iter(QueryContext.java:357) at org.basex.query.QueryProcessor.iter(QueryProcessor.java:95) # schnapp Do I have to create the Node differently now, or is this a bug? nsMap. a map of namespace I guess. In my case the w3c.dom.Node has no namespace. Thanks a lot Andreas

2 4

"Expressions must be all updating" puzzle
by Eliot Kimber 21 Aug '23

21 Aug '23

In BaseX 10: In an updating function I have a call to another updating function that uses update:output() to log messages. If that function call is included, BaseX says that expressions must be all updating, but if I remove the call, BaseX is happy. The calling function is: declare %updating %rest:path('/now/rest/analytics/store-latest-acrolinx-users' ) function now:storeAcrolinxUsers() { let $acrolinxMap as map(*)? := try { acrolinx:getUserDataMap() } catch * { util:logToConsole( 'storeAcrolinxUsers' ,``[`{$err:code}` - `{$err:description}`]`` ,'warn' ) } return if (exists($acrolinxMap)) then let $doc as element() := acrolinx:serializeAcrolinxUserMap($acrolinxMap) let $fileName as xs:string := 'acrolinx-users_' || replace($doc/@timestamp, ':', '~') || '.xml' return db:put(analyticsmgmt:getAnalyticsDatabase(), $doc, relpath:newFile('acrolinx/users', $fileName)) }; The offending function is util:logToConsole(), which is declared as: declare %updating function util:logToConsole( $functionName as xs:string, $message as xs:string, $logLevel as xs:string ) { let $logMessage as xs:string := '[' || upper-case($logLevel) || '] ' || $functionName || '(): ' || $message return ( update:output(prof:dump($logMessage)) , update:output($logMessage) ) }; This is declared as an updating function and returns “updating” functions. If I comment out the call to util:logToConsole() in the first function, then BaseX is happy. If I use the non-updating version of logToConsole(), util:logToLog(), then BaseX is also happy. So my question is: why is BaseX complaining about the use of this updating function in an updating function? There must be some subtlety that I’m missing but I have not yet figured out what it is. Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 4

Problem with uploading a file larger than 512 MB .......
by Dmitry Gorskiy 21 Aug '23

21 Aug '23

Dear Sirs, What could be the problem? Big file upload problem. Below is the listing from the console. My computer: ------------------- CPU: 4 cores RAM: 8G OS: Linux BaseX: 107 and BaseX108-20230816.163639 Java: Adoptium idk-17.0.8+7 I'm creating files: dd if=/dev/random of=file_512M bs=1M count=512 …………………………file_530M …………………………file_1G …………………………file_2G and etc. Launching “basexhttp" ( default ...-Xmx2g … ) Let's go in … http://192.168.1.120:8080/dba/files <http://192.168.1.120:8080/dba/files> Downloading files - no problem. Uploading: file 512MB size - no problem. Uploading file 530MB and more - PROBLEM ( looking console output below ) OK, relaunching basexhttp whith -Xmx4G ….. Uploading file 512MB - ok Uploading file 1GB - ok More size - crashes What could be the problem? Thank you very much. Dmitriy Gorskiy. HTTP ERROR 500 java.lang.OutOfMemoryError: Java heap space URI: /dba/file-upload STATUS: 500 MESSAGE: java.lang.OutOfMemoryError: Java heap space SERVLET: RESTXQ CAUSED BY: java.lang.OutOfMemoryError: Java heap space Caused by: java.lang.OutOfMemoryError: Java heap space at java.base/java.util.Arrays.copyOf(Arrays.java:3537) at org.basex.util.list.ByteList.add(ByteList.java:43) at org.basex.io.in.BufferInput.content(BufferInput.java:216) at org.basex.http.RequestContext.body(RequestContext.java:119) at org.basex.http.RequestContext.addMultipart(RequestContext.java:137) at org.basex.http.RequestContext.formValues(RequestContext.java:103) at org.basex.http.restxq.RestXqFunction.bind(RestXqFunction.java:223) at org.basex.http.restxq.RestXqResponse.init(RestXqResponse.java:60) at org.basex.http.web.WebResponse.create(WebResponse.java:54) at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:72) at org.basex.http.BaseXServlet.service(BaseXServlet.java:69) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel$$Lambda$140/0x00007f5974196000.dispatch(Unknown Source) Powered by Jetty:// 9.4.51.v20230217 <https://eclipse.org/jetty>

2 1

Use Bash to construct multi-file form submission
by Eliot Kimber 18 Aug '23

18 Aug '23

This is not strictly a BaseX question but it is in the context of communicating with a BaseX-implemented REST service and I’m hoping someone in this community can answer my question as I’ve not found a solution the interwebs. I’ve implemented a POST method that expects a multi-part form request with multiple files that then loads those files into a database. The files are all related such that I want something close to an atomic commit for the set of files. Thus the multi-file form approach. The REST method works and I can use from the command line using curl, i.e: curl --location http://localhost:9984/now/rest/validation/reports?debug=false \ --form 'files=@"/Users/eliot.kimber/git-basex/product-content-analytics/validation-reports/vancouver/bundle-strategic-portfolio-mgmt_errors_vancouver_2023-08-14T22~11~28Z_1ae692f8_1692054936.xml"' \ --form 'files=@"/Users/eliot.kimber/git-basex/product-content-analytics/validation-reports/vancouver/bundle-tech-technology-industry_errors_vancouver_2023-08-14T22~11~28Z_1ae692f8_1692054971.xml"' What I haven’t been able to figure out is how to construct the equivalent curl command in a bash script, where the script gets a list of flies to load and then constructs the –form parameters. Everything I’ve tried results in a “Warning: Trailing data after quoted form parameter” message or some more utter failure. It must be some subtlety of how to escape the quotes—my googling around produces lots of stack overflow discussions of how to manage the escapes, but nothing I’ve tried has so far worked. My bash approach is simply: curlcmd=”curl --location …” for file in ${files[@]}; do curlcmd+=” --form ‘files=@\”${file}\”’ done And then execute the resulting string: bash -c $curlcmd I have a separate script that successfully constructs a curl call to a companion end point that takes one file: curl \ "--location" "${serverUrl}:${serverPort}/now/rest/validation/report/${fileName}?debug=${debug}" \ "--header" "Content-Type: text/xml" \ "--data-binary" "@${filePath}" And I’m currently just calling this for each file, which is OK. But I’d really like to be able to send a set of files as a single REST call. Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 2

Re: [basex-talk] mapping functions
by Imsieke, Gerrit, le-tex 17 Aug '23

17 Aug '23

Hi Graydon, Replying to the list for posterity. I hope you don’t mind. When you dynamically evaluate an expression (xquery:eval(), job:eval(), …), you start with the default values of the static context [1]. Nothing of the XQuery module from which you evaluate the expression is known (unless supplied by binding, but ordinary function declarations are not amenable to binding), therefore no risk of circularity. As stated in the table in [1], you can use module import [2] to augment this clean slate. In particular the module’s function declarations/signatures will be available then. If you don’t want the dynamically evaluated expression to import the whole module, you can store functions into variables and supply these variables to the evaluation through bindings. Haven’t tried it yet. Gerrit [1] https://www.w3.org/TR/xquery-31/#id-xq-static-context-components [2] https://www.w3.org/TR/xquery-31/#id-module-import On 16.08.2023 14:02, Graydon Saunders wrote: > Thank you for the specific example! > > I'm croggled that works without giving me a circular reference error of some kind, but it does work. > > Much appreciated! > > Graydon > > On Tue, Aug 15, 2023 at 11:34 PM <gerrit.imsieke(a)le-tex.de <mailto:gerrit.imsieke@le-tex.de>> wrote: > > You can import the library module in the evaluated expression. Example: https://github.com/transpect/control-backend/blob/main/webapp/control-backe… <https://github.com/transpect/control-backend/blob/main/webapp/control-backe…> > > Gerrit > > > Sent from MailDroid <https://goo.gl/ODgwBb> > > -----Original Message----- > From: Graydon Saunders <graydonish(a)gmail.com <mailto:graydonish@gmail.com>> > To: BaseX <BaseX-Talk(a)mailman.uni-konstanz.de <mailto:BaseX-Talk@mailman.uni-konstanz.de>> > Sent: Wed, 16 Aug 2023 4:42 > Subject: [basex-talk] mapping functions > > Hello -- > > I have various databases that will be run through the same code after their contents is regularized. So the only different function is the one used for regularization. > > This can be handled by writing a wrapper function and passing in a label to say which function to use, but I was wondering if there's a way to do it in the configuration map directly. The configuration map exists as a text file and gets run through xquery:eval() at the start of the process. > > declare namespace abc='http://some/uri <http://some/uri>'; > 'regularizationFunction': map { > 'typeOne': abc:regularizationOne#1, > 'typeTwo': abc:regularizationTwo#1 > } > > doesn't work, because the abc functions aren't known at evaluation time. These functions are defined in the library module that's evaluating the configuration file but xquery:eval() doesn't know that. I don't see a way to use bindings to get around this. > > Is there a way to map a locally-defined function reference so xquery:eval() will recognize it? > > thanks! > Graydon > > -- > Graydon Saunders | graydonish(a)fastmail.com <mailto:graydonish@fastmail.com> > Þæs oferéode, ðisses swá mæg. > -- Deor ("That passed, so may this.") > -- Gerrit Imsieke Geschäftsführer / Managing Director le-tex publishing services GmbH Weissenfelser Str. 84, 04229 Leipzig, Germany Phone +49 341 355356 110, Fax +49 341 355356 510 gerrit.imsieke(a)le-tex.de, http://www.le-tex.de Registergericht / Commercial Register: Amtsgericht Leipzig Registernummer / Registration Number: HRB 24930 Geschäftsführer / Managing Directors: Gerrit Imsieke, Svea Jelonek, Thomas Schmidt ---------------------------------------------- Besuchen Sie uns auf der Frankfurter Buchmesse in Halle 4.0, G94.

2 3