- BaseX-Talk - mailman.uni-konstanz.de

Docker query
by Joseph Szili 13 Nov '19

13 Nov '19

Hello all, I'm trying to use the docker image for 9.x but when I surf to http://localhost:8984/dba/ per the documentation ... I get the following response in the browser No function found that matches the request. Using this command in linux (Ubuntu 19.04) ›$ docker run -d \ --name basexhttp \ --publish 1984:1984 \ --publish 8984:8984 \ --volume "$HOME/Projects/basex-dev/data":/srv/basex/data \ basex/basexhttp:latest container log >>>>>>>>>> /srv/basex/.basex: writing new configuration file. BaseX 9.3 beta [HTTP Server] [main] INFO org.eclipse.jetty.util.log - Logging initialized @248ms to org.eclipse.jetty.util.log.Slf4jLog [main] INFO org.eclipse.jetty.server.Server - jetty-9.4.21.v20190926; built: 2019-09-26T16:41:09.154Z; git: 72970db61a2904371e1218a95a3bef5d79788c33; jvm 1.8.0_212-b04 [main] INFO org.eclipse.jetty.util.TypeUtil - JVM Runtime does not support Modules [main] INFO org.eclipse.jetty.webapp.StandardDescriptorProcessor - NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet [main] INFO org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0 [main] INFO org.eclipse.jetty.server.session - No SessionScavenger set, using defaults [main] INFO org.eclipse.jetty.server.session - node0 Scavenging every 600000ms Server was started (port: 1984). java.io.FileNotFoundException: /srv/basex/data/.logs/2019-11-08.log (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at org.basex.server.LogFile.create(LogFile.java:31) at org.basex.server.Log.write(Log.java:128) at org.basex.server.Log.writeServer(Log.java:70) at org.basex.BaseXServer.<init>(BaseXServer.java:122) at org.basex.http.HTTPContext.init(HTTPContext.java:101) at org.basex.http.BaseXServlet.init(BaseXServlet.java:37) at org.eclipse.jetty.servlet.ServletHolder$WrapperServlet.init(ServletHolder.java:1287) at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:599) at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:425) at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:751) at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:352) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312) at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:743) at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:361) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1443) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1407) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:821) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:276) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:407) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106) at org.eclipse.jetty.server.Server.doStart(Server.java:371) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.basex.BaseXHTTP.<init>(BaseXHTTP.java:120) at org.basex.BaseXHTTP.main(BaseXHTTP.java:52) [main] INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@545997b1{BaseX: The XML Database and XQuery Processor,/,file:///srv/basex/webapp/,AVAILABLE}{/srv/basex/webapp} [main] INFO org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@3c756e4d{HTTP/1.1,[http/1.1]}{0.0.0.0:8984} [main] INFO org.eclipse.jetty.server.Server - Started @678ms HTTP STOP Server was started (port: 8985). java.io.FileNotFoundException: /srv/basex/data/.logs/2019-11-08.log (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at org.basex.server.LogFile.create(LogFile.java:31) at org.basex.server.Log.write(Log.java:128) at org.basex.server.Log.writeServer(Log.java:70) at org.basex.BaseXHTTP.<init>(BaseXHTTP.java:150) at org.basex.BaseXHTTP.main(BaseXHTTP.java:52) HTTP Server was started (port: 8984).

3 2

file:list and symbolic links
by Marco Lettere 09 Nov '19

09 Nov '19

Dear all, what is the rationale behind the fact that file:list has been rewritten to not following symbolic links [1]? We had some jobs related code that relied on symlinks and stopped working after moving to new version of BaseX. Is there any work around? By using children or the new descendants functions maybe? Thanks for your support. Marco. [1] https://github.com/BaseXdb/basex/blob/bbb62d1ec3e10b4fa1cc48bd4e16911fa73e4…

2 3

Backups
by Christoph Gaukel 09 Nov '19

09 Nov '19

Hi, the following handling of backups is surprising: * db:alter changes the name of the database but does not alter the names of the corresponding backups * db:drop deletes both, the database and the corresponding backups I would prefer, if there was an option in db:drop to keep the backups, which would allow to restore a deleted database from their backups. Is it possible to add options to db:drop, which allow that? For me, the behavior of db:alter looks like a bug. There is another minor handling, i would like to have changed: * db:info($db)/resourceproperties/inputdate is a UTC dateTime with a time zone offset * db:backups($db)/@date is a local dateTime without a time zone offset If I compare these two, it works correctly if the local time zone offset is the same as for the creation of the backup E.g.: if ( not(db:backups($db)) or max(db:backups($db)/@date/(. cast as xs:dateTime)) < db:info($db)/resourceproperties/inputdate/. cast as xs:dateTime ) then db:create-backup($db) else () Adding a time zone offset +hh:mm or -hh:mm to db:backups($db)/@date would solve this. Thanks, Christoph

2 2

"Out of Memory" when inserting data from one DB to another
by BIRKNER Michael 30 Oct '19

30 Oct '19

Hi to all, I get an "Out of Memory" error (using the BaseX GUI on Ubuntu Linux) when I try to insert quite a lot of data into a BaseX database. The use case: I have a database (size is about 2600 MB, 13718400 nodes) with information in <record> elements that should be added to <record> elements in another database. The <record>s have a 1 to 1 connection identified by an ID that is available in both databases. An example (simplified) of the DB with the information I want to add to the other DB: <collection> <record> <id>1</id> <data>Some data</data> <data>More data</data> <data>More data</data> ... </record> <record> <id>2</id> <data>Some data</data> <data>More data</data> <data>More data</data> ... </record> <record> <id>3</id> <data>Some data</data> <data>More data</data> <data>More data</data> ... </record> ... many many more <record>s </collection> Here an example (simplified) of the DB to which the above <data> elements should be added: <collection> <record> <id>1</id> <mainData>Main data</mainData> <mainData>More main data</mainData> <mainData>More main data</mainData> ...  </record> <record> <id>2</id> <mainData>Main data</mainData> <mainData>More main data</mainData> <mainData>More main data</mainData> ...  </record> <record> <id>3</id> <mainData>Main data</mainData> <mainData>More main data</mainData> <mainData>More main data</mainData> ...  </record> ... many many more <record>s </collection> This is the XQuery I use to insert the given <data> elements from the <record> in one database to the corresponding <record> in the other database. It results in an "Out of Memory" error: let $infoRecs := db:open('db-with-data')/collection/record let $mainRecs := db:open('db-to-insert-data')/collection/record for $infoRec in $infoRecs let $id := data($infoRec/id) let $mainRec := $mainRecs[id=$id] let $dataToInsert := $infoRec/*[not(name()='id')] return insert node ($dataToInsert) into $mainRec I assume that the error is a result of the large amount of data that is processed. My question is if a strategy exists to work with such an amount of data without getting an "Out of Memory" error? Thanks very much to everyone in advance for any hint and advice. If you need more information about DB setup or options just let me know. Best regrads, Michael Dieses Mail ist ausschließlich für die Verwendung durch die/den darin genannten AdressatInnen bestimmt und kann vertrauliche bzw rechtlich geschützte Informationen enthalten, deren Verwendung ohne Genehmigung durch den/ die AbsenderIn rechtswidrig sein kann. Falls Sie dieses Mail irrtümlich erhalten haben, informieren Sie uns bitte und löschen Sie die Nachricht. UID: ATU 16209706 I https://wien.arbeiterkammer.at/datenschutz

2 2

http:request parsing question
by Bridger Dyson-Smith 29 Oct '19

29 Oct '19

Hi all - I was hoping that someone on the list could help with my thinking about parsing the http:response values from a request; I've tried the following, but I'm not thinking about this correctly. declare namespace oai = "http://www.openarchives.org/OAI/2.0/"; declare namespace h = "http://expath.org/ns/http-client"; http:send-request(<http:request method='get' href=' http://export.arxiv.org/oai2?verb=Identify'/>)//@status/data() returns '200', but trying http:send-request(<http:request method='get' href=' http://export.arxiv.org/oai2?verb=Identify'/>)/h:response/@status fails. I'm guessing (but am not sure) that the combination of the <http:response> with the body of the response is causing my problem, but I'm not sure how to wiggle my way around it. Note: this is not in a RESTXQ context (yet), but for my simple trials is there a way to get at the http -namespace elements? Thank you for your time and trouble. Best, Bridger

3 5

Binding a variable
by Ben Engbers 29 Oct '19

29 Oct '19

Hi, When experimenting with my RBaseX-package (I had hoped to submit it to CRAN today), I use the following pattern: 1 Define a query 2 Create a query-object 3 Bind variables (optional) 4 Execute the query When used this pattern on the following query, everything functions as expected: declare variable $name external; for $i in 1 to 3 return element { $name } { $i } The following query is also functioning: paste("declare variable $greet external;", "declare variable $friend external;", "declare variable $into external;", "let $greet := 'Greetings, my '", "let $friend := 'friend'", "for $i in 1 to 3", "let $friend_num := $greet || $friend || $i", "return insert nodes element { $friend } { $friend_num }", "into root", sep = " ") When I modify "into root" to "into $p" and bind $p to root Bind(query_obj, "$p", "root") I get the following error: [XPST0008] Undeclared variable $p The Bind-function returns with code \00, indicating that it is executed without errors. Does this mean that there is a bug in my code or am I violating XQuery-syntax? Ben Engbers

2 2

Understanding the path index
by Beck, Julia 29 Oct '19

29 Oct '19

Hi, first of all: thank you, the fix for [1] did the trick and in 9.2.4 the query is working as expected. Today, I come back to you with another challenge in performance which again seems to have something to do with indexing(?). So here's the situation: I have two databases "abc" and "def". "abc" contains 1 xml doc with about 150.000 nodes and "def" contains 1 xml doc with about 400.000 nodes. Both are similarly strutured and have their up-to-date text and attr indexes. The xml docs look both (simplified) like the following: <rdf:RDF> <ore:Aggregation rdf:about="123"> <edm:object rdf:resource="urn1"/> <...> </ore:Aggregation> <edm:WebResource rdf:about="urn1"> <...> </edm:WebResource> <ore:Aggregation rdf:about="124"> <edm:object rdf:resource="urn2"/> <...> </ore:Aggregation> <edm:WebResource rdf:about="urn2"> <...> </edm:WebResource> <ore:Aggregation rdf:about="125"> <edm:object rdf:resource="urn2"/> <edm:object rdf:resource="urn3"/> <...> </ore:Aggregation> <edm:WebResource rdf:about="urn3"> <...> </edm:WebResource> <...> </rdf:RDF> So one aggregation refers to one (or more) web resources. I boiled down my original query to the following purpose to keep it simple: for each aggregation give me the corresponding web resource. for $agg in db:open($db_name)/rdf:RDF/ore:Aggregation return for $urn in $agg/edm:object/@rdf:resource return (# db:enforceindex #) {db:open($db_name)/rdf:RDF/edm:WebResource[@rdf:about=$urn]} For both databases the query gives me the required result and the query info tells me that the attribute index for $urn is applied in both cases (this is also the case if I leave out the pragma). However, oddly enough, for the "larger" database "def" with a larger attribute index it takes roughly 1 second while the "smaller" database "abc" with a smaller attribute index takes 20 seconds. This is not very long but the original query is more complicated and I have bigger databases with the same structure where it starts to matter. The only (and I think important) difference between "abc" and "def" is that "abc" contains only 3 web resources that all 5.000 aggregations refer to. While in "def" each aggregation refers to a particular web resource (== 7.000 aggregations and 7.000 web resources). With index:facets I had a look at the facet values and learned that there is a "maximum number of distinct values to store per name". Is there a difference in performance because of that? Maybe I do not get the index structures but it feels strange that it takes longer to find the correct attribute in a range of 3 different values than in a range of 7.000. Maybe there is also another problem in my query, databases or my reasoning that I do not see? Either way, I need help in understanding this phenomenon :-) I hope you could follow, please don't hesitate to ask if you need anything to reproduce this situation (I am using BaseX 9.2.4). Julia [1] https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-July/014511.html

3 6

http:send-request - problem with response
by Bogdan Bogucki 29 Oct '19

29 Oct '19

Hello, I encounter problem with http:send-request function. I need handle multiple forward requests (3) with cookies. First request returns cookies information which are required with next requests. Output from http:send-request dosen't contains all fields which are returned by server. Response from browser is flowing: cache-control private cf-cache-status DYNAMIC cf-ray 52b3cedc5b9bcc9f-WAW content-type text/html; charset=utf-8 date Fri, 25 Oct 2019 11:21:37 GMT expect-ct max-age=604800, report-uri="ht….com/cdn-cgi/beacon/expect-ct" location http://www.pracuj.pl/praca/jav…eloper-warszawa,oferta,7171988 server cloudflare set-cookie __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly set-cookie _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/ set-cookie _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly x-aspnet-version 4.0.30319 x-aspnetmvc-version 5.2 X-Firefox-Spdy h2 x-powered-by ASP.NET x-ua-compatible IE=edge Request: http:send-request( <http:request method='get' follow-redirect='false'> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"/> <http:header name="Accept" value="text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"/> <http:header name="Accept-Languag" value="pl,en-US;q=0.7,en;q=0.3"/> <http:header name="DNT" value="1"/> <http:header name="Connection" value="keep-alive"/> <http:header name="Upgrade-Insecure-Requests" value="1"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> </http:request>, 'https://www.pracuj.pl/praca/java-developer-warszawa,oferta,7171988') Response form http:send-request is: <http:response xmlns:http="http://expath.org/ns/http-client" status="302" message="Found"> <http:header name="Transfer-Encoding" value="chunked"/> <http:header name="CF-RAY" value="52b3d0de097fb839-WAW"/> <http:header name="Server" value="cloudflare"/> <http:header name="Connection" value="keep-alive"/> <http:header name="Date" value="Fri, 25 Oct 2019 11:23:00 GMT"/> <http:header name="X-AspNetMvc-Version" value="5.2"/> <http:header name="X-UA-Compatible" value="IE=edge"/> <http:header name="CF-Cache-Status" value="DYNAMIC"/> <http:header name="Cache-Control" value="private"/> <http:header name="X-AspNet-Version" value="4.0.30319"/> <http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/> <http:header name="Expect-CT" value="max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct…"/> <http:header name="X-Powered-By" value="ASP.NET"/> <http:header name="Content-Type" value="text/html; charset=utf-8"/> <http:body media-type="text/html"/> </http:response> Two set-cookie fields is missing. How should I invoke http:send-request to receive proper result ? Regards Bogdan

2 1

Optimization issue?
by Johannes Echterhoff 25 Oct '19

25 Oct '19

Hello, I'm experiencing some strange behavior when executing the following query in the BaseX 9.2.4 GUI (nevermind the awkward query structure - that is just to be able to inspect the results of the individual evaluation steps): let $t := <x>200.0000</x> let $num := number($t) let $gt5000 := $num gt 5000 let $ltMinus5000 := $num lt -5000 return $gt5000 or $ltMinus5000 The result is true, although it should be false. The compilation and optimization logs for this query in the GUI tell us the following: ---------- Compiling: - rewrite > comparison to range comparison: ($num_1 > 5000) -> ($num_1 >= 5000.000000000001) - rewrite 'gt' comparison to range comparison: ($num_1 gt 5000) -> ($num_1 >= 5000.000000000001) - pre-evaluate unary expression to xs:integer item: -5000 - rewrite < comparison to range comparison: ($num_1 < -5000) -> ($num_1 <= -5000.000000000001) - rewrite 'lt' comparison to range comparison: ($num_1 lt -5000) -> ($num_1 <= -5000.000000000001) - inline $gt5000_2 - inline $ltMinus5000_3 - simplify or: (true()) - pre-evaluate or to xs:boolean item: (true()) -> true() - remove unused variable $num_1 - remove unused variable $t_0 - simplify FLWOR expression: true() Optimized Query: true() ---------- When the return statement is changed to "return ($num,$gt5000,$ltMinus5000,$gt5000 or $ltMinus5000)" then the result of the boolean operation (last element of the result sequence) is false, as expected. The GUI log then is: ---------- Compiling: - rewrite > comparison to range comparison: ($num_1 > 5000) -> ($num_1 >= 5000.000000000001) - rewrite 'gt' comparison to range comparison: ($num_1 gt 5000) -> ($num_1 >= 5000.000000000001) - pre-evaluate unary expression to xs:integer item: -5000 - rewrite < comparison to range comparison: ($num_1 < -5000) -> ($num_1 <= -5000.000000000001) - rewrite 'lt' comparison to range comparison: ($num_1 lt -5000) -> ($num_1 <= -5000.000000000001) Optimized Query: let $t_0 := element x { ("200.0000") } let $num_1 := number($t_0) let $gt5000_2 := ($num_1 >= 5000.000000000001) let $ltMinus5000_3 := ($num_1 <= -5000.000000000001) return ($num_1, $gt5000_2, $ltMinus5000_3, ($gt5000_2 or $ltMinus5000_3)) ---------- Am I missing something, or could this be a bug in BaseX 9.2.4? With BaseX 9.0.2, both queries (with the different return statements) returned the expected value false for the boolean operation. Best regards, Johannes

2 3

Question about WITHDB option
by France Baril 24 Oct '19

24 Oct '19

I am looking at the documentation for WITHDB and I want to ask for further explanation 'If db:open is consistently used to access documents', does it means as opposed to having a DEFAULTDB or does it mean that we use db:open but not local queries on the document obtained (example below), or since there is a notion of lock, do you mean db:open as opposed to db:add or db:replace? In short, I lack context to interpret the explanation. let $current-doc := db:open($lang)/*[@id=x] (: Query document further without db:opened :) let $avaliable-sections := $current-doc/descendant::*[name()=''section][@status !='archived'] >From the documentation: WITHDB Introduced with Version 9.3. *Signature* WITHDB *Default* true *Summary* By default, URIs specified in the fn:doc <http://docs.basex.org/wiki/Databases#XML_Documents> and fn:collection <http://docs.basex.org/wiki/Databases#XML_Documents> functions will also be resolved against existing databases. If db:open <http://docs.basex.org/wiki/Database_Module#db:open> is consistently used to access database documents, it is recommendable to disable this option: - Access to local and external resources will be faster, as the database lookup will be skipped. - No locks will be created by the two functions (see limitations of database locking <http://docs.basex.org/wiki/Transaction_Management#Limitations> for more details). -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 3