November 2019 - BaseX-Talk - mailman.uni-konstanz.de

Is there an API that provides XQuery compilation results?
by Peter Villadsen 15 Nov '19

15 Nov '19

When I use the GUI application I can see some error description (even if it a little terse) when my queries are incorrect. For instance I might get: Stopped at tableFields.xq, 27/50: [XQST0118] Different start and end tag: <Artifact>...</Artiact>. If I submit a query like <Artifact></Artiact> However, I cannot seem to find an API that brings me that information? I tried: xquery:parse("<B></C>", map {'pass':true()}) but that did not get the result I expected. Also, while we're at at it, it would be nice to also get the optimized query. Best Regards Peter Villadsen Principal Architect Microsoft Business Applications Group

3 2

Re: [basex-talk] REPLACE vs STORE
by Christian Grün 14 Nov '19

14 Nov '19

> Thank you for your answer. Do you plan something like that in future? We face the problem we need to upload tens thousands of files in 4 hours per week. We are able to do it with replace with folder, but current store per file is too slow. :-( Have you already tried the XQuery way? If so, and if it’s too slow, feel free to share your code with us. > Odesláno z mého chytrého telefonu Samsung Galaxy. > > -------- Původní zpráva -------- > Od: Christian Grün <christian.gruen(a)gmail.com> > Datum: 14.11.19 18:30 (GMT+01:00) > Komu: Radim Havlicek <radim-havlicek(a)post.cz> > Cc: BaseX <basex-talk(a)mailman.uni-konstanz.de> > Předmět: Re: [basex-talk] REPLACE vs STORE > > Hi Radim, > > The STORE command is restricted to single files [1]. XQuery and > db:store (e.g. combined with file:list) gives you more flexibility > here. > > Hope this helps, > Christian > > [1] http://docs.basex.org/wiki/Commands#STORE > > > > > On Thu, Nov 14, 2019 at 10:53 AM <radim-havlicek(a)post.cz> wrote: > > > > Hello, > > I would like to store new reports into existing database folder (with other existing files already) after opening. > > > > Help: > > REPLACE > > Syntax REPLACE [path] [input] > > XML Syntax <replace path='...'>[input]</replace> > > > > STORE > > Syntax STORE (TO [path]) [input] > > XML Syntax <store (path='...')>[input]</store> > > > > But actually I encountered this problem: > > > open mydb > > Database 'mydb' was opened in 3.0 ms. > > > replace /Report/ c:/Share/REPORTS/ > > 903 resource(s) replaced in 11077.73 ms. > > > store to /Report/ c:/Share/REPORTS/ > > Resource "C:/Share/REPORTS/" not found. > > > > What am I doing wrong? Is it possible to use store to instead of replace to just add files to existing ones? > > > > Thanks in advance for your answer, > > > > Radim Havlicek

1 0

REPLACE vs STORE
by radim-havlicek＠post.cz 14 Nov '19

14 Nov '19

Hello, I would like to store new reports into existing database folder (with other existing files already) after opening. Help: REPLACE Syntax REPLACE [path] [input] XML Syntax <replace path='...'>[input]</replace> STORE Syntax STORE (TO [path]) [input] XML Syntax <store (path='...')>[input]</store> But actually I encountered this problem: > open mydb Database 'mydb' was opened in 3.0 ms. > replace /Report/ c:/Share/REPORTS/ 903 resource(s) replaced in 11077.73 ms. > store to /Report/ c:/Share/REPORTS/ Resource "C:/Share/REPORTS/" not found. What am I doing wrong? Is it possible to use store to instead of replace to just add files to existing ones? Thanks in advance for your answer, Radim Havlicek

2 1

Feature request: leveraging db:open-id and db:node-id
by France Baril 14 Nov '19

14 Nov '19

Hi, any chance we could make the db indexed ids match the xml doc ids? - db:open-id() could match the id at the root of then documents (when one is provided) - db:node-id() could match other nodes match internal ids (when they are provided). It would make indexing and querying a lot more efficient for any type of navigation/linking purposes. Options: - I'd suggest that this could be an option that can be turned on/off for each doc level ids and node level ids - I'd suggest that node id has the option to be unique within the db or within a document when the value for the entire db would then have to become something like doc-id#node-id to stay unique within the db Then I can easily imagine also expending to db:save-id to reduce the workload of having to pass through base-uri() once the doc exists in the db under its path let $db-name := 'en-us' let $doc := db:open-id($db-name, $doc-id) let $new-doc := copy, modify, return copy db:replace-id('en-us', $doc-id, $new-doc) instead of let $db-name := 'en-us' let $doc := db:open($db-name)/*[@id=$doc-id] let $new-doc := copy, modify, return copy let $path := substring-after($doc/base-uri(), $db-name) return db:replace($db-name, $path, $new-doc) Is that something that can be considered?

2 4

I am looking for the fastest way to sort 2.4 Mio tags by two attribute ascending and descending
by Omar Siam 14 Nov '19

14 Nov '19

Hi, I have a custom index that looks like this (one db, different files): <_:dryed xmlns:_="https://www.oeaw.ac.at/acdh/tools/vle/util" db_name="z881_qdb-TEI-02n" order="none"> <_:d pre="15627" db_name="z881_qdb-TEI-02n" xml:id="z881_qdbn-d16e2" vutlsk="tsįttr Ziter [Subst]" vutlsk-archiv="HK 881, z8810118.sch#1"/> <_:d pre="15673" db_name="z881_qdb-TEI-02n" xml:id="z881_qdbn-d16e21" vutlsk="tsįttr Ziter [Subst]" vutlsk-archiv="HK 881, z8810118.sch#1"/> ... </_:dryed> <_:dryed xmlns:_="https://www.oeaw.ac.at/acdh/tools/vle/util" db_name="f227_qdb-TEI-02n" order="none"> <_:d pre="467" db_name="f227_qdb-TEI-02n" xml:id="f237_qdb-d1e29398" vutlsk="(aus)faren [Verb]" vutlsk-archiv="HK 327, f227#944.1 = fare0126.eck#1.1"/> <_:d pre="591" db_name="f227_qdb-TEI-02n" xml:id="f237_qdb-d1e29438" vutlsk="(aus)faren [Verb]" vutlsk-archiv="HK 327, f227#945.1 = fare0126.eck#2.1"/> ... </_:dryed> There are about 2.4 Mio _:d tags in this db. I need to sort them by the @vutlsk* attributes alphabetically in ascending and descending order. With the code I have now: declare namespace _ = "https://www.oeaw.ac.at/acdh/tools/vle/util"; let $sorted-ascending := subsequence(for $d in collection('_qdb-TEI-02__cache')//*[@order="none"]/_:d order by $d/@vutlsk ascending return $d/(@ID, @xml:id)/data(), 1, 10000) let $sorted-descending := subsequence(for $d in collection('_qdb-TEI-02__cache')//*[@order="none"]/_:d order by $d/@vutlsk descending return $d/(@ID, @xml:id)/data(), 1, 10000) let $sorted-ascending-archiv := subsequence(for $d in collection('_qdb-TEI-02__cache')//*[@order="none"]/_:d order by $d/@vutlsk-archiv ascending return $d/(@ID, @xml:id)/data(), 1, 10000) let $sorted-descending-archiv := subsequence(for $d in collection('_qdb-TEI-02__cache')//*[@order="none"]/_:d order by $d/@vutlsk-archiv descending return $d/(@ID, @xml:id)/data(), 1, 10000) return (db:replace("_qdb-TEI-02__cache", 'ascending_cache.xml', <_:dryed order="ascending" ids="{string-join($sorted-ascending, ' ')}"/>), db:replace("_qdb-TEI-02__cache", 'descending_cache.xml', <_:dryed order="descending" ids="{string-join($sorted-descending, ' ')}"/>), db:replace("_qdb-TEI-02__cache", 'ascending-archiv_cache.xml', <_:dryed order="ascending" label="archiv" ids="{string-join($sorted-ascending-archiv, ' ')}"/>), db:replace("_qdb-TEI-02__cache", 'descending-archiv_cache.xml', <_:dryed order="descending" label="archiv" ids="{string-join($sorted-descending-archiv, ' ')}"/>)) This takes 30 s to about a minute depending on the subsequence I choose. I did experiments with doing multithreading and not. Multiple jobs or fork-join make it worse. Worst case I need to do it every time I save a change to the original DBs for which I maintain that index. Any ideas how to speed this up? Best regards Omar Siam

2 5

Re: [basex-talk] http:send-request - problem with response
by Christian Grün 13 Nov '19

13 Nov '19

Hi Bogdan (cc to the list), Thanks for digging deeper. I noticed that the standard Java function that we used returned only one value per header field, and dropped the others [1]. I managed to fix this in the latest stable snapshot; could you give it a try [2]? BaseX 9.3 will be released end of November. Cheers, Christian [1] http://files.basex.org/releases/latest/ [2] https://github.com/BaseXdb/basex/issues/1751 On Wed, Nov 13, 2019 at 4:26 PM Bogdan Bogucki <bbogucki(a)hussar.pl> wrote: > > Hi Christian, > Lack of many redirection is not a big problem. I can handle it manually. > Problem is not complete response header form first request. > > Browser make 3 redirection I am talking about first. > > Please take a look. > > > Response header from browser (first request) - contains three fields Set > Cookies: > > set-cookie __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly > > set-cookie _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/ > > set-cookie _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly > > but response header form http:send-request contains only last one: > > <http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; > expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/> > > I need information from previous set-cookie fields to make manual > redirection passing information about sessions to next requests. > > Regards > > Bogdan Bogucki > > W dniu 29.10.2019 o 03:06, Christian Grün pisze: > > Hi Bogdan, > > > > The current http:send-request implementation is based on the default > > Java HttpURLConnection, which does not resolve redirects that use > > different protocols [1]. This is the reason why your request will not > > be fully processed (as it e.g. happens when you use the browser). > > > > It seems that your initial request to the https protocol returns a 302 > > redirect to a (now unsafe) http URL, which returns another redirect to > > https. I don’t know who maintains the discussed web site, but it could > > be worth contacting the admins and asking them if they could update > > and simplify their redirect policy. > > > > Hope this helps, > > Christian > > > > [1] https://stackoverflow.com/a/1884427/1018790 > > > > > > > > > > On Fri, Oct 25, 2019 at 1:52 PM Bogdan Bogucki <bbogucki(a)hussar.pl> wrote: > >> > >> Hello, > >> > >> I encounter problem with http:send-request function. I need handle multiple forward requests (3) with cookies. > >> > >> First request returns cookies information which are required with next requests. Output from http:send-request dosen't contains all fields which are returned by server. > >> > >> Response from browser is flowing: > >> cache-control > >> private > >> cf-cache-status > >> DYNAMIC > >> cf-ray > >> 52b3cedc5b9bcc9f-WAW > >> content-type > >> text/html; charset=utf-8 > >> date > >> Fri, 25 Oct 2019 11:21:37 GMT > >> expect-ct > >> max-age=604800, report-uri="ht….com/cdn-cgi/beacon/expect-ct" > >> location > >> http://www.pracuj.pl/praca/jav…eloper-warszawa,oferta,7171988 > >> server > >> cloudflare > >> set-cookie > >> __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly > >> set-cookie > >> _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/ > >> set-cookie > >> _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly > >> x-aspnet-version > >> 4.0.30319 > >> x-aspnetmvc-version > >> 5.2 > >> X-Firefox-Spdy > >> h2 > >> x-powered-by > >> ASP.NET > >> x-ua-compatible > >> IE=edge > >> > >> > >> Request: > >> > >> http:send-request( > >> <http:request method='get' follow-redirect='false'> > >> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"/> > >> <http:header name="Accept" value="text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"/> > >> <http:header name="Accept-Languag" value="pl,en-US;q=0.7,en;q=0.3"/> > >> <http:header name="DNT" value="1"/> > >> <http:header name="Connection" value="keep-alive"/> > >> <http:header name="Upgrade-Insecure-Requests" value="1"/> > >> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> > >> </http:request>, 'https://www.pracuj.pl/praca/java-developer-warszawa,oferta,7171988') > >> > >> Response form http:send-request is: > >> > >> <http:response xmlns:http="http://expath.org/ns/http-client" status="302" message="Found"> > >> <http:header name="Transfer-Encoding" value="chunked"/> > >> <http:header name="CF-RAY" value="52b3d0de097fb839-WAW"/> > >> <http:header name="Server" value="cloudflare"/> > >> <http:header name="Connection" value="keep-alive"/> > >> <http:header name="Date" value="Fri, 25 Oct 2019 11:23:00 GMT"/> > >> <http:header name="X-AspNetMvc-Version" value="5.2"/> > >> <http:header name="X-UA-Compatible" value="IE=edge"/> > >> <http:header name="CF-Cache-Status" value="DYNAMIC"/> > >> <http:header name="Cache-Control" value="private"/> > >> <http:header name="X-AspNet-Version" value="4.0.30319"/> > >> <http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/> > >> <http:header name="Expect-CT" value="max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct…"/> > >> <http:header name="X-Powered-By" value="ASP.NET"/> > >> <http:header name="Content-Type" value="text/html; charset=utf-8"/> > >> <http:body media-type="text/html"/> > >> </http:response> > >> > >> Two set-cookie fields is missing. > >> > >> How should I invoke http:send-request to receive proper result ? > >> > >> Regards > >> > >> Bogdan

2 1

proc:system
by Giuseppe G. A. Celano 13 Nov '19

13 Nov '19

Hi, When I try to run proc:system("/usr/local/bin/tesseract") I get the error [proc:code0001] Similarly: proc:system("tesseract") returns [proc:error] Cannot run program "tesseract": error=2, No such file or directory Similarly: proc:system("tesseract", (), map {"dir" : "/usr/local/bin/"}) returns [proc:error] Cannot run program "tesseract" (in directory "/usr/local/bin"): error=2, No such file or directory The command "tesseract" works at the command line. I suspect there may be a problem with permissions: is there a way to overcome this error? Thanks. Best, Giuseppe

2 5

Docker query
by Joseph Szili 13 Nov '19

13 Nov '19

Hello all, I'm trying to use the docker image for 9.x but when I surf to http://localhost:8984/dba/ per the documentation ... I get the following response in the browser No function found that matches the request. Using this command in linux (Ubuntu 19.04) ›$ docker run -d \ --name basexhttp \ --publish 1984:1984 \ --publish 8984:8984 \ --volume "$HOME/Projects/basex-dev/data":/srv/basex/data \ basex/basexhttp:latest container log >>>>>>>>>> /srv/basex/.basex: writing new configuration file. BaseX 9.3 beta [HTTP Server] [main] INFO org.eclipse.jetty.util.log - Logging initialized @248ms to org.eclipse.jetty.util.log.Slf4jLog [main] INFO org.eclipse.jetty.server.Server - jetty-9.4.21.v20190926; built: 2019-09-26T16:41:09.154Z; git: 72970db61a2904371e1218a95a3bef5d79788c33; jvm 1.8.0_212-b04 [main] INFO org.eclipse.jetty.util.TypeUtil - JVM Runtime does not support Modules [main] INFO org.eclipse.jetty.webapp.StandardDescriptorProcessor - NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet [main] INFO org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0 [main] INFO org.eclipse.jetty.server.session - No SessionScavenger set, using defaults [main] INFO org.eclipse.jetty.server.session - node0 Scavenging every 600000ms Server was started (port: 1984). java.io.FileNotFoundException: /srv/basex/data/.logs/2019-11-08.log (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at org.basex.server.LogFile.create(LogFile.java:31) at org.basex.server.Log.write(Log.java:128) at org.basex.server.Log.writeServer(Log.java:70) at org.basex.BaseXServer.<init>(BaseXServer.java:122) at org.basex.http.HTTPContext.init(HTTPContext.java:101) at org.basex.http.BaseXServlet.init(BaseXServlet.java:37) at org.eclipse.jetty.servlet.ServletHolder$WrapperServlet.init(ServletHolder.java:1287) at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:599) at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:425) at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:751) at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:352) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312) at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:743) at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:361) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1443) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1407) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:821) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:276) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:407) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106) at org.eclipse.jetty.server.Server.doStart(Server.java:371) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.basex.BaseXHTTP.<init>(BaseXHTTP.java:120) at org.basex.BaseXHTTP.main(BaseXHTTP.java:52) [main] INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@545997b1{BaseX: The XML Database and XQuery Processor,/,file:///srv/basex/webapp/,AVAILABLE}{/srv/basex/webapp} [main] INFO org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@3c756e4d{HTTP/1.1,[http/1.1]}{0.0.0.0:8984} [main] INFO org.eclipse.jetty.server.Server - Started @678ms HTTP STOP Server was started (port: 8985). java.io.FileNotFoundException: /srv/basex/data/.logs/2019-11-08.log (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at org.basex.server.LogFile.create(LogFile.java:31) at org.basex.server.Log.write(Log.java:128) at org.basex.server.Log.writeServer(Log.java:70) at org.basex.BaseXHTTP.<init>(BaseXHTTP.java:150) at org.basex.BaseXHTTP.main(BaseXHTTP.java:52) HTTP Server was started (port: 8984).

3 2

file:list and symbolic links
by Marco Lettere 09 Nov '19

09 Nov '19

Dear all, what is the rationale behind the fact that file:list has been rewritten to not following symbolic links [1]? We had some jobs related code that relied on symlinks and stopped working after moving to new version of BaseX. Is there any work around? By using children or the new descendants functions maybe? Thanks for your support. Marco. [1] https://github.com/BaseXdb/basex/blob/bbb62d1ec3e10b4fa1cc48bd4e16911fa73e4…

2 3

Backups
by Christoph Gaukel 09 Nov '19

09 Nov '19

Hi, the following handling of backups is surprising: * db:alter changes the name of the database but does not alter the names of the corresponding backups * db:drop deletes both, the database and the corresponding backups I would prefer, if there was an option in db:drop to keep the backups, which would allow to restore a deleted database from their backups. Is it possible to add options to db:drop, which allow that? For me, the behavior of db:alter looks like a bug. There is another minor handling, i would like to have changed: * db:info($db)/resourceproperties/inputdate is a UTC dateTime with a time zone offset * db:backups($db)/@date is a local dateTime without a time zone offset If I compare these two, it works correctly if the local time zone offset is the same as for the creation of the backup E.g.: if ( not(db:backups($db)) or max(db:backups($db)/@date/(. cast as xs:dateTime)) < db:info($db)/resourceproperties/inputdate/. cast as xs:dateTime ) then db:create-backup($db) else () Adding a time zone offset +hh:mm or -hh:mm to db:backups($db)/@date would solve this. Thanks, Christoph

2 2