- BaseX-Talk - mailman.uni-konstanz.de

Re: [basex-talk] HTTPServer + gzip compression
by Stefan Koch 20 Nov '19

20 Nov '19

3 5

executing windows command
by Dieter Zanzinger 20 Nov '19

20 Nov '19

Hello all, I tried to execute a windows command with proc:execute(). See my module: ——— module namespace _ ='process'; declare %rest:path('treez/proc') function _:process() { prof:dump( proc:execute('C:\Program Files\Microsoft Office 15\root\office15\winword.exe','/q') ) }; declare %rest:path('treez/proc_dir') function _:process2() { proc:execute('dir','\’) ———— While function process() works as expected and opens winword, the function process2() (example from the docs) leads to following error: <result> <error> Cannot run program "dir": CreateProcess error=2, Das System kann die angegebene Datei nicht finden </error> </result> Any hints? As process() works and opens winword, I prosume that the error does not come from my installation (Win10, Tomcat 9.0, BaseX 9 - war version). Starting Tomcat as admin does not change anything. Best, Dieter Zanzinger Finanzgericht München

2 1

Exception shown on fn:doc with XML document
by Stefan Pöschel 18 Nov '19

18 Nov '19

Hello, the XPath function "fn:doc" in BaseX can also be used to open a database (instead of an XML document). However when I open an XML document and have debug messages enabled, an exception is shown that seems to be related to an (unsuccessful) attempt to open a database (meanwhile the fn:doc command itself succeeds): ----- org.basex.core.BaseXException: Database 'test.xml' was not found. at org.basex.core.cmd.Open.open(Open.java:92) at org.basex.query.QueryResources.data(QueryResources.java:405) at org.basex.query.QueryResources.doc(QueryResources.java:207) at org.basex.query.func.fn.Docs.doc(Docs.java:61) at org.basex.query.func.fn.FnDoc.item(FnDoc.java:17) at org.basex.query.func.StandardFunc.optimize(StandardFunc.java:82) at org.basex.query.expr.Arr.compile(Arr.java:43) at org.basex.query.func.StaticFunc.comp(StaticFunc.java:69) at org.basex.query.QueryCompiler.compile(QueryCompiler.java:114) at org.basex.query.QueryCompiler.compile(QueryCompiler.java:105) at org.basex.query.QueryContext.compile(QueryContext.java:312) at org.basex.query.QueryContext.iter(QueryContext.java:331) at org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:73) at org.basex.http.web.WebResponse.create(WebResponse.java:63) at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:53) at org.basex.http.BaseXServlet.service(BaseXServlet.java:65) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) at java.lang.Thread.run(Thread.java:748) ----- Please find attached a minimal test XQM and a corresponding XML document. Both files have to be stored in the "webapp" subfolder of the folder from which BaseX is called. With the option "WITHDB = false" in ".basex", the exception doesn't occur (which is not surprising). However I think that the exception shouldn't show up in the debug output here at all, as the fn:doc command actually succeeds (and as it "pollutes" the output). I use BaseX 9.2.4 on Linux; the used cmdline is "basexhttp -d". Thanks and best regards, Stefan -- Stefan Pöschel IRT - Institut für Rundfunktechnik GmbH - https://www.irt.de ------------------------------------------------------------ Research and Development Institute of ARD, ZDF, DRadio, ORF and SRG/SSR Floriansmühlstr. 60, 80939 München, Germany Registration court Munich Commercial, Register No. B 5191 Managing Director: Michael Hagemeyer

2 4

Is there an API that provides XQuery compilation results?
by Peter Villadsen 15 Nov '19

15 Nov '19

When I use the GUI application I can see some error description (even if it a little terse) when my queries are incorrect. For instance I might get: Stopped at tableFields.xq, 27/50: [XQST0118] Different start and end tag: <Artifact>...</Artiact>. If I submit a query like <Artifact></Artiact> However, I cannot seem to find an API that brings me that information? I tried: xquery:parse("<B></C>", map {'pass':true()}) but that did not get the result I expected. Also, while we're at at it, it would be nice to also get the optimized query. Best Regards Peter Villadsen Principal Architect Microsoft Business Applications Group

3 2

Re: [basex-talk] REPLACE vs STORE
by Christian Grün 14 Nov '19

14 Nov '19

> Thank you for your answer. Do you plan something like that in future? We face the problem we need to upload tens thousands of files in 4 hours per week. We are able to do it with replace with folder, but current store per file is too slow. :-( Have you already tried the XQuery way? If so, and if it’s too slow, feel free to share your code with us. > Odesláno z mého chytrého telefonu Samsung Galaxy. > > -------- Původní zpráva -------- > Od: Christian Grün <christian.gruen(a)gmail.com> > Datum: 14.11.19 18:30 (GMT+01:00) > Komu: Radim Havlicek <radim-havlicek(a)post.cz> > Cc: BaseX <basex-talk(a)mailman.uni-konstanz.de> > Předmět: Re: [basex-talk] REPLACE vs STORE > > Hi Radim, > > The STORE command is restricted to single files [1]. XQuery and > db:store (e.g. combined with file:list) gives you more flexibility > here. > > Hope this helps, > Christian > > [1] http://docs.basex.org/wiki/Commands#STORE > > > > > On Thu, Nov 14, 2019 at 10:53 AM <radim-havlicek(a)post.cz> wrote: > > > > Hello, > > I would like to store new reports into existing database folder (with other existing files already) after opening. > > > > Help: > > REPLACE > > Syntax REPLACE [path] [input] > > XML Syntax <replace path='...'>[input]</replace> > > > > STORE > > Syntax STORE (TO [path]) [input] > > XML Syntax <store (path='...')>[input]</store> > > > > But actually I encountered this problem: > > > open mydb > > Database 'mydb' was opened in 3.0 ms. > > > replace /Report/ c:/Share/REPORTS/ > > 903 resource(s) replaced in 11077.73 ms. > > > store to /Report/ c:/Share/REPORTS/ > > Resource "C:/Share/REPORTS/" not found. > > > > What am I doing wrong? Is it possible to use store to instead of replace to just add files to existing ones? > > > > Thanks in advance for your answer, > > > > Radim Havlicek

1 0

REPLACE vs STORE
by radim-havlicek＠post.cz 14 Nov '19

14 Nov '19

Hello, I would like to store new reports into existing database folder (with other existing files already) after opening. Help: REPLACE Syntax REPLACE [path] [input] XML Syntax <replace path='...'>[input]</replace> STORE Syntax STORE (TO [path]) [input] XML Syntax <store (path='...')>[input]</store> But actually I encountered this problem: > open mydb Database 'mydb' was opened in 3.0 ms. > replace /Report/ c:/Share/REPORTS/ 903 resource(s) replaced in 11077.73 ms. > store to /Report/ c:/Share/REPORTS/ Resource "C:/Share/REPORTS/" not found. What am I doing wrong? Is it possible to use store to instead of replace to just add files to existing ones? Thanks in advance for your answer, Radim Havlicek

2 1

Feature request: leveraging db:open-id and db:node-id
by France Baril 14 Nov '19

14 Nov '19

Hi, any chance we could make the db indexed ids match the xml doc ids? - db:open-id() could match the id at the root of then documents (when one is provided) - db:node-id() could match other nodes match internal ids (when they are provided). It would make indexing and querying a lot more efficient for any type of navigation/linking purposes. Options: - I'd suggest that this could be an option that can be turned on/off for each doc level ids and node level ids - I'd suggest that node id has the option to be unique within the db or within a document when the value for the entire db would then have to become something like doc-id#node-id to stay unique within the db Then I can easily imagine also expending to db:save-id to reduce the workload of having to pass through base-uri() once the doc exists in the db under its path let $db-name := 'en-us' let $doc := db:open-id($db-name, $doc-id) let $new-doc := copy, modify, return copy db:replace-id('en-us', $doc-id, $new-doc) instead of let $db-name := 'en-us' let $doc := db:open($db-name)/*[@id=$doc-id] let $new-doc := copy, modify, return copy let $path := substring-after($doc/base-uri(), $db-name) return db:replace($db-name, $path, $new-doc) Is that something that can be considered?

2 4

I am looking for the fastest way to sort 2.4 Mio tags by two attribute ascending and descending
by Omar Siam 14 Nov '19

14 Nov '19

Hi, I have a custom index that looks like this (one db, different files): <_:dryed xmlns:_="https://www.oeaw.ac.at/acdh/tools/vle/util" db_name="z881_qdb-TEI-02n" order="none"> <_:d pre="15627" db_name="z881_qdb-TEI-02n" xml:id="z881_qdbn-d16e2" vutlsk="tsįttr Ziter [Subst]" vutlsk-archiv="HK 881, z8810118.sch#1"/> <_:d pre="15673" db_name="z881_qdb-TEI-02n" xml:id="z881_qdbn-d16e21" vutlsk="tsįttr Ziter [Subst]" vutlsk-archiv="HK 881, z8810118.sch#1"/> ... </_:dryed> <_:dryed xmlns:_="https://www.oeaw.ac.at/acdh/tools/vle/util" db_name="f227_qdb-TEI-02n" order="none"> <_:d pre="467" db_name="f227_qdb-TEI-02n" xml:id="f237_qdb-d1e29398" vutlsk="(aus)faren [Verb]" vutlsk-archiv="HK 327, f227#944.1 = fare0126.eck#1.1"/> <_:d pre="591" db_name="f227_qdb-TEI-02n" xml:id="f237_qdb-d1e29438" vutlsk="(aus)faren [Verb]" vutlsk-archiv="HK 327, f227#945.1 = fare0126.eck#2.1"/> ... </_:dryed> There are about 2.4 Mio _:d tags in this db. I need to sort them by the @vutlsk* attributes alphabetically in ascending and descending order. With the code I have now: declare namespace _ = "https://www.oeaw.ac.at/acdh/tools/vle/util"; let $sorted-ascending := subsequence(for $d in collection('_qdb-TEI-02__cache')//*[@order="none"]/_:d order by $d/@vutlsk ascending return $d/(@ID, @xml:id)/data(), 1, 10000) let $sorted-descending := subsequence(for $d in collection('_qdb-TEI-02__cache')//*[@order="none"]/_:d order by $d/@vutlsk descending return $d/(@ID, @xml:id)/data(), 1, 10000) let $sorted-ascending-archiv := subsequence(for $d in collection('_qdb-TEI-02__cache')//*[@order="none"]/_:d order by $d/@vutlsk-archiv ascending return $d/(@ID, @xml:id)/data(), 1, 10000) let $sorted-descending-archiv := subsequence(for $d in collection('_qdb-TEI-02__cache')//*[@order="none"]/_:d order by $d/@vutlsk-archiv descending return $d/(@ID, @xml:id)/data(), 1, 10000) return (db:replace("_qdb-TEI-02__cache", 'ascending_cache.xml', <_:dryed order="ascending" ids="{string-join($sorted-ascending, ' ')}"/>), db:replace("_qdb-TEI-02__cache", 'descending_cache.xml', <_:dryed order="descending" ids="{string-join($sorted-descending, ' ')}"/>), db:replace("_qdb-TEI-02__cache", 'ascending-archiv_cache.xml', <_:dryed order="ascending" label="archiv" ids="{string-join($sorted-ascending-archiv, ' ')}"/>), db:replace("_qdb-TEI-02__cache", 'descending-archiv_cache.xml', <_:dryed order="descending" label="archiv" ids="{string-join($sorted-descending-archiv, ' ')}"/>)) This takes 30 s to about a minute depending on the subsequence I choose. I did experiments with doing multithreading and not. Multiple jobs or fork-join make it worse. Worst case I need to do it every time I save a change to the original DBs for which I maintain that index. Any ideas how to speed this up? Best regards Omar Siam

2 5

Re: [basex-talk] http:send-request - problem with response
by Christian Grün 13 Nov '19

13 Nov '19

Hi Bogdan (cc to the list), Thanks for digging deeper. I noticed that the standard Java function that we used returned only one value per header field, and dropped the others [1]. I managed to fix this in the latest stable snapshot; could you give it a try [2]? BaseX 9.3 will be released end of November. Cheers, Christian [1] http://files.basex.org/releases/latest/ [2] https://github.com/BaseXdb/basex/issues/1751 On Wed, Nov 13, 2019 at 4:26 PM Bogdan Bogucki <bbogucki(a)hussar.pl> wrote: > > Hi Christian, > Lack of many redirection is not a big problem. I can handle it manually. > Problem is not complete response header form first request. > > Browser make 3 redirection I am talking about first. > > Please take a look. > > > Response header from browser (first request) - contains three fields Set > Cookies: > > set-cookie __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly > > set-cookie _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/ > > set-cookie _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly > > but response header form http:send-request contains only last one: > > <http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; > expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/> > > I need information from previous set-cookie fields to make manual > redirection passing information about sessions to next requests. > > Regards > > Bogdan Bogucki > > W dniu 29.10.2019 o 03:06, Christian Grün pisze: > > Hi Bogdan, > > > > The current http:send-request implementation is based on the default > > Java HttpURLConnection, which does not resolve redirects that use > > different protocols [1]. This is the reason why your request will not > > be fully processed (as it e.g. happens when you use the browser). > > > > It seems that your initial request to the https protocol returns a 302 > > redirect to a (now unsafe) http URL, which returns another redirect to > > https. I don’t know who maintains the discussed web site, but it could > > be worth contacting the admins and asking them if they could update > > and simplify their redirect policy. > > > > Hope this helps, > > Christian > > > > [1] https://stackoverflow.com/a/1884427/1018790 > > > > > > > > > > On Fri, Oct 25, 2019 at 1:52 PM Bogdan Bogucki <bbogucki(a)hussar.pl> wrote: > >> > >> Hello, > >> > >> I encounter problem with http:send-request function. I need handle multiple forward requests (3) with cookies. > >> > >> First request returns cookies information which are required with next requests. Output from http:send-request dosen't contains all fields which are returned by server. > >> > >> Response from browser is flowing: > >> cache-control > >> private > >> cf-cache-status > >> DYNAMIC > >> cf-ray > >> 52b3cedc5b9bcc9f-WAW > >> content-type > >> text/html; charset=utf-8 > >> date > >> Fri, 25 Oct 2019 11:21:37 GMT > >> expect-ct > >> max-age=604800, report-uri="ht….com/cdn-cgi/beacon/expect-ct" > >> location > >> http://www.pracuj.pl/praca/jav…eloper-warszawa,oferta,7171988 > >> server > >> cloudflare > >> set-cookie > >> __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly > >> set-cookie > >> _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/ > >> set-cookie > >> _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly > >> x-aspnet-version > >> 4.0.30319 > >> x-aspnetmvc-version > >> 5.2 > >> X-Firefox-Spdy > >> h2 > >> x-powered-by > >> ASP.NET > >> x-ua-compatible > >> IE=edge > >> > >> > >> Request: > >> > >> http:send-request( > >> <http:request method='get' follow-redirect='false'> > >> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"/> > >> <http:header name="Accept" value="text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"/> > >> <http:header name="Accept-Languag" value="pl,en-US;q=0.7,en;q=0.3"/> > >> <http:header name="DNT" value="1"/> > >> <http:header name="Connection" value="keep-alive"/> > >> <http:header name="Upgrade-Insecure-Requests" value="1"/> > >> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> > >> </http:request>, 'https://www.pracuj.pl/praca/java-developer-warszawa,oferta,7171988') > >> > >> Response form http:send-request is: > >> > >> <http:response xmlns:http="http://expath.org/ns/http-client" status="302" message="Found"> > >> <http:header name="Transfer-Encoding" value="chunked"/> > >> <http:header name="CF-RAY" value="52b3d0de097fb839-WAW"/> > >> <http:header name="Server" value="cloudflare"/> > >> <http:header name="Connection" value="keep-alive"/> > >> <http:header name="Date" value="Fri, 25 Oct 2019 11:23:00 GMT"/> > >> <http:header name="X-AspNetMvc-Version" value="5.2"/> > >> <http:header name="X-UA-Compatible" value="IE=edge"/> > >> <http:header name="CF-Cache-Status" value="DYNAMIC"/> > >> <http:header name="Cache-Control" value="private"/> > >> <http:header name="X-AspNet-Version" value="4.0.30319"/> > >> <http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/> > >> <http:header name="Expect-CT" value="max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct…"/> > >> <http:header name="X-Powered-By" value="ASP.NET"/> > >> <http:header name="Content-Type" value="text/html; charset=utf-8"/> > >> <http:body media-type="text/html"/> > >> </http:response> > >> > >> Two set-cookie fields is missing. > >> > >> How should I invoke http:send-request to receive proper result ? > >> > >> Regards > >> > >> Bogdan

2 1

proc:system
by Giuseppe G. A. Celano 13 Nov '19

13 Nov '19

Hi, When I try to run proc:system("/usr/local/bin/tesseract") I get the error [proc:code0001] Similarly: proc:system("tesseract") returns [proc:error] Cannot run program "tesseract": error=2, No such file or directory Similarly: proc:system("tesseract", (), map {"dir" : "/usr/local/bin/"}) returns [proc:error] Cannot run program "tesseract" (in directory "/usr/local/bin"): error=2, No such file or directory The command "tesseract" works at the command line. I suspect there may be a problem with permissions: is there a way to overcome this error? Thanks. Best, Giuseppe

2 5