February 2015 - BaseX-Talk - mailman.uni-konstanz.de

Lock issue while editing a .xql script over WebDAV
by Andreas Jung 20 Feb '15

20 Feb '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi there, the following bug occurs while re-editing a .xql script. - - start cadaver against the WebDAV port - - inside cadaver: edit some.xsql and save an empty file - - re-edit the same file again and this LOCK related issue pops up Cheers Andreas [qtp43256131-15] INFO com.bradmcevoy.http.HttpManager - PROPFIND :: http://localhost:8984/webdav/scripts/foo.xql - http://localhost:8984/webdav/scripts/foo.xql [qtp43256131-15] INFO com.bradmcevoy.http.HttpManager - PROPFIND :: http://localhost:8984/webdav/scripts/foo.xql - http://localhost:8984/webdav/scripts/foo.xql [qtp43256131-15] INFO com.bradmcevoy.http.HttpManager - OPTIONS :: http://localhost:8984/webdav/scripts/foo.xql - http://localhost:8984/webdav/scripts/foo.xql [qtp43256131-20] INFO com.bradmcevoy.http.HttpManager - LOCK :: http://localhost:8984/webdav/scripts/foo.xql - http://localhost:8984/webdav/scripts/foo.xql com.bradmcevoy.http.exceptions.BadRequestException [qtp43256131-20] ERROR com.bradmcevoy.http.StandardFilter - process java.lang.NullPointerException at com.bradmcevoy.http.webdav.LockHandler.processNewLock(LockHandler.java:234) at com.bradmcevoy.http.webdav.LockHandler.processExistingResource(LockHandler.java:110) at com.bradmcevoy.http.webdav.LockHandler.process(LockHandler.java:86) at com.bradmcevoy.http.StandardFilter.process(StandardFilter.java:52) at com.bradmcevoy.http.FilterChain.process(FilterChain.java:40) at com.bradmcevoy.http.HttpManager.process(HttpManager.java:228) at org.basex.http.webdav.WebDAVServlet.run(WebDAVServlet.java:43) at org.basex.http.BaseXServlet.service(BaseXServlet.java:64) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Unknown Source) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJU4NDtAAoJEADcfz7u4AZjrYYLvA5ssGdFWLFhxjcH+h+2HzUd Q/GXKDuHoz1cpzBxYahaur2VbAuVysTWbb1Cr5PkOQMtXgqoNh0IbP/FDprk8VZW mqtpIVfPqSEhYtsxzr+5lo2iuppco+K5WOo16UV4+R2tRKi9yOtq5vwhp8mNOBZ1 Ottv6Q2iBON6G1DBg6qStNGBJWl1AD+HWEgvc6p0SKL8IWUnp+iaalowK00AFb6y Ie2xXZd/5iF9ukSKEs2TeSXyXF6Hx6dZqKRDUFjDZSdvIqcR4w3Z/3AQdseM3oh4 gdPMTM7HVen5mqX0DNookE4OPQY2yurNMHWrhAiHNdAisU0PkjdqZ9wjN2zl8rMy 7NnSiZwkCRL3Fk0SP2F8YrQTMFhZTUaJbIwOHclHIUZw/Jl6ULOJTQSVu34YCjBK cmwe7h9k2TBsq/LchHYLWwjtppoUNKmlMkbaKvH+7cSdwMumX7Vg2M5GApZ7xeYD 0ZjV2xXCil+Y4eyNE/Dwk+QKFO4rxp8= =rjkk -----END PGP SIGNATURE-----

2 3

Design pattern ...
by Marco Lettere 20 Feb '15

20 Feb '15

Hi all, I start from the following condition: - 2 dbs (let's say A and B) with around 200K documents each - every document in B refers to exactly one document in A (many to one is possible) - I have to apply a filter on every document of A and depending on the boolean outcome "move" the document to anoter db A'. - According to this the correlated documents of B need to be "moved" to a database B'. - The solution should be made automatic in the mid term What is the best design pattern and tool (I use the GUI by now) for this finding the right compromise for limiting "Out of memory (huge PUL)", "too much time", "complexity introduced by chunking" and "absence of a _db:move-document_ operation for moving one document in one single operation? Every suggestion is more then warmly welcome! Thanks and best regards, Marco

2 2

schema validation
by pmenso57＠comcast.net 20 Feb '15

20 Feb '15

Hi all. Is it possible to enforce schema validation on update in BaseX? Regards, Paul Mensonides

2 1

Performance on batch processes with http requests.
by France Baril 20 Feb '15

20 Feb '15

Hi, I have an item that I would like to bring to attention. We have developed a web controller to let users manages translation processes for BaseX content. Our process is something like this: - Users select content to translation (1 to 500 small files) + languages to translation to (1 to 32 selections). - For each language, for each files: - The system transforms the content to xliff and sets the segments to translate="yes" if they have changed since the last translation for this language. Content is saved because we'll need to query it, we redirect to the next task. - File without new segment to translate, content is processed and saved to the target languages (because attributes might have changed and/or segments might have been deleted). The xliff-file is deleted, we redirect to the next task because we'll need the new information for the next query. - We query the server to offer the users stats on items to translate. This is a simplification of the process, but it shows the logics. Redirects occur after each step for each language. We have grouped operations to limit commits/redirects to a minimum. We apply them: - After each language to avoid running out of memory. - Before each operation that needs to query files based on the changes from the previous steps. We have also split groups of tasks into smaller groups where too many tasks have led us to run out of memory in the past. Our request would be for a way to force changes to commit without having to redirect. Refreshing the browser has a big impact on performance. Or maybe you have suggestions to improve batch processing when using a web interface for process management. Thank you in advance for you input! -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 4

Repo and development workflow
by Marc van Grootel 19 Feb '15

19 Feb '15

Hi, As part of the basex-gradle-starter scripts [1] I am now looking at how to deal with the repo. Until now I have always developed by putting modules under webapp. It allows to just edit a file and query away. I like to have the same workflow during development and take advantage of having modules not depend on a fixed import path. When working with the repo modules have to be installed first. Just copying them under the repo doesn't make basex aware of them. Of course I could arrange it so that I run a build task that does this via repo:install but that involves the extra step of copying files from the install location into the repo instead of just registering the files that are already in the repo location. I would like to have, for example, a 'src' directory with the repo module directory structure. Is there a way to say to basex that my repo lives in 'src' and that it auto registers what it finds there. Say a command like repo:register('dir') or maybe even an auto-register option. Once a module is already installed editing and saving is enough for basex to pick up the change. No issue there. Of course for production I probably want to compile the code to jar or xar in order to install it and share it. But during development I would like to be able to just have the code be picked up so that I can edit, save and query. I am not sure if what I'm asking is a feature or if I just have to do things differently. BTW one more thing I noticed. When I sometimes get into a situation where individual empty directories are also recognized as packages (but I cannot delete them from the GUI package manager). With repo:list() I then see something like this. <package name="org." type="Internal"/> <package name="org.basex." type="Internal"/> <package name="org.xokomola." type="Internal"/> <package name="org.xokomola.xq.Hello" type="Internal"/> [1] https://github.com/theapsgroup/basex-gradle-starter --Marc

2 8

Static files in other places than /static/* without modifying WEB-INF/web.xml ?
by Maud Ingarao 19 Feb '15

19 Feb '15

Hi everybody Thanks a lot for this wonderful BaseX 8 release and for distributing the dba module : Not only is it very practical, but it is also very inspiring to look carefully at how you guys wrote it and it gives us thousands of ideas to complete our own SynopsX module :) But it doesn't mean we always manage to imitate you... For example, exactly as you do, we want to distribute our static files inside our module, in a synopsx/files/ directory, and of course we don't want to ask people to modify the static path declaration in their WEB-INF/web.xml file to use our module. It works perfectly in your dba module : for exemple in dba/modules/tmpl.xqm, the created html element calls files/js.js or files/style.css and it simply works... But not in ours : we get this "No function matches the request" error because these files pathes are interpreted as restxq pathes (which sounds reasonable, but why not in yours ?) We are know trying more and more complicated things that don't work neither, like in https://github.com/mingarao/synopsx/blob/dev/webapp/synopsx/_restxq/files.x… line 37... So I think we should better stop there and ask you : what's the trick in dba ? All the best ! Maud and the AHN/SynopsX people -- /« Ne travaillez jamais »/ *Maud Ingarao* IHPC (UMR 5037) Institut d’histoire de la pensée classique, de l’Humanisme aux Lumières Ecole Normale Supérieure de Lyon - Site Descartes - Bureau R322 15 Parvis René Descartes - BP7000 - 69342 Lyon CEDEX 07 +33 4 37 37 65 79 - maud.ingarao(a)ens-lyon.fr /Je suis absente le jeudi | Out of office on Thursdays/ http://pensee-classique.ens-lyon.fr http://institutdesanti.ens-lyon.fr http://ahn.ens-lyon.fr http://www.mutec-shs.fr http://cahier.hypotheses.org

3 4

Issue with dtd validation
by France Baril 18 Feb '15

18 Feb '15

Hi, Since my last upgrade to BaseX 8.0 beta 9cb3a37 (also just tried BaseX 8.0.1 beta 161c4fd), I noticed that the dtd validation errors are cut when to long, so I'm getting messages like this one, that don't help my users much: File 'html-review/dh4/device-help/3338/50/global/en-us/T1401261601.html' doesn't validate against BaseX-DTDs/xhtml/xhtml1-transitional.dtd. Error [bxerr:BXVA0001]: Validation failed. "Error:archie, 5:28: Attribute "... Plus, in previous versions, I was removing some errors that we don't care about, which I can't do anymore. Is there a new option to get the full messages like I used to with $err:description? Here is my code snippet (sorry for the ugly color coding in there... never had time to upgrade them to @class): *let* *$errors* := <ul>{ *for **$file* *in **$files* *return try*{ *let **$validate* := *validate:dtd*(*$file*, *$dtd-file-path*) *return *<li *style*=*"color:green"*>Ye!</li> } *catch* *** { *let **$error-description* := *$err:description* *return if* (*contains* (*normalize-space*( *$error-description*), *normalize-space*('Attribute ""xmlns"" must be declared for element type')) or *contains* (*normalize-space*( *$error-description*), *normalize-space*('Attribute ""xmlns:xsi"" must be declared for element type'))) (: The 'xmlns' error is introduced by BaseX, and does not get exported to static content, so we ignore it :) *then *() *else if* (*contains* (*normalize-space*( *$error-description*), *normalize-space*('Attribute ""content"" is required and must be specified for element type ""meta""'))) *then *() (: No longer true in HTML5 :) *else *<li *style*=*"color:red;"*> <p>File '{*$file*/*base-uri*()}' doesn't validate against {*$dtd-file-path*}.</p> <p>{'Error [' || *$err:code* || ']: ' || *$err:description*}</p> </li> } }</*ul*> (:let $debug := file:write('debug-in-fix-validation.xml', $errors):) *return* <ul>{*$errors*//*li*[*contains*(*@style*, 'color:red;')]}</ul> -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

3 4

Bug in XQuery archive module
by alxarch 18 Feb '15

18 Feb '15

Hi, I stumbled upon a bug in the archive module on BaseX 8.0. The zip files created by the archive module are not valid. ``` let $found := file:list("/tmp/test/media/") return file:write-binary("/tmp/test.zip", archive:create( $found, $found ! file:read-binary( "/tmp/test/media/" || .) , map { "format" : "zip", "algorithm" : "deflate"})) ``` The resultting zip file is invalid. Trying to unzip in cli returns: ``` Archive: test.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of test.zip or test.zip.zip, and cannot find test.zip.ZIP, period. ``` The same code works in BaseX7.9 Cheers, Alex

2 1

Debugging RESTXQ scripts
by Andreas Jung 18 Feb '15

18 Feb '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi there, I moved the following (under existdb 2.2 working) script to BaseX https://gist.github.com/zopyx/eee05f7a421a6c0b6b80 Calling the script gives me a 404. The hello world example script from the BaseX docs however works. So general question: does BaseX log related errors somewhere? At least the standard logs don't contain anything related. Andreas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJU4jgnAAoJEADcfz7u4AZjcxALwIvmRvRz3m9SXsAGncDh0DTt sVaBk+s4CMzwTrz9VraZyjiV87y5VpL5KWio3WhPGet3jjR3u74C09swljo07h5c 8H75VAoWpf7TjaxqaqAOI2I+VZCZtkFfwwW34zFTBlYXkly6uRMEwppofcX+zc2+ ZEfIxF7g7B/mmW37zXtJVN2/pQMbfXEJisi2hDM/87QoIKyGyBN4NqSIMCucGLto fTYElkfQILUEb8CWzGDbJBnwOLB6MrHGShEZg/YVlSsvOoalsNsSagQ4yrUP2oBb G3d9g2CT7wAjs3K6fJROd9ZS5/Vw96+jsoECayW+8SqT+phXe/tmahDbBTrjEGxJ jnhkifviGhn4eRNukPf/3ZHAa5CI96X/ANtJrsyL3zMJ5KPresMUtJZ9sftit9kx bq2oqA8E+hmDzO2snFpPflDjdOUd5ABUcYMrGx7lqOVQNTOzRatBAH3dqpmQjZLb ha5S7yApyT7+3Lk841bNiTUH3zItPJQ= =iWtX -----END PGP SIGNATURE-----

3 20

Regression? Unknown parameter: 'wrap' in 8.0.x
by Gioele Barabucci 18 Feb '15

18 Feb '15

Hello, I am toying around with BaseX 8.0.1 (beta 161c4fd) and it looks like there is a regression from BaseX 7.9. In BaseX 7.9 I could issue the following POST query via the REST interface. <query xmlns="http://basex.org/rest"> <text>...</text> <parameter name='wrap' value='yes'/> </query> Now, the same query raises an exception in the server: org.basex.http.HTTPException: Unknown parameter: 'wrap'. at org.basex.http.HTTPCode.get(HTTPCode.java:55) at org.basex.http.rest.RESTPost.get(RESTPost.java:65) at org.basex.http.rest.RESTServlet.code(RESTServlet.java:43) Has something been changed in the way REST requests are processed? What is the proper way in 8.0.x to wrap results? Regards, -- Gioele Barabucci <gioele(a)svario.it>

2 3