August 2016 - BaseX-Talk - mailman.uni-konstanz.de

retrieve a sequence of all values within an attribute index
by Alex Muir 10 Oct '16

10 Oct '16

Greetings, I need to generate a list of all the distinct values within an index. Is there a way to retrieve a sequence of all values within an attribute index? Regards Alex tech.jahtoe.com bafila.jahtoe.com

3 9

8.5.3 and xquery:invoke ...
by Marco Lettere 11 Sep '16

11 Sep '16

Hello again, at the end I managed to reproduce the issue we are currently facing with xquery:invoke in 8.5.3 (and also in latest snapshot). Please consider the three queries [1,2,3]. The store [2] into a subdirectory src of the directory containing [1] and store [3] into a directory src2 of src. You should get the error message [4]. If instead you change the parameter of xquery:invoke to "src/src2/q2.xqm" it works. Seems like the reference for static-base-uri in [2] remains the one of [1] even if it is reported by static-base-uri() to be correctly "/home/user/src/q1.xqm"(I've double-checked this). Sorry it's not clear to me whether this is meant to be the correct behaviour ... Thanks for any help, Marco. [1] q.xqm import module namespace q1="urn:q1" at "src/q1.xqm"; q1:f() [2] q1.xqm module namespace q1 = "urn:q1"; declare function q1:f(){ xquery:invoke("src2/q2.xqm") }; [3] q2.xqm 1 = 1 [4] Stopped at /home/user/src/q1.xqm, 4/16: [FODC0002] Resource 'src2/q2.xqm' does not exist.

3 4

fork-join and recursive function
by Carl Leitner 11 Sep '16

11 Sep '16

Hi All, I have hierarchical information encoded an XML document that looks something like: <organization entityID=“1”> <organization entityID=“2"> <parent entityID=“1”> </organization> <organization entityID=“3"> <parent entityID=“2”> </organization> <organization entityID=“4"> <parent entityID=“1”> </organization> There are around 80,000 entries like this and I need to regular do extractions of sub-hiearchies (see the commented out version of the below function). The commented-out version runs in about a minute using a modest amount of memory. Hoping to take advantage of this <https://github.com/BaseXdb/basex/commit/ac86bbbc3cd1f71461ce94d803cab46f21e…>, I modified the function (the uncommented version) which tries to use xquery:fork-join in 8.5.2beta (the July 12th snapshot). The parallelized one chews up the 3g of available memory, unceremoniously throws exceptions (Exception in thread "qtp198198276-19” ), with the occasional: java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1857) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2073) Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp198198276-19" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "qtp198198276-14" java.lang.OutOfMemoryError: GC overhead limit exceeded and runs for tens of minutes (perhaps more - I always kill the process). Any ideas on what I can do to improve the situation? Thanks in advance. Cheers, -carl declare function csd_bl:get_child_orgs($orgs,$org) { let $org_id := $org/@entityID return if (functx:all-whitespace($org_id)) then () else let $c_orgs := $orgs[./parent[@entityID = $org_id]] let $t0 := trace($org_id, "creating func for ") let $t1 := trace(count($c_orgs), " func checks children: ") let $c_org_funcs:= for $c_org in $c_orgs return function() { ( trace($org_id, "executing child func for ") , $c_org, csd_bl:get_child_orgs($orgs,$c_org))} return xquery:fork-join($c_org_funcs) (: let $c_orgs := if (functx:all-whitespace($org_id)) then () else $orgs[./parent[@entityID = $org_id]] return for $c_org in $c_orgs let $t0 := trace($org_id, "processing children for ") return ($c_org,csd_bl:get_child_orgs($orgs,$c_org)) :) };

2 18

Serialize a string to unencode XML entities?
by Hondros, Constantine (ELS-AMS) 11 Sep '16

11 Sep '16

Ok, this must be possible. Can someone point me towards the correct serialization $params such that let $doi := <doi>10.1002/1521-4141(200207)32:7<2021::AID-IMMU2021>3.0.CO;2-J</doi> return serialize($doi, $params) will return (to a calling Java application) the UTF-8 string 10.1002/1521-4141(200207)32:7<2021::AID-IMMU2021>3.0.CO;2-J where the entities < and > have been replaced with the UTF-8 byte representation for U+003C and U+003E respectively? Thanks in advance, C. ________________________________ Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.

2 1

Using the same database in many BaseX instances
by cmarchand＠oxiane.com 11 Sep '16

11 Sep '16

Hello, as discussed here few monthes ago, I do this : Install many instances (8) of BaseX on the same computer Start one instance, here called "main-instance" Create a database "library" , and load a filesystem directory into "library" Stop main-instance For each "other-instance", create a sym-link to main-instance/data/library Start "other-instances" On each instance, create 2 new databases "input" and "output", load a file in "input", process it and store result in "output". The process queries a lot "library", but never modifies "library". Then export "output" and drop databases "input" and "output". I have about 100 files to process; and I distribute them on the various instances I had a first test environment, with only 4 isntances, and everything was working perfectly. Now, I'm working on target environment, a bigger computer with 8 instances. Process fails, at least one time per instance, so between 8 and 12 times on 100 process. The exception is java.lang.NullPointerException at org.basex.data.DiskData.write(DiskData.java:136) at org.basex.data.DiskData.close(DiskData.java:151) at org.basex.core.Datas.unpin(Datas.java:54) at org.basex.core.cmd.Close.close(Close.java:45) at org.basex.query.QueryResources.close(QueryResources.java:110) at org.basex.query.QueryContext.close(QueryContext.java:596) at org.basex.query.QueryProcessor.close(QueryProcessor.java:251) at org.basex.core.cmd.AQuery.query(AQuery.java:124) at org.basex.core.cmd.XQuery.run(XQuery.java:22) at org.basex.core.Command.run(Command.java:398) at org.basex.core.Command.execute(Command.java:100) at org.basex.server.ClientListener.run(ClientListener.java:136) If I change the sym-links by a true copy (cp -r main/data/library other/data/), it now works correctly, but the copy takes a long time (database is about 30gb), and it consumes disk space. The computer I use has many available cores, and many RAM to use, so I'd love to add other instance, many 16 more to reach 24 isntances, but if database grow, I'm going to have disk-space troubles... 1 - is it legal to share a database (and files) between many instances as far as database is access read-only ? 2 - is there a way to configure a database to be read-only ? 3 - why symbolic links were perfectly usable with 4 instances, and do not with 8 instances ? (Hum I have to try symlinks with only 4 instances on the new computer...) 4 - are there other specific things to check to do what I need ? Best regards, Christophe

3 2

Re: [basex-talk] REST API not working
by E. Wray Johnson 01 Sep '16

01 Sep '16

I apologize, by 12 I meant Windows Server 2012. BaseX version is 8.5.3 By TCP vs REST I mean for REST I was using default port 8984 instead of the default server TCP port 1984 For example: http://*admin*:*admin*@localhost:8984/*rest *OR http://localhost:8984/ *rest* I run basexserver from DOS Command Prompt *.* I am using a web browser to test as I have many times before with version 8.4 I know the server is listening because it hangs and does not return, which does not occur when the server is stopped. On Wed, Aug 31, 2016 at 9:15 AM, Dirk Kirsten <dk(a)basex.org> wrote: > Hello, > > Windows 12? I am an long-time linux-only user, but isn't the most > current windows version 10 or am I missing the news completely? > > You will have to be much more specific. Please specify which BaseX > version you use, please post how you start the basexserver and the > console output and how you try to reach the REST endpoint. Also, I don't > know what you mean by you are not confusing TCP and REST port. This > statement doesn't really make sense, as REST uses TCP... > > Cheers > > Dirk > > > On 08/31/2016 02:09 PM, E. Wray Johnson wrote: > > Windows 12 server, running basexserver. Server is listening but hangs > and never returns anything. When server is killed behavior is different so > I can tell the server was listening on the REST port when running. I am > not confusing the TCP port with REST port. > > -- > Dirk Kirsten, BaseX GmbH, http://basexgmbh.de > |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz > |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: > | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle > `-- Phone: 0049 7531 91 68 276, Fax: 0049 7531 20 05 22 > >

2 1

Problems with 8.5.3
by Marco Lettere 31 Aug '16

31 Aug '16

Hi all, after passing to 8.5.3 we have several points in our current application that stop working. The first one refers to the use of sql module. The following query worked neatly in previous version 8.5.1 beta .... let $init := sql:init("com.mysql.jdbc.Driver") let $connection := sql:connect("jdbc:mysql://localhost:3306/adb?useSSL=false", "user", "pass") return (sql:execute($connection, "select 1 + 1 from dual"), sql:close($connection)) returns: <sql:row xmlns:sql="http://basex.org/modules/sql"> <sql:column name="1 + 1">2</sql:column> </sql:row> In 8.5.3 we get the following stacktrace: Version: BaseX 8.5.3 Java: Oracle Corporation, 1.8.0_91 OS: Windows 7, x86 Stack Trace: java.lang.RuntimeException: Class org.basex.query.QueryResources can not access a member of class org.basex.query.func.sql.JDBCConnections with modifiers "" at org.basex.util.Util.notExpected(Util.java:62) at org.basex.query.QueryResources.index(QueryResources.java:121) at org.basex.query.func.sql.SqlFn.jdbc(SqlFn.java:36) at org.basex.query.func.sql.SqlConnect.item(SqlConnect.java:35) [….] We have also an issue with executing xquery:invoke on -xqm files that are stored in subdirectories of the invoking module. But I still have to repdroduce a SSCE first. Thanks for any support. Regards, Marco.

2 2

REST API not working
by E. Wray Johnson 31 Aug '16

31 Aug '16

Windows 12 server, running basexserver. Server is listening but hangs and never returns anything. When server is killed behavior is different so I can tell the server was listening on the REST port when running. I am not confusing the TCP port with REST port.

3 3

Basex server ports
by Bram Vanroy 31 Aug '16

31 Aug '16

Hi all I am trying to initialise multiple server on a single machine. This will probably be possibly by listening to different ports. The question that I have is that I am not sure which values to change in the .basex file. Candidates are: PORT SERVERPORT EVENTPORT STOPPORT Which of these cannot be the same when running a BaseX server on the same machine? Thank you in advance Bram Vanroy

3 3

REST API performance, massive disk IO
by Braunschweig, Björn 29 Aug '16

29 Aug '16

Hello everybody, Im using the basexhttp docker image: root@koala:/tmp# docker images | grep basexhttp basex/basexhttp 8.4.1 54d1e0001449 3 months ago 628.1 MB On ubuntu 14.04 [1] and the python client library: <https://github.com/lucalianas/pyBaseX> https://github.com/lucalianas/pyBaseX I have a performance problem with the setup. The database gets filled up by up to 64 parallel clients using the REST api. Currently the database has a size of 2GB, 74899811 Nodes, 305756 Documents. For every document I add, I can see a lot of disk I/O on my host with up to 80MB/s [2] by the org.basex.BaseXHTTP class. The size of the xml documents cannot be the cause, they are < 10KB. The corresponding response time of the api is in the range of 10-30 seconds [3]. Furthermore the basex docker container log shows a HTTPException and a strange Creating Database... for every request [4]. strace of the basex process shows massive open/write calls [5]. What am I doing wrong? :-) Thx, Björn Braunschweig [1] root@koala:~# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS" root@koala:~# uname -a Linux koala 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux [2] accumulated disk io after a couple of seconds root@koala:/tmp# iotop -a -P Total DISK READ : 3.88 K/s | Total DISK WRITE : 53.39 M/s Actual DISK READ: 3.88 K/s | Actual DISK WRITE: 52.84 M/s PID PRIO USER DISK READ DISK WRITE> SWAPIN IO COMMAND 62061 be/4 1984 8.00 K 2.20 G 0.00 % 0.01 % java -cp /usr/src/basex/basex-api/target~.2.jar: -Xmx2048m org.basex.BaseXHTTP d [3] my application log: 18 seconds till response 2016-08-15 14:52:00,337 | dmp043 | 46912496392256 | INFO | mdqi | add_document | 241 | saving document: urn:diasid:ast:kop01:0201502181219395 340000/mets.xml to db: mets 2016-08-15 14:52:18,164 | dmp043 | 46912496392256 | INFO | mdqi | add_document | 243 | RESPONSE (status code 201): 0 resource(s) replaced in 62.67 ms. [5] docker logs basex Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: _ REQUEST _________________________________ Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: (PUT /rest/mets/urn:diasid:ast:kop01:0201502150401114120000/mets.xml)@843076472 <mailto:org.eclipse.jetty.server.Request@32405378> org.eclipse.jetty.server.Request@32405378 Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: - Host: koala.gwdg.de:8984 Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: - Accept: */* Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: - Authorization: Basic xxx Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: - Content-Length: 7558 Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: - User-Agent: python-requests/2.10.0 Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: - Connection: keep-alive Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: - Accept-Encoding: gzip, deflate Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: _ RESPONSE ________________________________ Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: HTTP/1.1 201 Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: Content-Type: text/plain;charset=UTF-8#015 Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: #015 Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: Creating Database... Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: . 1.98 ms (458 MB) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: org.basex.http.HTTPException: 0 resource(s) replaced in 52.02 ms. Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.basex.http.HTTPCode.get(HTTPCode.java:57) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.basex.http.rest.RESTServlet.run(RESTServlet.java:31) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.basex.http.BaseXServlet.service(BaseXServlet.java:64) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137 ) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java :231) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java :1086) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java: 193) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java: 1020) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135 ) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:1 16) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.Server.handle(Server.java:370) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpCo nnection.java:494) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnecti on.java:982) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(Abstr actHttpConnection.java:1043) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:236) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java :82) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint. java:696) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.j ava:53) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java: 608) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:5 43) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at java.lang.Thread.run(Thread.java:745) [4] root@koala:/tmp# strace -t -f -p 9675 -e trace=open -o /tmp/trace.log root@koala:/tmp# root@koala:/tmp# head trace.log 38813 14:23:12 open("/srv/BaseXData/mets/pth.basex", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 129 38797 14:23:12 open("/srv/BaseXData/mets/inf.basex", O_RDONLY) = 76 38797 14:23:12 open("/srv/BaseXData/mets/tbli.basex", O_RDONLY) = 76 38797 14:23:12 open("/srv/BaseXData/mets/tbl.basex", O_RDWR|O_CREAT, 0666) = 76 38797 14:23:12 open("/srv/BaseXData/mets/txt.basex", O_RDWR|O_CREAT, 0666) = 121 38797 14:23:12 open("/srv/BaseXData/mets/atv.basex", O_RDWR|O_CREAT, 0666) = 127 38797 14:23:12 open("/srv/BaseXData/mets/upd.basex", O_WRONLY|O_CREAT|O_EXCL, 0666) = 128 38797 14:23:12 open("/srv/BaseXData/mets/pth.basex", O_RDONLY) = 128 38797 14:23:12 open("/srv/BaseXData/mets/tbli.basex", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 128 38797 14:23:12 open("/srv/BaseXData/mets/inf.basex", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 128

4 4