- BaseX-Talk - mailman.uni-konstanz.de

Re: [basex-talk] REST API performance, massive disk IO
by Jens Erat 06 Sep '16

06 Sep '16

Hi Benedikt, sorry for not answering for a week. > our storage driver is the default aufs: > > Storage Driver: aufs > Root Dir: /var/lib/docker/aufs > Backing Filesystem: extfs > Dirs: 189 > Dirperm1 Supported: false > > ... based on ext4, also with default option (journaling only on metadata). That should be fine. Copy on write file systems in general require some attention when using databases (and other applications with massive small writes), but you don't use one (so you shouldn't have an issue with the file system here). > However, we have a persistent volume for the BaseX database files pointed to a regular folder on the host (ext4). To my understanding the storage driver isn't used in this case. You're right. > Is there a recommendation for a basex filesystem? The file system is not too important, as long as no copy on write is applied (chattr +t for BTRFS, but read up on that before using it). Ext4, XFS, ... should not require any special attention. I'm not aware anybody every considered different file systems and BaseX performance in a reasonable benchmark yet, but BaseX' access patterns do not differ substantially from other database systems. Regards, Jens -- Jens Erat Universität Konstanz Kommunikations-, Infomations-, Medienzentrum (KIM) Abteilung Basisdienste D-78457 Konstanz Mail: jens.erat(a)uni-konstanz.de

1 0

question on starting multiple instance of basex
by Wang, Genneva 04 Sep '16

04 Sep '16

Hi I’m trying to set up to having multiple instance of basex in my local dev environment, presumably to simulate production environment where our application is pointing to two different basex that running a different server. I’ve running basex as client server application in dev environment. Is there a way to do that? I’ve tried to start basex server with 2 different ports A and B, and also starting client with port A and B, but both seemed to be pointing to the same list of the databases that was created. Any tips would be greatly appreciated. Thanks! Regards, -Genneva

1 0

Fwd: REST API not working
by E. Wray Johnson 03 Sep '16

03 Sep '16

C:\Users\Administrator>basexserver -d BaseX 8.5.3 [Server] Server was started (port: 1984). On Thu, Sep 1, 2016 at 5:13 AM, Dirk Kirsten <dk(a)basex.org> wrote: > Hello, > > How do you start the basex server? Could you please start it in debug mode > and post us the console output? > Cheers, Dirk > > > On 08/31/2016 04:41 PM, E. Wray Johnson wrote: > > I apologize, by 12 I meant Windows Server 2012. > BaseX version is 8.5.3 > By TCP vs REST I mean for REST I was using default port 8984 instead of > the default server TCP port 1984 > For example: http://*admin*:*admin*@localhost:8984/*rest *OR > http://localhost:8984/ > *rest * > I run basexserver from DOS Command Prompt > *. * > I am using a web browser to test as I have many times before with version > 8.4 > I know the server is listening because it hangs and does not return, which > does not occur when the server is stopped. > > On Wed, Aug 31, 2016 at 9:15 AM, Dirk Kirsten <dk(a)basex.org> wrote: > >> Hello, >> >> Windows 12? I am an long-time linux-only user, but isn't the most >> current windows version 10 or am I missing the news completely? >> >> You will have to be much more specific. Please specify which BaseX >> version you use, please post how you start the basexserver and the >> console output and how you try to reach the REST endpoint. Also, I don't >> know what you mean by you are not confusing TCP and REST port. This >> statement doesn't really make sense, as REST uses TCP... >> >> Cheers >> >> Dirk >> >> >> On 08/31/2016 02:09 PM, E. Wray Johnson wrote: >> > Windows 12 server, running basexserver. Server is listening but hangs >> and never returns anything. When server is killed behavior is different so >> I can tell the server was listening on the REST port when running. I am >> not confusing the TCP port with REST port. >> >> -- >> Dirk Kirsten, BaseX GmbH, http://basexgmbh.de >> |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz >> |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: >> | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle >> `-- Phone: 0049 7531 91 68 276, Fax: 0049 7531 20 05 22 >> >> > > > -- > Dirk Kirsten, BaseX GmbH, http://basexgmbh.de > |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz > |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: > | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle > `-- Phone: 0049 7531 91 68 276, Fax: 0049 7531 20 05 22 > >

1 2

post about basex and parse trees
by Stefan Petrea 01 Sep '16

01 Sep '16

Hey, Recently worked on a project using BaseX [2]. I wrote a blog post about it here [1]. Something that might be interesting is building a Merkle tree [3] [4] in XQuery and storing/querying it in BaseX. [1] https://blog.garage-coding.com/2016/09/01/detecting-potentially-malicious-p… [2] https://github.com/wsdookadr/mdetect [3] https://github.com/wsdookadr/mdetect/blob/master/resources/mast.xql [4] https://en.wikipedia.org/wiki/Merkle_tree

1 0

Re: [basex-talk] REST API not working
by E. Wray Johnson 01 Sep '16

01 Sep '16

I apologize, by 12 I meant Windows Server 2012. BaseX version is 8.5.3 By TCP vs REST I mean for REST I was using default port 8984 instead of the default server TCP port 1984 For example: http://*admin*:*admin*@localhost:8984/*rest *OR http://localhost:8984/ *rest* I run basexserver from DOS Command Prompt *.* I am using a web browser to test as I have many times before with version 8.4 I know the server is listening because it hangs and does not return, which does not occur when the server is stopped. On Wed, Aug 31, 2016 at 9:15 AM, Dirk Kirsten <dk(a)basex.org> wrote: > Hello, > > Windows 12? I am an long-time linux-only user, but isn't the most > current windows version 10 or am I missing the news completely? > > You will have to be much more specific. Please specify which BaseX > version you use, please post how you start the basexserver and the > console output and how you try to reach the REST endpoint. Also, I don't > know what you mean by you are not confusing TCP and REST port. This > statement doesn't really make sense, as REST uses TCP... > > Cheers > > Dirk > > > On 08/31/2016 02:09 PM, E. Wray Johnson wrote: > > Windows 12 server, running basexserver. Server is listening but hangs > and never returns anything. When server is killed behavior is different so > I can tell the server was listening on the REST port when running. I am > not confusing the TCP port with REST port. > > -- > Dirk Kirsten, BaseX GmbH, http://basexgmbh.de > |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz > |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: > | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle > `-- Phone: 0049 7531 91 68 276, Fax: 0049 7531 20 05 22 > >

2 1

Problems with 8.5.3
by Marco Lettere 31 Aug '16

31 Aug '16

Hi all, after passing to 8.5.3 we have several points in our current application that stop working. The first one refers to the use of sql module. The following query worked neatly in previous version 8.5.1 beta .... let $init := sql:init("com.mysql.jdbc.Driver") let $connection := sql:connect("jdbc:mysql://localhost:3306/adb?useSSL=false", "user", "pass") return (sql:execute($connection, "select 1 + 1 from dual"), sql:close($connection)) returns: <sql:row xmlns:sql="http://basex.org/modules/sql"> <sql:column name="1 + 1">2</sql:column> </sql:row> In 8.5.3 we get the following stacktrace: Version: BaseX 8.5.3 Java: Oracle Corporation, 1.8.0_91 OS: Windows 7, x86 Stack Trace: java.lang.RuntimeException: Class org.basex.query.QueryResources can not access a member of class org.basex.query.func.sql.JDBCConnections with modifiers "" at org.basex.util.Util.notExpected(Util.java:62) at org.basex.query.QueryResources.index(QueryResources.java:121) at org.basex.query.func.sql.SqlFn.jdbc(SqlFn.java:36) at org.basex.query.func.sql.SqlConnect.item(SqlConnect.java:35) [….] We have also an issue with executing xquery:invoke on -xqm files that are stored in subdirectories of the invoking module. But I still have to repdroduce a SSCE first. Thanks for any support. Regards, Marco.

2 2

REST API not working
by E. Wray Johnson 31 Aug '16

31 Aug '16

Windows 12 server, running basexserver. Server is listening but hangs and never returns anything. When server is killed behavior is different so I can tell the server was listening on the REST port when running. I am not confusing the TCP port with REST port.

3 3

Basex server ports
by Bram Vanroy 31 Aug '16

31 Aug '16

Hi all I am trying to initialise multiple server on a single machine. This will probably be possibly by listening to different ports. The question that I have is that I am not sure which values to change in the .basex file. Candidates are: PORT SERVERPORT EVENTPORT STOPPORT Which of these cannot be the same when running a BaseX server on the same machine? Thank you in advance Bram Vanroy

3 3

REST API performance, massive disk IO
by Braunschweig, Björn 29 Aug '16

29 Aug '16

Hello everybody, Im using the basexhttp docker image: root@koala:/tmp# docker images | grep basexhttp basex/basexhttp 8.4.1 54d1e0001449 3 months ago 628.1 MB On ubuntu 14.04 [1] and the python client library: <https://github.com/lucalianas/pyBaseX> https://github.com/lucalianas/pyBaseX I have a performance problem with the setup. The database gets filled up by up to 64 parallel clients using the REST api. Currently the database has a size of 2GB, 74899811 Nodes, 305756 Documents. For every document I add, I can see a lot of disk I/O on my host with up to 80MB/s [2] by the org.basex.BaseXHTTP class. The size of the xml documents cannot be the cause, they are < 10KB. The corresponding response time of the api is in the range of 10-30 seconds [3]. Furthermore the basex docker container log shows a HTTPException and a strange Creating Database... for every request [4]. strace of the basex process shows massive open/write calls [5]. What am I doing wrong? :-) Thx, Björn Braunschweig [1] root@koala:~# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS" root@koala:~# uname -a Linux koala 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux [2] accumulated disk io after a couple of seconds root@koala:/tmp# iotop -a -P Total DISK READ : 3.88 K/s | Total DISK WRITE : 53.39 M/s Actual DISK READ: 3.88 K/s | Actual DISK WRITE: 52.84 M/s PID PRIO USER DISK READ DISK WRITE> SWAPIN IO COMMAND 62061 be/4 1984 8.00 K 2.20 G 0.00 % 0.01 % java -cp /usr/src/basex/basex-api/target~.2.jar: -Xmx2048m org.basex.BaseXHTTP d [3] my application log: 18 seconds till response 2016-08-15 14:52:00,337 | dmp043 | 46912496392256 | INFO | mdqi | add_document | 241 | saving document: urn:diasid:ast:kop01:0201502181219395 340000/mets.xml to db: mets 2016-08-15 14:52:18,164 | dmp043 | 46912496392256 | INFO | mdqi | add_document | 243 | RESPONSE (status code 201): 0 resource(s) replaced in 62.67 ms. [5] docker logs basex Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: _ REQUEST _________________________________ Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: (PUT /rest/mets/urn:diasid:ast:kop01:0201502150401114120000/mets.xml)@843076472 <mailto:org.eclipse.jetty.server.Request@32405378> org.eclipse.jetty.server.Request@32405378 Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: - Host: koala.gwdg.de:8984 Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: - Accept: */* Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: - Authorization: Basic xxx Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: - Content-Length: 7558 Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: - User-Agent: python-requests/2.10.0 Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: - Connection: keep-alive Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: - Accept-Encoding: gzip, deflate Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: _ RESPONSE ________________________________ Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: HTTP/1.1 201 Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: Content-Type: text/plain;charset=UTF-8#015 Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: #015 Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: Creating Database... Aug 12 13:44:21 koala docker/basex/ead004c4ce42[1167]: . 1.98 ms (458 MB) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: org.basex.http.HTTPException: 0 resource(s) replaced in 52.02 ms. Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.basex.http.HTTPCode.get(HTTPCode.java:57) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.basex.http.rest.RESTServlet.run(RESTServlet.java:31) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.basex.http.BaseXServlet.service(BaseXServlet.java:64) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137 ) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java :231) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java :1086) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java: 193) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java: 1020) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135 ) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:1 16) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.Server.handle(Server.java:370) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpCo nnection.java:494) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnecti on.java:982) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(Abstr actHttpConnection.java:1043) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:236) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java :82) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint. java:696) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.j ava:53) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java: 608) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:5 43) Aug 12 13:44:22 koala docker/basex/ead004c4ce42[1167]: #011at java.lang.Thread.run(Thread.java:745) [4] root@koala:/tmp# strace -t -f -p 9675 -e trace=open -o /tmp/trace.log root@koala:/tmp# root@koala:/tmp# head trace.log 38813 14:23:12 open("/srv/BaseXData/mets/pth.basex", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 129 38797 14:23:12 open("/srv/BaseXData/mets/inf.basex", O_RDONLY) = 76 38797 14:23:12 open("/srv/BaseXData/mets/tbli.basex", O_RDONLY) = 76 38797 14:23:12 open("/srv/BaseXData/mets/tbl.basex", O_RDWR|O_CREAT, 0666) = 76 38797 14:23:12 open("/srv/BaseXData/mets/txt.basex", O_RDWR|O_CREAT, 0666) = 121 38797 14:23:12 open("/srv/BaseXData/mets/atv.basex", O_RDWR|O_CREAT, 0666) = 127 38797 14:23:12 open("/srv/BaseXData/mets/upd.basex", O_WRONLY|O_CREAT|O_EXCL, 0666) = 128 38797 14:23:12 open("/srv/BaseXData/mets/pth.basex", O_RDONLY) = 128 38797 14:23:12 open("/srv/BaseXData/mets/tbli.basex", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 128 38797 14:23:12 open("/srv/BaseXData/mets/inf.basex", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 128

4 4

Docker run basexhttp and permission issues
by Michael Seiferle 28 Aug '16

28 Aug '16

Hi BaseX-Talk, in case any of you is using our docker images, in recent builds there seems to be a bug related to the ENTRYPOINT and the PARENT mvn Image we are using. If you encounter the following error message (in fact you should if you docker pulled recently ;-)): > ⇒ docker run basex/basexhttp > touch: cannot touch ‘/root/.m2/copy_reference_file.log’: Permission denied > Can not write to /root/.m2/copy_reference_file.log. Wrong volume permissions? You might want to override the ENTRYPOINT yourself: > docker run --entrypoint=basexhttp basex/basexhttp I already proposed a fix but I’d like someone with docker-expertise have a look, maybe s/o might want to add her or his 5cts ;-) https://github.com/BaseXdb/basex/pull/1346 <https://github.com/BaseXdb/basex/pull/1346> Best Michael -- Michael Seiferle, BaseX GmbH, http://www.basexgmbh.de |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Fon: +49 7531 916 82 77

2 1