Hi,
I seem to have a concurrency problem and I have a hard time fixing it. Our BaseX db is a very modest database, for testing we load a 138Kb file. We use JMeter to do performance testing on our Web Application deployment of BaseX 7.3. Randomly GET and PUT commands are sent to BaseX. When we use only one thread (simulated user) everything is fine, when we increase the number of threads we quickly end up with a corrupted database - two threads is generally slower to crash than three threads, with three threads it crashes fast.
Here are some stacktraces I get:
[INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 27 org.basex.util.list.ByteList.add(ByteList.java:39) org.basex.util.Compress.push(Compress.java:72) org.basex.util.Compress.pack(Compress.java:52) org.basex.data.DiskData.index(DiskData.java:387) org.basex.data.Data.attr(Data.java:1051) org.basex.data.Data.insert(Data.java:835) org.basex.query.up.primitives.InsertInto.apply(InsertInto.java:32) org.basex.query.up.NodeUpdates.makePrimitivesEffective(NodeUpdates.java:216) org.basex.query.up.DatabaseUpdates.apply(DatabaseUpdates.java:152) org.basex.query.up.ContextModifier.apply(ContextModifier.java:66) org.basex.query.up.Updates.apply(Updates.java:183) org.basex.query.QueryContext.update(QueryContext.java:276) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:76) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348) org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894) org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:851) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620) org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679)
I am also getting:
[INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 22 org.basex.util.Compress.pull(Compress.java:132) org.basex.util.Compress.unpack(Compress.java:109) org.basex.data.DiskData.txt(DiskData.java:262) org.basex.data.DiskData.text(DiskData.java:229) org.basex.data.Data.atom(Data.java:208) org.basex.query.value.node.DBNode.string(DBNode.java:115) org.basex.query.value.node.ANode.string(ANode.java:50) org.basex.query.value.item.Str.eq(Str.java:80) org.basex.query.value.node.ANode.eq(ANode.java:61) org.basex.query.expr.CmpV$OpV$5.eval(CmpV.java:84) org.basex.query.expr.CmpG.eval(CmpG.java:258) org.basex.query.expr.CmpG.item(CmpG.java:213) org.basex.query.expr.CmpG.item(CmpG.java:25) org.basex.query.expr.ParseExpr.ebv(ParseExpr.java:81) org.basex.query.expr.ParseExpr.test(ParseExpr.java:94) org.basex.query.expr.Preds.preds(Preds.java:129) org.basex.query.path.IterStep$1.next(IterStep.java:48) org.basex.query.path.IterStep$1.next(IterStep.java:32) org.basex.query.path.IterPath$1.next(IterPath.java:67) org.basex.query.path.IterPath$1.next(IterPath.java:39) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:329) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:304) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.Constr.add(Constr.java:65) org.basex.query.expr.CElem.item(CElem.java:84) org.basex.query.expr.CElem.item(CElem.java:21) org.basex.query.expr.ParseExpr.value(ParseExpr.java:69) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.func.TailFuncCall.iter(TailFuncCall.java:55) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.flwor.FLWR$1.next(FLWR.java:59) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:75) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348) org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884) org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620) org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679)
and also this:
14:52:02,452 - WARN - qtp384018721-20 - oup.nl.mms.plib.server.rest.AbstractEntityResource # HTTP response is not OK: code 500, ETag: empty, content: org.apache.http.conn.EofSensorInputStream@4acfed14mailto:org.apache.http.conn.EofSensorInputStream@4acfed14 Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException 2012-08-27 14:52:02.959:WARN:oejs.ServletHandler:/basex/restxq/plib/platforms/ java.lang.ArrayIndexOutOfBoundsException: 0 at org.basex.util.Util.message(Util.java:166) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:327) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:345) at org.basex.util.Util.inf(Util.java:218) at org.basex.util.Util.infohttp://org.basex.util.Util.info(Util.java:207) at org.basex.http.BaseXServlet.service(BaseXServlet.java:53) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:348) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) at java.lang.Thread.run(Thread.java:679)
What I did to try to fix it:
1 --- Set org.basex.mode to "server". I set it in the "override.xml" file and later also set it in the .war file's web.xml. It seems to be running in "server" mode now. The INFO command output looks like:
============================== General Information Version: 7.3 Used Memory: 27080 KB
Main Options DBPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/data DEBUG: true EVENTPORT: 1985 HOST: localhost HTTPPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/http HTTPPORT: 8984 KEEPALIVE: 600 LANG: English LANGKEYS: false NONPROXYHOSTS: PARALLEL: 8 PORT: 1984 PROXYHOST: PROXYPORT: 80 REPOPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/repo SERVERHOST: SERVERPORT: 1984 STOPPORT: 8985 TIMEOUT: 30
Options ADDARCHIVES: true ADDRAW: false ATTRINDEX: true AUTOFLUSH: true BINDINGS: CACHEQUERY: false CASESENS: false CATFILE: CHOP: true COMPPLAN: true CREATEFILTER: *.xml DIACRITICS: false DOTCOMPACT: false DOTDISPLAY: true DOTPLAN: false DOTTY: dotty DTD: false EXPORTER: FORCECREATE: false FTINDEX: false HTMLOPT: html=false,omit-xml-declaration=false,method=xml,nons=false,nobogons=false,nodefaults=false,nocolons=false,norestart=false,ignorable=false,emptybogons=false,any=false,norootbogons=false,nocdata=false,lexical=false,encoding=utf-8 INTPARSE: true LANGUAGE: en LSERROR: 0 MAINMEM: false MAXCATS: 100 MAXHITS: -1 MAXLEN: 96 MAXSTAT: 30 PARSER: xml PARSEROPT: encoding=UTF-8,flat=false,format=verbose,header=false,jsonml=false,lines=true,separator=comma QUERYINFO: false QUERYPATH: RUNS: 1 SERIALIZE: true SERIALIZER: SKIPCORRUPT: false STEMMING: false STOPWORDS: TAILCALLS: 42 TEXTINDEX: true UPDINDEX: false WRITEBACK: false XMLPLAN: false XQUERY3: true ==============================
2 --- I also tried using 7.4-SNAPSHOT (downloaded this morning) but it shows the same behaviour. It reports (I'm copying this by hand from another PC):
========================== Unexpected error: org.basex.util.Util: java.lang.RunTimeException: Data Access out of bounds:
- Pre value: 7290
- #used blocks: 31
- #total blocks: 31
- Access 30 (31 > 30) ==========================
Some more remarks:
- I would much appreciate a wiki-per-version, e.g. a 7.3 wiki, a 7.4 wiki, etc. The adaptations for a new version are slightly confusing - BaseX changes a lot (new options, renames, what option to put in what config file, etc.). Also for people using an older version this can be handy.
- the mailing list search service on the basex.org website (links from the "Open Source" page) crashes.
- (7.4) I am confused about the org.basex.restxqpath vs. org.basex.webpath settings, I put them both in and then it seemed to work. (Perhaps db:system() is also inconsistent wrt these settings.) RESTXQPATH is not mentioned in the OPTIONS list.
- (7.4) perhaps I missed it and the math:uuid() function was deprecated already, but I found that I needed to adapt my .xqm script for 7.4 because math:uuid() was renamed to random:uuid(). It seemed a little abrupt to me, but as I said, perhaps it missed the deprecation. I also used db:system() and had to adapt that too because org.basex.httppath no longer exists.
I suspect I missed some configuration option or something like that and will appreciate any help!
Thanks,
Huib Verweij.
------------------------------------------------------------------------------------------------------------ Disclaimer:
If you are not the intended recipient of this email, please notify the sender and delete it. Any unauthorized copying, disclosure or distribution of this email or its attachment(s) is forbidden. Thales Nederland BV will not accept liability for any damage caused by this email or its attachment(s). Thales Nederland BV is seated in Hengelo and is registered at the Chamber of Commerce under number 06061578. ------------------------------------------------------------------------------------------------------------
Hi Huib,
we have recently fixed some concurrency issues that have been related to the (un)compression of texts, so I'd be glad if you could test our latest stable snapshot:
http://files.basex.org/releases/latest/
Thanks, Christian ___________________________
On Mon, Aug 27, 2012 at 3:09 PM, VERWEIJ Huib huib.verweij@nl.thalesgroup.com wrote:
Hi,
I seem to have a concurrency problem and I have a hard time fixing it. Our BaseX db is a very modest database, for testing we load a 138Kb file. We use JMeter to do performance testing on our Web Application deployment of BaseX 7.3. Randomly GET and PUT commands are sent to BaseX. When we use only one thread (simulated user) everything is fine, when we increase the number of threads we quickly end up with a corrupted database – two threads is generally slower to crash than three threads, with three threads it crashes fast.
Here are some stacktraces I get:
[INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 27 org.basex.util.list.ByteList.add(ByteList.java:39) org.basex.util.Compress.push(Compress.java:72) org.basex.util.Compress.pack(Compress.java:52) org.basex.data.DiskData.index(DiskData.java:387) org.basex.data.Data.attr(Data.java:1051) org.basex.data.Data.insert(Data.java:835) org.basex.query.up.primitives.InsertInto.apply(InsertInto.java:32)
org.basex.query.up.NodeUpdates.makePrimitivesEffective(NodeUpdates.java:216) org.basex.query.up.DatabaseUpdates.apply(DatabaseUpdates.java:152) org.basex.query.up.ContextModifier.apply(ContextModifier.java:66) org.basex.query.up.Updates.apply(Updates.java:183) org.basex.query.QueryContext.update(QueryContext.java:276) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:76) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894)
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:851) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620)
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679)
I am also getting:
[INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 22 org.basex.util.Compress.pull(Compress.java:132) org.basex.util.Compress.unpack(Compress.java:109) org.basex.data.DiskData.txt(DiskData.java:262) org.basex.data.DiskData.text(DiskData.java:229) org.basex.data.Data.atom(Data.java:208) org.basex.query.value.node.DBNode.string(DBNode.java:115) org.basex.query.value.node.ANode.string(ANode.java:50) org.basex.query.value.item.Str.eq(Str.java:80) org.basex.query.value.node.ANode.eq(ANode.java:61) org.basex.query.expr.CmpV$OpV$5.eval(CmpV.java:84) org.basex.query.expr.CmpG.eval(CmpG.java:258) org.basex.query.expr.CmpG.item(CmpG.java:213) org.basex.query.expr.CmpG.item(CmpG.java:25) org.basex.query.expr.ParseExpr.ebv(ParseExpr.java:81) org.basex.query.expr.ParseExpr.test(ParseExpr.java:94) org.basex.query.expr.Preds.preds(Preds.java:129) org.basex.query.path.IterStep$1.next(IterStep.java:48) org.basex.query.path.IterStep$1.next(IterStep.java:32) org.basex.query.path.IterPath$1.next(IterPath.java:67) org.basex.query.path.IterPath$1.next(IterPath.java:39) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:329) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:304) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.Constr.add(Constr.java:65) org.basex.query.expr.CElem.item(CElem.java:84) org.basex.query.expr.CElem.item(CElem.java:21) org.basex.query.expr.ParseExpr.value(ParseExpr.java:69) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.func.TailFuncCall.iter(TailFuncCall.java:55) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.flwor.FLWR$1.next(FLWR.java:59) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:75) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884)
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620)
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679)
and also this:
14:52:02,452 - WARN - qtp384018721-20 - oup.nl.mms.plib.server.rest.AbstractEntityResource # HTTP response is not OK: code 500, ETag: empty, content: org.apache.http.conn.EofSensorInputStream@4acfed14 Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException 2012-08-27 14:52:02.959:WARN:oejs.ServletHandler:/basex/restxq/plib/platforms/ java.lang.ArrayIndexOutOfBoundsException: 0 at org.basex.util.Util.message(Util.java:166) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:327) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:345) at org.basex.util.Util.inf(Util.java:218) at org.basex.util.Util.info(Util.java:207) at org.basex.http.BaseXServlet.service(BaseXServlet.java:53) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:348) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) at java.lang.Thread.run(Thread.java:679)
What I did to try to fix it:
1 ---
Set org.basex.mode to “server”. I set it in the “override.xml” file and later also set it in the .war file’s web.xml. It seems to be running in “server” mode now. The INFO command output looks like:
============================== General Information Version: 7.3 Used Memory: 27080 KB
Main Options DBPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/data DEBUG: true EVENTPORT: 1985 HOST: localhost HTTPPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/http HTTPPORT: 8984 KEEPALIVE: 600 LANG: English LANGKEYS: false NONPROXYHOSTS: PARALLEL: 8 PORT: 1984 PROXYHOST: PROXYPORT: 80 REPOPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/repo SERVERHOST: SERVERPORT: 1984 STOPPORT: 8985 TIMEOUT: 30
Options ADDARCHIVES: true ADDRAW: false ATTRINDEX: true AUTOFLUSH: true BINDINGS: CACHEQUERY: false CASESENS: false CATFILE: CHOP: true COMPPLAN: true CREATEFILTER: *.xml DIACRITICS: false DOTCOMPACT: false DOTDISPLAY: true DOTPLAN: false DOTTY: dotty DTD: false EXPORTER: FORCECREATE: false FTINDEX: false HTMLOPT: html=false,omit-xml-declaration=false,method=xml,nons=false,nobogons=false,nodefaults=false,nocolons=false,norestart=false,ignorable=false,emptybogons=false,any=false,norootbogons=false,nocdata=false,lexical=false,encoding=utf-8 INTPARSE: true LANGUAGE: en LSERROR: 0 MAINMEM: false MAXCATS: 100 MAXHITS: -1 MAXLEN: 96 MAXSTAT: 30 PARSER: xml PARSEROPT: encoding=UTF-8,flat=false,format=verbose,header=false,jsonml=false,lines=true,separator=comma QUERYINFO: false QUERYPATH: RUNS: 1 SERIALIZE: true SERIALIZER: SKIPCORRUPT: false STEMMING: false STOPWORDS: TAILCALLS: 42 TEXTINDEX: true UPDINDEX: false WRITEBACK: false XMLPLAN: false XQUERY3: true
==============================
2 ---
I also tried using 7.4-SNAPSHOT (downloaded this morning) but it shows the same behaviour. It reports (I’m copying this by hand from another PC):
==========================
Unexpected error: org.basex.util.Util: java.lang.RunTimeException: Data Access out of bounds:
Pre value: 7290
#used blocks: 31
#total blocks: 31
Access 30 (31 > 30)
==========================
Some more remarks:
I would much appreciate a wiki-per-version, e.g. a 7.3 wiki, a
7.4 wiki, etc. The adaptations for a new version are slightly confusing – BaseX changes a lot (new options, renames, what option to put in what config file, etc.). Also for people using an older version this can be handy.
the mailing list search service on the basex.org website (links
from the “Open Source” page) crashes.
(7.4) I am confused about the org.basex.restxqpath vs.
org.basex.webpath settings, I put them both in and then it seemed to work. (Perhaps db:system() is also inconsistent wrt these settings.) RESTXQPATH is not mentioned in the OPTIONS list.
(7.4) perhaps I missed it and the math:uuid() function was
deprecated already, but I found that I needed to adapt my .xqm script for 7.4 because math:uuid() was renamed to random:uuid(). It seemed a little abrupt to me, but as I said, perhaps it missed the deprecation. I also used db:system() and had to adapt that too because org.basex.httppath no longer exists.
I suspect I missed some configuration option or something like that and will appreciate any help!
Thanks,
Huib Verweij.
Disclaimer:
If you are not the intended recipient of this email, please notify the sender and delete it. Any unauthorized copying, disclosure or distribution of this email or its attachment(s) is forbidden. Thales Nederland BV will not accept liability for any damage caused by this email or its attachment(s). Thales Nederland BV is seated in Hengelo and is registered at the Chamber of Commerce under number 06061578.
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Hi Huib,
an intermediate update: yes, I should have parsed your mail until the very bottom..
[...] When we use only one thread (simulated user) everything is fine, when we increase the number of threads we quickly end up with a corrupted database [...]
I know it’s no easy task, but if you manage to create a reproducible use case (the easier/smaller the better), that would be great.
I would much appreciate a wiki-per-version, e.g. a 7.3 wiki, a
7.4 wiki, etc.
Copying the whole Wiki with every release feels like a greater effort, which is why we have decided to create PDF versions, which can also be retrieved for older versions [1,2,..]. Does this help? The most elegant solution, in my point of view, would be if you could browse past „snapshot” of the documentation. Unfortunately, although it uses revisions, the MediaWiki software does not allow this.
the mailing list search service on the basex.org website (links
from the “Open Source” page) crashes.
True, and thanks, I'll have a look at that. We're about to switch to a new version of the homepage, which will be completely built in XQuery; the problems should then be gone.
[1] http://files.basex.org/releases/7.2/BaseX72.pdf [2] http://files.basex.org/releases/7.2.1/BaseX721.pdf
(7.4) I am confused about the org.basex.restxqpath vs.
org.basex.webpath settings, I put them both in and then it seemed to work. (Perhaps db:system() is also inconsistent wrt these settings.) RESTXQPATH is not mentioned in the OPTIONS list.
Those changes are very fresh, which is why we haven’t updated all pages of the documentation yet. In future, WEBPATH will point to the webapp directory, whereas the additional RESTXQPATH variable can point to an alternative path.
I also used db:system() and had to adapt that too because org.basex.httppath no longer exists.
Out of interest: For what reason do you access/utilize this particular option? Christian
Hi Christian,
[...] When we use only one thread (simulated user) everything is fine, when we increase the number of threads we quickly end up with a corrupted database [...]
I know it's no easy task, but if you manage to create a reproducible use case (the easier/smaller the better), that would be great.
[VERWEIJ Huib - Contractor] I'll see if I can produce a SSCCE.
I would much appreciate a wiki-per-version, e.g. a 7.3 wiki, a
7.4 wiki, etc.
Copying the whole Wiki with every release feels like a greater effort, which is why we have decided to create PDF versions, which can also be retrieved for older versions [1,2,..]. Does this help? [VERWEIJ Huib - Contractor] Of course! I missed the PDFs :-(.
[1] http://files.basex.org/releases/7.2/BaseX72.pdf [2] http://files.basex.org/releases/7.2.1/BaseX721.pdf
(7.4) I am confused about the org.basex.restxqpath vs.
org.basex.webpath settings, I put them both in and then it seemed to work. (Perhaps db:system() is also inconsistent wrt these settings.) RESTXQPATH is not mentioned in the OPTIONS list.
Those changes are very fresh, which is why we haven't updated all pages of the documentation yet. In future, WEBPATH will point to the webapp directory, whereas the additional RESTXQPATH variable can point to an alternative path. [VERWEIJ Huib - Contractor] OK.
I also used db:system() and had to adapt that too because org.basex.httppath no longer exists.
Out of interest: For what reason do you access/utilize this particular option? [VERWEIJ Huib - Contractor] I am trying to locate XSD files that are used to verify incoming data. I am trying to locate them relative to the "http" dir where my .xqm file resides, e.g. "../xsd/book.xsd"
[VERWEIJ Huib - Contractor] Huib (sorry for the horrible reply-format, it just started doing that...).
------------------------------------------------------------------------------------------------------------ Disclaimer:
If you are not the intended recipient of this email, please notify the sender and delete it. Any unauthorized copying, disclosure or distribution of this email or its attachment(s) is forbidden. Thales Nederland BV will not accept liability for any damage caused by this email or its attachment(s). Thales Nederland BV is seated in Hengelo and is registered at the Chamber of Commerce under number 06061578. ------------------------------------------------------------------------------------------------------------
[VERWEIJ Huib - Contractor] I'll see if I can produce a SSCCE.
[CG] perfect, thanks ;)
[VERWEIJ Huib - Contractor] I am trying to locate XSD files that are used to verify incoming data. I am trying to locate them relative to the "http" dir where my .xqm file resides, e.g. "../xsd/book.xsd"
static-base-uri() may also do the job; just try.
[VERWEIJ Huib - Contractor] I am trying to locate XSD files that are used to verify incoming data. I am trying to locate them relative to the "http" dir where my .xqm file resides, e.g. "../xsd/book.xsd"
static-base-uri() may also do the job; just try.
[VERWEIJ Huib - Contractor] I tried it, it worked. A case of "a few days in the field will often save a few minutes in the library" ;-). Thanks for your help.
------------------------------------------------------------------------------------------------------------ Disclaimer:
If you are not the intended recipient of this email, please notify the sender and delete it. Any unauthorized copying, disclosure or distribution of this email or its attachment(s) is forbidden. Thales Nederland BV will not accept liability for any damage caused by this email or its attachment(s). Thales Nederland BV is seated in Hengelo and is registered at the Chamber of Commerce under number 06061578. ------------------------------------------------------------------------------------------------------------
Hi Huib,
thanks for sending me your case, I'll have a look at soon.
A note for everyone: all previous PDF versions of the documentation are now listed in our Wiki [1]. Next, we’ve added some words on the new HTTP Options, such as RESTXQPATH [2]. Some more general notes on the jetty.xml and web.xml configuration files will follow; those files have got more important for creating web pages, both in standalone mode and as web applications.
Christian
[1] http://docs.basex.org/wiki/Documentation [2] http://docs.basex.org/wiki/Options#HTTP_Options ___________________________
On Mon, Aug 27, 2012 at 3:09 PM, VERWEIJ Huib huib.verweij@nl.thalesgroup.com wrote:
Hi,
I seem to have a concurrency problem and I have a hard time fixing it. Our BaseX db is a very modest database, for testing we load a 138Kb file. We use JMeter to do performance testing on our Web Application deployment of BaseX 7.3. Randomly GET and PUT commands are sent to BaseX. When we use only one thread (simulated user) everything is fine, when we increase the number of threads we quickly end up with a corrupted database – two threads is generally slower to crash than three threads, with three threads it crashes fast.
Here are some stacktraces I get:
[INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 27 org.basex.util.list.ByteList.add(ByteList.java:39) org.basex.util.Compress.push(Compress.java:72) org.basex.util.Compress.pack(Compress.java:52) org.basex.data.DiskData.index(DiskData.java:387) org.basex.data.Data.attr(Data.java:1051) org.basex.data.Data.insert(Data.java:835) org.basex.query.up.primitives.InsertInto.apply(InsertInto.java:32)
org.basex.query.up.NodeUpdates.makePrimitivesEffective(NodeUpdates.java:216) org.basex.query.up.DatabaseUpdates.apply(DatabaseUpdates.java:152) org.basex.query.up.ContextModifier.apply(ContextModifier.java:66) org.basex.query.up.Updates.apply(Updates.java:183) org.basex.query.QueryContext.update(QueryContext.java:276) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:76) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894)
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:851) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620)
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679)
I am also getting:
[INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 22 org.basex.util.Compress.pull(Compress.java:132) org.basex.util.Compress.unpack(Compress.java:109) org.basex.data.DiskData.txt(DiskData.java:262) org.basex.data.DiskData.text(DiskData.java:229) org.basex.data.Data.atom(Data.java:208) org.basex.query.value.node.DBNode.string(DBNode.java:115) org.basex.query.value.node.ANode.string(ANode.java:50) org.basex.query.value.item.Str.eq(Str.java:80) org.basex.query.value.node.ANode.eq(ANode.java:61) org.basex.query.expr.CmpV$OpV$5.eval(CmpV.java:84) org.basex.query.expr.CmpG.eval(CmpG.java:258) org.basex.query.expr.CmpG.item(CmpG.java:213) org.basex.query.expr.CmpG.item(CmpG.java:25) org.basex.query.expr.ParseExpr.ebv(ParseExpr.java:81) org.basex.query.expr.ParseExpr.test(ParseExpr.java:94) org.basex.query.expr.Preds.preds(Preds.java:129) org.basex.query.path.IterStep$1.next(IterStep.java:48) org.basex.query.path.IterStep$1.next(IterStep.java:32) org.basex.query.path.IterPath$1.next(IterPath.java:67) org.basex.query.path.IterPath$1.next(IterPath.java:39) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:329) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:304) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.Constr.add(Constr.java:65) org.basex.query.expr.CElem.item(CElem.java:84) org.basex.query.expr.CElem.item(CElem.java:21) org.basex.query.expr.ParseExpr.value(ParseExpr.java:69) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.func.TailFuncCall.iter(TailFuncCall.java:55) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.flwor.FLWR$1.next(FLWR.java:59) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:75) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884)
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620)
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679)
and also this:
14:52:02,452 - WARN - qtp384018721-20 - oup.nl.mms.plib.server.rest.AbstractEntityResource # HTTP response is not OK: code 500, ETag: empty, content: org.apache.http.conn.EofSensorInputStream@4acfed14 Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException 2012-08-27 14:52:02.959:WARN:oejs.ServletHandler:/basex/restxq/plib/platforms/ java.lang.ArrayIndexOutOfBoundsException: 0 at org.basex.util.Util.message(Util.java:166) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:327) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:345) at org.basex.util.Util.inf(Util.java:218) at org.basex.util.Util.info(Util.java:207) at org.basex.http.BaseXServlet.service(BaseXServlet.java:53) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:348) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) at java.lang.Thread.run(Thread.java:679)
What I did to try to fix it:
1 ---
Set org.basex.mode to “server”. I set it in the “override.xml” file and later also set it in the .war file’s web.xml. It seems to be running in “server” mode now. The INFO command output looks like:
============================== General Information Version: 7.3 Used Memory: 27080 KB
Main Options DBPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/data DEBUG: true EVENTPORT: 1985 HOST: localhost HTTPPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/http HTTPPORT: 8984 KEEPALIVE: 600 LANG: English LANGKEYS: false NONPROXYHOSTS: PARALLEL: 8 PORT: 1984 PROXYHOST: PROXYPORT: 80 REPOPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/repo SERVERHOST: SERVERPORT: 1984 STOPPORT: 8985 TIMEOUT: 30
Options ADDARCHIVES: true ADDRAW: false ATTRINDEX: true AUTOFLUSH: true BINDINGS: CACHEQUERY: false CASESENS: false CATFILE: CHOP: true COMPPLAN: true CREATEFILTER: *.xml DIACRITICS: false DOTCOMPACT: false DOTDISPLAY: true DOTPLAN: false DOTTY: dotty DTD: false EXPORTER: FORCECREATE: false FTINDEX: false HTMLOPT: html=false,omit-xml-declaration=false,method=xml,nons=false,nobogons=false,nodefaults=false,nocolons=false,norestart=false,ignorable=false,emptybogons=false,any=false,norootbogons=false,nocdata=false,lexical=false,encoding=utf-8 INTPARSE: true LANGUAGE: en LSERROR: 0 MAINMEM: false MAXCATS: 100 MAXHITS: -1 MAXLEN: 96 MAXSTAT: 30 PARSER: xml PARSEROPT: encoding=UTF-8,flat=false,format=verbose,header=false,jsonml=false,lines=true,separator=comma QUERYINFO: false QUERYPATH: RUNS: 1 SERIALIZE: true SERIALIZER: SKIPCORRUPT: false STEMMING: false STOPWORDS: TAILCALLS: 42 TEXTINDEX: true UPDINDEX: false WRITEBACK: false XMLPLAN: false XQUERY3: true
==============================
2 ---
I also tried using 7.4-SNAPSHOT (downloaded this morning) but it shows the same behaviour. It reports (I’m copying this by hand from another PC):
==========================
Unexpected error: org.basex.util.Util: java.lang.RunTimeException: Data Access out of bounds:
Pre value: 7290
#used blocks: 31
#total blocks: 31
Access 30 (31 > 30)
==========================
Some more remarks:
I would much appreciate a wiki-per-version, e.g. a 7.3 wiki, a
7.4 wiki, etc. The adaptations for a new version are slightly confusing – BaseX changes a lot (new options, renames, what option to put in what config file, etc.). Also for people using an older version this can be handy.
the mailing list search service on the basex.org website (links
from the “Open Source” page) crashes.
(7.4) I am confused about the org.basex.restxqpath vs.
org.basex.webpath settings, I put them both in and then it seemed to work. (Perhaps db:system() is also inconsistent wrt these settings.) RESTXQPATH is not mentioned in the OPTIONS list.
(7.4) perhaps I missed it and the math:uuid() function was
deprecated already, but I found that I needed to adapt my .xqm script for 7.4 because math:uuid() was renamed to random:uuid(). It seemed a little abrupt to me, but as I said, perhaps it missed the deprecation. I also used db:system() and had to adapt that too because org.basex.httppath no longer exists.
I suspect I missed some configuration option or something like that and will appreciate any help!
Thanks,
Huib Verweij.
Disclaimer:
If you are not the intended recipient of this email, please notify the sender and delete it. Any unauthorized copying, disclosure or distribution of this email or its attachment(s) is forbidden. Thales Nederland BV will not accept liability for any damage caused by this email or its attachment(s). Thales Nederland BV is seated in Hengelo and is registered at the Chamber of Commerce under number 06061578.
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Hi Huib,
your test case seems to run fine on my machine. I'll send you a note in private, containing more details.
Best, Christian ___________________________
On Tue, Aug 28, 2012 at 3:15 PM, Christian Grün christian.gruen@gmail.com wrote:
Hi Huib,
thanks for sending me your case, I'll have a look at soon.
A note for everyone: all previous PDF versions of the documentation are now listed in our Wiki [1]. Next, we’ve added some words on the new HTTP Options, such as RESTXQPATH [2]. Some more general notes on the jetty.xml and web.xml configuration files will follow; those files have got more important for creating web pages, both in standalone mode and as web applications.
Christian
[1] http://docs.basex.org/wiki/Documentation [2] http://docs.basex.org/wiki/Options#HTTP_Options ___________________________
On Mon, Aug 27, 2012 at 3:09 PM, VERWEIJ Huib huib.verweij@nl.thalesgroup.com wrote:
Hi,
I seem to have a concurrency problem and I have a hard time fixing it. Our BaseX db is a very modest database, for testing we load a 138Kb file. We use JMeter to do performance testing on our Web Application deployment of BaseX 7.3. Randomly GET and PUT commands are sent to BaseX. When we use only one thread (simulated user) everything is fine, when we increase the number of threads we quickly end up with a corrupted database – two threads is generally slower to crash than three threads, with three threads it crashes fast.
Here are some stacktraces I get:
[INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 27 org.basex.util.list.ByteList.add(ByteList.java:39) org.basex.util.Compress.push(Compress.java:72) org.basex.util.Compress.pack(Compress.java:52) org.basex.data.DiskData.index(DiskData.java:387) org.basex.data.Data.attr(Data.java:1051) org.basex.data.Data.insert(Data.java:835) org.basex.query.up.primitives.InsertInto.apply(InsertInto.java:32)
org.basex.query.up.NodeUpdates.makePrimitivesEffective(NodeUpdates.java:216) org.basex.query.up.DatabaseUpdates.apply(DatabaseUpdates.java:152) org.basex.query.up.ContextModifier.apply(ContextModifier.java:66) org.basex.query.up.Updates.apply(Updates.java:183) org.basex.query.QueryContext.update(QueryContext.java:276) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:76) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894)
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:851) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620)
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679)
I am also getting:
[INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 22 org.basex.util.Compress.pull(Compress.java:132) org.basex.util.Compress.unpack(Compress.java:109) org.basex.data.DiskData.txt(DiskData.java:262) org.basex.data.DiskData.text(DiskData.java:229) org.basex.data.Data.atom(Data.java:208) org.basex.query.value.node.DBNode.string(DBNode.java:115) org.basex.query.value.node.ANode.string(ANode.java:50) org.basex.query.value.item.Str.eq(Str.java:80) org.basex.query.value.node.ANode.eq(ANode.java:61) org.basex.query.expr.CmpV$OpV$5.eval(CmpV.java:84) org.basex.query.expr.CmpG.eval(CmpG.java:258) org.basex.query.expr.CmpG.item(CmpG.java:213) org.basex.query.expr.CmpG.item(CmpG.java:25) org.basex.query.expr.ParseExpr.ebv(ParseExpr.java:81) org.basex.query.expr.ParseExpr.test(ParseExpr.java:94) org.basex.query.expr.Preds.preds(Preds.java:129) org.basex.query.path.IterStep$1.next(IterStep.java:48) org.basex.query.path.IterStep$1.next(IterStep.java:32) org.basex.query.path.IterPath$1.next(IterPath.java:67) org.basex.query.path.IterPath$1.next(IterPath.java:39) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:329) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:304) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.Constr.add(Constr.java:65) org.basex.query.expr.CElem.item(CElem.java:84) org.basex.query.expr.CElem.item(CElem.java:21) org.basex.query.expr.ParseExpr.value(ParseExpr.java:69) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.func.TailFuncCall.iter(TailFuncCall.java:55) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.flwor.FLWR$1.next(FLWR.java:59) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:75) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884)
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620)
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679)
and also this:
14:52:02,452 - WARN - qtp384018721-20 - oup.nl.mms.plib.server.rest.AbstractEntityResource # HTTP response is not OK: code 500, ETag: empty, content: org.apache.http.conn.EofSensorInputStream@4acfed14 Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException 2012-08-27 14:52:02.959:WARN:oejs.ServletHandler:/basex/restxq/plib/platforms/ java.lang.ArrayIndexOutOfBoundsException: 0 at org.basex.util.Util.message(Util.java:166) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:327) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:345) at org.basex.util.Util.inf(Util.java:218) at org.basex.util.Util.info(Util.java:207) at org.basex.http.BaseXServlet.service(BaseXServlet.java:53) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:348) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) at java.lang.Thread.run(Thread.java:679)
What I did to try to fix it:
1 ---
Set org.basex.mode to “server”. I set it in the “override.xml” file and later also set it in the .war file’s web.xml. It seems to be running in “server” mode now. The INFO command output looks like:
============================== General Information Version: 7.3 Used Memory: 27080 KB
Main Options DBPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/data DEBUG: true EVENTPORT: 1985 HOST: localhost HTTPPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/http HTTPPORT: 8984 KEEPALIVE: 600 LANG: English LANGKEYS: false NONPROXYHOSTS: PARALLEL: 8 PORT: 1984 PROXYHOST: PROXYPORT: 80 REPOPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/repo SERVERHOST: SERVERPORT: 1984 STOPPORT: 8985 TIMEOUT: 30
Options ADDARCHIVES: true ADDRAW: false ATTRINDEX: true AUTOFLUSH: true BINDINGS: CACHEQUERY: false CASESENS: false CATFILE: CHOP: true COMPPLAN: true CREATEFILTER: *.xml DIACRITICS: false DOTCOMPACT: false DOTDISPLAY: true DOTPLAN: false DOTTY: dotty DTD: false EXPORTER: FORCECREATE: false FTINDEX: false HTMLOPT: html=false,omit-xml-declaration=false,method=xml,nons=false,nobogons=false,nodefaults=false,nocolons=false,norestart=false,ignorable=false,emptybogons=false,any=false,norootbogons=false,nocdata=false,lexical=false,encoding=utf-8 INTPARSE: true LANGUAGE: en LSERROR: 0 MAINMEM: false MAXCATS: 100 MAXHITS: -1 MAXLEN: 96 MAXSTAT: 30 PARSER: xml PARSEROPT: encoding=UTF-8,flat=false,format=verbose,header=false,jsonml=false,lines=true,separator=comma QUERYINFO: false QUERYPATH: RUNS: 1 SERIALIZE: true SERIALIZER: SKIPCORRUPT: false STEMMING: false STOPWORDS: TAILCALLS: 42 TEXTINDEX: true UPDINDEX: false WRITEBACK: false XMLPLAN: false XQUERY3: true
==============================
2 ---
I also tried using 7.4-SNAPSHOT (downloaded this morning) but it shows the same behaviour. It reports (I’m copying this by hand from another PC):
==========================
Unexpected error: org.basex.util.Util: java.lang.RunTimeException: Data Access out of bounds:
Pre value: 7290
#used blocks: 31
#total blocks: 31
Access 30 (31 > 30)
==========================
Some more remarks:
I would much appreciate a wiki-per-version, e.g. a 7.3 wiki, a
7.4 wiki, etc. The adaptations for a new version are slightly confusing – BaseX changes a lot (new options, renames, what option to put in what config file, etc.). Also for people using an older version this can be handy.
the mailing list search service on the basex.org website (links
from the “Open Source” page) crashes.
(7.4) I am confused about the org.basex.restxqpath vs.
org.basex.webpath settings, I put them both in and then it seemed to work. (Perhaps db:system() is also inconsistent wrt these settings.) RESTXQPATH is not mentioned in the OPTIONS list.
(7.4) perhaps I missed it and the math:uuid() function was
deprecated already, but I found that I needed to adapt my .xqm script for 7.4 because math:uuid() was renamed to random:uuid(). It seemed a little abrupt to me, but as I said, perhaps it missed the deprecation. I also used db:system() and had to adapt that too because org.basex.httppath no longer exists.
I suspect I missed some configuration option or something like that and will appreciate any help!
Thanks,
Huib Verweij.
Disclaimer:
If you are not the intended recipient of this email, please notify the sender and delete it. Any unauthorized copying, disclosure or distribution of this email or its attachment(s) is forbidden. Thales Nederland BV will not accept liability for any damage caused by this email or its attachment(s). Thales Nederland BV is seated in Hengelo and is registered at the Chamber of Commerce under number 06061578.
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Hi Christian,
I downloaded the latest SNAPSHOT this morning (20120903) and the problem is indeed fixed now. Using both the Oracle JDK and OpenJDK everything's running smoothly.
Thanks a lot for your excellent support!
Huib Verweij.
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Sunday, 02 September, 2012 19:13 To: VERWEIJ Huib - Contractor Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Concurreny troubles (?) and some remarks.
Hi Huib,
your test case seems to run fine on my machine. I'll send you a note in private, containing more details.
Best, Christian ___________________________
On Tue, Aug 28, 2012 at 3:15 PM, Christian Grün christian.gruen@gmail.com wrote:
Hi Huib,
thanks for sending me your case, I'll have a look at soon.
A note for everyone: all previous PDF versions of the documentation are now listed in our Wiki [1]. Next, we've added some words on the new HTTP Options, such as RESTXQPATH [2]. Some more general notes on the jetty.xml and web.xml configuration files will follow; those files have got more important for creating web pages, both in standalone mode and as web applications.
Christian
[1] http://docs.basex.org/wiki/Documentation [2] http://docs.basex.org/wiki/Options#HTTP_Options ___________________________
On Mon, Aug 27, 2012 at 3:09 PM, VERWEIJ Huib huib.verweij@nl.thalesgroup.com wrote:
Hi,
I seem to have a concurrency problem and I have a hard time fixing it. Our BaseX db is a very modest database, for testing we load a 138Kb file. We use JMeter to do performance testing on our Web Application deployment of BaseX 7.3. Randomly GET and PUT commands are sent to BaseX. When we use only one thread (simulated user) everything is fine, when we increase the number of threads we quickly end up with a corrupted database - two threads is generally slower to crash than three threads, with three threads it crashes fast.
Here are some stacktraces I get:
[INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 27 org.basex.util.list.ByteList.add(ByteList.java:39) org.basex.util.Compress.push(Compress.java:72) org.basex.util.Compress.pack(Compress.java:52) org.basex.data.DiskData.index(DiskData.java:387) org.basex.data.Data.attr(Data.java:1051) org.basex.data.Data.insert(Data.java:835) org.basex.query.up.primitives.InsertInto.apply(InsertInto.java:32)
org.basex.query.up.NodeUpdates.makePrimitivesEffective(NodeUpdates.java:216) org.basex.query.up.DatabaseUpdates.apply(DatabaseUpdates.java:152) org.basex.query.up.ContextModifier.apply(ContextModifier.java:66) org.basex.query.up.Updates.apply(Updates.java:183) org.basex.query.QueryContext.update(QueryContext.java:276) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:76) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java :479)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j ava:119)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.jav a:524)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandl er.java:225)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java: 406)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandle r.java:186)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandle r.java:965)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j ava:117)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Cont extHandlerCollection.java:250)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerColl ection.java:149)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(Abstrac tHttpConnection.java:452)
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpC onnection.java:894)
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:851)
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnecti on.java:77)
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEn dPoint.java:620)
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEnd Point.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPoo l.java:603)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679)
I am also getting:
[INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 22 org.basex.util.Compress.pull(Compress.java:132) org.basex.util.Compress.unpack(Compress.java:109) org.basex.data.DiskData.txt(DiskData.java:262) org.basex.data.DiskData.text(DiskData.java:229) org.basex.data.Data.atom(Data.java:208) org.basex.query.value.node.DBNode.string(DBNode.java:115) org.basex.query.value.node.ANode.string(ANode.java:50) org.basex.query.value.item.Str.eq(Str.java:80) org.basex.query.value.node.ANode.eq(ANode.java:61) org.basex.query.expr.CmpV$OpV$5.eval(CmpV.java:84) org.basex.query.expr.CmpG.eval(CmpG.java:258) org.basex.query.expr.CmpG.item(CmpG.java:213) org.basex.query.expr.CmpG.item(CmpG.java:25) org.basex.query.expr.ParseExpr.ebv(ParseExpr.java:81) org.basex.query.expr.ParseExpr.test(ParseExpr.java:94) org.basex.query.expr.Preds.preds(Preds.java:129) org.basex.query.path.IterStep$1.next(IterStep.java:48) org.basex.query.path.IterStep$1.next(IterStep.java:32) org.basex.query.path.IterPath$1.next(IterPath.java:67) org.basex.query.path.IterPath$1.next(IterPath.java:39) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:329) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:304) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.Constr.add(Constr.java:65) org.basex.query.expr.CElem.item(CElem.java:84) org.basex.query.expr.CElem.item(CElem.java:21) org.basex.query.expr.ParseExpr.value(ParseExpr.java:69) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.func.TailFuncCall.iter(TailFuncCall.java:55) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.flwor.FLWR$1.next(FLWR.java:59) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:75) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java :479)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j ava:119)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.jav a:524)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandl er.java:225)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java: 406)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandle r.java:186)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandle r.java:965)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j ava:117)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Cont extHandlerCollection.java:250)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerColl ection.java:149)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(Abstrac tHttpConnection.java:452)
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(Abstra ctHttpConnection.java:884)
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630)
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnecti on.java:77)
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEn dPoint.java:620)
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEnd Point.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPoo l.java:603)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679)
and also this:
14:52:02,452 - WARN - qtp384018721-20 - oup.nl.mms.plib.server.rest.AbstractEntityResource # HTTP response is not OK: code 500, ETag: empty, content: org.apache.http.conn.EofSensorInputStream@4acfed14 Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException 2012-08-27 14:52:02.959:WARN:oejs.ServletHandler:/basex/restxq/plib/platforms/ java.lang.ArrayIndexOutOfBoundsException: 0 at org.basex.util.Util.message(Util.java:166) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:327) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:345) at org.basex.util.Util.inf(Util.java:218) at org.basex.util.Util.info(Util.java:207) at org.basex.http.BaseXServlet.service(BaseXServlet.java:53) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:348) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) at java.lang.Thread.run(Thread.java:679)
What I did to try to fix it:
1 ---
Set org.basex.mode to "server". I set it in the "override.xml" file and later also set it in the .war file's web.xml. It seems to be running in "server" mode now. The INFO command output looks like:
============================== General Information Version: 7.3 Used Memory: 27080 KB
Main Options DBPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/data DEBUG: true EVENTPORT: 1985 HOST: localhost HTTPPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/http HTTPPORT: 8984 KEEPALIVE: 600 LANG: English LANGKEYS: false NONPROXYHOSTS: PARALLEL: 8 PORT: 1984 PROXYHOST: PROXYPORT: 80 REPOPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/repo SERVERHOST: SERVERPORT: 1984 STOPPORT: 8985 TIMEOUT: 30
Options ADDARCHIVES: true ADDRAW: false ATTRINDEX: true AUTOFLUSH: true BINDINGS: CACHEQUERY: false CASESENS: false CATFILE: CHOP: true COMPPLAN: true CREATEFILTER: *.xml DIACRITICS: false DOTCOMPACT: false DOTDISPLAY: true DOTPLAN: false DOTTY: dotty DTD: false EXPORTER: FORCECREATE: false FTINDEX: false HTMLOPT: html=false,omit-xml-declaration=false,method=xml,nons=false,nobogons= false,nodefaults=false,nocolons=false,norestart=false,ignorable=false ,emptybogons=false,any=false,norootbogons=false,nocdata=false,lexical =false,encoding=utf-8 INTPARSE: true LANGUAGE: en LSERROR: 0 MAINMEM: false MAXCATS: 100 MAXHITS: -1 MAXLEN: 96 MAXSTAT: 30 PARSER: xml PARSEROPT: encoding=UTF-8,flat=false,format=verbose,header=false,jsonml=false,li nes=true,separator=comma QUERYINFO: false QUERYPATH: RUNS: 1 SERIALIZE: true SERIALIZER: SKIPCORRUPT: false STEMMING: false STOPWORDS: TAILCALLS: 42 TEXTINDEX: true UPDINDEX: false WRITEBACK: false XMLPLAN: false XQUERY3: true
==============================
2 ---
I also tried using 7.4-SNAPSHOT (downloaded this morning) but it shows the same behaviour. It reports (I'm copying this by hand from another PC):
==========================
Unexpected error: org.basex.util.Util: java.lang.RunTimeException: Data Access out of bounds:
Pre value: 7290
#used blocks: 31
#total blocks: 31
Access 30 (31 > 30)
==========================
Some more remarks:
I would much appreciate a wiki-per-version, e.g. a 7.3 wiki, a
7.4 wiki, etc. The adaptations for a new version are slightly confusing - BaseX changes a lot (new options, renames, what option to put in what config file, etc.). Also for people using an older version this can be handy.
the mailing list search service on the basex.org website (links
from the "Open Source" page) crashes.
(7.4) I am confused about the org.basex.restxqpath vs.
org.basex.webpath settings, I put them both in and then it seemed to work. (Perhaps db:system() is also inconsistent wrt these settings.) RESTXQPATH is not mentioned in the OPTIONS list.
(7.4) perhaps I missed it and the math:uuid() function was
deprecated already, but I found that I needed to adapt my .xqm script for 7.4 because math:uuid() was renamed to random:uuid(). It seemed a little abrupt to me, but as I said, perhaps it missed the deprecation. I also used db:system() and had to adapt that too because org.basex.httppath no longer exists.
I suspect I missed some configuration option or something like that and will appreciate any help!
Thanks,
Huib Verweij.
Disclaimer:
If you are not the intended recipient of this email, please notify the sender and delete it. Any unauthorized copying, disclosure or distribution of this email or its attachment(s) is forbidden. Thales Nederland BV will not accept liability for any damage caused by this email or its attachment(s). Thales Nederland BV is seated in Hengelo and is registered at the Chamber of Commerce under number 06061578.
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
------------------------------------------------------------------------------------------------------------ Disclaimer:
If you are not the intended recipient of this email, please notify the sender and delete it. Any unauthorized copying, disclosure or distribution of this email or its attachment(s) is forbidden. Thales Nederland BV will not accept liability for any damage caused by this email or its attachment(s). Thales Nederland BV is seated in Hengelo and is registered at the Chamber of Commerce under number 06061578. ------------------------------------------------------------------------------------------------------------
basex-talk@mailman.uni-konstanz.de