- BaseX-Talk - mailman.uni-konstanz.de

Concurreny troubles (?) and some remarks.
by VERWEIJ Huib 03 Sep '12

03 Sep '12

Hi, I seem to have a concurrency problem and I have a hard time fixing it. Our BaseX db is a very modest database, for testing we load a 138Kb file. We use JMeter to do performance testing on our Web Application deployment of BaseX 7.3. Randomly GET and PUT commands are sent to BaseX. When we use only one thread (simulated user) everything is fine, when we increase the number of threads we quickly end up with a corrupted database - two threads is generally slower to crash than three threads, with three threads it crashes fast. Here are some stacktraces I get: [INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de> Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 27 org.basex.util.list.ByteList.add(ByteList.java:39) org.basex.util.Compress.push(Compress.java:72) org.basex.util.Compress.pack(Compress.java:52) org.basex.data.DiskData.index(DiskData.java:387) org.basex.data.Data.attr(Data.java:1051) org.basex.data.Data.insert(Data.java:835) org.basex.query.up.primitives.InsertInto.apply(InsertInto.java:32) org.basex.query.up.NodeUpdates.makePrimitivesEffective(NodeUpdates.java:216) org.basex.query.up.DatabaseUpdates.apply(DatabaseUpdates.java:152) org.basex.query.up.ContextModifier.apply(ContextModifier.java:66) org.basex.query.up.Updates.apply(Updates.java:183) org.basex.query.QueryContext.update(QueryContext.java:276) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:76) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348) org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894) org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:851) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620) org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679) I am also getting: [INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de> Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 22 org.basex.util.Compress.pull(Compress.java:132) org.basex.util.Compress.unpack(Compress.java:109) org.basex.data.DiskData.txt(DiskData.java:262) org.basex.data.DiskData.text(DiskData.java:229) org.basex.data.Data.atom(Data.java:208) org.basex.query.value.node.DBNode.string(DBNode.java:115) org.basex.query.value.node.ANode.string(ANode.java:50) org.basex.query.value.item.Str.eq(Str.java:80) org.basex.query.value.node.ANode.eq(ANode.java:61) org.basex.query.expr.CmpV$OpV$5.eval(CmpV.java:84) org.basex.query.expr.CmpG.eval(CmpG.java:258) org.basex.query.expr.CmpG.item(CmpG.java:213) org.basex.query.expr.CmpG.item(CmpG.java:25) org.basex.query.expr.ParseExpr.ebv(ParseExpr.java:81) org.basex.query.expr.ParseExpr.test(ParseExpr.java:94) org.basex.query.expr.Preds.preds(Preds.java:129) org.basex.query.path.IterStep$1.next(IterStep.java:48) org.basex.query.path.IterStep$1.next(IterStep.java:32) org.basex.query.path.IterPath$1.next(IterPath.java:67) org.basex.query.path.IterPath$1.next(IterPath.java:39) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:329) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:304) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.Constr.add(Constr.java:65) org.basex.query.expr.CElem.item(CElem.java:84) org.basex.query.expr.CElem.item(CElem.java:21) org.basex.query.expr.ParseExpr.value(ParseExpr.java:69) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.func.TailFuncCall.iter(TailFuncCall.java:55) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.flwor.FLWR$1.next(FLWR.java:59) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:75) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348) org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884) org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620) org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679) and also this: 14:52:02,452 - WARN - qtp384018721-20 - oup.nl.mms.plib.server.rest.AbstractEntityResource # HTTP response is not OK: code 500, ETag: empty, content: org.apache.http.conn.EofSensorInputStream@4acfed14<mailto:org.apache.http.conn.EofSensorInputStream@4acfed14> Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de> Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException 2012-08-27 14:52:02.959:WARN:oejs.ServletHandler:/basex/restxq/plib/platforms/ java.lang.ArrayIndexOutOfBoundsException: 0 at org.basex.util.Util.message(Util.java:166) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:327) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:345) at org.basex.util.Util.inf(Util.java:218) at org.basex.util.Util.info<http://org.basex.util.Util.info>(Util.java:207) at org.basex.http.BaseXServlet.service(BaseXServlet.java:53) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:348) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) at java.lang.Thread.run(Thread.java:679) What I did to try to fix it: 1 --- Set org.basex.mode to "server". I set it in the "override.xml" file and later also set it in the .war file's web.xml. It seems to be running in "server" mode now. The INFO command output looks like: ============================== General Information Version: 7.3 Used Memory: 27080 KB Main Options DBPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/data DEBUG: true EVENTPORT: 1985 HOST: localhost HTTPPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/http HTTPPORT: 8984 KEEPALIVE: 600 LANG: English LANGKEYS: false NONPROXYHOSTS: PARALLEL: 8 PORT: 1984 PROXYHOST: PROXYPORT: 80 REPOPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/repo SERVERHOST: SERVERPORT: 1984 STOPPORT: 8985 TIMEOUT: 30 Options ADDARCHIVES: true ADDRAW: false ATTRINDEX: true AUTOFLUSH: true BINDINGS: CACHEQUERY: false CASESENS: false CATFILE: CHOP: true COMPPLAN: true CREATEFILTER: *.xml DIACRITICS: false DOTCOMPACT: false DOTDISPLAY: true DOTPLAN: false DOTTY: dotty DTD: false EXPORTER: FORCECREATE: false FTINDEX: false HTMLOPT: html=false,omit-xml-declaration=false,method=xml,nons=false,nobogons=false,nodefaults=false,nocolons=false,norestart=false,ignorable=false,emptybogons=false,any=false,norootbogons=false,nocdata=false,lexical=false,encoding=utf-8 INTPARSE: true LANGUAGE: en LSERROR: 0 MAINMEM: false MAXCATS: 100 MAXHITS: -1 MAXLEN: 96 MAXSTAT: 30 PARSER: xml PARSEROPT: encoding=UTF-8,flat=false,format=verbose,header=false,jsonml=false,lines=true,separator=comma QUERYINFO: false QUERYPATH: RUNS: 1 SERIALIZE: true SERIALIZER: SKIPCORRUPT: false STEMMING: false STOPWORDS: TAILCALLS: 42 TEXTINDEX: true UPDINDEX: false WRITEBACK: false XMLPLAN: false XQUERY3: true ============================== 2 --- I also tried using 7.4-SNAPSHOT (downloaded this morning) but it shows the same behaviour. It reports (I'm copying this by hand from another PC): ========================== Unexpected error: org.basex.util.Util: java.lang.RunTimeException: Data Access out of bounds: - Pre value: 7290 - #used blocks: 31 - #total blocks: 31 - Access 30 (31 > 30) ========================== Some more remarks: - I would much appreciate a wiki-per-version, e.g. a 7.3 wiki, a 7.4 wiki, etc. The adaptations for a new version are slightly confusing - BaseX changes a lot (new options, renames, what option to put in what config file, etc.). Also for people using an older version this can be handy. - the mailing list search service on the basex.org website (links from the "Open Source" page) crashes. - (7.4) I am confused about the org.basex.restxqpath vs. org.basex.webpath settings, I put them both in and then it seemed to work. (Perhaps db:system() is also inconsistent wrt these settings.) RESTXQPATH is not mentioned in the OPTIONS list. - (7.4) perhaps I missed it and the math:uuid() function was deprecated already, but I found that I needed to adapt my .xqm script for 7.4 because math:uuid() was renamed to random:uuid(). It seemed a little abrupt to me, but as I said, perhaps it missed the deprecation. I also used db:system() and had to adapt that too because org.basex.httppath no longer exists. I suspect I missed some configuration option or something like that and will appreciate any help! Thanks, Huib Verweij. ------------------------------------------------------------------------------------------------------------ Disclaimer: If you are not the intended recipient of this email, please notify the sender and delete it. Any unauthorized copying, disclosure or distribution of this email or its attachment(s) is forbidden. Thales Nederland BV will not accept liability for any damage caused by this email or its attachment(s). Thales Nederland BV is seated in Hengelo and is registered at the Chamber of Commerce under number 06061578. ------------------------------------------------------------------------------------------------------------

2 8

Bug in XPath expression
by Hans-Juergen Rennau 31 Aug '12

31 Aug '12

Hello, if I am not mistaken, the following code demonstrates a bug. The code extracts from a schema a type definition, referenced by an element declaration. The second line determines the element whose type definition is sought. In mode 1, the type definition is not found, in mode 2 it is found. With Saxon, modes 1 and 2 are successful. Strangely, at least with my XSD the error occurs only if the type to be found is xs:simpleType - it does not occur if it is xs:complexType. Kind regards, Hans-Juergen =================================== declare variable $schemas := /*; declare variable $elem := ($schemas//xs:element[@type eq 't:Room'])[1]; declare variable $mode := 1; declare function local:getTypeDef($elem as element(xs:element)) as element()? { let $type := $elem/@type return if (empty($type)) then $elem/(xs:simpleType, xs:complexType) else let $qname := resolve-QName($type, $elem) let $lname := local-name-from-QName($qname) let $uri := namespace-uri-from-QName($qname) return if (empty($uri)) then let $actSchemas := $schemas[not(@targetNamespace)] return $actSchemas/(xs:complexType, xs:simpleType)[@name eq $lname] else if ($mode eq 1) then $schemas[@targetNamespace eq $uri]/(xs:complexType, xs:simpleType)[@name eq $lname] else let $actSchemas := $schemas[@targetNamespace eq $uri] let $types := $actSchemas/(xs:complexType, xs:simpleType) return $types[@name eq $lname] }; <typeDef>{ local:getTypeDef($elem) }</typeDef> ==============================================

2 2

limit to the number of entity expansions
by Fabrice Etanchaud 30 Aug '12

30 Aug '12

Dear all at BaseX, Is there any solution when limit to the number of entity expansions is reached ? file:/C:/FE/Data/FR.old/maj/maj2005-01_1.xml" (Line 1): The parser has encountered more than "64.000" entity expansions in this document; this is the limit imposed by the application. Best regards from yet another BaseX addict Fabrice ETANCHAUD Senior Software Engineer edital Berkenlaan 1 B-1831 Brussels, Belgium <skype:fabrice.etanchaud?call> +32 2 716 32 32 general +32 2 716 32 20 fax fetanchaud(a)edital.com corsearch.com <http://www.corsearch.com/> | edital.com <http://www.edital.com/> <outbind://899-00000000E439C83B50BF17409D8C27B7AE0425AF07005AA868536216F 543BAB9422EAEA5338B00000011020900005AA868536216F543BAB9422EAEA5338B00000 13AA36B0000/cid:image001.gif@01CC0F02.2C91B6C0> Confidentiality Notice: This email and its attachments (if any) contain confidential information of the sender. The information is intended only for the use by the direct addressees of the original sender of this email. If you are not an intended recipient of the original sender (or responsible for delivering the message to such person), you are hereby notified that any review, disclosure, copying, distribution or the taking of any action in reliance of the contents of and attachments to this email is strictly prohibited. If you have received this email in error, please immediately notify the sender at the address shown herein and permanently delete any copies of this email (digital or paper) in your possession.

2 1

OAuth with BaseX, next release
by Christian Grün 29 Aug '12

29 Aug '12

Hi all, if you need an OAuth implementation in XQuery, feel free to check out Andy’s XQuery-OAuth project in GitHub: https://github.com/apb2006/XQuery-OAuth Cheers, Christian PS: BaseX 7.4 will most probably be finalized next month.

1 0

Re: [basex-talk] profiling function execution order
by Christian Grün 29 Aug '12

29 Aug '12

> plz find the sample code below. the particular URI will create a simple new > default xml under 'example' database. This was working fine in the previous > versions, but with the latest release(BaseX74-20120829.045851.zip), its just > hanging and never producing any response. Is there anywhere I need to do > configuration? I’m currently revising the RESTXQ locking mechanism. Do you work with a single BaseX instance, or does "10.18.48.180" refer to another machine? In the first case, your query might not be supported anymore in future versions, because it is conceptually dangerous to call an updating query from a reading query in the same instance. The reason is that your first query could reference database contents that are modified by the send-request() call, which would make them invalid in your query. – The preferred solution here is to use XQuery Update for all your updating operations. If you want to create new databases, we will soon offer a db:create() command. In the second case (if you work with two BaseX instances), I’m not sure what could be the reason. Christian

1 0

profiling function execution order
by Seenivasan Gunabalan 29 Aug '12

29 Aug '12

Dear all, We have an question regarding the prof:current-ns() / prof:current-ms() in basex7.4 beta. our need is to calculate the execution time of the calling function. consider the example below, let $o1 := prof:current-ns() let $resp := http:send-request(<http:request method="GET" href="http://starling.us/cgi-bin/gus_atom_xsl.pl?atom=starling_us_atom.xml!sort_o…" status-only = "false" username = "admin" password = "admin" send-authorization="true"> </http:request>) let $o2 := prof:current-ns() let $v := (validate:xsd($resp[2],"../http/hdatatest/xsd/atom.xsd")) let $o3 := prof:current-ns() return ($o2 - $o1, $o3 - $o2, $v) After spending some time in trying the understand why the timings were not coherent ( validation time is almost 0 when compared to the total time), we noticed that the query plan was modified in the following way, Query: let $o1 := prof:current-ns() let $resp := http:send-request(<http:request method="GET" href="http://starling.us/cgi-bin/gus_atom_xsl.pl?atom=starling_us_atom.xml!sort_o…" status-only = "false" username = "admin" password = "admin" send-authorization="true"> </http:request>) let $o2 := prof:current-ns() let $v := (validate:xsd($resp[2],"../http/hdatatest/xsd/atom.xsd")) let $o3 := prof:current-ns() return ($o2 - $o1, $o3 - $o2, $v) Compiling: - moving for/let clauses Result: let $o1 := prof:current-ns() let $resp := http:send-request(element { Q{http://expath.org/ns/http-client}request } { attribute { method } { "GET" }, attribute { href } { "http://starling.us/cgi-bin/gus_atom_xsl.pl?atom=starling_us_atom.xml!sort_o…" }, attribute { status-only } { "false" }, attribute { username } { "admin" }, attribute { password } { "admin" }, attribute { send-authorization } { "true" } }) let $v := validate:xsd(($resp)[2], "../http/hdatatest/xsd/atom.xsd") let $o2 := prof:current-ns() let $o3 := prof:current-ns() return ($o2 - $o1, $o3 - $o2, $v) Timing: - Parsing: 0.56 ms - Compiling: 0.28 ms - Evaluating: 3929.59 ms - Printing: 0.04 ms - Total Time: 3930.5 ms Result: - Hit(s): 2 Items - Updated: 0 Items - Printed: 15 Bytes Query plan: <QueryPlan> <FLWR> <Let var="$o1"> <FNProf name="current-ns()"/> </Let> <Let var="$resp"> <FNHttp name="send-request(request[,href,[bodies]])"> <CElem> <QNm value="http:request" type="xs:QName"/> <CAttr> <QNm value="method" type="xs:QName"/> <Str value="GET" type="xs:string"/> </CAttr> <CAttr> <QNm value="href" type="xs:QName"/> <Str value="http://starling.us/cgi-bin/gus_atom_xsl.pl?atom=starling_us_atom.xml!sort_o…" type="xs:string"/> </CAttr> <CAttr> <QNm value="status-only" type="xs:QName"/> <Str value="false" type="xs:string"/> </CAttr> <CAttr> <QNm value="username" type="xs:QName"/> <Str value="admin" type="xs:string"/> </CAttr> <CAttr> <QNm value="password" type="xs:QName"/> <Str value="admin" type="xs:string"/> </CAttr> <CAttr> <QNm value="send-authorization" type="xs:QName"/> <Str value="true" type="xs:string"/> </CAttr> </CElem> </FNHttp> </Let> <Let var="$v"> <FNValidate name="xsd(input[,schema])"> <IterPosFilter> <VarRef> <Var name="$resp" id="1"/> </VarRef> <Pos min="2" max="2"/> </IterPosFilter> <Str value="../http/hdatatest/xsd/atom.xsd" type="xs:string"/> </FNValidate> </Let> <Let var="$o2"> <FNProf name="current-ns()"/> </Let> <Let var="$o3"> <FNProf name="current-ns()"/> </Let> <Return> <List> <Arith op="-"> <VarRef> <Var name="$o2" id="2"/> </VarRef> <VarRef> <Var name="$o1" id="0"/> </VarRef> </Arith> <Arith op="-"> <VarRef> <Var name="$o3" id="4"/> </VarRef> <VarRef> <Var name="$o2" id="2"/> </VarRef> </Arith> <VarRef> <Var name="$v" id="3"/> </VarRef> </List> </Return> </FLWR> </QueryPlan> and then after modifying the validation line into the following, we noticed that the plan was not modified and the profile timings are as expected. let $v := (validate:xsd($resp[2],"../http/hdatatest/xsd/atom.xsd"),$o2) Is it possible to skip the profiling functions from query optimization process in order to avoid those kind of issues?. Thanks and kind regards, Seenivasan & Marco

2 2

goat whole milk powder
by Ms. Chocolate (Vitacore Co., Ltd.) 27 Aug '12

27 Aug '12

Ms. Chocolate (Vitacore Co., Ltd.) Dear friend, We are glad to be partner of ‘Meiling’ brand goat milk. Goat whole milk powder FOB $6100/MT, 20KG,25KG per bag, 16MT/20’FCL. Thanks for attention. Best wishes, Ms. Chocolate Vitacore Co., Ltd. Hebei Branch ADD: No.9,Unit 2,Building 17, Fengnanfu, Lubei Street, Tangshan, Hebei, China Tel: 0086 - 131 8018 4008 E-mail: chocolate(a)vitacore.cn Http://www.vitacore.cn

1 0

CPU bottleneck?
by Gilles CARRY 27 Aug '12

27 Aug '12

Hi, I'm currently setting up a prototype to store test log results in a XML DB and let users do reports with XQUERY. I'm experiencing performance issues and am trying to identify where the bottleneck is. The status: - I use a single database. - The computer is dedicated to the DB server. (all clients are on distant machines) - Each applications insert bursts of 100-400 xml nodes in each insert query. Between inserts there is a random delay. (seconds to minutes) - UPDINDEX=true - Autoflush is on - The delays start being unacceptable when CPU average reaches ~40% (on a dual core which means ~80% of a single CPU) - Disk i/o seems fine: iostat says write kB_wrtn/s is about 10% of max iorate supported by disk. (I also tried with an SSD with no better results) - After some trials, it seems the more nodes in the DB the more delays to insert. The delays seem proportional to the number of nodes in the DB. - The delays are also proportional to the number of inserter client application (it looks like the transaction monitor is responsible for this) So I suppose the bottleneck is the CPU. Is there a way to check where in Basex server the bottleneck is? Which functions crunch up all cpu? How can I be sure basex uses both CPU? Is there any chance to have multiple simultaneous write access in future versions? Thank-you, Best regards, Gilles. By the way, the output of 'info database': [BaseX]: info database Database Properties Name: logs2 Size: 202 GB Nodes: 81534781 Documents: 1 Binaries: 0 Timestamp: 24.08.2012 15:07:11 Resource Properties Timestamp: 24.08.2012 15:07:10 Encoding: UTF-8 Whitespace Chopping: ON Indexes Up-to-date: false Text Index: ON Attribute Index: ON Full-Text Index: OFF

2 1

I/O Read bytes information on Windows TaskManager
by Fabrice Etanchaud 24 Aug '12

24 Aug '12

Dear all at BaseX, I'm working with a 30 Go database, containing nearly 10 Millions records. When I iterate on all these record, and get only a few informations on each, I can see in the Windows TaskManager that I/O Read bytes is nearly the database's size... but I don't read all the informations, only a few elements. Do you think this is a 'bug' in the manager, or is BaseX really reading all the data ? Maybe seeking in a file is just considered like reading in that file ? best regards, Fabrice ETANCHAUD Senior Software Engineer edital Berkenlaan 1 B-1831 Brussels, Belgium <skype:fabrice.etanchaud?call> +32 2 716 32 32 general +32 2 716 32 20 fax fetanchaud(a)edital.com corsearch.com <http://www.corsearch.com/> | edital.com <http://www.edital.com/> <outbind://899-00000000E439C83B50BF17409D8C27B7AE0425AF07005AA868536216F 543BAB9422EAEA5338B00000011020900005AA868536216F543BAB9422EAEA5338B00000 13AA36B0000/cid:image001.gif@01CC0F02.2C91B6C0> Confidentiality Notice: This email and its attachments (if any) contain confidential information of the sender. The information is intended only for the use by the direct addressees of the original sender of this email. If you are not an intended recipient of the original sender (or responsible for delivering the message to such person), you are hereby notified that any review, disclosure, copying, distribution or the taking of any action in reliance of the contents of and attachments to this email is strictly prohibited. If you have received this email in error, please immediately notify the sender at the address shown herein and permanently delete any copies of this email (digital or paper) in your possession.

2 1

XQJ: "Error: [BXDB0007] Database 'factbook' is opened by another process" on server is not propagated to Java code?
by Arjan van Bentem 24 Aug '12

24 Aug '12

Hi all, Maybe this is related to http://xqj.net/basex/ instead of BaseX itself, but as I don't know allow me to ask here first. Whenever I have somehow locked a BaseX database, like by using the standalone command line utility or the GUI, then I get something like the following in the BaseX log files: QUERY(1) for $c in collection('factbook')//country[name='Albania']return( replace value of node $c/@population with xs:integer($c/@population + 1)) OK 39.92 ms QUERY(1) OK 0.62 ms FULL(1) Error: [BXDB0007] Database 'factbook' is opened by another process. CLOSE(1) OK 39.68 ms Good. However, I don't see that error in my Java code. The Java code just continues like all was fine. Can I somehow make the Java code fail when this happens? See example below, and the same thing in the attachment. Thanks, Arjan. private static final String DRIVER = "net.xqj.basex.BaseXXQDataSource"; private static final String SELECT = "xs:long(collection('factbook')//country[name='Albania']/(a)population[1]/data(.))"; private static final String UPDATE = "for $c in collection('factbook')//country[name='Albania']" + "return(" + " replace value of node $c/@population with xs:integer($c/@population + 1)" + ")"; @Test public void testUpdate() { try { XQDataSource xqd = (XQDataSource) Class.forName(DRIVER) .newInstance(); XQConnection xqc = xqd.getConnection("admin", "admin"); XQExpression xqe = xqc.createExpression(); XQSequence xqs = xqe.executeQuery(SELECT); xqs.next(); long before = xqs.getLong(); System.out.println("Initial value: " + before); System.out.println("Updating..."); xqs = xqe.executeQuery(UPDATE); xqs = xqe.executeQuery(SELECT); xqs.next(); long after = xqs.getLong(); System.out.println("Resulting value: " + after); xqc.close(); Assert.assertEquals("Update should have succeeded or failed", before + 1, after); } catch (Exception e) { e.printStackTrace(); } }

3 5