August 2012 - BaseX-Talk - mailman.uni-konstanz.de

CPU bottleneck?
by Gilles CARRY 27 Aug '12

27 Aug '12

Hi, I'm currently setting up a prototype to store test log results in a XML DB and let users do reports with XQUERY. I'm experiencing performance issues and am trying to identify where the bottleneck is. The status: - I use a single database. - The computer is dedicated to the DB server. (all clients are on distant machines) - Each applications insert bursts of 100-400 xml nodes in each insert query. Between inserts there is a random delay. (seconds to minutes) - UPDINDEX=true - Autoflush is on - The delays start being unacceptable when CPU average reaches ~40% (on a dual core which means ~80% of a single CPU) - Disk i/o seems fine: iostat says write kB_wrtn/s is about 10% of max iorate supported by disk. (I also tried with an SSD with no better results) - After some trials, it seems the more nodes in the DB the more delays to insert. The delays seem proportional to the number of nodes in the DB. - The delays are also proportional to the number of inserter client application (it looks like the transaction monitor is responsible for this) So I suppose the bottleneck is the CPU. Is there a way to check where in Basex server the bottleneck is? Which functions crunch up all cpu? How can I be sure basex uses both CPU? Is there any chance to have multiple simultaneous write access in future versions? Thank-you, Best regards, Gilles. By the way, the output of 'info database': [BaseX]: info database Database Properties Name: logs2 Size: 202 GB Nodes: 81534781 Documents: 1 Binaries: 0 Timestamp: 24.08.2012 15:07:11 Resource Properties Timestamp: 24.08.2012 15:07:10 Encoding: UTF-8 Whitespace Chopping: ON Indexes Up-to-date: false Text Index: ON Attribute Index: ON Full-Text Index: OFF

2 1

I/O Read bytes information on Windows TaskManager
by Fabrice Etanchaud 24 Aug '12

24 Aug '12

Dear all at BaseX, I'm working with a 30 Go database, containing nearly 10 Millions records. When I iterate on all these record, and get only a few informations on each, I can see in the Windows TaskManager that I/O Read bytes is nearly the database's size... but I don't read all the informations, only a few elements. Do you think this is a 'bug' in the manager, or is BaseX really reading all the data ? Maybe seeking in a file is just considered like reading in that file ? best regards, Fabrice ETANCHAUD Senior Software Engineer edital Berkenlaan 1 B-1831 Brussels, Belgium <skype:fabrice.etanchaud?call> +32 2 716 32 32 general +32 2 716 32 20 fax fetanchaud(a)edital.com corsearch.com <http://www.corsearch.com/> | edital.com <http://www.edital.com/> <outbind://899-00000000E439C83B50BF17409D8C27B7AE0425AF07005AA868536216F 543BAB9422EAEA5338B00000011020900005AA868536216F543BAB9422EAEA5338B00000 13AA36B0000/cid:image001.gif@01CC0F02.2C91B6C0> Confidentiality Notice: This email and its attachments (if any) contain confidential information of the sender. The information is intended only for the use by the direct addressees of the original sender of this email. If you are not an intended recipient of the original sender (or responsible for delivering the message to such person), you are hereby notified that any review, disclosure, copying, distribution or the taking of any action in reliance of the contents of and attachments to this email is strictly prohibited. If you have received this email in error, please immediately notify the sender at the address shown herein and permanently delete any copies of this email (digital or paper) in your possession.

2 1

XQJ: "Error: [BXDB0007] Database 'factbook' is opened by another process" on server is not propagated to Java code?
by Arjan van Bentem 24 Aug '12

24 Aug '12

Hi all, Maybe this is related to http://xqj.net/basex/ instead of BaseX itself, but as I don't know allow me to ask here first. Whenever I have somehow locked a BaseX database, like by using the standalone command line utility or the GUI, then I get something like the following in the BaseX log files: QUERY(1) for $c in collection('factbook')//country[name='Albania']return( replace value of node $c/@population with xs:integer($c/@population + 1)) OK 39.92 ms QUERY(1) OK 0.62 ms FULL(1) Error: [BXDB0007] Database 'factbook' is opened by another process. CLOSE(1) OK 39.68 ms Good. However, I don't see that error in my Java code. The Java code just continues like all was fine. Can I somehow make the Java code fail when this happens? See example below, and the same thing in the attachment. Thanks, Arjan. private static final String DRIVER = "net.xqj.basex.BaseXXQDataSource"; private static final String SELECT = "xs:long(collection('factbook')//country[name='Albania']/(a)population[1]/data(.))"; private static final String UPDATE = "for $c in collection('factbook')//country[name='Albania']" + "return(" + " replace value of node $c/@population with xs:integer($c/@population + 1)" + ")"; @Test public void testUpdate() { try { XQDataSource xqd = (XQDataSource) Class.forName(DRIVER) .newInstance(); XQConnection xqc = xqd.getConnection("admin", "admin"); XQExpression xqe = xqc.createExpression(); XQSequence xqs = xqe.executeQuery(SELECT); xqs.next(); long before = xqs.getLong(); System.out.println("Initial value: " + before); System.out.println("Updating..."); xqs = xqe.executeQuery(UPDATE); xqs = xqe.executeQuery(SELECT); xqs.next(); long after = xqs.getLong(); System.out.println("Resulting value: " + after); xqc.close(); Assert.assertEquals("Update should have succeeded or failed", before + 1, after); } catch (Exception e) { e.printStackTrace(); } }

3 5

how to speed up //myTag requests ?
by Fabrice Etanchaud 23 Aug '12

23 Aug '12

dear all at Basex, In a /a/b/c example schema, Is there any index that could speed up //c request ? best regards, Fabrice ETANCHAUD Senior Software Engineer edital Berkenlaan 1 B-1831 Brussels, Belgium <skype:fabrice.etanchaud?call> +32 2 716 32 32 general +32 2 716 32 20 fax fetanchaud(a)edital.com corsearch.com <http://www.corsearch.com/> | edital.com <http://www.edital.com/> <outbind://899-00000000E439C83B50BF17409D8C27B7AE0425AF07005AA868536216F 543BAB9422EAEA5338B00000011020900005AA868536216F543BAB9422EAEA5338B00000 13AA36B0000/cid:image001.gif@01CC0F02.2C91B6C0> Confidentiality Notice: This email and its attachments (if any) contain confidential information of the sender. The information is intended only for the use by the direct addressees of the original sender of this email. If you are not an intended recipient of the original sender (or responsible for delivering the message to such person), you are hereby notified that any review, disclosure, copying, distribution or the taking of any action in reliance of the contents of and attachments to this email is strictly prohibited. If you have received this email in error, please immediately notify the sender at the address shown herein and permanently delete any copies of this email (digital or paper) in your possession.

2 1

Index auto update
by Gilles CARRY 22 Aug '12

22 Aug '12

Hello, I'm a little bit confused with indexes auto update mechanism. Having created a DB with UPDINDEX=true, and after inserting nodes, 'info database' shows : Indexes Up-to-date: false Text Index: ON Attribute Index: ON Full-Text Index: OFF whereas UPDINDEX=true. After issuing 'optimize', Up-to-date come back to true and FT index to ON. Does this mean index are not automatically updated if 'optimize' is not run? I use BaseX73-20120528.232116.jar Is there any comprehensive documentation about index update mechanism? Thank-you, Gilles.

2 1

Basex editing performance test
by Yoann Maingon 21 Aug '12

21 Aug '12

Hi, Continuing on my recent question about using multiple databases, we've been running some performance test on basex. I don't have the detail of the computer that hosted these test but they were all done on the same computer. Test was to edit a node's value from a php script in a database a million time, the database being choosen randomly in X amount of databases created. (this simulates multiple users accessing our app) *2 DB* Total time : 1517.95 seconds for 1000000 iterations with 2 DB Min time : 1.17 ms with database 1 Max time : 704.72 ms with database 0 Mean time : 1.52 ms *10 000 DB* Total time : 1515.75 seconds for 1000000 iterations with 10 000 DB Min time : 1.21 ms with database 8879 Max time : 645.16 ms with database 3822 Mean time : 1.52 ms *20 000 DB* Total time : 1680.29 seconds for 1000000 iterations with 20 000 DB Min time : 1.18 ms with database 3749 Max time : 285.49 ms with database 6518 Mean time : 1.68 ms *40 000 DB* Total time : 1813.53 seconds for 1000000 iterations with 40 000 DB Min time : 1.04 ms with database 786 Max time : 212.2 ms with database 6949 Mean time : 1.81 ms *80 000 db - test 1* Total time : 24728.94 seconds for 1000000 iterations with 80 000 DB Min time : 1.16 ms with database 25693 Max time : 2433.44 ms with database 22021 Mean time : 24.73 ms *80 000 db - test 2* Total time : 18661.74 seconds for 1000000 iterations with 80 000 DB Min time : 1.68 ms with database 5979 Max time : 1936.4 ms with database 30239 Mean time : 18.66 ms We can just see that there is an important difference from 40k to 80k databases. We haven't checked other mean method to see if this was due to a few edit actions. Does any one tried to have many databases and at some point reached a certain limit? In oder to do server sizing what is key for these actions? processor?ram? Thanks for your help! -- Yoann Maingon mydatalinx 0664324966

2 1

large number of xml files
by sateesh 21 Aug '12

21 Aug '12

Hi Basex, I have a requirement of querying on large number of xml files some where around 10,000 xml files , I have written the query and while executing the query it is taking huge amount of memory and time some where around 700mb memory and time around 4 -5 minutes .Is there a way to execute the query with less memory and with in short time. Thanks & Regards Sateesh.A

2 9

the stack - what are your current favorites?
by Colin McEnearney 21 Aug '12

21 Aug '12

Dear basex xml-ers, What is your stack of choice these days? As a newcomer, interested in learning how to develop for the web using x* from end to end, it's tricky to see how quickly this space is evolving, and what is the latest and greatest. Lots of info on the basex, exist, and marklogic sites, + Adam Retter's site, + some very evangelical blog posts on XRX from 2007-09... And more recently the implementation of restxq by basex and exist... And XQuery in the browser. So many possibilities, but no one stack/framework to rule them all... ? Very interested to hear what basex users are developing with. Thanks, Colin

1 0

restxq
by Colin McEnearney 20 Aug '12

20 Aug '12

This is a newbie email. I've been using basex for some time for its xquery engine (which is awesome) but never in production or for web development. I have some small projects coming up and looking at it for the db side of a web app I just noticed RESTXQ - looks really cool! I'm not really a developer but I use xquery every day at work and so would love to use it - exclusively if possible. Does restxq mean that you can serve a site with no client (php, ruby, etc) in between the db and the html? basex <-> html That would be so simple and fun, and quite useful for a blog or other not-too-huge type of thing. Or is that totally crazy for security reasons? Colin

3 3

xquery grouping
by sateesh 20 Aug '12

20 Aug '12

Hi Basex, I have written a query on an xml file which contains 13000 nodes ,query does some grouping action on that xml,when I run that query it is taking huge amount of memory(1.5gb) and also processing time. PFA of xml file and the query.I have removed the nodes from xml for easy transfer,but in the original xml there will be around 13k <row/> tags Please suggest whether is there a way to reduce the memoru consumption. Thanks & Regards Sateesh.A

1 0