Hello Christian,

we have started working in this small project as I informed you. We have set up server/client problem and we have run many test queries on the standard Xmark database. I have some questions though. For example I run:

for $b in basex:db('standard')/site/open_auctions/open_auction return <increase>{$b/bidder[1]/increase/text()}</increase>

With options -z -r100 and I get:

Parsing: 0.15 ms (avg) Compiling: 2.49 ms (avg) Evaluating: 0.0 ms (avg) Printing: 140.66 ms (avg) Total Time: 143.31 ms (avg) Results: 12000 Items Printed: 69 Bytes Memory: 6354 KB

Do you think this output is normal? I do not understand why printing time is not 0 since I use '-z'. Should we include printing in timing our benchmarks?

In addition what exactly does OPTIMIZE? If we just create a DB based on an .xml file and we dont actually UPDATE anything but we only perform queries, then optimize does not change anything, right? I mean when created a DB is already 'optimized'?

I tried to find all this answers in documentation but I couldn't. Thanks for the help!

Best regards, Antonis

On 06/28/2010 01:34 AM, Christian Grün wrote:

Dear Antonis,

thanks for your e-mail and your interest to use BaseX for your class.

We have performed numerous comparisons with other XML databases by ourselves, but your results could include helpful hints as well, so your focus sounds fine to us. Keep in mind, however, that the given time frame is very tough, as long as you haven't done serious benchmarking before: you will most likely encounter side effects, which can lead to completely wrong interpretations of performance results.

As first hint, I'd recommend you to use the latest BaseX version and the client/server architecture for all tests, and the -v option to display the BaseX timing output:

$ java -Xmx1g -cp basex-6.1.7.jar org.basex.BaseXServer BaseX 6.1.7 [Server] Server was started.

$ java -cp basex-6.1.7.jar org.basex.BaseXClient -Uadmin -Padmin -v query.xq

<xml/> Parsing: 0.04 ms Compiling: 0.02 ms Evaluating: 0.01 ms Printing: 0.05 ms Total Time: 0.19 ms Results: 1 Item Printed: 42 Bytes Memory: 2180 KB

You might use the hidden "-r" option to increase the number of runs:

$ java -cp basex-6.1.7.jar org.basex.BaseXClient -Uadmin -Padmin -r100 -v query.xq

<xml/> Parsing: 0.02 ms (avg) Compiling: 0.0 ms (avg) Evaluating: 0.0 ms (avg) Printing: 0.02 ms (avg) Total Time: 0.04 ms (avg) Results: 1 Item Printed: 42 Bytes Memory: 2604 KB

You can also use BaseX commands:

$ java -cp basex-6.1.7.jar org.basex.BaseXClient -Uadmin -Padmin -c "set info on; set runs 100; run query.xq" …

Check out http://basex.org/server for details on using c/s. Next, the following post from Michael Kay on benchmarking is well worth reading:

http://x-query.com/pipermail/talk/2009-October/003613.html

Best, Christian

On Sun, Jun 27, 2010 at 4:06 PM, Antonis Thomas iorulezz@gmail.com wrote:

...

Hello all,

we are two master students at the university of Utrecht. We have a DB Architecture class that includes a small project (15%*7.5 ECTS= 1.125 ECTS). So, BaseX looks like a good chance to work on such a project.

Our idea so far is based on some benchmarking experiments and small summary on how it works (maybe its relation with Monet) including any experiments we run (5-6 pages). Do you have anything better for us to do (maybe something you can get advantage of yourselves) or any guidelines/hints/whatever?

Take into account that out time limit is approximately a week (full-time) and that we cannot do more work that what deserves 2.25ECTS for the 2 of us.

I look forward to an answer (so we can decide and start working -the deadline is 9/7-.

Best regards,

Antonis Thomas Information & Computing Sciences Utrecht University Utrecht, The Netherlands

---

BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

Hello Christian,

I would indeed suggest to use the total times for benchmarking, and to minimize the time for printing the result. You could e.g. move all output to /dev/null, or to a temporary file on disk, which is ok with the performance of today's hard disk. To play fair with the competition, you could omit the -z flag for BaseX.

This is fine. We will run some query examples based on Xmark. We will compare the results of several queries with eXist. It will be similar to here (unfortunately smaller scale): http://monetdb.cwi.nl/XQuery/Benchmark/XMark/

I am thinking to run it on three different sizes of Xmark generated database: 1.1MB, 11MB, 110MB. Any hints/suggestions/requests here? Note that both systems are running on client/server mode.

So far our mini-report-paper structure will have the following structure:

. small summary of how BaseX works (based on first couple research papers): 2-3 pages incl some intro . Small summary of how to benchmark XML DBs (based on XMark relative papers): 1-2 pages . Results of mini-benchmark and comparison with eXist: 1-2 pages _____________________________Total: 4-7 p

Any hints/suggestions/requests here?

thanks again, Antonis

Hello Christian,

So first of all (due to my struggling with my personal research) we got extension to Sunday night for our report.

Regarding the actual experiments, I managed to run 1.1MB, 11MB, 111MB, 555MB Xmark instances for baseX, Monet, eXist. The queries are chosen from this set: http://www.ins.cwi.nl/projects/xmark/Assets/xmlquery.txt Since there was not enough time to run them all, I chose queries 1,3,6,8,13,14 as an indicative subset.

In addition I tried to run on 1g instance and baseX was the only one to successfully run all the queries. For both eXist & Monet my 3gb of ram was not enough to open such a large instance. Do you think I should report that? (I still I am not exactly sure that I configured Monet the best way possible.)

In general, baseX seems to win the race by far. That's besides query 8 that performs an expensive join, at which monet is better in the large dbs. Of course my computer is not the best to run benchmarks on -core2duo 2.4, with 3g of ram and 64bit linux- but it is like a small business that runs a db and I guess 0.5gb of Xml db is already enough for that.

If you want to review the benchmark results before the final version -maybe you spot an anomaly and some query needs rerun- ask for it and I will send you asap.

regards, Antonis

On 07/06/2010 08:18 PM, Christian Grün wrote:

To get new insights, it would be nice if you could include some other DBs as well, but the given time restrictions won't probably allow to do more.

You could think about including at least 1GB instances (as a side note, we'd created XMark instances up to 55GB).

On 07/09/2010 12:17 PM, Christian Grün wrote:

It might be interesting to swap the order in which you're running the queries. A query might be executed faster in Java if it's executed at the end of the test, as the JIT compiler will delay initial processes; garbage collection might influence runtimes as well, etc. If you don't restart the database server every time you run new queries, the differences should be less significant.

I wasn't turning up and down the servers after each query. What I did every time before testing each query I was running a couple of random ones. This way the initial execution of each query took more time and it smoothed out the more I executed the same one. In addition I didnt use the '-r' of basex because the results were always much faster -is there any kind of pipelining teke place there?- and since no other system had such an option I went with normal runs. Usually basex wanted 10 executions of a query in a row to reach its minimum -then it flunctuates reasonably- similarly for exist. Monet was more stable in the sense that it could reach its minimum sometimes from the first query. That might be differences in C and Java though. Note that I had to compile Monet on my system, no binary worked for me.

Feel free to send me a preview.

I am including the results. The files are simple CSVs. There is an info file that explains a bit.

We are at the stage of finishing our writing. Again, please do not expect much from our project, it's only a small one and databases is not the main interest for neither of us. Although I can say it was kind of enjoyable!

greetings, Antonis

Thanks for the details. Your test environment sounds reasonable, as well as the results (mabe except for the xmall.txt-results, which seem to be somewhat influenced by other factors. Side effects are easy to see if the results are visualized on a logarithmic scale). If you're using the -r option in BaseX, the results will always be much faster, as the same code is run again and again, and numerous side-effects are this minimized. In fact, the results wouldn't be comparable if the option had been used; it should only be used for tests inside BaseX.

Just a hint: in the small.txt file, one result for eXist is missing.

Looking forward to your final results, Christian

...

...

It might be interesting to swap the order in which you're running the queries. A query might be executed faster in Java if it's executed at the end of the test, as the JIT compiler will delay initial processes; garbage collection might influence runtimes as well, etc. If you don't restart the database server every time you run new queries, the differences should be less significant.

I wasn't turning up and down the servers after each query. What I did every time before testing each query I was running a couple of random ones. This way the initial execution of each query took more time and it smoothed out the more I executed the same one. In addition I didnt use the '-r' of basex because the results were always much faster -is there any kind of pipelining teke place there?- and since no other system had such an option I went with normal runs. Usually basex wanted 10 executions of a query in a row to reach its minimum -then it flunctuates reasonably- similarly for exist. Monet was more stable in the sense that it could reach its minimum sometimes from the first query. That might be differences in C and Java though. Note that I had to compile Monet on my system, no binary worked for me.

Do you think the best option to visualize the results is in logarithmic scale anyways? And if yes how do you justify this?

Logarithm scales are particularly useful if scalability is tested, such as in your scenario. You might check out some online information first, such as e.g.:

http://en.wikipedia.org/wiki/Logarithmic_scale

Christian