Hi Christian,
Thank you so much for the feedback. Yes, we should try to use several separating "collections" instead.
Your advice for our script is very informative, too. Thanks again.
Regards, Kento
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Tuesday, April 24, 2012 11:40 AM To: Tarui, Kento Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] BaseXClient.py raised IOError
Kento san,
Our data consists of hundreds of thousands xml documents and each of them has several thousands of nodes. The error occurred while we added it to a database.
thanks for the feedback. In this case, you probably need to store your data in several database instances. As "databases" are pretty light-weight data structures in BaseX [1], we may rename them to "collections", and add another indirection in a future version to increase the maximum number of supported documents (nodes) per database.
Hope this helps, Christian
A little PS reg. your import script: the "add" command will be faster than "replace" in most cases. Next, when doing bulk updates, it is recommended to turn off the "autoflush" option to improve performance.
[1] http://docs.basex.org/wiki/Databases [2] http://docs.basex.org/wiki/Options#AUTOFLUSH
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Tuesday, April 24, 2012 1:37 AM To: Tarui, Kento Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] BaseXClient.py raised IOError
Hi Kento,
what's the total size of your XML documents? As Mattijs indicated, it may be that you have reached the id limit of 2^31 entries. In this case, you can distribute your data to multiple database instances, all of which can be queried by a single XQuery.
Our documentation contains some statistics on large databases that have been created with BaseX [1].
Hope this helps, your feedback is welcome, Christian
[1] http://docs.basex.org/wiki/Statistics ___________________________
On Mon, Apr 23, 2012 at 5:30 PM, Mattijs Ugen m.ugen@student.utwente.nl wrote:
Is it a coincidence that this value is exactly 2 * 31 - 1 (Integer.MAX_VALUE) ?
java.lang.ArrayIndexOutOfBoundsException: 2147483647
*snip*
Mattijs
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk