corruption when adding documents via the GUI in a almost full database
Dear all at BaseX, Under the GUI (7.5.1 beta), When adding documents in a almost full database (Properties/Resources panel), An error is raised, and then the node count gets negative (Properties/General panel). Is there a way to safely add documents until the maximum node number is reached, Or do I have to add logic based on db:info()/databaseproperties/nodes of the input and destination data ? Best regards, Fabrice
When adding documents in a almost full database (Properties/Resources panel),
An error is raised, and then the node count gets negative (Properties/General panel).
Is there a way to safely add documents until the maximum node number is reached,
Currently, there is no such option, but it might be easy to add – provided that the number of new nodes can be safely pre-calculated for all update operations. We’ll have a look. Best, Christian
Thank you Christian. It will be very useful, mainly to avoid restoring databases that size. Do you still plan to break the 2 147 483 648 nodes limit in version 8 ? Best regards, Fabrice -----Message d'origine----- De : Christian Grün [mailto:christian.gruen@gmail.com] Envoyé : mardi 5 février 2013 23:53 À : Fabrice Etanchaud Cc : basex-talk@mailman.uni-konstanz.de Objet : Re: [basex-talk] corruption when adding documents via the GUI in a almost full database
When adding documents in a almost full database (Properties/Resources panel),
An error is raised, and then the node count gets negative (Properties/General panel).
Is there a way to safely add documents until the maximum node number is reached,
Currently, there is no such option, but it might be easy to add - provided that the number of new nodes can be safely pre-calculated for all update operations. We'll have a look. Best, Christian
Do you still plan to break the 2 147 483 648 nodes limit in version 8 ?
Yes, this is still planned. I can’t give any info, though, if this will happen in Version 8 or later. Best, Christian
Best regards,
Fabrice
-----Message d'origine----- De : Christian Grün [mailto:christian.gruen@gmail.com] Envoyé : mardi 5 février 2013 23:53 À : Fabrice Etanchaud Cc : basex-talk@mailman.uni-konstanz.de Objet : Re: [basex-talk] corruption when adding documents via the GUI in a almost full database
When adding documents in a almost full database (Properties/Resources panel),
An error is raised, and then the node count gets negative (Properties/General panel).
Is there a way to safely add documents until the maximum node number is reached,
Currently, there is no such option, but it might be easy to add - provided that the number of new nodes can be safely pre-calculated for all update operations. We'll have a look.
Best, Christian _______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Dear BaseX Team, dear BaseX users, Corrupting a collection when adding a document past the 2 147 483 648 nodes limit is leading to big backup/restore issues. Are there other BaseX users facing the same problem ? I faced a strange behavior after adding several files (each one sizing around 100 Mo) in a collection without any indexes. Opening the collection was leading to memory overflow. 'optimize all' solved the problem at the end. Could you please explain what is going on behind the scenes ? Do I have to 'optimize all' the collection after each 'add' sequence ? I would like to have your advice on 'add only' collections strategies, regarding both updating and querying. What sounds best to you ? Strategy #1 : For each new documents' set, add it to the current collection (and optimize it all). When collection is full, continue with a new collection : very slow updates because of optimization times. Strategy #2 : For each new documents' set, create a new collection : fast updates. But when it comes to querying, is it better to aggregate results of a few indexes on big collections, or of several indexes on little collections ? Finally, do you think that improving the ADD/db:add functions to let them take a collection path as input could be a desirable feature ? For example : ADD (TO mytargetpath) collection('mysourcecollection/mysourcepath[/mysourcedoc]'). I know I could do that inlining the desired xml in a <add> command, but at a extra serialization cost. In our use case, it would be very useful, as we often try to ingest xml files as soon as possible in the workflow to get rid of raw xml files. In general it would be a great feature to move data from a collection to another without having to export/import xml. Best regards, And once again thank you for your fantastic XNDB. Fabrice Etanchaud Questel-Orbit -----Message d'origine----- De : Fabrice Etanchaud Envoyé : mercredi 6 février 2013 13:54 À : basex-talk@mailman.uni-konstanz.de Objet : RE: [basex-talk] corruption when adding documents via the GUI in a almost full database Thank you Christian. It will be very useful, mainly to avoid restoring databases that size. Do you still plan to break the 2 147 483 648 nodes limit in version 8 ? Best regards, Fabrice -----Message d'origine----- De : Christian Grün [mailto:christian.gruen@gmail.com] Envoyé : mardi 5 février 2013 23:53 À : Fabrice Etanchaud Cc : basex-talk@mailman.uni-konstanz.de Objet : Re: [basex-talk] corruption when adding documents via the GUI in a almost full database
When adding documents in a almost full database (Properties/Resources panel),
An error is raised, and then the node count gets negative (Properties/General panel).
Is there a way to safely add documents until the maximum node number is reached,
Currently, there is no such option, but it might be easy to add - provided that the number of new nodes can be safely pre-calculated for all update operations. We'll have a look. Best, Christian
participants (2)
-
Christian Grün -
Fabrice Etanchaud