Hi, I wanted to understand more on how deleting documents impact the storage layout. I did go through https://docs.basex.org/wiki/Storage_Layout but didn't find all the required information. I observed that size of files like "txt.basex" and "atv.basex" doesn't shrink on deleting documents. Is that expected?
PS: Thanks for the great product.
Thank you, Deepak
Hi Deepak,
Some more time would have to be spent to explain this in full detail.
In a nutshell, deleted space is not immediately reduced to a minimum size because inserts are more regular. If you want to minimize your database structure after extensive updates, you can call OPTIMIZE (ALL).
Hope this helps, Christian
On Mon, Feb 5, 2024 at 1:09 PM Deepak Dinakara deepukalmane@gmail.com wrote:
Hi, I wanted to understand more on how deleting documents impact the storage layout. I did go through https://docs.basex.org/wiki/Storage_Layout but didn't find all the required information. I observed that size of files like "txt.basex" and "atv.basex" doesn't shrink on deleting documents. Is that expected?
PS: Thanks for the great product.
Thank you, Deepak
Hi Christian,
Thank you for such a quick turnaround. It's indeed well documented but would be great to have these aspects also covered in detail. I avoided "OPTIMIZE" since it caused OOM issues and I was fine without the index which looks like a double edged sword : ). QQ, Is there some stat on how much RAM is needed for maintaining basex DB of size 'X'GB (with regular inserts and delete) so that "optimize" could be called without worrying about OOM?
-Deepak
On Mon, Feb 5, 2024 at 5:42 PM Christian Grün christian.gruen@gmail.com wrote:
Hi Deepak,
Some more time would have to be spent to explain this in full detail.
In a nutshell, deleted space is not immediately reduced to a minimum size because inserts are more regular. If you want to minimize your database structure after extensive updates, you can call OPTIMIZE (ALL).
Hope this helps, Christian
On Mon, Feb 5, 2024 at 1:09 PM Deepak Dinakara deepukalmane@gmail.com wrote:
Hi, I wanted to understand more on how deleting documents impact the storage layout. I did go through https://docs.basex.org/wiki/Storage_Layout but didn't find all the required information. I observed that size of files like "txt.basex" and "atv.basex" doesn't shrink on deleting documents. Is that expected?
PS: Thanks for the great product.
Thank you, Deepak
I avoided "OPTIMIZE" since it caused OOM issues and I was fine without the index which looks like a double edged sword : ).
If you don’t need the indexes, you can disable them when running db:optimize, and setting textindex and attrindex to false.
QQ, Is there some stat on how much RAM is needed for maintaining basex DB
of size 'X'GB (with regular inserts and delete) so that "optimize" could be called without worrying about OOM?
Hardly possible to say in general; it depends a lot on the “regular inserts and deletes” ;) If you cannot solve the optimization problem, feel free to share the OOM stack trace with us.
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
Deepak Dinakara