Hi Christian,
Thank you for the suggestions! I'm creating my backups through java (actually scala) code but I ended up using an approach where I:
1. Run job:eval with a custom job ID and a query that runs a dummy update followed by a prof:sleep for a long period of time 2. Wait for the job to be running 3. Create the backup 4. Run job:remove to cancel the job
Next time I have a slow few days at work I'll take a closer look at contributing the .tar.zst backup behavior along with tests and documentation :)
Best, Matt
On Thu, Jun 12, 2025 at 6:16 AM Christian Grün christian.gruen@gmail.com wrote:
Hi Matt,
I figured it might be ill-advised, just as my reason to do it might be… :)
…the reason sounds reasonable ;)
Thanks for your offer to contribute code. This is always appreciated, but note in advance that the time consuming part could become me persisting on the notorious 20% percent to ensure consistent support of both archive types for all backup features, to include tests, to update the documentation, etc. ;)
We try to keep the number of code dependencies as small as possible, so the best option would be to make the dependencies optional (see e.g. [1] for how this is done for HTML parsing), and use .tar.zst compression whenever the library is in the classpath.
Other pragmatic approaches:
- Use db:create-backup with compress:false() [2]. This will at least
speed up the time for creating the backup. 2. With XQuery locks [3], you can at least ensure that no other process with the same lock key will be executed in parallel 3. Not sure to recommend this, but it should work: To enforce a write lock, you can specify a dummy update operation on a database that won’t change your data, and do the actual backup with proc:system:
delete node db:get('factbook')//DUMMY, proc:system('zstd', ('-r', db:option('dbpath' || $db-name, '-o', ...)), proc:system('upload', 'to' 's3')
Hope this helps, Christian
[1] https://github.com/BaseXdb/basex/blob/main/basex-core/src/main/java/org/base... [2] https://docs.basex.org/main/Database_Functions#db:create-backup [3] https://docs.basex.org/main/Transaction_Management#xquery_locks
I’m creating database backups manually because I want to compress them
with tar and zstd instead of zip. I also upload them to S3. I was previously creating a backup through BaseX, unzipping it, recompressing it, and then uploading it, but I can achieve the same result in a much faster way by creating and uploading the .tar.zst file by hand. While doing so, I was hoping to lock the database.
As an alternative, I would be happy to contribute code that lets users select how to compress their backups — the .tar.zst approach requires a couple extra dependencies but the compression ratio is worth it for me.
Let me know what you think.
Thanks, Matt
On Wed, Jun 11, 2025 at 4:44 PM Christian Grün christian.gruen@gmail.com wrote:
Hi Matt,
Locking is a low-level mechanism of the database system. It should be avoided to try to influence the behavior from outside.
What would be your reason for (un)locking databases manually?
Best, Christian
Matt Dziuban mrdziuban@gmail.com schrieb am Mi., 11. Juni 2025, 21:55:
Hello everyone,
Is there any function or command I can use to manually lock and unlock a database? Based on the documentation on File-System Locks https://docs.basex.org/12/Transaction_Management#file-system_locks it sounds like maybe just creating an upd.basex file in the database directory would be sufficient?
Thanks in advance, Matt