…thanks for the summary. I am glad to hear you managed to restore your data. Out of space issues can be tricky to catch, in particular in cases where inserted data is streamed into the database (in those cases, it will be difficult to guess in advance how many bytes the database will grow). Some large DBMS simply tackle that by rejecting any update operation if there is not at leas 10% or 20% free disk space. This seemed too patronizing to us, and I am not sure it’s the best option. Maybe we could check the disk space before the actual updates are performed (i.e., the pending update list is processed), and reject operations if an out-of-space situation will be ovious. I’ll have more thoughts on it. For productive systems, we use and recommend OS-wide monitoring solutions, as low disk space can also be caused by other applications that are running on the same system. This has at least protected us from losing any productive data from the very start of our company (…fingers crossed ;·). All the best, Christian On Mon, May 12, 2025 at 5:13 PM Omar Siam <Omar.Siam@oeaw.ac.at> wrote:

So we think we got the data back. The people that created it will check in the next few days.

The reason for these problems almost certain an out of diskspace situation. I don't know if this is something that BaseX could be better protected against in the future. It would certainly be hard to incorporate in automatet tests.

I think this was just luck and maybe it wont work for anyone who tries that for their data.

Here is what I did: * I got the source code for BaseX 9 branch and had a look what hapens at org.basex.data.DiskData.write(DiskData.java:144). * This pointed me to the inf.basex file. inf.basex ins only 319 bytes. * I compared that to a (way to old) backup of that same database where inf.basex is 63569 bytes. * So I replaced that too small inf.basex file with the old version. * I tried to open the db in basex running the IDE debugger. I could do a "xquery ." until it stopped with a Util.notExpected Data Access out of bounds exception. * I disabled that check in TabelDiskAccess.java and used "xquery ." on the open database again which worked. * I did an export to some directory and the result had only one obvieous defect: a missing attribute name on the root tag.

So maybe there is a way to improve the redundancy of the database structure files like this inf.basex? Or have a tool that can recreate it from other files?

We activated an external backup process today so hopefully this will never happen again.

Best regards

Am 12.05.2025 um 14:19 schrieb Christian Grün:

Hi Omar,

I am sorry to hear that. BaseX provides some basic recovery mechanisms, but it does not cover all cases that may happen (and of course it is always too late to give the advise to do regular backups when a failure occurs…).

If OPTIMIZE ALL fails, a simple OPTIMIZE works sometimes, but I guess not in this case?

In general it's a good sign if INSPECT reports no errors. I could have a look at the database this week (for free, as I cannot promise anything).

Best, Christian

Omar Siam <Omar.Siam@oeaw.ac.at> schrieb am Mo., 12. Mai 2025, 14:04:

...

Hi!

We had some serious problems with our storage last week and unfortunately that killed one of our important BaseX databases.

Is there any way to restore the data?

Does BaseX GmbH provide paid data rescue services?

The symptoms are:

There is data in that database:

Mode LastWriteTime Length Name ---- ------------- ------ ---- -a--- 12.05.2025 13:16 25214377 atv.basex -a--- 12.05.2025 13:16 451580 atvl.basex -a--- 12.05.2025 13:16 106065 atvr.basex -a--- 12.05.2025 13:16 131 ftxx.basex -a--- 12.05.2025 13:16 449615 ftxy.basex -a--- 12.05.2025 13:16 697581 ftxz.basex -a--- 12.05.2025 13:59 319 inf.basex -a--- 12.05.2025 13:16 39 pth.basex -a--- 12.05.2025 13:16 28 swl.basex -a--- 12.05.2025 13:16 29798400 tbl.basex -a--- 12.05.2025 13:16 23650 tbli.basex -a--- 12.05.2025 13:16 693232 tokl.basex -a--- 12.05.2025 13:16 105495 tokr.basex -a--- 12.05.2025 13:16 53480811 txt.basex -a--- 12.05.2025 13:16 669153 txtl.basex -a--- 12.05.2025 13:16 166050 txtr.basex -a--- 12.05.2025 13:59 0 upd.basex

Inspect reports 0 documents.

Checking main table (0 nodes): - 0 invalid node kinds - 0 invalid parent references - 0 wrong parent/descendant relationships No inconsistencies found. 'dc_shawi_eng_000' inspected in 0.63 ms.

Optimize all yields:

Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 9.3.2 Java: Azul Systems, Inc., 21.0.4 OS: Windows 11, amd64 Stack Trace: java.lang.NullPointerException: Cannot invoke "org.basex.index.path.PathIndex.write(org.basex.io.out.DataOutput)" because "this.paths" is null at org.basex.data.DiskData.write(DiskData.java:144) at org.basex.data.DiskData.close(DiskData.java:160) at org.basex.core.Datas.unpin(Datas.java:52) at org.basex.core.cmd.Close.close(Close.java:45) at org.basex.core.cmd.OptimizeAll.optimizeAll(OptimizeAll.java:126) at org.basex.core.cmd.OptimizeAll$1.run(OptimizeAll.java:42) at org.basex.core.cmd.ACreate.update(ACreate.java:90) at org.basex.core.cmd.OptimizeAll.run(OptimizeAll.java:38) at org.basex.core.Command.run(Command.java:257) at org.basex.core.Command.execute(Command.java:93) at org.basex.api.client.LocalSession.execute(LocalSession.java:132) at org.basex.api.client.Session.execute(Session.java:36) at org.basex.core.CLI.execute(CLI.java:92) at org.basex.core.CLI.execute(CLI.java:76) at org.basex.BaseX.console(BaseX.java:177) at org.basex.BaseX.<init>(BaseX.java:152) at org.basex.BaseX.main(BaseX.java:43)

Best regards

-- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295omar.siam@oeaw.ac.at | www.oeaw.ac.at/acdh