Hi all,
I’m searching for a way to apply “incremental updates” to a database. By “incremental updates”, I refer to the following situation:
1. Starting point is a database containing several xml resources, e.g.
/coll1/res1.xml /coll1/res2.xml /coll2/res3.xml.
2. The update consists of a file structure like
/coll1/res2.xml /coll2/res4.xml /coll3/res5.xml
- so I want to update the existing resource /coll1/res2.xml and add some additional resources /coll2.res4.xml, /coll3/res5.xml.
The desired result is a structure like this:
/coll1/res1.xml (untouched) /coll1/res2.xml (updated) /coll2/res3.xml (untouched) /coll2/res4.xml (added) /coll3/res5.xml (added)
Using the command line client and the “ADD” command, I’m able to apply the update content in a single step, but (as the documentation predicted) /coll1/res2.xml now appears twice. This isn’t the desired result - /coll1.res2.xml should be overwritten by the content of / coll1.res2.xml from the update.
Using the command line client and the “PUT” command replaces / coll1.res2.xml and adds the additional resource. But it deletes / coll1/res1.xml and /coll2/res3.xml - again, not the desired result, since resources missing in the update should simply be kept untouched.
In my real use case, the initial dataset contains about 18,000 resources, updates may contain hundreds of updated or new resources, so separating ADD and PUT operations manually isn't possible.
A simply but ugly solution would be to keep the initial set of resources in the file system, "apply" the update there and import the complete set of files as new Database. Another solution would be to write a dedicated xquery that checks for each single resource to apply a PUT or ADD operation.
Is there a better way for such a use case?
Florian
Hi Florian,
The most flexible solution is an XQuery script that traverses all files:
let $root := 'local-root-folder/' let $db := 'your-fb' for $path in file:list($root, true()) return db:put($db, doc($root || $path), $path)
If you like, you can do different things if a file exists or not:
let $root := 'local-root-folder/' let $db := 'your-fb' for $path in file:list($root, true()) return if(db:exists($db, $path)) then ( db:put($db, doc($root || $path), $path) ) else ( db:add($db, doc($root || $path), $path) )
Hope this helps, Christian
On Wed, Mar 12, 2025 at 3:34 PM Florian Schmitt < ml-basex-talk@florian-schmitt.net> wrote:
Hi all,
I’m searching for a way to apply “incremental updates” to a database. By “incremental updates”, I refer to the following situation:
- Starting point is a database containing several xml resources, e.g.
/coll1/res1.xml /coll1/res2.xml /coll2/res3.xml.
- The update consists of a file structure like
/coll1/res2.xml /coll2/res4.xml /coll3/res5.xml
- so I want to update the existing resource /coll1/res2.xml and add
some additional resources /coll2.res4.xml, /coll3/res5.xml.
The desired result is a structure like this:
/coll1/res1.xml (untouched) /coll1/res2.xml (updated) /coll2/res3.xml (untouched) /coll2/res4.xml (added) /coll3/res5.xml (added)
Using the command line client and the “ADD” command, I’m able to apply the update content in a single step, but (as the documentation predicted) /coll1/res2.xml now appears twice. This isn’t the desired result - /coll1.res2.xml should be overwritten by the content of / coll1.res2.xml from the update.
Using the command line client and the “PUT” command replaces / coll1.res2.xml and adds the additional resource. But it deletes / coll1/res1.xml and /coll2/res3.xml - again, not the desired result, since resources missing in the update should simply be kept untouched.
In my real use case, the initial dataset contains about 18,000 resources, updates may contain hundreds of updated or new resources, so separating ADD and PUT operations manually isn't possible.
A simply but ugly solution would be to keep the initial set of resources in the file system, "apply" the update there and import the complete set of files as new Database. Another solution would be to write a dedicated xquery that checks for each single resource to apply a PUT or ADD operation.
Is there a better way for such a use case?
Florian
Hi Christian,
thanks a lot, looks fine! I've just added a where clause to filter for files only, leaving out directories:
let $root := '/mnt/baseX_files/iu_test2/' let $db := 'IU_Test' for $path in file:list($root, true()) *where file:is-file($root || $path) * return if(db:exists($db, $path)) then ( db:put($db, doc($root || $path), $path) ) else ( db:add($db, doc($root || $path), $path) )
Works like a charm!
Yours Florian
basex-talk@mailman.uni-konstanz.de