In my Mirabel system, I create a link database that records all the links made in a set of documents. This becomes a “where used” index over the content.
We have on the order if 200K links for one content set, so at 0.1 second per link it takes about 7 hours to build this index.
I’m currently doing this in one process that builds the whole index and then stores it in a database. This is failing in hard-to-diagnose ways, for example, because a database has a write lock on it when I go to rename it from it’s temp name to it’s production name (to replace the current production version).
The data is such that I could parallelize the processing but I’m not sure how I would do that in BaseX so that I can safely write to a single database from multiple threads.
The fork-join() docs clearly say “non-updating” functions, so that doesn’t seem to be an option.
I have multiple BaseX HTTP servers running so I could farm processing across them, but I think I would then run into write lock issues.
I could create separate databases for each thread of operation and then combine those at the end—that seems like it might be the best option.
Have I missed anything?
Thanks,
Eliot _____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow