Database 'name' is being updated, or update was not completed.
I have some processes that update databases, taking a long time to do so. I would like other processes to be able to read the last complete snapshot while this happens. Is that possible? Jonathan
The approach I’ve been taking is to have two databases: a “production” database that is used for reads by the consuming code (i.e, RESTXQ-served web pages that provide reports or views of the content) and a “temp” database where updates happen. I then swap the temp for the production when the data processing is done. If I understand your requirement it sounds like you’d need a query that does the work to make a “snapshot”, commit that state, copy it to the production database, and then run the query to make the next snapshot, etc. I use two or more BaseX instances to do the database processing, all sharing the same data directory (and thus the same set of databases). The main instance, running on the default ports, serves the web application and only reads from the databases. The “worker” instances, started on other ports of course, do the writing to the temp databases. The main logistical challenge I’ve faced is how to orchestrate the database swapping and data processing. If your set of database names is static, then it’s easy to do with simple basex scripts that run individual XQueries, i.e.: DROP DB temp COPY prod temp XQUERY (: Call to code that does some work to put data in temp :) ALTER prod prod_backup ALTER temp prod My challenge has been that I have a number of databases created dynamically, so I need a bit more flexibility, thus my work to do the orchestration with BaseX jobs directly within XQuery. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: BaseX-Talk <basex-talk-bounces@mailman.uni-konstanz.de> on behalf of Jonathan Robie <jonathan.robie@gmail.com> Date: Saturday, February 19, 2022 at 3:03 PM To: BaseX <basex-talk@mailman.uni-konstanz.de> Subject: [basex-talk] Database 'name' is being updated, or update was not completed. [External Email] I have some processes that update databases, taking a long time to do so. I would like other processes to be able to read the last complete snapshot while this happens. Is that possible? Jonathan
I’ve created the GitHub project ekimbernow/basex-orchestration: https://github.com/ekimbernow/basex-orchestration This contains the orchestration module and a “database administration” module that serves as a test bed for orchestration and as a set of utilities for manipulating databases with error checking and logging. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: Eliot Kimber <eliot.kimber@servicenow.com> Date: Sunday, February 20, 2022 at 7:55 AM To: Jonathan Robie <jonathan.robie@gmail.com>, BaseX <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Database 'name' is being updated, or update was not completed. The approach I’ve been taking is to have two databases: a “production” database that is used for reads by the consuming code (i.e, RESTXQ-served web pages that provide reports or views of the content) and a “temp” database where updates happen. I then swap the temp for the production when the data processing is done. If I understand your requirement it sounds like you’d need a query that does the work to make a “snapshot”, commit that state, copy it to the production database, and then run the query to make the next snapshot, etc. I use two or more BaseX instances to do the database processing, all sharing the same data directory (and thus the same set of databases). The main instance, running on the default ports, serves the web application and only reads from the databases. The “worker” instances, started on other ports of course, do the writing to the temp databases. The main logistical challenge I’ve faced is how to orchestrate the database swapping and data processing. If your set of database names is static, then it’s easy to do with simple basex scripts that run individual XQueries, i.e.: DROP DB temp COPY prod temp XQUERY (: Call to code that does some work to put data in temp :) ALTER prod prod_backup ALTER temp prod My challenge has been that I have a number of databases created dynamically, so I need a bit more flexibility, thus my work to do the orchestration with BaseX jobs directly within XQuery. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow> From: BaseX-Talk <basex-talk-bounces@mailman.uni-konstanz.de> on behalf of Jonathan Robie <jonathan.robie@gmail.com> Date: Saturday, February 19, 2022 at 3:03 PM To: BaseX <basex-talk@mailman.uni-konstanz.de> Subject: [basex-talk] Database 'name' is being updated, or update was not completed. [External Email] I have some processes that update databases, taking a long time to do so. I would like other processes to be able to read the last complete snapshot while this happens. Is that possible? Jonathan
participants (2)
-
Eliot Kimber -
Jonathan Robie