I've been doing some more thinking about this.
As the database and the back-ups are in the same directory, I can sync them all using rsync in one go.
In the vast majority of cases, the active database will be fine; only if the server failure occurs during the rsync will it be corrupt. Our application is much heavier on reads than writes, so the data changes relatively slowly.
My plan is therefore to create a back-up relatively often (say once an hour) and rsync the database more frequently (say once every five minutes).
If the Master fails, I will start the slave, check the database. If the database is corrupt, I want to restore the back-up.
This leaves me with the problem, how do I detect a corrupt database and automatically do a restore?
I've tried deliberately corrupting some of the database files (on a test system). Depending on what I corrupt, most commands run fine; OPTIMIZE seems to be the most prone to failing.
Any suggestions on how to detect a corrupt database?
On 24/07/2013 12:01, Andrew Miller wrote:
Hi Folks,
We are looking at deploying BaseX as a resilient deployment. If the BaseX server fails, then a stand-by server is to take over. My problem is how to replicate the data from the active Master to the passive slave.
I found a thread from July 2012 proposing replicating the data directory using rsync. However, I have concerns about this.
Firstly, if BaseX is writing to the database at the time of failure then the database on the Master would be corrupted, wouldn't it? Secondly, rsync does not guarantee a consistent file set is transferred. If BaseX updates the database during the rsync transfer, or the server fails during the rsync transfer, the database on the Slave would be corrupt, wouldn't it?
I am considering using a cron job to do a "CREATE BACKUP" on the database at regular intervals and push the resulting zip file to the slave. If the Master fails during this process, we can detect the failure (at best we get an incomplete zip file, which will not be valid). If this happens we can fall back to the previous backup.
If the Master fails, we detect this (using monit), start BaseX on the slave and do a database restore.
I think this should give us a resilient deployment. In the worse case scenario (master dying during a replication) we lose changes since the last back-up, but at least we get a working database.
Does this sound like a sound strategy? When I do a database restore, does BaseX automatically cope if the latest backup is corrupt?