How to recover from failed Mysql schema update, with replication?
- by OmerGertel
I have two MySQL servers configured with master-slave replication.
Before we deploy a new application version we:
1) STOP SLAVE
2) Take a MySQL dump of the slave.
However, if a mistake is done during the deployment of the new schema version (a table is dropped by mistake, for example), having the slave intact doesn't help. Our service is write-intensive, so we can't turn it back up until we have a master working.
If we now load the mysql dump back into the master, it will take a long time during which our service remains down.
What is the best-practice to recover from such a mistake?
How can I setup the system so I can easily promote the slave, turn on our service and only then tend to the broken database? Mainly, I'm worried with re-syncing the slave and the master after changes are done on the slave.