How to recover from failed Mysql schema update, with replication?

Posted by OmerGertel on Server Fault See other posts from Server Fault or by OmerGertel
Published on 2010-04-12T15:18:13Z Indexed on 2010/04/13 1:13 UTC
Read the original article Hit count: 303

Filed under:

mysql

|

database-replication

|

recovery

I have two MySQL servers configured with master-slave replication.

Before we deploy a new application version we:

1) STOP SLAVE

2) Take a MySQL dump of the slave.

However, if a mistake is done during the deployment of the new schema version (a table is dropped by mistake, for example), having the slave intact doesn't help. Our service is write-intensive, so we can't turn it back up until we have a master working.

If we now load the mysql dump back into the master, it will take a long time during which our service remains down.

What is the best-practice to recover from such a mistake? How can I setup the system so I can easily promote the slave, turn on our service and only then tend to the broken database? Mainly, I'm worried with re-syncing the slave and the master after changes are done on the slave.

© Server Fault or respective owner

Related posts about mysql

How to remove MySQL completely with config and library files on ubuntu 12.04 gnome 3.0

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried everything till now: sudo apt-get remove mysql-server mysql-client mysql-common sudo apt-get purge mysql-server mysql-client mysql-common sudo apt-get autoremove and even more commands... But whenever I am trying to locate mysql. I get a no. of files related to mysql command: shell>>… >>> More
mysql: Cannot load from mysql.proc. The table is probably corrupted

as seen on Server Fault - Search for 'Server Fault'
Mysql was started: /usr/bin/mysqld_safe --datadir=/srv/mysql/myDB --log-error=/srv/mysql/logs/mysqld-myDB.log --pid-file=/srv/mysql/pids/mysqld-myDB.pid --user=mysql --socket=/srv/mysql/sockets/mysql-myDB.sock --port=3700 but when I'm trying to do something: ERROR 1548 (HY000) at line 1: Cannot… >>> More
Why is there a /etc/init.d/mysql file on this Slackware machine? How could it have gotten there?

as seen on Server Fault - Search for 'Server Fault'
A client of my IT-consulting service owns a web-development shop. He's been having problems with a Slackware 12.0 server running MySQL 5.0.67. The machine was set up by the client's sysadmin, who left on bad terms. My client no longer employs a sysadmin. As far as I can tell, the only copy of MySQL… >>> More
mysql: Bind on unix socket: Permission denied

as seen on Server Fault - Search for 'Server Fault'
Can't start mysql with: sudo /usr/bin/mysqld_safe --datadir=/srv/mysql/myDB --log-error=/srv/mysql/logs/mysqld-myDB.log --pid-file=/srv/mysql/pids/mysqld-myDB.pid --user=mysql --socket=/srv/mysql/sockets/mysql-myDB.sock --port=3700 120222 13:40:48 mysqld_safe Starting mysqld daemon with databases… >>> More
MySQL – Learning MySQL Online in 6 Hours – MySQL Fundamentals in 320 Minutes

as seen on SQL Authority - Search for 'SQL Authority'
MySQL is one of the most popular database language and I have been recently working with it a lot. Data have no barrier and every database have their own place. I have been working with MySQL for quite a while and just like SQL Server, I often find lots of people asking me if I have a tutorial which… >>> More

Related posts about database-replication

Database replication with composite keys?

as seen on Stack Overflow - Search for 'Stack Overflow'
According to the MySQL documentation on database replication, "All tables replicated using row-based replication must have explicit primary keys." Is it safe to assume that database replication will work with a composite keys? >>> More
Setup database replication between 2 servers.

as seen on Server Fault - Search for 'Server Fault'
Is it possible to setup replication of oracle databases, with one server outside the firewall? If so any suggestions on how to setup would be great. >>> More
ShoreTel Database Replication And Manipulation Of MAXDBQUERIES!

as seen on Article City - Search for 'Article City'
It really doesn?t matter what VoIP system you installed they all generally have one architectural characteristic in common; the configuration database. Depending on the system, you might find a datab... [Author: Peter Buswell - Computers and Internet - May 20, 2010] >>> More
PostgreSQL 9.1 Database Replication Between Two Production Environments with Load Balancer

as seen on Server Fault - Search for 'Server Fault'
I'm investigating different solutions for database replication between two PostgreSQL 9.1 databases. The setup will include two production servers on the cloud (Amazon EC2 X-Large Instances), with an elastic load balancer. What is the typical database implementation for for this type of setup? … >>> More
Database Replication

as seen on Stack Overflow - Search for 'Stack Overflow'
I have tried to follow the steps outlined in http://www.howtoforge.com/mysql_database_replication for Database Replication. I have created the database exampledb, and create some tables and load them with values. But when I execute USE exampledb; FLUSH TABLES WITH READ LOCK; SHOW MASTER STATUS; I… >>> More