Replacing a faulty disk in a software RAID1 array on Linux

Tags: linux, raid1, software raid, mdstat, md0
Last update: Jan 2021

Assumptions

We have a running software RAID1 at moint point /dev/md0 consisting of the following partitions:

• /dev/sda1
• /dev/sdb1

Showing details

You can get some detailed information about the fitness of /dev/md0 by issuing:

cat /proc/mdstat

An exemplary entry looks like this:

Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
      136448 blocks [2/2] [UU]

and tells us that

• it’s a raid1 array
• md0 consists of sda1 (first part) and sdb1 (second part)
• the raid1 is running synchronously > [UU]

If one partition of your raid fails then you will see underscores instead of the character ‘U’: [U_] or [_U] (depends on which partition is faulty).

Replacing a failed partition

For explanatory purposes we assume /dev/sdb1 to be faulty. If you want to replace the disk you have to follow the following steps:

(Optional) If your partition isn’t already marked failed you have to mark it failed:

sudo mdadm --manage /dev/md0 --fail /dev/sdb1

Then you can remove the faulty partition from the array md0:

sudo mdadm --manage /dev/md0 --remove

Remember: In our example we just have one partition per disk. If you have more than one partition belonging to multiple arrays, be sure to remove all partitions of that one particular disk which you want to replace. Example: another additional /dev/sdb2 could belong to /dev/md1.

Once all your partitions from the faulty disk are removed from all raid arrays you can replace the disk by replacing it with a new one. You should shutdown the system before 😉 Since this new one won’t be partitioned like the old one, we need to copy the partition structure from the disk which is still active in our array (/dev/sda). For copying a GTP partition structure from /dev/sda to our new /dev/sdb and creating a new UUID we can use:

# copy sda to sdb (GPT)
sudo sgdisk -R /dev/sdb /dev/sda
# create UUID
sudo sgdisk -G /dev/sdb

If you use classical MBR for disks below 2TB:

# copy sda to sdb (MBR)
sudo sfdisk -d /dev/sda | sudo sfdisk /dev/sdb

After having copied the partition structure we need to add the partition /dev/sdb1 back again to our array /dev/md0:

sudo mdadm --manage /dev/md0 --add /dev/sdb1

Now you can watch the progress of rebuilding your software array which is running completely transparent in the background until the target state of [UU] is reached again:

Personalities : [raid1]
md0   : active raid1 sdb1[1] sda1[0]
      1464725760 blocks level 5, 64k chunk, algorithm 2 [6/5] [U_]
      [==>..................]  recovery = 12.6% (37043392/292945152) finish=127.5min speed=33440K/sec

If you replaced a bootable disk (for instance /dev/sda) think about installing GRUB again. It might be helpful to copy the full boot partition (let’s say /dev/sda0) to /dev/sdb0 and to install GRUB also on /dev/sdb in order to allow booting from both disks.

# installing GRUB
sudo grub-install /dev/sda