How to resize a soft RAID using mdadm (by replacing the old disks)
15 min read | by Jordi Prats
If you have a linux software RAID (in this case RAID 5) and you want to resize it, you can do it by replacing underlying disks with larger ones. This is a step-by-step guide on how to do it without loosing data and without having to copy data across.
The RAID device (md0) that we are going to use is configured as a RAID 5 array with 4 disks of 3TB each. We are going to replace them with 8TB disks. With the resulting array we are using LVM to manage the disks, so we will also need to extend the LVM volume group to use the full capacity of the new disks.
# pvdisplay
--- Physical volume ---
PV Name /dev/md0
VG Name raid
PV Size <8.19 TiB / not usable 5.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 2146093
Free PE 0
Allocated PE 2146093
PV UUID h68yNc-5MPU-5lWd-3p7m-RqCi-weLs-18yhsY
# vgdisplay
--- Volume group ---
VG Name raid
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 25
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 7
Open LV 5
Max PV 0
Cur PV 1
Act PV 1
VG Size <8.19 TiB
PE Size 4.00 MiB
Total PE 2146093
Alloc PE / Size 2146093 / <8.19 TiB
Free PE / Size 0 / 0
VG UUID HRKcbj-VoNI-w1jr-Ksu0-cbp9-lFAe-MkuiAn
Initial check
Let's assume we have a RAID 5 array with 4 disks of 3TB each, and we want to replace them with 8TB disks. This is the device:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sdc1[1] sdd1[2] sde1[3] sdb1[0]
8790402048 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/22 pages [0KB], 65536KB chunk
unused devices: <none>
Using lsblk we can see the it's serial numbers so that we can identify and replace them later:
# lsblk -do +VENDOR,MODEL,SERIAL
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT VENDOR MODEL SERIAL
(...)
sdb 8:16 0 2.7T 0 disk ATA WDC_WD30EFRX-68EUZN0 WD-WCC4N4CZNE48
sdc 8:32 0 2.7T 0 disk ATA WDC_WD30EFRX-68EUZN0 WD-WCC4N5LNHRP8
sdd 8:48 0 2.7T 0 disk ATA WDC_WD30EFRX-68EUZN0 WD-WCC4N5LNHFN6
sde 8:64 0 2.7T 0 disk ATA WDC_WD30EFRX-68EUZN0 WD-WCC4N2AHPJ7J
To make sure we have the array consistent, we can perform a data-check:
echo check > /sys/block/md0/md/sync_action
Depending on the size of the array, this can take a while. You can check the progress using cat /proc/mdstat:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sdc1[1] sdd1[2] sde1[3] sdb1[0]
8790402048 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
[===>.................] check = 17.5% (513533708/2930134016) finish=275.1min speed=146361K/sec
bitmap: 0/22 pages [0KB], 65536KB chunk
unused devices: <none>
As soon as this is completed, we should check for any errors in dmesg. If we find any, we should replace that disk first to make sure it doesn't fail meanwhile we are replacing some other disk.
Assuming we don't find any errors, we can proceed to replace the disks.
Replace each disk
Now we can start replacing the disks. We will replace them one by one, and we will use mdadm to mark the disk as faulty and remove it from the array. This will allow us to replace the disk without loosing any data.
We can now use smartctl to check the health of the disks. This is a good practice to do before replacing any disk, as it will help us identify any potential issues with the disks.
# smartctl -a /dev/sdc
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-208-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD30EFRX-68EUZN0
Serial Number: WD-WCC4N5LNHRP8
LU WWN Device Id: 5 0014ee 2b7b9458e
Firmware Version: 82.00A82
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Mar 24 19:14:12 2025 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
(...)
The are two things we should check:
- The overall health of the disk. If it says "PASSED" we are good to go.
# for i in sdc sdd sde sdb; do echo $i; smartctl -a /dev/$i | grep overall; done
sdc
SMART overall-health self-assessment test result: PASSED
sdd
SMART overall-health self-assessment test result: PASSED
sde
SMART overall-health self-assessment test result: PASSED
sdb
SMART overall-health self-assessment test result: PASSED
- The number of errors. We should check, at least, the "Raw_Read_Error_Rate" and "Seek_Error_Rate" values. If they are high, we should consider replacing that disk first.
# for i in sdc sdd sde sdb; do echo $i; smartctl -a /dev/$i | grep _Error; done
sdc
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 14
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0
sdd
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 28
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0
sde
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 20
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0
sdb
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 74
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0dmes
Each brand can report in a different way, in some cases a high number of read errors can not necessarily mean that the disk is failing: A good example of that would be Seagate disks, which encode the number of actual errors with the total number of operations, making it very difficult to interpret. In this case, we are using Western Digital disks, which are easier to interpret. We can also use the SMART parser to simplify the process of interpreting the SMART data, we just need to paste the output of smartctl -a and it will show us the most relevant data.
Without any other data to use, we can just pick the disk with the highest number of read errors and replace it first. In this case, we can see that sdb has the highest number of read errors (74), so we will replace it first and continue in descending order.
To do so, we are going to use mdadm to mark the disk as faulty and then remove it from the array. This will allow us to replace the disk without loosing any data.
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sdc1[1] sde1[3] sdd1[2] sdb1[0]
8790402048 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/22 pages [0KB], 65536KB chunk
unused devices: <none>
# mdadm --manage /dev/md0 --fail /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md0
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sdc1[1] sde1[3] sdd1[2] sdb1[0](F)
8790402048 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
bitmap: 0/22 pages [0KB], 65536KB chunk
unused devices: <none>
# mdadm --manage /dev/md0 --remove /dev/sdb1
mdadm: hot removed /dev/sdb1 from /dev/md0
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sdc1[1] sde1[3] sdd1[2]
8790402048 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
bitmap: 0/22 pages [0KB], 65536KB chunk
unused devices: <none>
At this point, since we are doing it in a consumer-grade machine, we can shutdown the machine and replace the disk. In this case, physically replacing sdb with a new 8TB disk.
When the system comes back up, we can add the new disk to the array. Using lsblk we can see that the new disk is still detected as sdb:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sdc1[1] sde1[3] sdd1[2]
8790402048 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
bitmap: 2/22 pages [8KB], 65536KB chunk
unused devices: <none>
# lsblk -do +VENDOR,MODEL,SERIAL
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT VENDOR MODEL SERIAL
(...)
sda 8:0 0 447.1G 0 disk ATA SanDisk_SSD_PLUS_480GB 2014J6460214
sdb 8:16 0 7.3T 0 disk ATA ST8000DM004-2U9188 WSC2RGHW
sdc 8:32 0 2.7T 0 disk ATA WDC_WD30EFRX-68EUZN0 WD-WCC4N5LNHRP8
sdd 8:48 0 2.7T 0 disk ATA WDC_WD30EFRX-68EUZN0 WD-WCC4N5LNHFN6
sde 8:64 0 2.7T 0 disk ATA WDC_WD30EFRX-68EUZN0 WD-WCC4N2AHPJ7J
# mdadm --add /dev/md0 /dev/sdb
mdadm: added /dev/sdb
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sdb[4] sdc1[1] sde1[3] sdd1[2]
8790402048 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
[>....................] recovery = 0.0% (318076/2930134016) finish=307.0min speed=159038K/sec
bitmap: 2/22 pages [8KB], 65536KB chunk
unused devices: <none>
Please notice how it is going to change from [4/3] to [4/4]. After recovery, it will automatically proceed to to a data-check:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sdb[4] sdc1[1] sde1[3] sdd1[2]
8790402048 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
[=================>...] check = 86.7% (2542327552/2930134016) finish=76.4min speed=84543K/sec
bitmap: 0/22 pages [0KB], 65536KB chunk
unused devices: <none>
Again, we'll have to check dmesg for any errors. If we find any, we should replace that disk back:
# dmesg
(...)
[ 22.599212] aufs 5.4.3-20200302
[ 113.986883] md: recovery of RAID array md0
[27856.422639] md: md0: recovery done.
[34228.300717] md: data-check of RAID array md0
[55830.808107] md: md0: data-check done.
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sdb[4] sdc1[1] sde1[3] sdd1[2]
8790402048 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/22 pages [0KB], 65536KB chunk
unused devices: <none>
At this point, we can proceed to replace the next disk. We can repeat the process for the rest of the disks.
Extending the RAID
Once we have replaced all the disks, we can extend the RAID array to use the full capacity of the new disks. Using mdadm with the --detail option, we can see the current size of the array:
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Fri May 13 23:41:22 2016
Raid Level : raid5
Array Size : 8790402048 (8383.18 GiB 9001.37 GB)
Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Sun Apr 13 11:23:05 2025
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : bitmap
Name : shuvak:0 (local to host shuvak)
UUID : 799bf904:a38acec7:d86bfde7:39011a4a
Events : 325722
Number Major Minor RaidDevice State
4 8 16 0 active sync /dev/sdb
5 8 32 1 active sync /dev/sdc
7 8 48 2 active sync /dev/sdd
6 8 64 3 active sync /dev/sde
Array Size is the total size of the array, while Used Dev Size is the amount of space used by each disk. In this case, we can see that the array is still using 3TB disks, so the total size of the array is 8TB.
Since we have replaced all the disks with bigger disks, we can now extend the array to use the full capacity of the new disks. To do so, we can use the --grow option with mdadm. We'll need to also specify the new active size of the array, which is the total size of the new disks. In this case, we can use --size=max to use the maximum size of the disks.
# mdadm --grow /dev/md0 --size max
mdadm: component size of /dev/md0 has been set to 7813895168K
If we recheck the size of the array, we can see that it has been updated, but it needs to resync the array to use the new size:
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Fri May 13 23:41:22 2016
Raid Level : raid5
Array Size : 23441685504 (22355.73 GiB 24004.29 GB)
Used Dev Size : 7813895168 (7451.91 GiB 8001.43 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Sun Apr 13 11:53:37 2025
State : clean, resyncing
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : bitmap
Resync Status : 37% complete
Name : shuvak:0 (local to host shuvak)
UUID : 799bf904:a38acec7:d86bfde7:39011a4a
Events : 325731
Number Major Minor RaidDevice State
4 8 16 0 active sync /dev/sdb
5 8 32 1 active sync /dev/sdc
7 8 48 2 active sync /dev/sdd
6 8 64 3 active sync /dev/sde
We can also keep an eye on the progress of the resync using cat /proc/mdstat:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sdb[4] sde[6] sdc[5] sdd[7]
23441685504 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
[=======>.............] resync = 37.7% (2953028220/7813895168) finish=680.1min speed=119102K/sec
bitmap: 10/15 pages [40KB], 262144KB chunk
unused devices: <none>
Once it is finished, we can run the commands again to validate the final size of the array and that the new disks are not faulty:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sdb[4] sdc[5] sde[6] sdd[7]
23441685504 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/15 pages [0KB], 262144KB chunk
unused devices: <none>
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Fri May 13 23:41:22 2016
Raid Level : raid5
Array Size : 23441685504 (22355.73 GiB 24004.29 GB)
Used Dev Size : 7813895168 (7451.91 GiB 8001.43 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon Apr 14 08:55:49 2025
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : bitmap
Name : shuvak:0 (local to host shuvak)
UUID : 799bf904:a38acec7:d86bfde7:39011a4a
Events : 333143
Number Major Minor RaidDevice State
4 8 16 0 active sync /dev/sdb
5 8 32 1 active sync /dev/sdc
7 8 48 2 active sync /dev/sdd
6 8 64 3 active sync /dev/sde
Finally, using pvresize we can instruct LVM to resize the physical volume to use the new size of the array. This will allow us to use the full capacity of the new disks in our LVM volumes:
# pvresize /dev/md0
Physical volume "/dev/md0" changed
1 physical volume(s) resized or updated / 0 physical volume(s) not resized
# pvdisplay
--- Physical volume ---
PV Name /dev/md0
VG Name raid
PV Size 21.83 TiB / not usable 0
Allocatable yes
PE Size 4.00 MiB
Total PE 5723067
Free PE 3576974
Allocated PE 2146093
PV UUID h68yNc-5MPU-5lWd-3p7m-RqCi-weLs-18yhsY
# vgdisplay
--- Volume group ---
VG Name raid
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 26
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 7
Open LV 6
Max PV 0
Cur PV 1
Act PV 1
VG Size 21.83 TiB
PE Size 4.00 MiB
Total PE 5723067
Alloc PE / Size 2146093 / <8.19 TiB
Free PE / Size 3576974 / <13.65 TiB
VG UUID HRKcbj-VoNI-w1jr-Ksu0-cbp9-lFAe-MkuiAn
Finally, we can extend the LVM volume to use the full capacity of the new disks. In this case, we are using lvextend to extend the logical volume to just 10TB:
# lvextend -L10TB -r /dev/raid/crap
Size of logical volume raid/crap changed from <7.69 TiB (2015533 extents) to 10.00 TiB (2621440 extents).
Logical volume raid/crap successfully resized.
resize2fs 1.45.5 (07-Jan-2020)
Filesystem at /dev/mapper/raid-crap is mounted on /var/crap/data; on-line resizing required
old_desc_blocks = 493, new_desc_blocks = 640
The filesystem on /dev/mapper/raid-crap is now 2684354560 (4k) blocks long.
# df -hP
Filesystem Size Used Avail Use% Mounted on
(...)
/dev/mapper/raid-crap 9.9T 6.6T 3.0T 69% /var/crap/data
Posted on 14/04/2025