Linux software raid notes
Monday, April 21st, 2008This is mostly for my own reference, but someone else may find it useful.
I currently have two raid volumes in my home server. They’re running on four disks, which are partitioned thusly:
root@bender:~# fdisk -l /dev/sd[abcd] Disk /dev/sda: 400.0 GB, 400088457216 bytes 255 heads, 63 sectors/track, 48641 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sda1 1 48641 390708801 83 Linux Disk /dev/sdb: 400.0 GB, 400088457216 bytes 255 heads, 63 sectors/track, 48641 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdb1 1 48641 390708801 83 Linux Disk /dev/sdc: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdc1 1 48641 390708801 83 Linux /dev/sdc2 48642 60801 97675200 83 Linux Disk /dev/sdd: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0xb7e306eb Device Boot Start End Blocks Id System /dev/sdd1 1 48641 390708832 83 Linux /dev/sdd2 48642 60801 97675200 83 Linux
Using the partitions as raid targets, I end up with two raid volumes:
md0 : active raid5 sdd1[3] sda1[0] sdc1[2] sdb1[1] md1 : active raid1 sdd2[1] sdc2[0]
On top of these raid volumes, I’m running an LVM so that they can be easily resized:
root@bender:~# pvdisplay --- Physical volume --- PV Name /dev/md1 VG Name vgraid1 PV Size 93.15 GB / not usable 9.88 MB Allocatable yes (but full) PE Size (KByte) 16384 Total PE 5961 Free PE 0 Allocated PE 5961 PV UUID NX3sku-sNGU-yqyw-3peE-Bd8W-KjBo-xsUol8 --- Physical volume --- PV Name /dev/md0 VG Name vgraid PV Size 1.09 TB / not usable 14.31 MB Allocatable yes (but full) PE Size (KByte) 16384 Total PE 71540 Free PE 0 Allocated PE 71540 PV UUID wtYFqQ-499L-BbuB-zaRb-EV16-Y5sT-ZcG65b
And now, the fun stuff. Here’s how the original raid5 volume was made (which was originally missing a 500G disk, because it died on me when I first got it):
root@bender:~# mdadm --create --verbose /dev/md0 --chunk=128 \ --level=5 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1 root@bender:~# pvcreate /dev/md0 root@bender:~# vgcreate -s 16M vgraid /dev/md0 root@bender:~# lvcreate -l 47693 vgraid -n bigdisk root@bender:~# mkfs.xfs /dev/vgraid/bigdisk
[note: this was done a while back, so i”ve lost the output. for output example see below when i create my raid1 array]
I finally received my new 500G drive from RMA, so I had to resize the raid5 to include the new disk. Here’s how it’s done.
First, create the new partitions to match the size on the other drive. (see above) Then, resize the raid array (takes a while…I started it then went to sleep, then it was magically done when I woke up this morning)
root@bender:~# mdadm --add /dev/md0 /dev/sdd1 mdadm: added /dev/sdd1 root@bender:~# mdadm --detail /dev/md0 /dev/md0: Version : 00.90.03 Creation Time : Fri Mar 21 09:52:31 2008 Raid Level : raid5 Array Size : 781417472 (745.22 GiB 800.17 GB) Used Dev Size : 390708736 (372.61 GiB 400.09 GB) Raid Devices : 3 Total Devices : 4 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Sun Apr 20 23:28:58 2008 State : clean Active Devices : 3 Working Devices : 4 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 64K UUID : 2621450e:bb4f9269:a0a0eea7:e933aaa5 (local to host bender) Events : 0.17564 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1 2 8 33 2 active sync /dev/sdc1 3 8 49 - spare /dev/sdd1 root@bender:~# mdadm --grow /dev/md0 --raid-devices=4 mdadm: Need to backup 384K of critical section.. mdadm: ... critical section passed. root@bender:~# mdadm --detail /dev/md0 /dev/md0: Version : 00.91.03 Creation Time : Fri Mar 21 09:52:31 2008 Raid Level : raid5 Array Size : 781417472 (745.22 GiB 800.17 GB) Used Dev Size : 390708736 (372.61 GiB 400.09 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Sun Apr 20 23:29:48 2008 State : clean, recovering Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Reshape Status : 0% complete Delta Devices : 1, (3->4) UUID : 2621450e:bb4f9269:a0a0eea7:e933aaa5 (local to host bender) Events : 0.17680 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1 2 8 33 2 active sync /dev/sdc1 3 8 49 3 active sync /dev/sdd1 root@bender:~# mdadm --detail /dev/md0 /dev/md0: Version : 00.90.03 Creation Time : Fri Mar 21 09:52:31 2008 Raid Level : raid5 Array Size : 1172126208 (1117.83 GiB 1200.26 GB) Used Dev Size : 390708736 (372.61 GiB 400.09 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Mon Apr 21 09:15:38 2008 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K UUID : 2621450e:bb4f9269:a0a0eea7:e933aaa5 (local to host bender) Events : 0.277372 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1 2 8 33 2 active sync /dev/sdc1 3 8 49 3 active sync /dev/sdd1 root@bender:~# pvresize /dev/md0 Physical volume "/dev/md0" changed 1 physical volume(s) resized / 0 physical volume(s) not resized root@bender:~# vgdisplay --- Volume group --- VG Name vgraid System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 1.09 TB PE Size 16.00 MB Total PE 71540 Alloc PE / Size 47693 / 745.20 GB Free PE / Size 23847 / 372.61 GB VG UUID JPEInC-5bNp-f0iC-3611-qIIn-Drhd-A3xInG root@bender:~# lvextend -l +23847 /dev/vgraid/bigdisk Extending logical volume bigdisk to 1.09 TB Logical volume bigdisk successfully resized root@bender:~# xfs_growfs /dev/vgraid/bigdisk meta-data=/dev/mapper/vgraid-bigdisk isize=256 agcount=32, agsize=6104704 blks = sectsz=512 attr=0 data = bsize=4096 blocks=195350528, imaxpct=25 = sunit=0 swidth=0 blks, unwritten=1 naming =version 2 bsize=4096 log =internal bsize=4096 blocks=32768, version=1 = sectsz=512 sunit=0 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0 data blocks changed from 195350528 to 293027840 root@bender:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/hda1 73G 63G 6.2G 91% / varrun 1.9G 264K 1.9G 1% /var/run varlock 1.9G 0 1.9G 0% /var/lock udev 1.9G 120K 1.9G 1% /dev devshm 1.9G 0 1.9G 0% /dev/shm /dev/sde1 280G 271G 8.6G 97% /export/myth /dev/mapper/vgraid-bigdisk 1.1T 453G 665G 41% /export/archive
Since I now have my two 500G disks, I have an extra 100G on each (since I made all the partitions match to raid5 with the 400G disks) so I can raid1 those for a wonderfully redundant backup array.
root@bender:~# mdadm --create --verbose /dev/md1 --chunk=128 \ --level=1 --raid-devices=2 /dev/sdc2 /dev/sdd2 mdadm: chunk size ignored for this level mdadm: size set to 97675136K mdadm: array /dev/md1 started. root@bender:~# pvcreate /dev/md1 Physical volume "/dev/md1" successfully created root@bender:~# vgcreate -s 16M vgraid1 /dev/md1 Volume group "vgraid1" successfully created root@bender:~# vgdisplay --- Volume group --- VG Name vgraid1 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 93.14 GB PE Size 16.00 MB Total PE 5961 Alloc PE / Size 0 / 0 Free PE / Size 5961 / 93.14 GB VG UUID Cxrf05-HaUJ-sPHx-Ai9P-eRYL-Ofoz-mnIsFx --- Volume group --- VG Name vgraid System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 4 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 1.09 TB PE Size 16.00 MB Total PE 71540 Alloc PE / Size 71540 / 1.09 TB Free PE / Size 0 / 0 VG UUID JPEInC-5bNp-f0iC-3611-qIIn-Drhd-A3xInG root@bender:~# lvcreate -l 5961 vgraid1 -n backup Logical volume "backup" created root@bender:~# mkfs.xfs /dev/vgraid1/backup meta-data=/dev/vgraid1/backup isize=256 agcount=16, agsize=1526016 blks = sectsz=512 attr=0 data = bsize=4096 blocks=24416256, imaxpct=25 = sunit=0 swidth=0 blks, unwritten=1 naming =version 2 bsize=4096 log =internal log bsize=4096 blocks=11922, version=1 = sectsz=512 sunit=0 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0 root@bender:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/hda1 73G 63G 6.2G 91% / varrun 1.9G 264K 1.9G 1% /var/run varlock 1.9G 0 1.9G 0% /var/lock udev 1.9G 128K 1.9G 1% /dev devshm 1.9G 0 1.9G 0% /dev/shm /dev/sde1 280G 271G 8.6G 97% /export/myth /dev/mapper/vgraid-bigdisk 1.1T 453G 665G 41% /export/archive /dev/mapper/vgraid1-backup 94G 544K 94G 1% /export/backup
Huzzah! Less likeliness of lost data due to my stupidity! Now to write some scripts to do automatic backups. :)
