blog.dknowles.org

This is mostly for my own reference, but someone else may find it useful.

I currently have two raid volumes in my home server. They’re running on four disks, which are partitioned thusly:

root@bender:~# fdisk -l /dev/sd[abcd]

Disk /dev/sda: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1       48641   390708801   83  Linux

Disk /dev/sdb: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       48641   390708801   83  Linux

Disk /dev/sdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1       48641   390708801   83  Linux
/dev/sdc2           48642       60801    97675200   83  Linux

Disk /dev/sdd: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xb7e306eb

Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1       48641   390708832   83  Linux
/dev/sdd2           48642       60801    97675200   83  Linux

Using the partitions as raid targets, I end up with two raid volumes:

md0 : active raid5 sdd1[3] sda1[0] sdc1[2] sdb1[1]
md1 : active raid1 sdd2[1] sdc2[0]

On top of these raid volumes, I’m running an LVM so that they can be easily resized:

root@bender:~# pvdisplay
--- Physical volume ---
PV Name               /dev/md1
VG Name               vgraid1
PV Size               93.15 GB / not usable 9.88 MB
Allocatable           yes (but full)
PE Size (KByte)       16384
Total PE              5961
Free PE               0
Allocated PE          5961
PV UUID               NX3sku-sNGU-yqyw-3peE-Bd8W-KjBo-xsUol8
--- Physical volume ---
PV Name               /dev/md0
VG Name               vgraid
PV Size               1.09 TB / not usable 14.31 MB
Allocatable           yes (but full)
PE Size (KByte)       16384
Total PE              71540
Free PE               0
Allocated PE          71540
PV UUID               wtYFqQ-499L-BbuB-zaRb-EV16-Y5sT-ZcG65b

And now, the fun stuff. Here’s how the original raid5 volume was made (which was originally missing a 500G disk, because it died on me when I first got it):

root@bender:~# mdadm --create --verbose /dev/md0 --chunk=128 \
  --level=5 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1

root@bender:~# pvcreate /dev/md0

root@bender:~# vgcreate -s 16M vgraid /dev/md0

root@bender:~# lvcreate -l 47693 vgraid -n bigdisk

root@bender:~# mkfs.xfs /dev/vgraid/bigdisk

[note: this was done a while back, so i”ve lost the output. for output example see below when i create my raid1 array]

I finally received my new 500G drive from RMA, so I had to resize the raid5 to include the new disk. Here’s how it’s done.

First, create the new partitions to match the size on the other drive. (see above) Then, resize the raid array (takes a while…I started it then went to sleep, then it was magically done when I woke up this morning)

root@bender:~# mdadm --add /dev/md0 /dev/sdd1
mdadm: added /dev/sdd1

root@bender:~# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Fri Mar 21 09:52:31 2008
Raid Level : raid5
Array Size : 781417472 (745.22 GiB 800.17 GB)
Used Dev Size : 390708736 (372.61 GiB 400.09 GB)
Raid Devices : 3
Total Devices : 4
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Sun Apr 20 23:28:58 2008
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 64K

UUID : 2621450e:bb4f9269:a0a0eea7:e933aaa5 (local to host bender)
Events : 0.17564

Number   Major   Minor   RaidDevice State
0       8        1        0      active sync   /dev/sda1
1       8       17        1      active sync   /dev/sdb1
2       8       33        2      active sync   /dev/sdc1

3       8       49        -      spare   /dev/sdd1

root@bender:~# mdadm --grow /dev/md0 --raid-devices=4
mdadm: Need to backup 384K of critical section..
mdadm: ... critical section passed.

root@bender:~# mdadm --detail /dev/md0
/dev/md0:
Version : 00.91.03
Creation Time : Fri Mar 21 09:52:31 2008
Raid Level : raid5
Array Size : 781417472 (745.22 GiB 800.17 GB)
Used Dev Size : 390708736 (372.61 GiB 400.09 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Sun Apr 20 23:29:48 2008
State : clean, recovering
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Reshape Status : 0% complete
Delta Devices : 1, (3->4)

UUID : 2621450e:bb4f9269:a0a0eea7:e933aaa5 (local to host bender)
Events : 0.17680

Number   Major   Minor   RaidDevice State
0       8        1        0      active sync   /dev/sda1
1       8       17        1      active sync   /dev/sdb1
2       8       33        2      active sync   /dev/sdc1
3       8       49        3      active sync   /dev/sdd1

root@bender:~# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Fri Mar 21 09:52:31 2008
Raid Level : raid5
Array Size : 1172126208 (1117.83 GiB 1200.26 GB)
Used Dev Size : 390708736 (372.61 GiB 400.09 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Mon Apr 21 09:15:38 2008
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

UUID : 2621450e:bb4f9269:a0a0eea7:e933aaa5 (local to host bender)
Events : 0.277372

Number   Major   Minor   RaidDevice State
0       8        1        0      active sync   /dev/sda1
1       8       17        1      active sync   /dev/sdb1
2       8       33        2      active sync   /dev/sdc1
3       8       49        3      active sync   /dev/sdd1

root@bender:~# pvresize /dev/md0
Physical volume "/dev/md0" changed
1 physical volume(s) resized / 0 physical volume(s) not resized

root@bender:~# vgdisplay
--- Volume group ---
VG Name               vgraid
System ID
Format                lvm2
Metadata Areas        1
Metadata Sequence No  3
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                1
Open LV               0
Max PV                0
Cur PV                1
Act PV                1
VG Size               1.09 TB
PE Size               16.00 MB
Total PE              71540
Alloc PE / Size       47693 / 745.20 GB
Free  PE / Size       23847 / 372.61 GB
VG UUID               JPEInC-5bNp-f0iC-3611-qIIn-Drhd-A3xInG

root@bender:~# lvextend -l +23847 /dev/vgraid/bigdisk
Extending logical volume bigdisk to 1.09 TB
Logical volume bigdisk successfully resized

root@bender:~# xfs_growfs /dev/vgraid/bigdisk
meta-data=/dev/mapper/vgraid-bigdisk isize=256    agcount=32, agsize=6104704 blks
=                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=195350528, imaxpct=25
=                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=1
=                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 195350528 to 293027840

root@bender:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1              73G   63G  6.2G  91% /
varrun                1.9G  264K  1.9G   1% /var/run
varlock               1.9G     0  1.9G   0% /var/lock
udev                  1.9G  120K  1.9G   1% /dev
devshm                1.9G     0  1.9G   0% /dev/shm
/dev/sde1             280G  271G  8.6G  97% /export/myth
/dev/mapper/vgraid-bigdisk
1.1T  453G  665G  41% /export/archive

Since I now have my two 500G disks, I have an extra 100G on each (since I made all the partitions match to raid5 with the 400G disks) so I can raid1 those for a wonderfully redundant backup array.

root@bender:~# mdadm --create --verbose /dev/md1 --chunk=128 \
  --level=1 --raid-devices=2 /dev/sdc2 /dev/sdd2
mdadm: chunk size ignored for this level
mdadm: size set to 97675136K
mdadm: array /dev/md1 started.

root@bender:~# pvcreate /dev/md1
Physical volume "/dev/md1" successfully created

root@bender:~# vgcreate -s 16M vgraid1 /dev/md1
Volume group "vgraid1" successfully created

root@bender:~# vgdisplay
--- Volume group ---
VG Name               vgraid1
System ID
Format                lvm2
Metadata Areas        1
Metadata Sequence No  1
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                0
Open LV               0
Max PV                0
Cur PV                1
Act PV                1
VG Size               93.14 GB
PE Size               16.00 MB
Total PE              5961
Alloc PE / Size       0 / 0
Free  PE / Size       5961 / 93.14 GB
VG UUID               Cxrf05-HaUJ-sPHx-Ai9P-eRYL-Ofoz-mnIsFx

--- Volume group ---
VG Name               vgraid
System ID
Format                lvm2
Metadata Areas        1
Metadata Sequence No  4
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                1
Open LV               1
Max PV                0
Cur PV                1
Act PV                1
VG Size               1.09 TB
PE Size               16.00 MB
Total PE              71540
Alloc PE / Size       71540 / 1.09 TB
Free  PE / Size       0 / 0
VG UUID               JPEInC-5bNp-f0iC-3611-qIIn-Drhd-A3xInG

root@bender:~# lvcreate -l 5961 vgraid1 -n backup
Logical volume "backup" created

root@bender:~# mkfs.xfs /dev/vgraid1/backup
meta-data=/dev/vgraid1/backup    isize=256    agcount=16, agsize=1526016 blks
=                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=24416256, imaxpct=25
=                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=11922, version=1
=                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

root@bender:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1              73G   63G  6.2G  91% /
varrun                1.9G  264K  1.9G   1% /var/run
varlock               1.9G     0  1.9G   0% /var/lock
udev                  1.9G  128K  1.9G   1% /dev
devshm                1.9G     0  1.9G   0% /dev/shm
/dev/sde1             280G  271G  8.6G  97% /export/myth
/dev/mapper/vgraid-bigdisk
1.1T  453G  665G  41% /export/archive
/dev/mapper/vgraid1-backup
94G  544K   94G   1% /export/backup

Huzzah! Less likeliness of lost data due to my stupidity! Now to write some scripts to do automatic backups. :)

A while back, I posted about a google reader function that I wanted–the ability to seamlessly share things from google reader to my blog. Lazyweb wasn’t much help, so I rolled out a quick-and-dirty greasemonkey script to do it. You can find it here: http://www.dknowles.org/other/google-reader-cross-share.user.js

Right now, it works by clicking on the share icon below a reader entry. I’m not catching keyboard shortcuts, so it requires a physical click. (fine with me, as I’m a keyboard junkie and can’t be bothered with constant naggings about sharing on my blog)

The regular “use-at-own-risk-may-break-at-any-time” disclaimer applies. Ideally, I’d eventually like this to make a greader-styled popup that asks for your blog entry, but that requires more work than I’m willing to put into this. Plus, this works as-is. If there’s interest in this, I can put it in SVN somewhere so people can give back and extend it and stuff.

My awesome obama shirt wasn’t even up for a whole 24 hours before it got flagged by the people at cafepress.  I sent a nice email back asking for a specific reason for why my shirt was taken down, and I actually received a human response outlining everything.

While I don’t agree with the auto-flagging of copyrighted material, I do appreciate the human response, and I understand why they won’t sell the shirt.  My problem with this is that it’s typically the copyright holder’s burden to flag infringements, and it feels icky for a company to take the task off the hands of the copyright holder.  Of course, these days it’s nearly impossible for a copyright holder to find every infringement.  And it’s also possible that the Obama campaign has already had some sort of agreement with cafepress regarding what is fair use and what isn’t.  I’ll just cover my eyes and ears and hope that’s the case.

So maybe it’s my mistake for using such a large company.  Maybe I’ll have better luck infringing copyrights with a smaller company (printfection, maybe?).  Not that infringing copyrights was my end-goal, but it’s something to keep in mind nonetheless.

On the upside, I did get a shipping confirmation from cafepress today.  So maybe I’ll still get the two shirts I ordered.  That’s all I wanted anyway–a cool shirt to wear on super Tuesday to Manuel’s.

O RLY? YA RLY.

This is an interesting endorsement.  Munroe (of xkcd fame) is typically not very political on his webcomic.  I think this just comes to show how important this election is.  Maybe this will at least get some of the young, tech-savvy (read: nerdy) vote to reconsider.  I get the impression we’ve lost a lot of this demographic to Ron Paul, which is a shame.

Vote!

I had this awesome idea for a t-shirt the other day. Luckily I have moderately okay photoshop skills and cafepress at my disposal. So, now I can actually have the shirt. Huzzah!

http://www.cafepress.com/dknowles2.219891180

Oh–and make sure to vote this year, kids.

A seamless google reader / wordpress integration.   

I want to be able to pull up the links I’ve shared in my google reader and write comments about them to post directly into my blog.  Most of the time, I read articles directly in reader so it’s a pain to have to click the links copy them, then paste them here to comment on them. I’m thinking either a greasemonkey script, or maybe a way of making a daily digest.   

Anyone in the lazyweb want to make that happen for me? :) 

24 people who mattered in 2007

A nice list of people who mattered (in the technological world) this year.  There’s a number of people on that list whose work I follow pretty closely, and a few more that I’ve heard of before.  Good stuff.

• From a coworker, a great article on how django processes a request.  I’d actually figured most of this out from digging through the django codebase in the past.  Would’ve been nice to have this reference then. :) 
• A good primer on javascript keyboard events in different browsers.  I’ve been fighting with this one at work recently.  Not fun.
• An interesting javascript pattern for handling real public/private variables.
• Seven rules of unobtrusive javascript.
• Seven is a popular number:  Seven javascript techniques you should be using today
   

I’ve got a bug in one of my scripts somewhere that’s saving multiple entries for video files in mythtv’s videometadata table. Here’s a quick hacky SQL query to find them:

SELECT intid, filename, coverfile
FROM videometadata
WHERE filename IN
  (SELECT filename FROM
    (SELECT filename, COUNT(*) AS ttl FROM videometadata
     GROUP BY filename HAVING ttl > 1
    ) AS tmpfoo
  ) ORDER BY filename

Maybe one day, I’ll convince myself that using MySQL transactions is a good idea, and start using them whenever I do SQL writes.

In related news, I just NULLed out all my coverdata fields in my mythTV’s mythvideo table. Regenerating all of that crap is really tedious, as the website’s API I use gets flaky after I hit it with a few hundred requests in a matter of minutes. :(

Sigh.

Update:

All fixed.   Took ~90 lines of python.  Good script to have around anyway:  looks for entries with no/missing/invalid coverfile and tries to re-fetch it from thetvdb.com.  One day I’ll get all this crap into SVN.   I’m getting quite the collection of these handy scripts. :)