pitter patter on the keyboard

Recovering from a disk failure in a software RAID is a very straightforward and easy process. If your server, chipset and kernel module allow, it may be possible to replace the offending drive without downtime. This is generally not the case with cheap SATA setups, where the server will have to be powered down for the drive replacement. Although it might be safe to remove a SATA drive while the system is running your kernel module may not support recognising the new drive without reloading or rebooting.

The failed drive will have to be identified. This is a very easy process with hardware controllers (typically an indicator light will flash on the failed drive) but not quite as simple when using software RAID. The surefire way to identify a drive is by serial number; find the device node (either through cat /proc/mdstat or dmesg) then run either:

# hdparm -i /dev/{disk node}

# smartctl -a /dev/{disk node}

If your configuration does not support hot swapping power down the machine and replace the bad drive. If it does, you may need to remove other partitions on the target disk from their respective arrays:

# mdadm -f /dev/{md node} /dev/{partition node}

will mark the partition as FAILED, then it can be removed from the set:

# mdadm -r /dev/{md node} /dev/{partition node}

If you are running swap space on the failed drive be sure to disable it with swapoff before removing the disk.

It should be possible to boot the machine off of the remaining set but if you run in to trouble it is just as easy to perform these operations from a livecd.

First we need to copy the partition table from one of the healthy disks exactly. Let sda represent a healthy disk and sdb represent the new one:

# sfdisk -d /dev/sda | sfdisk /dev/sdb

Now we need to add the new partition(s) to our existing RAID set. If you are booting off a livecd and your sets were not automatically configured on boot (they should have been) use mdadm –assemble to assemble them. Then:

# mdadm /dev/md0 -a /dev/sdb1

for each set and partition which needs to be added.

We can watch the resynchronization status with:

# watch cat /proc/mdstat

Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid1 sdb2[2] sda2[1]
      243922816 blocks [2/1] [_U]
      [================>....]  recovery = 83.8% (204426880/243922816) finish=58.3min speed=11271K/sec

md0 : active raid1 sdb1[0] sda1[1]
      272960 blocks [2/2] [UU]

The resync process will slow down considerably if there is heavy disk i/o at the same time and you can expect below-average performance until recovery has completed.

I haven’t given macs much thought since they switched to Intel and my computer repair days are long behind me but one of my clients is a doll of an ol’ gal who eats, sleeps and breathes mailing lists and when her old mac started crapping out I could not bear to let her go without.

It did not take long to figure out the Maxtor (so much for the “choice components” argument, huh Apple?) drive which shipped with the mac years ago had fried.

Opening up this thing is an immense pain in the ass. The iMac models above and below the iSight G5 seem to all open by lifting out the back panel by the stand, exposing all the replaceable parts. That makes perfect sense and I can see working on one of them to be a dream. The people who designed this chassis, on the other hand, were clearly drunk.

Not only does the front bezel (which includes the outer sides) have to be removed from the back, the LCD display must be unscrewed and disconnected – all the while one must put up with copious amounts of easily torn tape. Tape which I eventually decided served its purpose better in the rubbish bin.

The worst part about taking this thing apart is the two strange latches at the top of the bezel. I’m still not sure how to correctly operate them, I just dug in with a butterknife and shimmied around a bit until the damn things unhinged.

Once the display has been lifted and some of the plugs disconnected you can remove the hard drive. You will probably need a “proper” Torx driver set to do this; the screws mounting the screen are sunk into wells which were not large enough for the shaft of my ratchet driver to fit down. If I did not happen to have a precision slot driver that “fit” these screws handy this mac would have ended up benched for weeks (exactly how long it takes for a Torx set to ship here from Hong Kong)

The drive has a mounting bracket and little headless screw-pegs that slide into the rubber grommets on the inward side of the enclosure.

Here’s a photo of those grommets:

The drive has a heat sensor mounted on its side. I’m not sure if it was by intention or if this mac was unique but the PCB was mounted with a corner covering half of the middle bolt hole on the drive. This let me pry it off easily and I took the hint to do the same thing when mounting it on the new drive.

It’s held on by a little goo-pad. If you pry gently it should stay mostly intact.

Her mac is now as healthy as a horse. Despite this, I told her next time she has a problem with it to toss it in the bin.

Payment? One homemade blueberry-blackberry-raspberry cheesecake and a 26er of rye.

Bring on the NWO, I’ve got my bartering skills in line.

Resources:

http://discussions.apple.com/thread.jspa?threadID=996617

This post in particular:

Remove all 5 screws on the bottom. The two for the ram slot are captive. The other 3 are not. One is longer than the others, note which slot it came from.

There are catch levers on the top corners that need to be released by sliding something in there. That post recommended a credit card or thin plastic. I used two thin table knives with non-serrated edges.

Once the top is off you can flip it up and lay it down or undo all the tape and unplug the iSight. I left mine intact.

Then the fun part (NOT!). Using my table knife I very carefully peeled up the thick black foil tape stuff along the two edges. It’s so sticky and tears easily. You need infinte patience. Once peeled up it reveals 4 torx screws holding the LCD in down on the bottom. You need a long Torx screwdriver and make sure it is magnetic—the screws are not captive and what a pain! Especially when you don’t have a magnetic driver!

The you can lift up the LCD. Again I left mine connected and just propped it up by sticking a can of compressed air I had sitting nearby in as a prop (obviously away from the boards). I did have to disconnect the video though (two white connectors on the left that are easy to disconnect).

In the last couple of weeks a wave of attacks has seen the snort packet logs on a client’s firewall fill the disc to capacity, causing all sorts of wonderful problems. Packet logging is optional and usually only worth the trouble if you are actively trying to solve an attack or false positive, in which cases it can be added at that time. For most folks it simply provides a hindrance on performance and, if your storage is not well diversified, a hazard as we have seen with this router:

Disable packet logging by editing /etc/init.d/snort to start the daemon with the -N flag:

  start)
        echo -n $"Starting $prog: "
        if test "x`/sbin/pidof snort`" != x; then
                failure
                echo ""
        else
                automagic
                # Add support for multiwan
                if [ -n "$EXTIF" ]; then
                                for INTERFACE in $EXTIF; do
                                                daemon snort -N -i $INTERFACE -D -c /etc/snort.conf
                                done
                else
                                daemon snort -N -D -c /etc/snort.conf
                fi
                RETVAL=$?
                echo
                [ $RETVAL -eq 0 ] && touch /var/lock/snort
        fi
        ;;

Restart snort via its init script:

# /etc/init.d/snort restart

If you take a look at the logrotate configuration file for snort at /etc/logrotate.d/snort you’ll see:

# A bit of a kludge here - the logrotate file is empty and
# created by /etc/rc.d/init.d/snort.
/var/log/snort/logrotate {
 missingok
 postrotate
 tar -czf /var/log/snort.tar.gz /var/log/snort 2> /dev/null
 rm -rf /var/log/snort/[0-9]* /var/log/snort/snort.log.[0-9]* 2> /dev/null
 killall -HUP snort 2> /dev/null || true
 endscript
}

I’m not sure why the ClearOS people are using a “kludge” here, at best guess it seems the point is to put the snort.tar.gz archive directly under /var/log rather than in its own directory. Maybe it has to do with accommodating snort’s built-in log rotation. I don’t know. I don’t really care.

If you’re concerned about aesthetics keep the init script from creating the blank:

        # Creates a dummy file for /etc/logrotate.d/snort script
#       if [ -d /var/log/snort ]; then
#               echo "Used for logrotate... do not delete" > /var/log/snort/logrotate
#       fi

If I read that right it’s saying “Used for logrotate… please delete.”

# yes | rm -r /var/log/snort/*