pitter patter on the keyboard

One of the oldest and simplest ways to back up UNIX-like systems is with rsync. Automated remote backups are a simple matter of setting up rsync as a daemon on the system you want to back up and running an appropriate rsync command as a cron job on the backup target.

Let’s create /etc/rsyncd.conf on the machine to be backed up:

[etc]
  path=/etc
  read only=TRUE
  hosts allow=192.168.0.0/24
  hosts deny=*
  uid = 0
  exclude = ssh/

Any number of modules can be defined with the [module_name] convention so different settings can be applied to different locations on the file system. Module names are used in the rsync:// protocol format insted of full path names. This block says, in English: Serve the contents of /etc/ except /etc/ssh/ to any host on the 192.168.0.0 subnet at the location rsync://localhost/etc as though root was reading the files but don’t allow any modifications to the local file system.

Read the rsyncd.conf man page for more configuration options. Your rsync package probably comes with a compatible init script, be sure to enable it for your default runlevel.

On the backup server we’ll drop this little bash script into /etc/cron.hourly:

#!/bin/bash

rsync -a -u --delete rsync://192.168.0.10/etc /mnt/backups/serverxyz/etc

Now every hour the backup target will connect to the rsync daemon and download any files which have been changed or do not already exist in the backup tree, also deleting files which no longer exist on that server to preserve space.

Recovering from a disk failure in a software RAID is a very straightforward and easy process. If your server, chipset and kernel module allow, it may be possible to replace the offending drive without downtime. This is generally not the case with cheap SATA setups, where the server will have to be powered down for the drive replacement. Although it might be safe to remove a SATA drive while the system is running your kernel module may not support recognising the new drive without reloading or rebooting.

The failed drive will have to be identified. This is a very easy process with hardware controllers (typically an indicator light will flash on the failed drive) but not quite as simple when using software RAID. The surefire way to identify a drive is by serial number; find the device node (either through cat /proc/mdstat or dmesg) then run either:

# hdparm -i /dev/{disk node}

# smartctl -a /dev/{disk node}

If your configuration does not support hot swapping power down the machine and replace the bad drive. If it does, you may need to remove other partitions on the target disk from their respective arrays:

# mdadm -f /dev/{md node} /dev/{partition node}

will mark the partition as FAILED, then it can be removed from the set:

# mdadm -r /dev/{md node} /dev/{partition node}

If you are running swap space on the failed drive be sure to disable it with swapoff before removing the disk.

It should be possible to boot the machine off of the remaining set but if you run in to trouble it is just as easy to perform these operations from a livecd.

First we need to copy the partition table from one of the healthy disks exactly. Let sda represent a healthy disk and sdb represent the new one:

# sfdisk -d /dev/sda | sfdisk /dev/sdb

Now we need to add the new partition(s) to our existing RAID set. If you are booting off a livecd and your sets were not automatically configured on boot (they should have been) use mdadm –assemble to assemble them. Then:

# mdadm /dev/md0 -a /dev/sdb1

for each set and partition which needs to be added.

We can watch the resynchronization status with:

# watch cat /proc/mdstat

Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid1 sdb2[2] sda2[1]
      243922816 blocks [2/1] [_U]
      [================>....]  recovery = 83.8% (204426880/243922816) finish=58.3min speed=11271K/sec

md0 : active raid1 sdb1[0] sda1[1]
      272960 blocks [2/2] [UU]

The resync process will slow down considerably if there is heavy disk i/o at the same time and you can expect below-average performance until recovery has completed.

Having trouble sending e-mail to @hotmail or @live addresses? Trouble like:

Diagnostic-Code: SMTP; 550 SC-001 (BAY0-MC4-F36) Unfortunately, messages from xxx.xxx.xxx.xxx weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.

The irony of course is that the URL provided gives you less information than the error message:

550 SC-001 Mail rejected by Hotmail for policy reasons. Reasons for rejection may be related to content with spam-like characteristics or IP/domain reputation. If you are not an email/network admin please contact your Email/Internet Service Provider for help.

Spam-like characteristics can mean anything but at least twice I’ve had this problem the issue was being on Hotmail’s internal IP blacklist. Like it said the first time.

First, you should check to see if you’re on any of the public RBLs: http://whatismyipaddress.com/blacklist-check. If you are, you have real problems and need to sort that out before bothering Microsoft. Chances are, however, you aren’t – especially if we’re dealing with a simple web server and not a dedicated mail server.

If you’re sure you’re really not spamming Hotmail fill out the Deliverability Issue Submission form at https://support.msn.com/eform.aspx?productKey=edfsmsbl2&ct=eformts. In a few hours the automated system will review your case and if successful you will receive an e-mail like this:

Dear Soey McSoandso

We have completed reviewing the IP(s) you submitted. The following table contains the results of our investigation.

Mitigated
xxx.xxx.xxx.xxx
The reported deliverability problem has been mitigated by the automated system. Mitigation may take 24 – 48 hours to replicate completely throughout our system. Please note that mitigating this issue does not guarantee that your email will be delivered to a user’s inbox.

Regardless of the deliverability status, Hotmail recommends that all senders join two free programs that provide visibility into the Hotmail traffic on your sending IP(s), the sending IP reputation with Hotmail and the Hotmail user complaint rates.

Junk Email Reporting program (JMRP) When a Hotmail user marks an email as “junk”, senders enrolled in this program get a copy of the mail forwarded to an address of their choice. It allows you to see which mails are being marked as junk and identify mail traffic you did not intend to send. To join, please visit http://support.msn.com/eform.aspx?productKey=edfsjmrpp&page=support_home_options_form_byemail&ct=eformts.

Smart Network Data Services program (SNDS) This program allows you to monitor the ‘health’ and reputation of your registered IPs by providing data about traffic such as mail volume and complaint rates seen originating from your IPs. To register, please visit http://postmaster.msn.com/snds/.

There is no silver bullet to maintaining or improving good IP reputation, but these programs help you proactively manage your email eco-system to help better ensure deliverability to Hotmail users.

Thank you,

Hotmail Deliverability Support Service

If not successful, being that the system is automated you may have to go hunting for a new subnet.

Both of the tools linked above are excellent sources of information if you administrate an ISP mail server or mail servers for third parties and JMRP is especially useful for tuning mass mail marketing campaigns.