Things I Learned the Hard Way Doing Full Disk Encryption on Gentoo with DM-Crypt LUKS

The documentation for setting up full disk encryption on Gentoo is specific at best, spotty at worst and confusing in general. Without re-living the immense pain it was to configure over again in detail, here are some of the things I wish I knew before I started:

  • The 2TB partition size limit is not a Windows-only limitation. It comes from the old BIOS style partition scheme. You can either use parted to create >2T GPT partitions or use the whole raw disk if you plan on using it for a single mount point or if you fancy putting DM/LVM on top. Obviously that isn't an option if you plan on booting from this device.
  • If you don't think you need LVM you probably don't, especially if this is a workstation or personal computer. It's way easier to skip all of that and use old-fashioned device nodes directly.
  • Having a newline character in your keyfile is deadly. The init script that genkernel rolls into your initramfs uses cryptsetup luksOpen /dev/whatever whatever -d - or --keyfile - instead of simply piping the output of gpg as the wiki article has you do when you luksFormat. The former stops reading the input at a newline, the latter incorporates it into your key. This is a problem if, say, you took the output of openssl rand -base64 96 because you wanted to generate a 512 bit or larger key. There is already a newline in the middle of the cleartext, so I thought I was clever when I removed it. Not so; nano and many other text editors will always leave a newline at the end of the file. If you cat the cleartext and your command prompt doesn't run on to the end of it you still have a newline in there. Pipe your keyfile through tr -d '\n' before you do anything to be safe.
  • genkernel will make mrproper unless you use the --no-clean or --no-mrproper flags. It will back up your .config and start building your kernel the way it wants to unless you specify ramdisk.
  • The plain64 IV doesn't take arguments, specifying sha512 as the hash is redundant.
  • Genkernel will not magically read /etc/conf.d/dmcrypt and import your keyfiles or their locations. You may need to add the following to your kernel command line, replacing {UUID} with the UUID of your /boot partition (or wherever you are keeping the keyfile). You can obtain the UUID by running blkid. Using a UUID instead of a path will allow you to store your keyfile on removable media which may not have the same device node from time to time.
    real_root=/dev/mapper/root crypt_root=/dev/sda2 root_key=root.gpg root_keydev=UUID={UUID}

    If you are using grub2 append this to your GRUB_CMDLINE_LINUX variable in /etc/default/grub and if you are booting Xen remember to ALSO append it to your GRUB_CMDLINE_LINUX_XEN_DEFAULT variable.

  • Non-root partitions will be luksOpened and mounted during bootup by the dmcrypt init script.

Zombie Disk Image is Loopback-Mounted in a Guest Domain and can't be Mounted

Chances are if you're reading this you've just pulled yourself out of a xend crash or a similar event which has left one or more zombie VM disk images attached to their loop device. You're trying to restart the VM but you keep getting this message:

Error: Device 2049 (vbd) could not be connected.
File /xen/vm/disk.img is loopback-mounted through /dev/loop/5,
which is mounted in a guest domain,
and so cannot be mounted now.

losetup will only tell you what you already know:

# losetup -d /dev/loop/5
loop: can't delete device /dev/loop/5: Device or resource busy

If you don't know the ID number of the VM before it tanked you can find it in your xend logs. Execute this command to free the loop device:

# xenstore-rm backend/vbd/{VM ID}

How to Disable PHP mail() Per-Domain/Vhost without Suhosin or FPM/suEXEC

So I'm tired of my goddamned WordPress blog (yep, this one) getting compromised and being used as a platform for spammers or phishers and so on.

Getting pwn't every so often is pretty much just a fact of using popular shrinkwrapware and being a lazy updater. But that's no reason to change our ways when we can just mitigate the damage - namely getting your web server's IP on to all those friendly RBLs and interrupting legit e-mail notification delivery.

The good news is you can use disable_functions in your php.ini to disable functions globally.

The bad news is you can't set disable_functions on a per-domain or vhost basis unless you're using FPM/suEXEC or the like.

The worse news is suhosin, a really sweet PHP security patch that I've written about before and which would give us the ability to do this has been abandoned for about two years now and there is no official support for php 5.4 and later.

So I'd like to give Dave Lachapelle a big round of applause for pointing out that even if you can't kill mail() you can at least cripple it. From http://www.davelachapelle.ca/2009/08/05/php-mail-abuse/:

...PHP doesn’t support setting disable_functions in the php_admin_value flag.

So, after a bit of searching, I decided to just add the following to each site’s .htaccess files:

php_admin_value sendmail_path "/dev/null"

Essentially sending all e-mail to /dev/null for that particular site. Perhaps not the most elegant solution, but it was effective...

Maybe not elegant, but surely classy.

Stay classy, Dave.

Zimbra < 8.0.6 Web Exploit, Bitcoin Slavery and Securing /tmp/

You may have noticed a bitcoin miner chugging along on your Zimbra server.

Doing a little searching, it seems you're not cool if you haven't.

A serious vulnerability (CVE-2013-7091) in the administration web interface was patched with the release of version 8.0.6. It was subsequently discovered and a PoC was crafted then released by rubina119 and marketed as 0day. While there has been some argument over whether that stretches the definition, I'm sad to say it was 0dh3y enough for me and countless other lazy buggers that never update their Zimbra. Go team!

If you were like me, you might have seen something like this:

top - 17:56:57 up 93 days, 15:06,  1 user,  load average: 6.09, 5.90, 5.87
 4489 zimbra    20   0  458m 2184  920 S 255.4  0.1   7731:52 minerd64

And you may have found this:

# lsof -i | grep minerd64
minerd64  4489  zimbra    4u  IPv4 47747967      0t0  TCP localhost:65535-> (ESTABLISHED)

# whois
% This is the RIPE Database query service.
org-name:       MediaServicePlus Ltd.
org-type:       LIR
address:        Novorogozhskaya 32c3, 212
address:        109029
address:        Moscow
address:        RUSSIAN FEDERATION
Well, OBVIOUSLY Russia. Right?
Well, OBVIOUSLY Russians. Right?

Then this:

# ls /tmp/
1  a  b  meep.pl  minerd32  minerd32.1  minerd32.2  minerd32.3  minerd32.4  minerd64  minerd64.1  minerd64.2  minerd64.3  xd.pl

And three of these things are not like the others:

# ls -lsah /opt/zimbra/zimlets-deployed/
total 84K
4.0K drwxr-xr-x. 21 zimbra zimbra 4.0K Jan 21 01:34 .
4.0K drwxr-xr-x. 51 zimbra zimbra 4.0K Aug 18 15:59 ..
4.0K drwxr-x---.  2 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_adminversioncheck
4.0K drwxr-x---.  2 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_attachcontacts
4.0K drwxr-x---.  2 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_attachmail
4.0K drwxr-x---.  2 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_bulkprovision
4.0K drwxr-x---.  2 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_cert_manager
4.0K drwxr-x---.  2 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_clientuploader
4.0K drwxr-x---.  2 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_date
4.0K drwxr-x---.  4 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_email
4.0K drwxr-x---   2 zimbra zimbra 4.0K Jan 21 01:34 com_zimbra_email_dns
4.0K drwxr-x---   2 zimbra zimbra 4.0K Dec 28 05:26 com_zimbra_example_simplejspaction
4.0K drwxr-x---   2 zimbra zimbra 4.0K Dec 31 16:37 com_zimbra_example_simplejspaction2
4.0K drwxr-x---.  4 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_phone
4.0K drwxr-x---.  2 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_proxy_config
4.0K drwxr-x---.  2 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_srchhighlighter
4.0K drwxr-x---.  2 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_tooltip
4.0K drwxr-x---.  2 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_url
4.0K drwxr-x---.  2 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_viewmail
4.0K drwxr-x---.  2 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_webex
4.0K drwxr-x---.  3 zimbra zimbra 4.0K Jan  9  2013 com_zimbra_ymemoticons

This is the order in which I recommend fixing things:

  • Locate and delete any unusual zimbra admin accounts.
  • Stop zimbra.
  • killall minerd(32|64)
  • Clear /tmp/
  • Mount /tmp/ with tmpfs, nodev,nosuid,noexec to prevent any future executables from running in your /tmp/ directory
  • Delete the bad zimlets
  • Make a backup
  • Download 8.0.6
  • Do an upgrade. Don't forget install.sh's annoying flags like --platform-override and -x.
  • Reset your LDAP and MySQL passwords.
  • Restart zimbra.
  • Check for any additional gifts that may have been left behind.

Obviously, you should have your admin interface listening on a private IP or restricted port wherever possible. Where it isn't, you might like to add some additional layer of security, for example HTTP auth.

This whole thing has me interested in Bitcoin mining again; I've got all sorts of servers that are mostly unused I'm not paying the hydro for. :p

At least we found something cute this time like hash crunching instead of something destructive like spamming or DoS. Right guys?

o/~ You've got to e-li-minate the negative... o/~

Fix e1000e: Detected Hardware Unit Hang

Having periodic connectivity issues and seeing this in your dmesg?

e1000e 0000:02:00.0: eth0: Detected Hardware Unit Hang:
  TDH                  <a2>
  TDT                  <8e>
  next_to_use          <8e>
  next_to_clean        <a2>
  time_stamp           <214ed55a5>
  next_to_watch        <a2>
  jiffies              <214ed6d98>
  next_to_watch.status <0>
MAC Status             <80080783>
PHY Status             <796d>
PHY 1000BASE-T Status  <7800>
PHY Extended Status    <3000>
PCI Status             <10>
e1000e 0000:02:00.0: eth0: Reset adapter
e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

This occurs during normal operation of some 82573-based NICs due to a problematic power saving feature. Fortunately, this can be fixed permanently by altering the NIC's EEPROM. If your card is affected you will see the value 0xDE in the second-last position of the second line when you run:

# ethtool -e eth0
Offset          Values
------          ------
0x0000          00 00 00 00 00 00 30 0b 46 f7 07 10 ff ff 00 24 
0x0010          ff ff ff ff 6b 22 f9 02 14 10 8c 10 86 80 de ac

This value must be changed to 0xDF to disable the feature, which can be accomplished using this bash script: http://e1000.sourceforge.net/files/fixeep-82573-dspd.sh.

# ./fixeep-82573-dspd.sh eth0
eth0: is a "82573E Gigabit Ethernet Controller"
This fixup is applicable to your hardware
executing command: ethtool -E eth0 magic 0x108c8086 offset 0x1e value 0xdf
Change made. You *MUST* reboot your machine before changes take effect!