=^.^=

Filling a Drive with Random Data: urandom, dd and Patience

karma

Filling a block device with random data is a good first step in file system encryption, particularly where there will be no partition table or the partition table itself is to be encrypted. The objective is to make encrypted data indistinguishable from free space, obscuring important metrics like file and partition sizes. Although a single pass is not enough to completely obliterate any trace of previously existing files, tools like shred can be used where this is necessary.

There are two sources of random numbers provided by the linux kernel: /dev/random and /dev/urandom. /dev/random is more "random" than urandom; it takes cues from hardware sources like sound cards and cursor movements to produce output which is not likely to be fingerprinted. Consequentially, random only generates output as entropy becomes available which rules it out for anything intensive like overwriting a hard drive. Urandom is a software-implemented pseudo-random generator which means it does its best to produce genuinely random output from mathematical algorithms. Being based on fixed formulas, the output of urandom is theoretically fingerprintable and reproducible given a set of known factors. The upside is that it is relatively fast while providing arguably more than enough randomness for disk encryption camouflage.

# dd if=/dev/urandom of=/dev/sdb bs=1M

dd is a low-level file swiss army knife. The command above instructs dd to take data from input file /dev/urandom and put it into output file /dev/sdb (our second SATA hard drive). The default block size is too small to efficiently make use of a hard drive's pipe so be sure to add bs=1M or more if you don't want it to take weeks.

Unfortunately, urandom is computationally intensive enough that it is not likely to pump out data as fast as your drive can take it without maxing out a CPU core. It is possible to take advantage of multi-core systems by running multiple instances of dd with clever use of the seek flag.

To give you an idea of the time frame you are looking at, to completely overwrite a 2TB drive on a 2.4GHz Core 2 Quad one core was at 99% utilization for 49 and a half hours:

# dd if=/dev/urandom of=/dev/sdd bs=1M
1907728+1 records in
1907727+1 records out
2000397885440 bytes (2.0 TB) copied, 177862 s, 11.2 MB/s

Contrast 11.2MB/s with the typical 70MB/s+ write speed of this drive. It should be noted that both shred and openssl can be used instead if you would prefer to trade off randomness for speed but 2 days for 2TB isn't all that bad - in my most humble of opinions - and doubles as a sort of burn-in test for new drives.

Documentary for Dinner: We. (2006)

karma

A stunning documentary about world politics, power, war, corporations, deception and exploitation illustrated by footage interspersed with Arundhati Roy's Come September speech against a backdrop of chilling downtempo music.

http://www.archive.org/download/resistcomauWehighquality/We_Arundhati_Roy__256k.wmv
http://www.archive.org/download/WeArundhatiRoyiTunesVersion/We_Unauthorized_Arundhati_Roy.mp4
http://thoughtmaybe.com/we/

http://www.weroy.org/

Gentoo Prefix Errors: Aborting due to QA concerns: there are files installed outside the prefix

karma

Just like it sounds, the package you are trying to emerge is attempting to write files outside of your prefix. Depending on the situation it may be that there is no fix, but it is possible that upgrading eselect or portage and re-syncing the portage tree will solve the problem - as was the case for me this morning when attempting to emerge -e system with the distributed portage snapshot:

>>> Completed installing ncurses-5.9-r02.1 into /Users/karma/gentoo/var/tmp/portage/sys-libs/ncurses-5.9-r02.1/image/

ecompressdir: bzip2 -9 /usr/share/man
 * QA Notice: the following files are outside of the prefix:
 * /usr
 * /usr/lib
 * /usr/lib/pkgconfig
 * /usr/lib/pkgconfig/form.pc
 * /usr/lib/pkgconfig/formw.pc
 * /usr/lib/pkgconfig/menu.pc
 * /usr/lib/pkgconfig/menuw.pc
 * /usr/lib/pkgconfig/ncurses++.pc
 * /usr/lib/pkgconfig/ncurses++w.pc
 * /usr/lib/pkgconfig/ncurses.pc
 * /usr/lib/pkgconfig/ncursesw.pc
 * /usr/lib/pkgconfig/panel.pc
 * /usr/lib/pkgconfig/panelw.pc
 * ERROR: sys-libs/ncurses-5.9-r02.1 failed:
 *   Aborting due to QA concerns: there are files installed outside the prefix
 * 
 * Call stack:
 *   misc-functions.sh, line 1877:  Called install_qa_check
 *   misc-functions.sh, line  253:  Called install_qa_check_prefix
 *   misc-functions.sh, line  900:  Called die
 * The specific snippet of code:
 *   			die 

 * 
 * If you need support, post the output of `emerge --info '=sys-libs/ncurses-5.9-r02.1'`,
 * the complete build log and the output of `emerge -pqv '=sys-libs/ncurses-5.9-r02.1'`.
 * The complete build log is located at '/Users/karma/gentoo/var/tmp/portage/sys-libs/ncurses-5.9-r02.1/temp/build.log'.
 * The ebuild environment file is located at '/Users/karma/gentoo/var/tmp/portage/sys-libs/ncurses-5.9-r02.1/temp/environment'.
 * Working directory: '/Users/karma/gentoo/var/tmp/portage/sys-libs/ncurses-5.9-r02.1/image'
 * S: '/Users/karma/gentoo/var/tmp/portage/sys-libs/ncurses-5.9-r02.1/work/ncurses-5.9'
!!! post install failed; exiting.

>>> Failed to emerge sys-libs/ncurses-5.9-r02.1, Log file:

>>>  '/Users/karma/gentoo/var/tmp/portage/sys-libs/ncurses-5.9-r02.1/temp/build.log'

 * Messages for package sys-libs/ncurses-5.9-r02.1:

 * ERROR: sys-libs/ncurses-5.9-r02.1 failed:
 *   Aborting due to QA concerns: there are files installed outside the prefix
 * 
 * Call stack:
 *   misc-functions.sh, line 1877:  Called install_qa_check
 *   misc-functions.sh, line  253:  Called install_qa_check_prefix
 *   misc-functions.sh, line  900:  Called die
 * The specific snippet of code:
 *   			die "Aborting due to QA concerns: there are files installed outside the prefix"
 * 
 * If you need support, post the output of `emerge --info '=sys-libs/ncurses-5.9-r02.1'`,
 * the complete build log and the output of `emerge -pqv '=sys-libs/ncurses-5.9-r02.1'`.
 * The complete build log is located at '/Users/karma/gentoo/var/tmp/portage/sys-libs/ncurses-5.9-r02.1/temp/build.log'.
 * The ebuild environment file is located at '/Users/karma/gentoo/var/tmp/portage/sys-libs/ncurses-5.9-r02.1/temp/environment'.
 * Working directory: '/Users/karma/gentoo/var/tmp/portage/sys-libs/ncurses-5.9-r02.1/image'
 * S: '/Users/karma/gentoo/var/tmp/portage/sys-libs/ncurses-5.9-r02.1/work/ncurses-5.9'

 * GNU info directory index is up-to-date.

 * IMPORTANT: 2 news items need reading for repository 'gentoo_prefix'.
 * Use eselect news to read news items.

ioss-macbook:~ karma$ 

Install slocate on OS X

karma

OS X used to ship with a version of locate which, when running locate.updatedb would throw a message warning that building the locate db as root could expose the location of all files to any user. As of Lion (10.7) this warning is no longer shown but I haven't found anything which would indicate it's no longer true. Combined with a few other limitations, a case can be made for installing slocate if Spotlight is just not for you.

Slocate has been the standard on Linux distributions for years, though it is losing favour to the more efficient mlocate. What they have in common is that the database is meant to be built as root and it includes ownership and permission information so lesser-priviledged users running m/slocate only get to see the files they have access to. To get slocate running on your mac you may choose to compile it from source or go the lazy route as I have and grab a binary distribution of it from MacPorts.

First, install MacPorts, available at http://www.macports.org/. Open a terminal session. If /opt/local/bin is not a part of your PATH cd there then run:

$ sudo ./port install slocate

Now that locate is installed it will be necessary to create the "slocate" group before creating the database or you will encounter an error like this:

slocate: warning: Could not find the group: slocate in the /etc/group file.
slocate: fatal error: This is a result of the group missing or a corrupted group file.

It is possible to create a new group from the command line with dscl but it requires you to choose a group number. To be tidy and use the internally incrementing group number let's create the group by opening System Preferences > Users & Groups. Click the lock to make changes. Click the + button under the user list. Select Group from the New dropdown. Type 'slocate' and click the "Create Group" button. Once the group has been created add your account to it by clicking the checkbox next to your avatar in the Membership list.

Now we need to create the initial database:

$ sudo slocate -u

If you want your slocate database to update automatically add the contents of /opt/local/etc/daily.slocate to /etc/daily.local or /etc/weekly.local, excluding the line #!/bin/sh - if one already exists.

If you try to use slocate as a regular user (which has been added to the slocate group) in any currently open shells you will encounter the error message:

slocate: fatal error: search_db: open: '/opt/local/var/db/slocate/slocate.db': Permission denied

terminate the shell and start a new one; the old instance of bash was not aware of your account being a part of the new slocate group.

SFTP File-System Abstraction in OS X with OSXFUSE and SSHFS

karma

One of the things I love about KDE is kio's built in, long-time support for sftp:// and fish:// SSH file system abstraction. It lets kdelibs applications like KATE and Konqueror manage files remotely as though they are on the local computer. This saves mountains of time when, for example, working on a live website.

The same kind of functionality can be implemented system-wide with FUSE on Linux and fortunately the same is true for OS X.

First, download and install FUSE for OS X at http://osxfuse.github.com/. FUSE for OS X is a descendent of the abandoned MacFUSE project, be sure to include the MacFUSE abstraction layer during installation.

Now download and install the SSHFS package at https://github.com/osxfuse/sshfs/downloads.

Open Terminal then create a mount point for the remote filesystem and mount it like so:

$ mkdir ~/Documents/remote
$ sshfs user@address:/path ~/Documents/remote

Now you can browse and modify the mounted filesystem in Finder or any other OS X application.