pitter patter on the keyboard

There appears to be a benign segfault from opendkim when Zimbra is shut down:

opendkim[4057]: segfault at 1f0 ip 0000003044209220 sp 00007fffa235bd48 error 4 in libpthread-2.12.so[3044200000+17000]
opendkim[10772]: segfault at 1f0 ip 0000003044209220 sp 00007fffbd4e47a8 error 4 in libpthread-2.12.so[3044200000+17000]

The only solid reference to this I could find was at http://www.zimbra.com/forums/administrators/58839-mta-logger-logswatch-wont-start-also-opendkim-issue.html. These users appear to be using CentOS 6.3 as well which may indicate the problem is with libpthread and not in fact opendkim.

I just encountered this while making a tar backup on one of my virtual filesystems:

lost page write due to I/O error on xvdb1
end_request: I/O error, dev xvdb1, sector 33707688
Buffer I/O error on device xvdb1, logical block 4213461

Unfortunately, my fs suffered corruption.

Some folks, particularly Ubuntu users with kernel 3.2.x seem to say the answer is adding

barrier=0

to your ext3/ext4 mount options.

In my case I’m pretty sure an OOM error killed something critical to the backend on dom0.

Be sure to check your dom0′s dmesg.

UPDATE Uhh yeah, something actually managed to overflow the dom0′s fs. >.> This is what I was getting in the dom0 dmesg:

loop: Write error at byte offset 17820364800, length 4096.

for which there is very little discussion, but a quick df told me what was going on.

After dealing with kernel panics (from what I am hoping was merely OOMs) by moving my new Zimbra VM to a different box and giving it 4 cores and 4 gigs of ram and 4 gigs of swap I was about 80 accounts deep into my second Courier-IMAP Maildir Migration when it started throwing this error:

service.FAILURE (system failure: ZimbraLdapContext)

dmesg shows:

[ 1299.261973] slapd[10678]: segfault at 444d2d54 eip b778f154 esp b1587e28 error 4

Upon restarting Zimbra everything seems to be fine. Unfortunately, the one reference to this error I have found (http://wiki.zimbra.com/wiki/Uninstall_Instructions_for_Unix_and_Windows_Account_Management_in_Admin_UI) states:

Once this is done, you’re almost out of the woods, but this last step is very, very important. You MUST run slapindex to update the indexes in your Zimbra LDAP database, or you run the risk of having segfault/protection errors that crash the slapd process. So far, this has mostly been observed using Zimbra Network Edition running on Ubuntu 8.04 Server LTS. The exact command is ‘/opt/zimbra/openldap-2.3.43.10z/sbin/slapindex’. The command will probably throw you an error message about “loglevel”. Open the referenced slapd file and temporarily change the log-level to an actual number (49152 is what I usually set it to). Then revert that change after slapindex has run. Start Zimbra again via ‘zmcontrol start’ and watch the processes for about 30 minutes to make sure nothing is amiss. If you get slapd errors, run slapindex again, it’s usually the cause of the problem.

The article has very little to do with what I’m trying to accomplish, but given the versions of Ubuntu and Zimbra I think it’s a fair bet that either/and:

• We’re having the same problem anyway
• Running slapindex might solve my problem too

Unfortunately, the migration moves quite slowly so adding slapindex to the script is out of the question.

I’ll be giving ZCS 8 a shot on RHEL 6 tomorrow. Hooray. >.>