pitter patter on the keyboard

A lot of AJAX seems to rely on ideal conditions: a server that is both running and in good health, a reliable high speed internet connection, a lack of packet loss etc. Unfortunately, this is not always the case – what happens when your user triggers an event and the daemon is restarting or they are hopping access points/cell towers? What happens if the server load is sky high and only half of the requests are getting through? AJAX isn’t a very robust technology but we can increase the odds of our XMLHttpRequest transactions succeeding with some rudimentary JavaScript.

Newer Msxml XMLHTTP implementations support runtime-definable connection time-outs but that covers at best half of your audience and is therefore irrelevant to our needs. We will be implementing our own cross-browser compatible time-outs with the setTimeout() function. Once the time-out expires we will kill the XHR and repeat the process again, with a slightly longer time-out. The process is repeated as many times as necessary; each time increasing the duration. This is called backoff – similar in concept to but much less complicated than the exponential backoff algorithm Ethernet uses to recover from packet collisions.

The tricky part is to set the initial time-out to a value which gives the browser as long as reasonably possible to send and receive the request under strained conditions. Depending on how long the round trip takes in ideal circumstances a safe value can range anywhere from (plus) less than one second to several seconds – this is going to be unique to your application, resources and network so test thoroughly. If the time-out is too low we will be doing more harm than good by stopping and re-sending the request before it has even had a chance to complete. It might be sensible for you to give the last attempt a huge time-out.

The other major factor to consider is whether “the user knows best” in the given situation or not. If you are giving a visual cue to the user that there is AJAX activity happening in the background or if they expect to see something on the page update once the request has completed they might notice that things are taking a little long to react and re-trigger the event (read: hammer on the button like a mindless idiot). If you don’t want them to interfere with the time-out and re-send procedure take that into account when setting up your trigger – the example we are going to use assumes the user knows best. If they hammer on the button the best case scenario is their request goes through faster than if they left it alone and the worst case scenario is once they stop hammering the time-out and re-load cycle will run its course from the top anyway.

// Initialize global variables
var attempts = 0;
var timeout = 1000;    // In miliseconds
var max_attempts = 5;  // Six from 0
var script_url = ''    // The location of the target processing script
var junk = '';         // The variables we're POSTing, probably set in trigger()
var t = '';

// Cross-Browser XMLHttpRequest()
function createXHR()
{
	try { return new XMLHttpRequest(); } catch(e) {}
	try { return new ActiveXObject("Msxml2.XMLHTTP.7.0"); } catch (e) {}
	try { return new ActiveXObject("Msxml2.XMLHTTP.6.0"); } catch (e) {}
	try { return new ActiveXObject("Msxml2.XMLHTTP.5.0"); } catch (e) {}
	try { return new ActiveXObject("Msxml2.XMLHTTP.4.0"); } catch (e) {}
	try { return new ActiveXObject("Msxml2.XMLHTTP.3.0"); } catch (e) {}
	try { return new ActiveXObject("Msxml2.XMLHTTP"); } catch (e) {}
	try { return new ActiveXObject("Microsoft.XMLHTTP"); } catch (e) {}
	alert("Please enable ActiveX or update your browser.");
	return null;
}

// User calls this function, i.e. onclick event handler
function trigger()
{	
	// Re-set the time-out in case the user interrupts the cycle
	clearTimeout(t);
	attempts = 0;
	timeout = 1000;

	driver();
}

// Handle the state changes of the XHR
function callback()
{
	if (xmlhttp.readyState==4 && xmlhttp.status==200)
	{
		// Success! Re-set the time-out for the next round.
		clearTimeout(t);
		attempts = 0;
		timeout = 1000;

		xmlhttp = '';

		// Perform whatever action we need to do on success.
	}
	else
	{
		// Error handling/Request status indication
	}
}

// The guts of the request
function driver()
{
	xmlhttp = createXHR();

	xmlhttp.open("POST",script_url,true);

	t = setTimeout(function(){timedout();}, timeout);
	xmlhttp.onreadystatechange=function(){callback();}

	xmlhttp.setRequestHeader("Content-type","application/x-www-form-urlencoded");
	xmlhttp.send(junk);
}

// Run this if we time out
function timedout()
{
	if(typeof(xmlhttp) == 'object')
		xmlhttp.abort();

	if(attempts < max_attempts)
	{
		// We timed out. Back off the time-out a bit.
		attempts++;
		timeout = timeout + 200;
		// We're adding 200ms to the timeout each iteration, YMMV.
		// Maybe if attempts == max_attempts make the timeout really huge.

		// Play it again, Sam!
		driver();
	}
	else
	{
		// We failed. Set the timeout back to default for the next round.
		timeout = 1000;
		attempts = 0;
		xmlhttp = '';
		alert("Timed out while sending request to server.");
	}
}

Connections in to and out of your network are working sporadically. Your router’s dmesg is flooded with “ip_conntrack: table full, dropping packet.” What do you do?

This condition occurs when the connection tracking table has reached its limit. Connection tracking is a function of Netfilter that stores information like the source and destination IP addresses, port numbers, protocol type, state and timeout of a two-way connection. This facility lets us create sophisticated and informed Netfilter rules in a way that is not possible to accurately derive on a packet header-by-header basis.

The conntrack table takes the form of a memory structure; if there were no constraints on the size of the table it could conceivably start knocking off userspace processes if it became too large (i.e. under DoS conditions). Entries in the conntrack table expire either when their timeout has been reached or the connection has been properly closed. In cases where connections are not being closed according to protocol (poor network connectivity, DoS, spoof attack, etc.) the table can fill rapidly causing an intermittent denial of service condition on your network.

The most prominent symptom of a full connection tracking table is that your old, running connections (secure shell sessions) will continue to function while it becomes impossible to establish new ones. Worse, as the entries continue to time out and the table keeps filling up you may “get lucky” and establish a new connection here and there, making the situation much more confusing.

Depending on the situation you may have one or two options. If you have gobs and gobs of RAM available or the (for example) attack is low-volume you can adjust the entry limit of the table. First, check what the current limit is:

# cat /proc/sys/net/ipv4/ip_conntrack_max
65536

You can see how full the table currently is by running:

# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
62168

ip_conntrack_max is determined as a multiple of how much RAM the system boots up with but generally stops at 65536 regardless. You may find that this isn’t even enough for a high volume network under normal conditions. We can adjust the limit temporarily thus:

# echo 131072 > /proc/sys/net/ipv4/ip_conntrack_max

If this turns out to be your magic bullet and you’re sure no other actions need to be taken to mitigate your particular situation add the following line to /etc/sysctl.conf:

net.ipv4.netfilter.ip_conntrack_max = 131072

To load the value from sysctl.conf run:

# sysctl -p

If you don’t have the option of throwing more RAM at the problem you may be forced to make an executive decision in the interest of preserving network services for legitimate clients. You can decrease the load on the conntrack table by removing rules that use stateful logic (i.e. containing “-t nat” or “-m state”). The brute force option is to rmmod the ip_conntrack module:

# rmmod ip_conntrack

However this may not be possible in all environments. The other option is to flush your rules and set the default policy to allow:

# iptables -P
# iptables -F

This is also typically the effect of

# /etc/init.d/iptables stop
or
# /etc/init.d/firewall stop