The simplest thing we can do to throw off an automated attack is to initially throw a failure state. While a lot of legitimate client software is designed to be tolerant (i.e. retry the connection if the first attempt fails) attackers have no reason to consider user friendliness when they design their bots and poking at a seemingly unresponsive host is a waste of time that could be better spent on the next target.
We can use Netfilter to introduce a crude delay (a single knock port knock, if you will...) to new connections and even if our attacker sticks around intentional delays severely limit the effectiveness of brute force cracking:
iptables -A INPUT -p tcp -i eth0 -m conntrack --ctstate NEW --dport 22 -m recent --update --seconds 15 -j DROP
iptables -A INPUT -p tcp -i eth0 -m conntrack --ctstate NEW --dport 22 -m recent --set -j ACCEPT
We can be more ideologically correct by monitoring new connections and drop them when there have been too many in a given time frame. By using two thresholds we can immediately block fast-acting attackers before they get more than a few tries in and take out rate-limited bots that generate a lot of attempts over a longer time without (hopefully) intercepting users that have forgotten their password:
iptables -N IN_SSH
iptables -A INPUT -p tcp --dport ssh -m conntrack --ctstate NEW -j IN_SSH
iptables -A IN_SSH -m recent --name sshbf --rttl --rcheck --hitcount 3 --seconds 10 -j DROP
iptables -A IN_SSH -m recent --name sshbf --rttl --rcheck --hitcount 10 --seconds 1800 -j DROP
iptables -A IN_SSH -m recent --name sshbf --set -j ACCEPT
We may need to limit the number of connections to a service, particularly if establishing a new connection is a very resource-intense proposition.
iptables -A INPUT -p tcp --dport 1935 -m conntrack --ctstate NEW -m limit --limit 5/second --limit-burst 20 -j ACCEPT
iptables -A input -p tcp --dport 1935 -m conntrack --ctstate NEW -j DROP
The limit module applies to packets in general and wouldn't apply to whole connections if we weren't specifying the NEW state; as the name implies it's more suited to rate limiting the speed of traffic flows. Our limit rule also doesn't care how many connections are coming from each individual address. A better way to implement whole-connection limiting might be with the connlimit module:
iptables -A INPUT -p tcp -m tcp --dport 1935 -m connlimit --connlimit-above 5 --connlimit-mask 32 -j REJECT --reject-with tcp-reset
--connlimit-mask refers to the subnet mask, meaning this rule will apply to indivudal IPs when set to 32. We can broaden it to apply to an entire class C subnet by changing it to 24.