The utility of analysing monitoring data remotely is limited; you end up spending more time waiting for images to be generated onscreen than you do actually analysing the data. So I downloaded the files onto my workstation over FTP to allow me to analyse them locally using Wireshark, and left the server recording. At first glance, the latest trace file looked just like any other web traffic. Web pages were being requested and returned. OK, so I wasn't looking at a botnet-initiated SYN flood.
Rather than just scrolling through the unending list of packets and hoping that something interesting would leap out at me, I turned to Wireshark's stats functions. They tend to keep their heads down, though God alone knows why – too many users with painful memories of unpleasant classroom encounters with statistics I guess. There were no obvious clusters in the endpoint statistics; not if you measured by number of packets nor by the volume of data transferred.
It came as no surprise that TCP port 80 was, by a large margin, at the top of the statistics. I turned to the next set of statistics, 'conversations' – connections, in this case of course for the TCP protocol. A total of 1,797 connections were listed and I quickly skimmed through the list to try to form an overall impression. I noticed an IP address which was generating a large number of very short-lived connections and working with more or less linearly increasing ports. I picked one out and used the convenient pop-up menu to filter out this connection.
Wireshark automatically generates an appropriate display filter and chews its way through the trace to isolate the packets associated with that connection. There were multiple SYN packets and, after having successfully established a connection, an HTTP GET request for the web root directory, which – in this trace at least – drew a blank. So far, nothing further out of the ordinary.
But further investigation of the large number of connections from this IP address showed that this node was repeatedly issuing the same request indiscriminately and at an extremely rapid rate. I filtered out the suspect IP address and looked for a match towards the end of the trace file. The packet list clearly showed how the attacker – it was now clear that was what we were dealing with – had bombarded the web server with meaningless requests.
In the normal run of things, a single user shouldn't be able to bring down a server using requests for static or simple dynamic web pages. The problem was that the commercial forum software was extensively interlinked with the gaming portal and gaming engine by means of its own internal code. Every time the forum was accessed, gaming statistics and internal game counters were loaded from the database. As a result, loading each page constituted a huge amount of work and the attacker was able to carry out a successful attack from just a single IP address.
This was good news and bad news all wrapped into one. Good news, because a single attacker IP address probably meant I would be able to identify the attacker, but bad news, because it meant that just one attacker was all it took to take out the server .
It was time to swing into action. First, I had to find a way to block the attack and make sure that normal users would still be able to use the forum. Secondly, I needed to preserve any evidence relating to the attack – that at least was going to be easy. Then I needed to discover as much as I could about the attacker, in order to take action against him.
First up, I took a look to see if there was a pattern to the attacks that might allow me to determine what attack tool the attacker was using. No dice. There was nothing out of the ordinary in the packets – it could just be as simple as a loop of wget calls.
To defend against the attack, the simplest measure was obviously to use the firewall to block the attacker's IP address. I took a look at www.dnstools.com and pulled up the Whois information for the IP address. It was an address from the AOL UK address space – the attacker was probably holed up in the UK.
Blocking the IP address was only going to give me a little breathing space at best, since as soon as he dialled back into the web, he was going to get a new dynamic IP address and the filter would be busted. Still, as an interim measure I asked the firewall administrator to block that IP address and restart the web server. As the web server filled back up with forum users, I saw from the recording system that the attacker was indeed being blocked and that his SYN requests were going unanswered, since they were not getting through to the server.
The next few days saw us engaged in a game of cat and mouse. As soon as the attacker got hold of a new IP address, he would bombard the forum server, until we could block the new address. I ran through my options. The online game was a small-scale project with very limited resources. That ruled out any kind of solution requiring the use of additional hardware. Rebuilding the web application, maybe with efficient caching of the game status, also, after a quick estimation of the work involved, had to be consigned to the long-term wish list.
The firewall wasn't much help either. With even a halfway up-to-date Linux system, by using
iptables -I INPUT -m state --state NEW -m recent --set
iptables -I INPUT -m state --state NEW -m recent \
--update --seconds 60 --hitcount 11 -j DROP
you could limit the number of attempts to establish a connection from any given IP address to 10 per minute. The firewall administrator assured me, however, that the stone age Debian system on which the firewall was running was just not up to the task (or maybe he just wasn't in the mood for experimenting). Finally, I was left with just one option – to, at least temporarily, block the whole AOL UK IP address space. In doing so we risked blocking honest customers, but constant downtime affecting all of our customers was definitely a much worse option. This countermeasure proved effective and the attacks were over as quickly as they had begun. The server was once again stable.