HELP! My server is under a DDOS attack!

Ok first just take a breath, you are getting nothing done by worrying. This guide is meant to try and help mitigate the effects of an attack that is in progresss. I have compiled a lot of the things that I do but it is very possible I have forgotten something. Please feel free to add suggestions at the bottom if you would do it a different way or if this works good for you. This guide is not meant to take the place of a professional looking at your server but it should give you a very good idea of what is going wrong and point you in the right direction to solving the problem. A lot of this requires you to have an idea of what is happening with your server on a normal basis so you can see what is abnormal with it. It would not be a bad idea to run a few of these tests to see what is normal under your normal busy periods.

First check the load of the server
-----command-----
uptime
-----command-----

If you have a dual processor server your server is going to start slowing down when the loads are above 4. If you only have a single processor server with no hyperthreading much above 1 and you will start having trouble. Chances are if you are looking at this you are already having trouble. If your load is not high but your server is slow it is some sort of a ping attack meant to use all of the available bandwidth.

The bwm-ng is a very handy script you may even want to use simply to monitor the server even when it is performing fine First install bwm-ng from http://www.gropp.org/ which is a very simple way to monitor the servers bandwidth.
-----command-----
cd /usr/local/src
wget http://www.gropp.org/bwm-ng/bwm-ng-0.5.tar.gz
tar -zxf bwm-ng-0.5.tar.gz
cd bwm-ng-0.5
./configure; make; make install
bwm-ng
-----command-----
In the bottom right is the total transfer in KB/s. Keep in mind some providers only provide a 10mbit uplink which is only 1024KB/s. Most servers are not going to be using much more then 800-1500KB/s. This is where knowing your server comes in handy. If you know that your server normally runs at 800k/sec and you see it using 3000k/sec something is obviously wrong. If the bandwidth is only 200-300KB/s the chances are very low you are under some sort of a bandwidth DOS attack.

Most of the attacks in the past few months that I have seen are not bandwidth attacks but rather they attack your services to slow the server down. This is when you will normally see the very high loads. First check how busy apache is:
-----command-----
httpd status
-----command-----
If this does not show anything you need to enable apache status in your httpd.conf

-----command-----
pico -w /etc/httpd/conf/httpd.conf
-----command-----

Look for the following:

<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from localhost
< /Location>

If will probably be commented out or in some way not look like the above, make it look like that. This will only be accessible via localhost so it must be done via the httpd status command listed above.


Now we are going to look at what this all means. Here is the important part of the status:

CPU Usage: u6047.55 s364.33 cu121.44 cs19.23 - 29.3% CPU load
25.5 requests/sec - 0.7 MB/second - 28.6 kB/request
130 requests currently being processed, 63 idle servers

This server in particular is doing 25.5 requests/second which is a decent number for most servers. A busy dual xeon can be doing 40-50 requests/second that all have mysql without too much trouble. If this number is very high you are having some sort of an attack against apache. Note this attack can also be unintentional. If for instance you are unfortunate enough to have your website listed on the frontpage of a news site like www.slashdot.org you site may be "slashdotted" which is simply legitimate users overwhelming your server. I woud also look at the current BW usage, if that number is high you may have a user that has postd a very popular file that is getting hit hard. If this number is very high I would install mod_dosevasive. The guide here will show you how to install and configure it. Another important thing to look at is the number of servers vs idle servers as you may want to adjust this in the httpd.conf later if you enough are not running.

Another very important thing to look at is how many active connetions your server is currently processing.
-----command-----
netstat -n | grep :80 |wc -l

netstat -n | grep :80 | grep SYN |wc -l
-----command-----

The first command will show the number of active connections that are open to your server. Many of the attacks typically seen work by starting a connection to the server and then not sending any reply making the server wait for it to time out. The number of active connections from the first command is going to vary widely but if you are much above 500 you are probably having problems. If the second command is over 100 you are having trouble with a syn attack.


Ok so now we have an idea of what is happening what to do about this. If you have a bandwidth related attack you are pretty much SOL unless your ISP filters it. Even if you block it with a firewall the traffic is still making it to your server which is going to bog it down. Imagine the ethernet cord going into your computer as a highway, once it is full there is very little you can do to go fast on it.

On the other hand if you are having an attack against apache there are a few things you can do to help. First install mod_dosevasive from my guide here. Next thing you will want to do is lower the timeout rates. As I mentioned above many attacks will start a connection then not respond to it will will open a lot of connections to your server and slowly bring it down. By lowering the timeouts the server will drop the connections quicker.

First make a backup copy of the apache config and start to edit it
-----command-----
cp /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf-GOOD
pico -w /etc/httpd/conf/httpd.conf
-----command-----

I would suggest commenting out the current lines in your config with a # and just adding mine right below. Though they are good for a DOS attack they are not really optimal for normal activity.

Timeout 15

KeepAlive Off

KeepAliveTimeout 5

Those are the 3 main configuration options, notice how much lower they are being put. You may have to play with them around a little but those should work fine. Now if you want to adjust the number of servers it would be a good time to adjust them. This is one of those tweaking things that will really depend on how busy your server is. Assuming the server is very busy I would set it at:

MinSpareServers 15
MaxSpareServers 20

If the server is not as busy you can lower the numbers to say 10/15. I would not set them much more then 10 apart unless you are sure of what you are doing. Basically this will help your server respond to a quick burst of traffic as it will not have to open up more processes.

It is also advisable to enable syn cookies which is a way to establish legitmate traffic from the traffic that is illegitimate.
-----command-----
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
-----command-----

That is all that I have for now on this guide. My hope is that it will point you in the right direction. Like I said above it is not supposed to substitute for hiring a professional but I know plenty of people like to do it themselves and want to learn or they just don't have the money. The things I have posted above may not help you in the end depending on the type and size of attack. There are some attacks which very little can be done other then waiting until the storm has passed. As always please post any success stories or questions/comments! :)