How to prevent DOS (denial of service) attack?

Let us assume you are hosting a web-application using a Apache Server on a Linux machine. With this configuration you can use one of these to prevent such an attack:

(i) IP Tables – this works at very low network level.

(ii) Mod Security module (with apache) – this ofcourse works at web-server level.

(iii) Mod Evasive (with apache) – this ofcourse works at web-server level. This is closest to what you may need.

(iv) Dos Deflate – this is a script that runs regularly to find offenders and block them via IP Tables.

IP Tables

IP Tables is a Linux module which is normally available by default on Linux machines, if not; you can always install it. It is capable of analyzing (for source IP Address, destination IP Address, destination Port etc.) incoming packets and doing various things with them, like letting them in, rejecting them etc.

Here are some useful commands.

To log all incoming requests

iptables -I INPUT 1 -j LOG –log-level debug –log-prefix “IPTablesLog: “
iptables -I OUTPUT 1 -j LOG –log-level debug –log-prefix “IPTablesLog: “
iptables -I FORWARD 1 -j LOG –log-level debug –log-prefix “IPTablesLog: “

Configure the system log (you need to do this to see logs):

Add this line to /etc/syslog.conf (NOTE: there is a tab between ‘*.debug’ and ‘/var/log/iptables.log’ – and not a space.):
*.debug /var/log/iptables.log

Restart log service:

Restart Syslog
service syslog restart

Flush IP Tables (resets IP tables such that no rules apply):

# iptables -F

Restart IP Tables (any rules added from command line will be lost):

# service iptables restart

Log packets from/to a particular IP:
iptables -I INPUT -s <TheIPHere> -j LOG –log-level debug –log-prefix “IPTablesLog: “
iptables -I OUTPUT -d <TheIPHere> -j LOG –log-level debug –log-prefix “IPTablesLog: “
iptables -I FORWARD d <TheIPHere> LOG –log-level debug –log-prefix “IPTablesLog: “

Log Packets for a particular destination port (always use this as the last statement):
iptables -I INPUT -p tcp –dport 80 -j LOG –log-level debug –log-prefix “IPTablesLog: “

Reject Packets from a particular source for a particular destination port:
iptables -I INPUT -s <TheIPHere> -p tcp  –dport 80 -j REJECT –reject-with icmp-port-unreachable

iptables -A INPUT -s 192.168.0.121 -p tcp  –dport 80 -j DROP

Limit Packets for a particular destination port:
iptables -I INPUT -p tcp –dport 80 -m limit –limit 10/sec –limit-burst 10 -j ACCEPT

If you want to permanently add any of the above rules you can add them to the file:/etc/sysconfig/iptables. And then restart iptables. Note that order of the commands could be important.

Mod Security

This is an module that allows you to configure Apache in a variety of useful ways. For example, preventing uploads of a certain kind, preventing access to pages with a certain extension, and so on…

URL: http://www.modsecurity.org
Download: http://www.modsecurity.org/download/direct.html

Most of the information regarding that is available on its website.
In brief the install instructions are:

a.) Download Mod Security gzip .
b.) Unzip it in some dir.
c.) Configure it by issuing command: # ./configure –with-apxs=<APACHE_HOME>/bin/apxs
d.) Compile: # make

e.) Install: # make install

f.) In <APACHE_HOME>/conf/httpd.conf add following lines (the second line is optional):

LoadFile /usr/lib/libxml2.so
#Next line is optional.
#LoadFile /usr/lib/liblua5.1.so

LoadModule security2_module modules/mod_security2.so

g.) Download the Core Rules (these are example config files for mod security)
unzip them such that they land up inside <APACHE_HOME>/conf/modsecurity/

h.) In <APACHE_HOME>/conf/httpd.conf add following lines:
Include conf/modsecurity/*.conf

The most important conf file in that dirs are:
modsecurity_crs_10_config.conf
modsecurity_crs_30_http_policy.conf

You can edit them as you see fit. And rest of the config files you can move to some other sub-dir, if you do not want them loaded.

i.) Now you can restart Apache, and you should be all set.

Now, that you have mod security, you will wonder where is the option for preventing DOS attacks? Infact, Mod Security by itself cannot prevent DOS; but it can do so in conjunction with another tool called ‘httpd guardian’.  See the SecGuardianLog option on this page: http://www.modsecurity.org/documentation/modsecurity-apache/2.1.0/modsecurity2-apache-reference.html .

This particular httpd guardian tool can also be used independently (in which case it analyzes apache logs) or in conjunction with Mod Security in which case it receives information from Mod Security; either way it analyzes input information to determine action to take. The download location of this particular tool is this:

It is available at: http://www.apachesecurity.net/tools/index.html and the download location of httpd-guardian is: http://apache-tools.cvs.sourceforge.net/viewvc/apache-tools/apache-tools/httpd-guardian?view=markup .

Mod Security is probably a great tool, but it does not seem right for the purpose of preventing DOS attacks – mainly because that is really not built into the tool, and it uses a yet another tool (httpd-guardian) to achieve it. So you get the picture, too many pieces involved etc. I have used Mod Security, but not in conjunction with httpd-guardian, so i may not be the best person to pass judgement in this regard.

Mod Evasive

This is the solution that i finally zeroed on and found to be the best solution. It is easy to configure and test.  And preventing DOS attacks is the primary purpose of this tool.

Install Instructions: http://wiki.leenox.in/index.php/Installing_mod_evasive

Download from: http://www.zdziarski.com/projects/mod_evasive/mod_evasive_1.10.1.tar.gz
Compile and Install: #<APACHE_HOME>/bin/apxs -cia mod_evasive20.c

Add to httpd.conf:
<IfModule mod_evasive.c>
DOSHashTableSize 3097
DOSPageCount 2
DOSSiteCount 50
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 300
DOSLogDir “<PathToYourApacheLogDirHere>”
DOSEmailNotify myemal@email.com
</IfModule>

NOTE: Do not forget that even a single web page can have many images and such things, so loading a single page can also cause many requests to be sent to the server(to load images etc). So, you will need to arriveat an appropriate values for the above params yourself.
Explanation of params (this link is useful http://www.directadmin.com/forum/showthread.php?s=&threadid=10957)

DOSHashTableSize
Size of the hash table. The greater this setting, the more memory is required for the look up table, but also the faster the look ups are processed. This option will automatically round up to the nearest prime number.

DOSPageCount
Number of requests for the same page within the ‘DOSPageInterval’ interval that will get an IP address added to the blocking list.

DOSSiteCount
Same as ‘DOSPageCount’, but corresponds to the number of requests for a given site, and uses the ‘DOSSiteInterval’ interval.

DOSPageInterval
Interval for the ‘DOSPageCount’ threshold in second intervals.

DOSSiteInterval
Interval for the ‘DOSSiteCount’ threshold in second intervals.

DOSBlockingPeriod
Blocking period in seconds if any of the thresholds are met. The user will recieve a 403 (Forbidden) when blocked, and the timer will be reset each time the site gets hit when the user is still blocked.

Logs:
You can still see the log messages here whenever a IP Address is blocked:
- <APACHE_HOME>/logs/error_log ( eg: [error] [client <IPAddressHere>] client denied by server configuration: )

- <APACHE_HOME>/logs/<dos_<IP>>( A file with IP Address in its name is created here if an IP Address is blocked)

- /var/log/messages – ( eg: mod_evasive[6480]: Blacklisting address <IPAddressHere>: possible DoS attack)

You can easily test Mod Evasive. Just keep F5 pressed on a web-page of your site (the browser will load the page again and again very fast), and when mod evasive kicks in it will return 403 error.

NOTE: One thing that did not work for me was that, even when a IP Address was correctly identified and blocked, i could still continue to randomly access the web-application.

DOS Deflate

This is a shell script that runs via cron every minute, and checks the processes on that machine. If there are too many http processes initiated by a particular IP Address, it blocks that IP Address using IPTables, and will unblock the IP Address again after some time (based on configuration i.e in ddos.conf file – see the link below).

http://deflate.medialayer.com/ - this is main URL you want to see.

After looking at it, i thought it required one improvement, i.e it should not just check for all processes but should check for httpd processes (assuming u r running Apache ofcourse). For this change the following line in instrall.sh from:

netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n

to:

netstat -aptn | grep httpd | awk ‘{print $5}’ | cut -d: -f4 | sort | uniq -c | sort -n

Remember that the concept is quite striaght forward, and you can even write your own java program (if you are unconformatable with shell scripts) to achieve the same.

How do i find that i am under DOS attack?

If you are using Mod Evasive, you can see its logs or you may receive a mail from it if that is configured correctly. But, if you are not using any such thing, read on…

The most obvious thing is that your application will become ‘unexpectedly’ overloaded (and therefore unresponsive). Remember, you could be genuinely overloaded too, because your web-application is so popular! So, one key thing is that it would be ‘unexpected’, another that, most of the requests would seem to come from some specific IPs (which would further confirm that this is an auotmated attack).

On Linux, find out if there are a huge number of http processes:

Find if you have huge number of http processes (>100 is not good):
#ps -aux|grep HTTP|wc -l

#ps -aux|grep HTTP

You have more than 30 connection from a single ip. Under normal cases there is no need for that many number of connection requests from a single IP. Try to identify such ips/networks from the list you get.
#netstat -lpn|grep :80 |awk ‘{print $5}’|sort

If you identify suspicious IPs, you can block them using iptables.

NOTE: In case of DDOS (distributed denial of service) there will be more than one IP that would be suspicious, and it would be much more difficult to stop it too, as the source IP may keep changing.

If you are running an Application Server (say tomcat) – all the threads will be occupied (busy), causing the application to become unresponsive.

Is there a even better and robust solution?

Yes ofcourse, the most robust solution to this problem is to be found in expensive Hardware Load Balancers, or FireWalls.

How can i monitor and improve performance of my Apache/Tomcat/MySQL application?

Monitor Apache

You can view status of your Apache Web Server by going to a URL of the kind:

This allows us to view status of Apache by going to a URL like:

http://host/server-status (http://host/server-status?refresh=10 - to refresh every 10 secs)

If the above URL does not work, your apache is probably not configured to return status. For information on how to set it up for this, see Apache Mod Status.

To view the JK Status, go to a URL of the kind:

http://host/jk-status

Monitor-Tomcat

You can view a lot of useful statistics of Tomcat by looking at its Admin Console:

http://<IP:port>/ > click on status link.

Things to look for here are under the approriate headings JVM, jk-8009 (assuming apache is configured to connect via jk on port 8009), http-8080.

JVM

Free Memory – Not sure how this works.

Total Memory – Not sure how this works.

Maximum Memory – this is the maximum amount of memory available to tomcat server, it is the value of the -mx setting in your tomcat startup java command (in catalina.sh). If no mx setting is done, a default value is assumed (which is different for windows and linux)

 

jk-8009 or http-8080

Here the important things to look at are:

Max threads: 200 Current thread count: 28 Current thread busy: 8
Request count: 19 Error count: 0

Max threads – this is maximum number of threads that tomcat can have to process incoming requests (and its internal functions).

Current thread count – this is total number of threads in tomcat’s threadpool at present. Threads will get garbage collected from this pool. And the pool size can grow to ‘Max threads’ size when necessary.

Currrent thread busy – this is total number of threads that are processing variour requests at present.

Request count – total number of http requests received by tomcat since the time it was started.

Improve -Tomcat

You have a performance problem in tomcat when:

Current Thread Count is near about Max Threads. This is a clear indicator that tomcat is receiving more requests faster than it can handle. The default Max Thread setting is 200 which is good enough for most applications, but you can change it here in <TOMCAT_HOME>/conf/server.xml – look for your connector element, here you can change (or add if not present) maxThreads attribute. See http://tomcat.apache.org/tomcat-5.5-doc/config/http.html . But beware, you cannot increase this value indiscrimnately(will affect memory and performance both), the optimum value is based on your hardware, and primarily the number of CPUs you have (more the better for increasing maxThreads). Thumb Rule: Do not change defaults, unless you really have to.

You see Out of Memory Exceptions in your catalina log. See the <TOMCAT_HOME>\logs\catalina.out log file. This is a clear indicator that your application needs more memory than what is available. The min,max memory available can be changed here: <TOMCAT_HOME>\bin\catalina.sh or .bat.  You can add a entry like this somewhere  (A good place would be immediately before this line: # Set juli LogManager if it is present)

(on linux): JAVA_OPTS=”$JAVA_OPTS -ms256M -mx512M”

(on windows): set JAVA_OPTS=-ms256M -mx512M

Monitor MySQL

You can check lots of important mysql params by going to mysql prompt and typing as follows:

mysql> SHOW VARIABLES;

One of things of interest in the response would be: max_connections. This will tell you how many max concurrent connections are allowed.

You can also find out how many concurrent MySQL connections exist at any point in time by using following command:

mysql> show processlist;
The number of rows in the response is equal to number of connections to MySQL at that time.

Improve MySQL

The most important thing you need to find out is if you are running out of db connections. This you can find out by using the above mentioned commands i.e show processlist shows equal number of connections as max_connections. If this happens you need to increase the max_connections.. HOW?

Ofcourse check the usual things in your db connection pool, you pool size cannot be greater than max_connection value ofcourse.

How to see Log of DB Queries being executed?

How to find how much time is being spent in execution of each query?

How to find the amount of data(bytes) being returned by each query?

How to configure Apache/Tomcat in Load Balanced configuration?

You will need to configure both Apache and Tomcat for this purpose. Let us assume you want to load balance two Tomcat Servers against an Apache instance. Apache, first tomcat, second tomcat, all can be on the same machine(hardware) OR they can be in three different machines (each on its own); this configuration will work.

The below mentioned configuration shows all three running on the same host(myhost.mycompany.com). (You must install tomcat twice ofcourse in different directories, so that you have two tomcats, ofcourse since they are on the same host both tomcats must run on different http ports, say first one on 8080, and second one on 9090. This config is present in <Tomcat_Home>/conf/server.xml file).

Apache Configuration

———-<Apache_Home>/conf/workers2.properties file————–

[shm]
info=Scoreboard. Requried for reconfiguration and status with multiprocess servers.
file=anon

# Defines a load balancer named lb. Use even if you only have one machine.
[lb:lb]
worker=ajp13:myhost.mycompany.com:8009
worker=ajp13:myhost.mycompany.com:9009
timeout=30
attempts=2
recovery=90
StickySession=1
noWorkersMsg=Server Busy please retry after some time.
noWorkerCodeMsg=503

# Example socket channel, override port and host.
[channel.socket:myhost.mycompany.com:8009]
port=8009
host=myhost.mycompany.com
tomcatId=server1
group=lb
lbfactor=100

[channel.socket:myhost.mycompany.com:9009]
port=9009
host=myhost.mycompany.com
tomcatId=server2
group=lb
lbfactor=100

[ajp13:myhost.mycompany.com:8009]
channel=channel.socket:server1

[ajp13:myhost.mycompany.com:9009]
channel=channel.socket:server2

# Map the Tomcat webapp to the Web server uri space

[uri:/myapp/*]
group=lb

[status:]
info=Status worker, displays runtime information

[uri:/jk-stinc/*]
info=The Tomcat /jkstatus handler
group=status:

———end of file——————-

Tomcat Confugration

The only file you will have to modify here is:<Tomcat_Home>/conf/server.xml, yes on both the tomcat servers you are trying to load balance. You have to set up connector port (8009 on first tomcat, 9009 on second tomcat), and jvmRoute (server1 on first tomcat, server2 on second tomcat). You will notice that these values you had defined earlier in workers2.properties file.

So, here is how you would update on first tomcat, the file:<Tomcat_Home>/conf/server.xml

So, AJP 1.3 Connector to be defined as follows(if it is commented out, uncomment it). The relevant line should look like this:

 <Connector port=”8009″ protocol=”AJP/1.3″ protocolHandlerClassName=”org.apache.jk.server.JkCoyoteHandler” redirectPort=”8443″ />

jvmRoute is set up as follows:

   <Engine name=”Catalina” defaultHost=”localhost” jvmRoute=”server1″>

Similar changes you must make to the second tomcat.

You are all set.

Tested on: Apache 2.2.6, Tomcat 6, Linux

Enjoy.

How can i (gzip) http responses (using Apache)?

Apache allows you to gzip responses provided the calling application (IE, or any other) is capable of handling zipped responses. The calling application sends information about its capability to handle zipped responses in accept-encoding http request header.

Following configuration is necessary in Apache for this.:

Create a deflate.conf file with following contents in <Apache_Home>/conf/extra

—deflate.conf file—-

<Location />
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE image/svg+xml
AddOutputFilterByType DEFLATE image/png
AddOutputFilterByType DEFLATE image/jpg
AddOutputFilterByType DEFLATE image/gif
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/atom_xml
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE application/x-httpd-php
AddOutputFilterByType DEFLATE application/x-httpd-fastphp
AddOutputFilterByType DEFLATE application/x-httpd-eruby
AddOutputFilterByType DEFLATE application/x-shockwave-flash
AddOutputFilterByType DEFLATE application/x-amf
AddOutputFilterByType DEFLATE text/javascript
</Location>

DeflateFilterNote Input input_info
DeflateFilterNote Output output_info
DeflateFilterNote Ratio ratio_info
LogFormat ‘”%r” %{output_info}n/%{input_info}n (%{ratio_info}n%%)’ deflate
CustomLog /usr/local/Apache-2.2.8/logs/deflate_log deflate

———–end of file ——-

In the above file comment out any response type you are not interested in zipping. So if you do not want images to be compressed, comment out the following lines as shown below:

#AddOutputFilterByType DEFLATE image/png
#AddOutputFilterByType DEFLATE image/jpg
#AddOutputFilterByType DEFLATE image/gif

In <Apache_Home>/conf/httpd.conf file:

Make sure following entry exists:

LoadModule deflate_module modules/mod_deflate.so

#Moddeflate Options file
Include conf/extra/deflate.conf

Thats all you are done. You can see the deflate log: /usr/local/Apache-2.2.8/logs/deflate_log deflate.

Tested in: Apache 2.2.8/Linux

Enjoy.

How can i have Apache forward requests to different app servers based on URL pattern?

Assuming you already know how to set up Apache and Tomcat to talk to each other using workers2.properties file.

The Apache workers2.properties file for such a configuration should look like (More explanation at the end):\

—-workers2.properties file—

[shm]
info=Scoreboard. Requried for reconfiguration and status with multiprocess servers.
file=anon

# Defines a load balancer named lb. Use even if you only have one machine.
[lb:lb]
worker=ajp13:166.208.0.11:8080
timeout=30
attempts=2
recovery=90
StickySession=1
noWorkersMsg=Server Busy please retry after some time.
noWorkerCodeMsg=503

# Defines a load balancer named lb. Use even if you only have one machine.
[lb:lbX]
worker=ajp13:166.208.0.11:9090
timeout=30
attempts=2
recovery=90
StickySession=0
noWorkersMsg=Server Busy please retry after some time.
noWorkerCodeMsg=503

# Example socket channel, override port and host.
[channel.socket:166.208.0.11:8080]
port=8080
host=166.208.0.11
tomcatId=server1
group=lb
lbfactor=100

# Example socket channel, override port and host.
[channel.socket:166.208.0.11:9090]
port=9090
host=166.208.0.11
tomcatId=serverX
group=lbX
lbfactor=100

[ajp13:166.208.0.11:8080]
channel=channel.socket:server1

[ajp13:166.208.0.11:9090]
channel=channel.socket:serverX

[uri:/myapp/craft/*]
group=lbX

[uri:/myapp/*]
group=lb

[status:]
info=Status worker, displays runtime information

[uri:/jk-stinc/*]
info=The Tomcat /jkstatus handler
group=status:

 

———————–

Notice that two load-balance groups lb1 and lbX have been defined. All URLs of the form /myapp/craft are handled by load balance group lbX. All URLs of the form /muapp are handled by load balance group lb. One tomcat server is running166.208.0.11:8080, and another on 166.208.0.11:9090. The latter one will handle all URLs of the form /myapp/craft. While the former one will handle all URLs of the form: /myapp.

Tested in: Apache 2.2.8/Linux