Blocking Abuse by IP Address

IP Abuse Detection Script

This shell script checks the access and error logs generated by apache for a particular domain, looking for the IP addresses that have connected to your site the most. It checks for IP addresses that trigger a Concurrent Connection Limit Exceeded error, which is a good sign they are an automated bot of some kind, making over 20 requests to your site at the same time. This script also checks for Internal Recursion Errors which can have very negative effects on your speed and resources, and are basically internal looping problems generally caused by improperly configured Htaccess setups.

Once the script finishes scanning your logs for those events, it automatically generates .htaccess code that you may add to your sites root .htaccess file to block those IP addresses the script identified as abusive. The only IP addresses included in the generated .htaccess file are those that have no reverse dns.

alt text

Installation

  1. Log in to your account using SSH
  2. Save this code in your $HOME directory as ip-abuse-lookup.sh
    1. Run pico $HOME/ip-abuse-lookup.sh
    2. Copy the code to the screen by clicking the right-mouse-button
    3. Hold down the Ctrl button and then press x to save
  3. Run the command dos2unix -dv $HOME/ip-abuse-lookup.sh to fix line break issues
  4. Run the command chmod -v 744 $HOME/ip-abuse-lookup.sh to make executable

Running the Script

From your $HOME directory (cd $HOME), run ./ip-abuse-lookup.sh to execute the program.

Example Generated .htaccess

This script will also generate code that you can place in your .htaccess file to block specific abusers.

## IP-ABUSE-LOOKUP
Order Allow,Deny
Allow from All
Deny from 6.132.177.129 27.67.117.178 6.135.166.102 8.93.225.133
Deny from 21.194.136.15 22.120.61.3 6.252.139.246 9.64.50.83
Deny from 8.123.144.98 21.249.83.87 29.85.238.28 25.214.237.62
Deny from 22.115.130.23 13.57.156.241 14.121.4.82 6.208.172.177

ip-abuse-lookup.sh

#!/bin/sh
# Version 0.2, 2008-04-20

# User-contributed script. Not sponsored by DreamHost.
# Script created 2008-01-16 by AskApache 

### SHELL OPTIONS
set +o noclobber  # allowed to clobber files
set +o noglob     # globbing on
set +o xtrace     # change to - to enable tracing
set +o verbose    # change to - to enable verbose debugging
set -e            # abort on first error

The full script is here, but the authors has an updated Ip Abuse Blocking with .htaccess page.

htpasswd Password Tutorial

Password Tutorial
If you would like to have a set of web pages that are protected, requiring a username/password to gain access, this tutorial will show you how to set it up. This is geared towards the Unix Apache httpd servers used on holly, lamar, and www.colostate.edu. If you are using another web server, you’ll need to check that server’s documentation to see how to do this.

Steps to Password-protect a Directory
First, create a subdirectory in your web area. For the sake of this tutorial, I have created the “protect” directory. Set the permissions on the directory so that the server has read/execute. I do this by using the local command chgrp-www to set the group to the www group. This is the group that the server runs under at Colorado State University for the lamar, holly and www servers. I have used the -sd flag which sets “set group id” for a directory. This will then force any files you create within the protect directory to the www group, so if you ftp files to this directory they will be automatically readable by the server but not by any other user on the system. I then cd into the protect directory.

cd ~ric/public_html
mkdir protect
chmod g+r,g+x,o-r,o-x protect
chgrp-www -sd protect
cd protect

Next you must create a .htaccess file inside the directory you want protected. You can use either the vi or pico editors on the supported systems mentioned above or ftp the file to this directory. If you are new to unix or know little about vi then I suggest you use the pico editor or ftp the .htaccess file. The command to edit with pico is “pico .htaccess”. The .htaccess file should contain the following lines. The items in bold are things you will want to change depending on the location of the AuthUserFile and content of AuthName.

AuthUserFile /z/ric/secret/.htpasswd
AuthGroupFile /dev/null
AuthName "Ric's protected files"
AuthType Basic

require valid-user

The AuthName is what the user will see when they’re prompted for a password – something to the effect of “Enter the username for Ric’s Protected files”. The AuthUserFile is location of the password file and should be not accessible with a url on the server for security reasons. This is a full unix path and the permissions should be set up like the “protect” directory using the chmod and chgrp-www commands above so the only one that can read this file is the owner and the server. To get the full path of a directory, cd to that directory and enter the command “pwd” to print the working directory path.

Now you’ll have to set up the password file. You’ll need to use the htpasswd program. It is included with the Apache httpd server.

First cd to the directory that contains the password file. In this example the password file is called .htpasswd and is in the directory /z/ric/secret/ as indicated by the AuthUserFile file entry in the .htaccess file. For every username you want to add to the password file, enter the following. (the -c is only required the first time; it indicates that you want to create the .htpasswd file).

cd
Read the rest of this entry »

Russian Style .htaccess rewrite

Forbidding all files:

Deny from all

Allow access from a certain IP address:

Order Allow Deny
Deny from all
Allow from this IP

your_IP is a specific IP

Order Allow Deny
Deny from all
Allow from 192.164.3.199

Forbid access from a certain IP address:

Order Allow Deny
Deny from all
Deny from this IP

Using this IP is similar to the example above.

Forbidding a group of files by mask:

<Files ~ "\.(inc|phps|scgi)$">
Order Allow,Deny
Deny from all
</Files>

Defines access to a file by its extension. For example, forbidding web visitors to access files with the “inc” extension:

<Files ~ "\.(inc)$">
Order Allow,Deny
Deny from all
</Files>

In this example the Apache server can access files with this extension.

Forbidding a particular file:

You can forbid a particular file using its name and extension.

<Files config.inc.php>
Order Allow,Deny
Deny from all
</Files>

This example forbids the file config.inc.php to be accessed. Setting a password

Password for a directory:

AuthName "Private zone"
AuthType Basic
AuthUserFile /home/site/.htpasswd
require valid-user

AuthName will be displayed for the user and can be used to explain authentication request. The value of AuthUserFile defines the location where the file with passwords for accessing this directory is stored. This file is created by a special tool named htpasswd or online at htpasswd generator.

For example, we create the following .htaccess file in the protected directory:

AuthName "Only Authenticated Users"
AuthType Basic
AuthUserFile /pub/site.com/.htpasswd
require valid-user

In this example, the user requesting this directory will read the message “Only Authenticated Users”, the file with passwords for access must be stored in the directory /pub/site.com/ and it must be named .htpasswd . The directory is specified from the server root. If you specify the directory incorrectly, Apache will not be able to read the .htpasswd file and nobody will get access to this directory.

Password for one file only:

Similar to protecting a whole directory with a password, you can set a password for one file only. An example of setting a password to the file private.zip:

<Files private.zip>
AuthName "Users zone"
AuthType Basic
AuthUserFile /home/site/.htpasswd
</Files>

Password for a group of files:

Similarly, you can use

<Files ~ "\.(inc|sql|...other_extensions...)$">

to set password for files by mask. An example of setting a password for accessing all files with the “sql” extension:

<Files ~ "\.(sql)$">
AuthName "Users zone"
AuthType Basic
AuthUserFile /home/site/.htpasswd
</Files>

Checking access rights

Task: there is a directory named a1 containing two subdirectories (a2, a3), there are two access levels for users. The first group can access only a1 and a2, the second group can access all three directories. You should perform authentication only once – when accessing a1, but observe access rights for а2 and а3.

The username and password are requested only once while accessing а1 – if the user has access to а2, the password it not requested again. If the user has no access to а3, he will see the message “Enter the password”.

www.site.com/a1
www.site.com/a1/а2
www.site.com/a1/a3
a1 - common and protected at the same time
а2 and а3 only for certain users.

The .htaccess file for the directory а1:

AuthName "Input password"
AuthType Basic
AuthUserFile "/home/site/htdocs/locked/.htpasswd"
<Files *.*>
require valid-user
</Files>

The .htaccess file for the directory а2:

AuthName "Input password"
AuthType Basic
AuthUserFile "/home/site/htdocs/locked/.htpasswd"
<Files *.*>
require user user1 user2 user3
</Files *.*>

The .htaccess file for the directory а3:

AuthName "Input password"
AuthType Basic
AuthUserFile "/home/site/htdocs/locked/.htpasswd"
<Files *.*>
require user user1 user4 user5
</Files *.*>

Redirecting Visitors

Redirecting to another URL:

To redirect a visitor to http://site.com, add the following to .htaccess

Redirect / http://www.site.com

Displaying different pages depending on the visitor’s IP address:

SetEnvIf REMOTE_ADDR <required_IP> REDIR="redir"
RewriteCond %{REDIR} redir
RewriteRule ^/$ /another_page.html

For example, redirecting visitors with IP 192.167.131.1 to the page index.html:

SetEnvIf REMOTE_ADDR 192.167.131.1 REDIR="redir"
RewriteCond %{REDIR} redir
RewriteRule ^/$ /index.html

Redirecting a visitor when he request certain pages:

It is already for all network viruses and scanners. Now any request with the address /_vti_bin will be automatically redirected to Microsoft:

redirect /_vti_bin http://www.microsoft.com
redirect /scripts http://www.microsoft.com
redirect /MSADC http://www.microsoft.com
redirect /c http://www.microsoft.com
redirect /d http://www.microsoft.com
redirect /_mem_bin http://www.microsoft.com
redirect /msadc http://www.microsoft.com
RedirectMatch (.*)\cmd.exe$ http://www.microsoft.com$1

How to change the default page?

To change the page that will be displayed when a visitor access a directory, write:

DirectoryIndex <necessary page>

It is possible to specify several pages:

DirectoryIndex index.shtml index.php index.php3 index.html index.htm

How to make Apache process SSI directives?

SSI Allows you to “assemble” a page using its parts. You have the code of the menu in one part, the code of the header in another part and the footer in a third part. And the visitor sees a usual page consisting of the code stored in your parts.

Some settings in httpd.conf are required.

Add

Options Includes

After that add the following to the .htaccess file:

AddHandler server-parsed .shtml .shtm .html .htm

We advise you to use the program htpasswd generator

How to process Apache errors yourself?

The most interesting and useful Apache errors are 403-404, 500.

403 – the user has not been authenticated, access denied (Forbidden).
404 – the requested document (file, directory) is not found.
500 – internal server error (for example, an error in the syntax of the .htaccess file).

For the user to see your own error messages for these error, add the following to .htaccess:

ErrorDocument 403 /errors/403.html
ErrorDocument 404 /errors/404.html
ErrorDocument 500 /errors/500.html

If error 404 occurs, the user receives the file errors/403.html.

It is convenient to create your own handler for some errors. Add the following to .htaccess:

ErrorDocument 403 /errors/error.php?403
ErrorDocument 404 /errors/error.php?404
ErrorDocument 500 /errors/error.php?500

How to forbid the contents of a directory to be displayed if it has no index file?

Suppose all graphics used on your site is stored in the ‘img’ directory. A visitor can type the address of this directory in his browser and see the list of all your image files. Of course, it will not cause any damage, but you might forbid the visitor to view this directory as well. Add the following to .htaccess:

Options -Indexes

Is it possible to specify the encoding of all file the browser receives documents in by default?

When the Internet only came to existence and first browsers appeared, it often happened that the browser could not automatically determine which of the Russian encodings a document was written in and the browser displayed a complete mess. To avoid it, specify that all pages will be encoded in Windows-1251:

AddDefaultCharset windows-1251

SOURCE