Blocking Abuse by IP Address

IP Abuse Detection Script

This shell script checks the access and error logs generated by apache for a particular domain, looking for the IP addresses that have connected to your site the most. It checks for IP addresses that trigger a Concurrent Connection Limit Exceeded error, which is a good sign they are an automated bot of some kind, making over 20 requests to your site at the same time. This script also checks for Internal Recursion Errors which can have very negative effects on your speed and resources, and are basically internal looping problems generally caused by improperly configured Htaccess setups.

Once the script finishes scanning your logs for those events, it automatically generates .htaccess code that you may add to your sites root .htaccess file to block those IP addresses the script identified as abusive. The only IP addresses included in the generated .htaccess file are those that have no reverse dns.

Installation

1. Log in to your account using SSH
2. Save this code in your $HOME directory as ip-abuse-lookup.sh
   1. Run pico $HOME/ip-abuse-lookup.sh
   2. Copy the code to the screen by clicking the right-mouse-button
   3. Hold down the Ctrl button and then press x to save
3. Run the command dos2unix -dv $HOME/ip-abuse-lookup.sh to fix line break issues
4. Run the command chmod -v 744 $HOME/ip-abuse-lookup.sh to make executable

Running the Script

From your $HOME directory (cd $HOME), run ./ip-abuse-lookup.sh to execute the program.

Example Generated .htaccess

This script will also generate code that you can place in your .htaccess file to block specific abusers.

## IP-ABUSE-LOOKUP
Order Allow,Deny
Allow from All
Deny from 6.132.177.129 27.67.117.178 6.135.166.102 8.93.225.133
Deny from 21.194.136.15 22.120.61.3 6.252.139.246 9.64.50.83
Deny from 8.123.144.98 21.249.83.87 29.85.238.28 25.214.237.62
Deny from 22.115.130.23 13.57.156.241 14.121.4.82 6.208.172.177

ip-abuse-lookup.sh

#!/bin/sh
# Version 0.2, 2008-04-20

# User-contributed script. Not sponsored by DreamHost.
# Script created 2008-01-16 by AskApache 

### SHELL OPTIONS
set +o noclobber  # allowed to clobber files
set +o noglob     # globbing on
set +o xtrace     # change to - to enable tracing
set +o verbose    # change to - to enable verbose debugging
set -e            # abort on first error

The full script is here, but the authors has an updated Ip Abuse Blocking with .htaccess page.

Simple .htpasswd Setup

When you click on a “Restricted Information”, your web browser should prompt you for a user name and password. If it does not you may be using a web browser which is not HTTP authentication capable.

---

NOTE: Please make sure you are setting this up in the directory that you want passwd protected. The .htpasswd file will end up in the wrong directory otherwise.

• The web document to be protected.Actually, access is restricted by directory so all files in the same directory will be protected.
• The file .htaccess.This file should be in the directory which contains the documents to restrict access to. The contents of this file specify the name of the password file.For example if you were user jdoe with web documents in the local directory “/home/je/public_html” that you wanted to restrict access to, your .htaccess might look like: AuthUserFile /home/je/public_html/.htpasswd
AuthName jdo
AuthType Basic
<Limit GET>
require valid-user
</Limit>

  Note that .htaccess will not work if there are extra spaces after AuthUserFile.

• The file .htpasswd.This file contains the passwords of the users.To create the .htpasswd file log in to this server using SSH, change directory to the directory you want to restrict access to, and type:htpasswd -c .htpasswd someuserfor the first user (where someuser is the username). You will then be prompted twice for the user’s password. The -c option causes the .htpasswd file to be created. For each additional user type:

  htpasswd .htpasswd someuser

NOTE: There is no correspondence between the usernames and passwords used for accounts on this server and usernames and passwords in any specific .htpasswd file. A user doesn’t need to have an account on this system in order to be validated for access to files protected by HTTP-based authentication.

---

Further Study

This example limits access by user and password, however you can also restrict access by domain. To find out more read the Mosaic User Authentication Tutorial.

from the article: 5 htaccess Tricks Every Webmaster Should Know

from the article: 5 htaccess Tricks Every Webmaster Should Know

If you’re new to htaccess, here’s a quick introduction. Otherwise, here are 5 sets of htaccess directives every webmaster should know:

1 – Redirect Visitors While You Update Your Site

Update and test your site while visitors are redirected to the page of your choice:

order deny,allow
deny from all
allow from 123.123.123.123

ErrorDocument 403 /page.html

<Files page.html>
allow from all
</Files>

Replace 123.123.123.123 with your IP address. Also replace page.html with the name of the page you want visitors to see.

2 – Display a Custom 404 Error Page

Your server displays a “404 File Not Found” error page whenever a visitor tries to access a page on your site that doesn’t exist.

You can replace the server’s default error page with one of your own that explains the error in plain language and links visitors to your home page. Here’s how to use your own page:

ErrorDocument 404 /404.html

Replace 404.html with the name of the page you want visitors to see.

3 – Handle Moved or Renamed Pages

You’ve moved or renamed a page on your site and you want visitors automatically sent to the new page when they try to access the old one. Use a 301 redirect:

Redirect 301 /old.html http://yoursite.com/new.html

Using a 301 redirect also ensures the page doesn’t lose its search engine ranking.

4 – Prevent Directory Browsing

When there’s no index page in a directory, visitors can look and see what’s inside. Some servers are configured to prevent directory browsing like this. If yours isn’t, here’s how to set it up:

Options All -Indexes

5 – Create User Friendly URLs

Which of the two URLs below looks friendlier?

http://yoursite.com/about
http://yoursite.com/pages/about.html

When it comes to URLs, as long as the meaning is clear, shorter is always better.

With htaccess and an Apache module called mod_rewrite, you can set up URLs however you want. Your server can show the contents of “/pages/about.html” whenever anyone visits “http://yoursite.com/about”. Here are a few examples:

RewriteEngine on
RewriteRule ^about/$    /pages/about.html [L]
RewriteRule ^features/$ /features.php [L]
RewriteRule ^buy/$      /buy.html [L]
RewriteRule ^contact/$  /pages/contact.htm [L]

There’s a lot more to mod_rewrite and htaccess. Check out the links below for more details and tricks.

htpasswd Password Tutorial

Password Tutorial
If you would like to have a set of web pages that are protected, requiring a username/password to gain access, this tutorial will show you how to set it up. This is geared towards the Unix Apache httpd servers used on holly, lamar, and www.colostate.edu. If you are using another web server, you’ll need to check that server’s documentation to see how to do this.

Steps to Password-protect a Directory
First, create a subdirectory in your web area. For the sake of this tutorial, I have created the “protect” directory. Set the permissions on the directory so that the server has read/execute. I do this by using the local command chgrp-www to set the group to the www group. This is the group that the server runs under at Colorado State University for the lamar, holly and www servers. I have used the -sd flag which sets “set group id” for a directory. This will then force any files you create within the protect directory to the www group, so if you ftp files to this directory they will be automatically readable by the server but not by any other user on the system. I then cd into the protect directory.

cd ~ric/public_html
mkdir protect
chmod g+r,g+x,o-r,o-x protect
chgrp-www -sd protect
cd protect

Next you must create a .htaccess file inside the directory you want protected. You can use either the vi or pico editors on the supported systems mentioned above or ftp the file to this directory. If you are new to unix or know little about vi then I suggest you use the pico editor or ftp the .htaccess file. The command to edit with pico is “pico .htaccess”. The .htaccess file should contain the following lines. The items in bold are things you will want to change depending on the location of the AuthUserFile and content of AuthName.

AuthUserFile /z/ric/secret/.htpasswd
AuthGroupFile /dev/null
AuthName "Ric's protected files"
AuthType Basic

require valid-user

The AuthName is what the user will see when they’re prompted for a password – something to the effect of “Enter the username for Ric’s Protected files”. The AuthUserFile is location of the password file and should be not accessible with a url on the server for security reasons. This is a full unix path and the permissions should be set up like the “protect” directory using the chmod and chgrp-www commands above so the only one that can read this file is the owner and the server. To get the full path of a directory, cd to that directory and enter the command “pwd” to print the working directory path.

Now you’ll have to set up the password file. You’ll need to use the htpasswd program. It is included with the Apache httpd server.

First cd to the directory that contains the password file. In this example the password file is called .htpasswd and is in the directory /z/ric/secret/ as indicated by the AuthUserFile file entry in the .htaccess file. For every username you want to add to the password file, enter the following. (the -c is only required the first time; it indicates that you want to create the .htpasswd file).

cd

Russian Style .htaccess rewrite

Forbidding all files:

Deny from all

Allow access from a certain IP address:

Order Allow Deny
Deny from all
Allow from this IP

your_IP is a specific IP

Order Allow Deny
Deny from all
Allow from 192.164.3.199

Forbid access from a certain IP address:

Order Allow Deny
Deny from all
Deny from this IP

Using this IP is similar to the example above.

Forbidding a group of files by mask:

<Files ~ "\.(inc|phps|scgi)$">
Order Allow,Deny
Deny from all
</Files>

Defines access to a file by its extension. For example, forbidding web visitors to access files with the “inc” extension:

<Files ~ "\.(inc)$">
Order Allow,Deny
Deny from all
</Files>

In this example the Apache server can access files with this extension.

Forbidding a particular file:

You can forbid a particular file using its name and extension.

<Files config.inc.php>
Order Allow,Deny
Deny from all
</Files>

This example forbids the file config.inc.php to be accessed. Setting a password

Password for a directory:

AuthName "Private zone"
AuthType Basic
AuthUserFile /home/site/.htpasswd
require valid-user

AuthName will be displayed for the user and can be used to explain authentication request. The value of AuthUserFile defines the location where the file with passwords for accessing this directory is stored. This file is created by a special tool named htpasswd or online at htpasswd generator.

For example, we create the following .htaccess file in the protected directory:

AuthName "Only Authenticated Users"
AuthType Basic
AuthUserFile /pub/site.com/.htpasswd
require valid-user

In this example, the user requesting this directory will read the message “Only Authenticated Users”, the file with passwords for access must be stored in the directory /pub/site.com/ and it must be named .htpasswd . The directory is specified from the server root. If you specify the directory incorrectly, Apache will not be able to read the .htpasswd file and nobody will get access to this directory.

Password for one file only:

Similar to protecting a whole directory with a password, you can set a password for one file only. An example of setting a password to the file private.zip:

<Files private.zip>
AuthName "Users zone"
AuthType Basic
AuthUserFile /home/site/.htpasswd
</Files>

Password for a group of files:

Similarly, you can use

<Files ~ "\.(inc|sql|...other_extensions...)$">

to set password for files by mask. An example of setting a password for accessing all files with the “sql” extension:

<Files ~ "\.(sql)$">
AuthName "Users zone"
AuthType Basic
AuthUserFile /home/site/.htpasswd
</Files>

Checking access rights

Task: there is a directory named a1 containing two subdirectories (a2, a3), there are two access levels for users. The first group can access only a1 and a2, the second group can access all three directories. You should perform authentication only once – when accessing a1, but observe access rights for а2 and а3.

The username and password are requested only once while accessing а1 – if the user has access to а2, the password it not requested again. If the user has no access to а3, he will see the message “Enter the password”.

www.site.com/a1
www.site.com/a1/а2
www.site.com/a1/a3

a1 - common and protected at the same time
а2 and а3 only for certain users.

The .htaccess file for the directory а1:

AuthName "Input password"
AuthType Basic
AuthUserFile "/home/site/htdocs/locked/.htpasswd"
<Files *.*>
require valid-user
</Files>

The .htaccess file for the directory а2:

AuthName "Input password"
AuthType Basic
AuthUserFile "/home/site/htdocs/locked/.htpasswd"
<Files *.*>
require user user1 user2 user3
</Files *.*>

The .htaccess file for the directory а3:

AuthName "Input password"
AuthType Basic
AuthUserFile "/home/site/htdocs/locked/.htpasswd"
<Files *.*>
require user user1 user4 user5
</Files *.*>

Redirecting Visitors

Redirecting to another URL:

To redirect a visitor to http://site.com, add the following to .htaccess

Redirect / http://www.site.com

Displaying different pages depending on the visitor’s IP address:

SetEnvIf REMOTE_ADDR <required_IP> REDIR="redir"
RewriteCond %{REDIR} redir
RewriteRule ^/$ /another_page.html

For example, redirecting visitors with IP 192.167.131.1 to the page index.html:

SetEnvIf REMOTE_ADDR 192.167.131.1 REDIR="redir"
RewriteCond %{REDIR} redir
RewriteRule ^/$ /index.html

Redirecting a visitor when he request certain pages:

It is already for all network viruses and scanners. Now any request with the address /_vti_bin will be automatically redirected to Microsoft:

redirect /_vti_bin http://www.microsoft.com
redirect /scripts http://www.microsoft.com
redirect /MSADC http://www.microsoft.com
redirect /c http://www.microsoft.com
redirect /d http://www.microsoft.com
redirect /_mem_bin http://www.microsoft.com
redirect /msadc http://www.microsoft.com
RedirectMatch (.*)\cmd.exe$ http://www.microsoft.com$1

How to change the default page?

To change the page that will be displayed when a visitor access a directory, write:

DirectoryIndex <necessary page>

It is possible to specify several pages:

DirectoryIndex index.shtml index.php index.php3 index.html index.htm

How to make Apache process SSI directives?

SSI Allows you to “assemble” a page using its parts. You have the code of the menu in one part, the code of the header in another part and the footer in a third part. And the visitor sees a usual page consisting of the code stored in your parts.

Some settings in httpd.conf are required.

Add

Options Includes

After that add the following to the .htaccess file:

AddHandler server-parsed .shtml .shtm .html .htm

We advise you to use the program htpasswd generator

How to process Apache errors yourself?

The most interesting and useful Apache errors are 403-404, 500.

403 – the user has not been authenticated, access denied (Forbidden).
404 – the requested document (file, directory) is not found.
500 – internal server error (for example, an error in the syntax of the .htaccess file).

For the user to see your own error messages for these error, add the following to .htaccess:

ErrorDocument 403 /errors/403.html
ErrorDocument 404 /errors/404.html
ErrorDocument 500 /errors/500.html

If error 404 occurs, the user receives the file errors/403.html.

It is convenient to create your own handler for some errors. Add the following to .htaccess:

ErrorDocument 403 /errors/error.php?403
ErrorDocument 404 /errors/error.php?404
ErrorDocument 500 /errors/error.php?500

How to forbid the contents of a directory to be displayed if it has no index file?

Suppose all graphics used on your site is stored in the ‘img’ directory. A visitor can type the address of this directory in his browser and see the list of all your image files. Of course, it will not cause any damage, but you might forbid the visitor to view this directory as well. Add the following to .htaccess:

Options -Indexes

Is it possible to specify the encoding of all file the browser receives documents in by default?

When the Internet only came to existence and first browsers appeared, it often happened that the browser could not automatically determine which of the Russian encodings a document was written in and the browser displayed a complete mess. To avoid it, specify that all pages will be encoded in Windows-1251:

AddDefaultCharset windows-1251

SOURCE

Basics of password protecting a directory

Here’s the basics of password protecting a directory on your server.

First, you need to create a password file. Exactly how you do this will vary depending on what authentication provider you have chosen. More on that later. To start with, we’ll use a text password file.

This file should be placed somewhere not accessible from the web. This is so that folks cannot download the password file. For example, if your documents are served out of /usr/local/apache/htdocs you might want to put the password file(s) in /usr/local/apache/passwd.

To create the file, use the htpasswd utility that came with Apache. This will be located in the bin directory of wherever you installed Apache. If you have installed Apache from a third-party package, it may be in your execution path.

To create the file, type:

htpasswd -c /usr/local/apache/passwd/passwords rbowen

htpasswd will ask you for the password, and then ask you to type it again to confirm it:

# htpasswd -c /usr/local/apache/passwd/passwords rbowen
New password: mypassword
Re-type new password: mypassword
Adding password for user rbowen

If htpasswd is not in your path, of course you’ll have to type the full path to the file to get it to run. With a default installation, it’s located at /usr/local/apache2/bin/htpasswd

Next, you’ll need to configure the server to request a password and tell the server which users are allowed access. You can do this either by editing the httpd.conf file or using an .htaccess file. For example, if you wish to protect the directory /usr/local/apache/htdocs/secret, you can use the following directives, either placed in the file /usr/local/apache/htdocs/secret/.htaccess, or placed in httpd.conf inside a <Directory /usr/local/apache/apache/htdocs/secret> section.

AuthType Basic
AuthName "Restricted Files"
# (Following line optional)
AuthBasicProvider file
AuthUserFile /usr/local/apache/passwd/passwords
Require user rbowen

Let’s examine each of those directives individually. The AuthType directive selects that method that is used to authenticate the user. The most common method is Basic, and this is the method implemented by mod_auth_basic. It is important to be aware, however, that Basic authentication sends the password from the client to the server unencrypted. This method should therefore not be used for highly sensitive data, unless accompanied by mod_ssl. Apache supports one other authentication method: AuthType Digest. This method is implemented by mod_auth_digest and is much more secure. Most recent browsers support Digest authentication.

The AuthName directive sets the Realm to be used in the authentication. The realm serves two major functions. First, the client often presents this information to the user as part of the password dialog box. Second, it is used by the client to determine what password to send for a given authenticated area.

So, for example, once a client has authenticated in the "Restricted Files" area, it will automatically retry the same password for any area on the same server that is marked with the "Restricted Files" Realm. Therefore, you can prevent a user from being prompted more than once for a password by letting multiple restricted areas share the same realm. Of course, for security reasons, the client will always need to ask again for the password whenever the hostname of the server changes.

The AuthBasicProvider is, in this case, optional, since file is the default value for this directive. You’ll need to use this directive if you are choosing a different source for authentication, such as mod_authn_dbm or mod_authn_dbd.

The AuthUserFile directive sets the path to the password file that we just created with htpasswd. If you have a large number of users, it can be quite slow to search through a plain text file to authenticate the user on each request. Apache also has the ability to store user information in fast database files. The mod_authn_dbm module provides the AuthDBMUserFile directive. These files can be created and manipulated with the dbmmanage program. Many other types of authentication options are available from third party modules in the Apache Modules Database.

Finally, the Require directive provides the authorization part of the process by setting the user that is allowed to access this region of the server. In the next section, we discuss various ways to use the Require directive.

Letting more than one person in

The directives above only let one person (specifically someone with a username of rbowen) into the directory. In most cases, you’ll want to let more than one person in. This is where the AuthGroupFile comes in.

If you want to let more than one person in, you’ll need to create a group file that associates group names with a list of users in that group. The format of this file is pretty simple, and you can create it with your favorite editor. The contents of the file will look like this:

GroupName: rbowen dpitts sungo rshersey

That’s just a list of the members of the group in a long line separated by spaces.

To add a user to your already existing password file, type:

htpasswd /usr/local/apache/passwd/passwords dpitts

You’ll get the same response as before, but it will be appended to the existing file, rather than creating a new file. (It’s the -c that makes it create a new password file).

Now, you need to modify your .htaccess file to look like the following:

AuthType Basic
AuthName "By Invitation Only"
# Optional line:
AuthBasicProvider file
AuthUserFile /usr/local/apache/passwd/passwords
AuthGroupFile /usr/local/apache/passwd/groups
Require group GroupName

Now, anyone that is listed in the group GroupName, and has an entry in the password file, will be let in, if they type the correct password.

There’s another way to let multiple users in that is less specific. Rather than creating a group file, you can just use the following directive:

Require valid-user

Using that rather than the Require user rbowen line will allow anyone in that is listed in the password file, and who correctly enters their password. You can even emulate the group behavior here, by just keeping a separate password file for each group. The advantage of this approach is that Apache only has to check one file, rather than two. The disadvantage is that you have to maintain a bunch of password files, and remember to reference the right one in the AuthUserFile directive.

Possible problems

Because of the way that Basic authentication is specified, your username and password must be verified every time you request a document from the server. This is even if you’re reloading the same page, and for every image on the page (if they come from a protected directory). As you can imagine, this slows things down a little. The amount that it slows things down is proportional to the size of the password file, because it has to open up that file, and go down the list of users until it gets to your name. And it has to do this every time a page is loaded.

A consequence of this is that there’s a practical limit to how many users you can put in one password file. This limit will vary depending on the performance of your particular server machine, but you can expect to see slowdowns once you get above a few hundred entries, and may wish to consider a different authentication method at that time.

Alternate password storage

Because storing passwords in plain text files has the above problems, you may wish to store your passwords somewhere else, such as in a database.

mod_authn_dbm and mod_authn_dbd are two modules which make this possible. Rather than selecting AuthBasicProvider file, instead you can choose dbm or dbd as your storage format.

To select a dbd file rather than a text file, for example:

<Directory /www/docs/private>
AuthName "Private"
AuthType Basic
AuthBasicProvider dbm
AuthDBMUserFile /www/passwords/passwd.dbm
Require valid-user </Directory>

Other options are available. Consult the mod_authn_dbm documentation for more details.

htpasswd information

You should also read the documentation for mod_auth_basic and mod_authz_host which contain some more information about how this all works. mod_authn_alias can also help in simplifying certain authentication configurations.

The various ciphers supported by Apache for authentication data are explained in Password Encryptions.

And you may want to look at the Access Control howto, which discusses a number of related topics.