How to Block 100,000+ Individual IP addresses

Question

Introduction

How do you Block large number of IP address from your web application/server. Obviously that can easily be done in PHP or any programming language

$ipList = []; // array list or from database
if (in_array(getIP(), $ipList)) {
    // Log IP & Access information
    header("https://www.google.com.ng/search?q=fool"); // redirect
    exit(); // exit
} 

Or Using htaccess

order allow,deny
deny from 123.45.6.7
deny from 012.34.5.
# .... the list continues
allow from all

The issues

• Am trying to block a whole 100k plus individual IPs not subnets
• Am trying to avoid user getting to PHP before blocking such IP
• 100000+ is over 1.5MB and that is a lot if information to be loading in htaccess all the time
• Database of IP still growing ... and they would be nee to dynamically add more values
• To set bans in iptables for 100000+ is just ridiculous (Might Be wrong)

Stupid Idea

order allow,deny
deny from database    <-------- Not sure if this is possible
allow from all

Question

• Is it possible for htaccess to get the list from database (Redis,Crunchbase,Mongo, MySQL or even Sqlite) ... any
• Is there a visible solution to manage such kind of issue in production
• I know the best solution is Block the IPs at the firewall level is there any way to pragmatically add/remove IP to the firewall

Finally

My approach might be totally wrong ... all I want is a visible solution since spammers and botnets are on the rise ...

Please this has nothing to do with DOS attack its a simple ... get lost response

Update

• Firewall : Cisco PIX 515UR

Answer 1

Something that you can try is keeping a list of the IP addresses you want to block in a text file or convert it to a dbm hash file, then use mod_rewrite's RewriteMap. You'd have to set this up in your server/vhost config. You cannot initialize a map in an htaccess file.

RewriteEngine On
RewriteMap deny_ips txt:/path/to/deny_ips.txt

RewriteCond ${deny_ips:%{REMOTE_ADDR}|0} !=0
RewriteRule ^ - [L,F]

The /path/to/deny_ips.txt file would look something like this:

12.34.56.78 1
11.22.33.44 1
etc.

Essentially, an IP that you want to deny and a space then a "1". Any IP in this text file will cause the server to return a 403 Forbidden. To speed things up a bit you can use the httxt2dbm to generate a dbm hash and then you'd define the mapping as so:

RewriteMap deny_ips dbm:/path/to/deny_ips.dbm

I'm not sure what the performance hit is for using mod_rewrite like this with a lot of IPs, but a quick benchmark test on apache 2.2 running on a 3Ghz i686 under linux, the difference between 5 IPs in the list versus 102418 is negligible. According to ab's output, they're nearly identical.

---

Addressing specific questions:

Is it possible for htaccess to get the list from database (Redis,Crunchbase,Mongo, MySQL or even Sqlite) ... any

Using a rewrite map, you can use the "prg" map type to run an external program for a mapping type. You can then write a perl, php, etc. script to talk to a database in order to look up an IP address. Also note that caveats listed under "Caution". You'd then use this map like you would any other map (RewriteCond ${deny_ips:%{REMOTE_ADDR}|0} !=0). This would essentially create a bottleneck for all requests. Not the best solution for talking to a database.

In apache 2.4 though, there is a dbd/fastdbd map type, which allows you to create queries through mod_dbd. This is a much better option and the mod_dbd module manages connections to the database, pools connections, etc. So the map definition would look something like:

RewriteMap deny_ips "fastdbd:SELECT active FROM deny_ips WHERE source = %s"

Assuming you have a table "deny_ips" with 2 columns "source" (the IP address) and "active" (1 for active, 0 for inactive).

Is there a visible solution to manage such kind of issue in production

If you are storing all of the blocked IPs in the database, it's a matter of managing the contents of your database table. If you are using the dbm map type, I know at least perl has a DBI for managing dbm files, so you can use that to add/remove IP entries from the deny list. I've never used it before so I can't really say much about it. Managing a flat text file is going to be a lot trickier, especially if you plan on removing entries, and not just append to it. Outside of using a database and apache 2.4's mod_dbd, I don't think any of these solutions are out of the box or production ready. It's going to require custom work.

I know the best solution is Block the IPs at the firewall level is there any way to pragmatically add/remove IP to the firewall

For IPtables, there is a perl interface that's marked as Beta, but I've never used it before. There's libiptc but according to netfilter's faq:

Is there an C/C++ API for adding/removing rules?

The answer unfortunately is: No.

Now you might think 'but what about libiptc?'. As has been pointed out numerous times on the mailinglist(s), libiptc was NEVER meant to be used as a public interface. We don't guarantee a stable interface, and it is planned to remove it in the next incarnation of linux packet filtering. libiptc is way too low-layer to be used reasonably anyway.

We are well aware that there is a fundamental lack for such an API, and we are working on improving that situation. Until then, it is recommended to either use system() or open a pipe into stdin of iptables-restore. The latter will give you a way better performance.

So I don't know how viable a libiptc solution is if there's no API stability.