Monthly Archives: March 2017

Using ipset with iptables

Published / by Taco Scheltema / Leave a Comment

Some time ago I noticed lots of hacking attempts on some of the servers I manage. Some of them are mail servers where hackers were brute forcing smtp user/password combinations, other servers are web servers with wordpress and magento sites where the logs showed lots of attempts to find vulnerabilities in those sites.

One way of dealing with those is to implement fail2ban which can be efficient if configured right, but I wanted to try and block the majority of those attempts at the firewall. So I started collecting addresses from the logs and started blocking them with normal iptables block rules. This worked for the first 50-60 addresses but soon became unmanageable. Then I found out about publicly available blacklists like blocklist.de and bruteforceblocker so I tried loading block rules based on those lists in the iptables firewalls but that caused iptables to take a few minutes to load(!), it also made the firewalls perform pretty poorly.

So after some investigation I found out about ipset. Ipset allows you to create tables that hold a large amount of ip addresses and or networks (amongst a few other things) that can be queried without a hit on performance.

To set it up you’ll need to install ipset. On Debian this is done as follows

~$ apt-get install ipset

on Yum based systems you’d use

~$ yum install ipset

A simple example of setting up an ipset table with some ip addresses and networks and a matching iptables rule. The list will be called example_list

~$ ipset create example_list hash:net family inet

This creates an empty table. Now we can add addresses and networks to the list, for this example I’ll use addresses from the non-public 10.x.x.x block

~$ ipset add example_list 10.1.1.1
~$ ipset add example_list 10.1.2.0/24
~$ ipset add example_list 10.1.3.2
~$ ipset add example_list 10.1.3.3

To see the contents of the table you use the following command

~$ ipset list example_list
Name: example_list
Type: hash:net
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 16880
References: 0
Members:
10.1.2.0/24
10.1.3.2
10.1.1.1
10.1.3.3

And to check if an address is matched in an ipset table you use this

~$ ipset test example_list 10.1.2.43
10.1.2.43 is in set example_list.

~$ ipset test example_list 10.20.2.100
10.20.2.100 is NOT in set example_list.

You’ll notice that the time for the test command to complete is minimal. Of course there are only 4 entries in the table but this command will perform just as well with 60,000 entries in the table.

For this example, you would use the following iptables rule to block addresses that are contained in the table

iptables -A INPUT -m set --match-set example_list src -j DROP -m comment --comment "ipset: example_list"

Now, adding the iptables rule before the ipset table is created will fail as iptables can’t reference a table that doesn’t exist. On the other hand, the ipset table can’t be removed as long as iptables references it. This also highlights the first issue when implementing this; iptables will fail to start/load when the ipset tables it references haven’t been created and this will cause iptables to not load at boot time. Also, ipset tables are loaded in memory and won’t survive a reboot which means we’ll need to create the ipset tables before iptables starts.

Another potential issue is that when using external blocklists like the one from blocklist.de, they’ll need to be updated regularly. So I’ve written a few scripts to take care of all of this. I’m not claiming that this is the best way of doing things but it’s working well for me.

First script is a script to retrieve ip blocklists, i’ve called it ‘getblocklist.sh’ and resides in /usr/local/sbin. It will work with most blocklists, all it expects is one ip address per line. It will store a local version of the list and only download a new one if the local version is over 24 hours old, this to avoid unnecessary load on the remote server. The script will create an ipset table if it doesn’t exist and flush the table if it does.

/usr/local/sbin/getblocklist.sh
#!/bin/bash
#
# Taco Scheltema Mon 31 Aug 2015 12:04:36 AM CEST
# Modified: Fri 09 Dec 2016 12:31:24 AM CET

# Path to store blocklists
CPATH=/etc/ipset
ipset=/usr/sbin/ipset

if [ ! -d $CPATH ]
then 
    echo "$CPATH does not exist, please create it first"
fi

usage() {
cat<<EOF

 $(basename $0) [-h] -u <url> -l <list>

 -h This help message
 -u blocklist url (txt format, i.e. http://lists.blocklist.de/lists/all.txt)
 -l ipset list name, defaults to ip_blocklist
 -q be quiet

to use the blocklist in your iptables firewall, add a rule like the following:

iptables -A INPUT -m set --match-set <blocklist name> src -j LOG --log-prefix "BLOCKLIST: " -m comment --comment "IP Blocklist match"
iptables -A INPUT -m set --match-set <blocklist name> src -j DROP -m comment --comment "IP Blocklist match"

EOF
}

cleanup(){
 LIST=$1
 URL=$2
 sed -i "s/^\([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\)\(\/[0-9]\{1,2\}\)\?.*/add $LIST \1\2/" $URL
 cat $URL | grep -v -e ':' -e '^#' -e '^$' | sort -u -o $URL
}

unset QUIET
while getopts "hu:l:q" option
do
 case $option in
 h ) usage; exit ;;
 u ) URL=$OPTARG;;
 l ) LIST=$OPTARG;;
 q ) QUIET=1;;
 esac
done

[ -z $URL ] && {
 usage
 exit 1
}
[ -z $LIST ] && {
 usage
 exit 1
}

$ipset -q flush $LIST
if [ $? -gt 0 ]
then
 $ipset create $LIST hash:net family inet
fi

if [ -e $URL ]
then
 cleanup $LIST $URL
 $ipset restore < $URL
else
 if test ! -e $CPATH/${LIST}
 then
 [ -z $QUIET ] && echo "no local file found, downloading..."
 curl -s $URL | grep -v -e '^;' -e '^#' -e '^$' > $CPATH/${LIST}
 elif test `find $CPATH/${LIST} -mmin +1440`
 then
 [ -z $QUIET ] && echo "local file is older than 24 hrs, downloading..."
 curl -s $URL | grep -v -e '^;' -e '^#' -e '^$' > $CPATH/${LIST}
 else
 [ -z $QUIET ] && echo "local file is less than 24 hrs old, using local file"
 fi
 cleanup $LIST $CPATH/${LIST}
 $ipset restore < $CPATH/${LIST} -q
 EXIT=$?
fi

[ -z $QUIET ] && {
 echo
 echo
 echo "Add the following to your iptables recipe, before any allow rules"
 echo "if you use UFW then add this to /etc/ufw/before.rules"
 echo
 echo "-A INPUT -m set --match-set $LIST src -j LOG --log-prefix \"BLOCKLIST: \" -m comment --comment \"IP Blocklist $URL\""
 echo "-A INPUT -m set --match-set $LIST src -j DROP -m comment --comment \"IP Blocklist $URL\""
}

exit $EXIT
# vim: ts=4 sw=4 et

Next file is the iptables rules file

/etc/ipset/rules
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:TRUSTED - [0:0]
:FILTERBEFORE - [0:0]
:PUBLIC - [0:0]
:LOGGER - [0:0]

-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 ! -i lo -j REJECT --reject-with icmp-port-unreachable
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

# Jump to FILTERBEFORE
# Add any service that you want exposed to the internet
# traffic to these ports will first go through the FILTERBEFORE chain before being
# allowed. in FILTERBEFORE the incoming address will be matched against our ipset 
# tables and blocked if a match is found
-A INPUT -p tcp -m tcp -m multiport --dports smtp,smtps,pop3s,imaps -j FILTERBEFORE

# Jump to TRUSTED
# add any ports that you only want to allow from trusted addresses, in the chain TRUSTED
# these are matched against the ipset allow tables that we've created
-A INPUT -p tcp -m tcp -m multiport --dports http,https -j TRUSTED -m comment --comment "Send http & https to chain TRUSTED"
-A INPUT -p tcp -m state --state NEW -m tcp -m multiport --dports ssh -j TRUSTED -m comment --comment "Send ssh to chain TRUSTED"
-A INPUT -p tcp -m state --state NEW -m tcp -m multiport --dports 5666 -j TRUSTED -m comment --comment "Send nrpe to chain TRUSTED"
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

# Stop the noise
-A INPUT -p tcp -m tcp -m multiport --dports telnet,bootps,bootpc -j DROP -m comment --comment "Do not log noisy protocols"

# Anything that gets to here should be blocked, logging is optional but if you do log
# it we'll use rate limiting to avoid our log file filling up
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "[ DENIED ] " --log-level 7
-A INPUT -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -j ACCEPT

###### TRUSTED - only allow known sources ######
## ipset allow rules will be inserted here by /usr/local/sbin/firewall
-A TRUSTED -s 10.1.1.0/24 -j ACCEPT -m comment --comment "Internal network"
-A TRUSTED -s 10.2.2.0/24 -j ACCEPT -m comment --comment "VPN network"
# Add a separate rule for the ip address that you are coming from to avoid being locked out
# replace 10.3.3.122 with the external address of your home connection
-A TRUSTED -s 10.3.3.122 -p tcp -m tcp -m multiport --dports ssh -j ACCEPT -m comment --comment "Allow myself"
# Optionally log anything that still makes it past here (should not happen)
-A TRUSTED -m limit --limit 5/min -j LOG --log-prefix "[ DENIED ] "
# Block anything that makes it past here (should not happen)
-A TRUSTED -j DROP


###### FILTERBEFORE - Block all bad people ######
## ipset rules will be inserted here by /usr/local/sbin/firewall
## anything that isn't matched in the ipset tables jumps to PUBLIC
-A FILTERBEFORE -j PUBLIC

###### PUBLIC - Allow from anywhere ######
-A PUBLIC -j ACCEPT

COMMIT

Next we’ll need a script to load the firewall rules and to fill the ipset tables;

/usr/local/sbin/firewall
#!/bin/bash

# Load iptables rules before interfaces are brought online
# This ensures that we are always protected by the firewall
# On debian, create a symlink as follows:
# ln -s /usr/local/sbin/firewall /etc/network/if-pre-up.d/firewall
# On Centos/RHEL systems, the symlink should be
# ln -s /usr/local/sbin/firewall /sbin/ifup-pre-local
#
# Note: if bad rules are inadvertently (or purposely) saved it could block
# access to the server except via the serial tty interface.
#

RULES=/etc/ipset/rules
CHAIN_ALLOW=TRUSTED
CHAIN_DENY=FILTERBEFORE

## If you use fail2ban on this system you may want to stop it before
## and start it after loading the firewall 
#service fail2ban stop
## load our rules
iptables-restore < $RULES
#service fail2ban start

# Define an array of block- and allow lists
# A list with a name ending on allowed will be added to the
# TRUSTED chain, all others to FILTERBEFORE
# Lists can be a local file with one ip address or cidr per line
# or a URL, the lists are parsed and converted to a format that can be passed
# to ipset 
declare -A ipset_lists=(
 [blocklist_de]="http://lists.blocklist.de/lists/all.txt"
 [bruteforce]="http://danger.rulez.sk/projects/bruteforceblocker/blist.php"
 [spamhaus]="http://www.spamhaus.org/drop/drop.lasso"
 [ci_badguys]="http://cinsscore.com/list/ci-badguys.txt"
 [trusted_allowed]="/etc/ipset/trusted_allowed"
 [localban]="/etc/ipset/localban"
 )

# Iterate over the array above and insert iptables rules accordingly
# for each list, the getblocklist.sh script is called which creates the ipset tables
for elem in ${!ipset_lists[@]}
do
 echo -n "$elem ${ipset_lists[$elem]}: "
 /usr/local/sbin/getblocklist.sh -u ${ipset_lists[$elem]} -l $elem -q
 echo
 if [[ $elem =~ allowed ]]
 then
 iptables -I $CHAIN_ALLOW 1 -m set --match-set $elem src -m state --state NEW -j LOG --log-prefix "[ALLOWED]: $elem: " -m limit -m comment --comment "IP Allowlist ${ipset_lists[$elem]}"
 iptables -I $CHAIN_ALLOW 2 -m set --match-set $elem src -j ACCEPT -m comment --comment "IP Allowlist ${ipset_lists[$elem]}"
 else
 iptables -I $CHAIN_DENY 1 -m set --match-set $elem src -j LOG --log-prefix "[BLOCKED]: $elem: " -m comment --comment "IP Blocklist ${ipset_lists[$elem]}"
 iptables -I $CHAIN_DENY 2 -m set --match-set $elem src -j DROP -m comment --comment "IP Blocklist ${ipset_lists[$elem]}"
 fi
done

# vim: ts=4 sw=4 et

Now create the /etc/ipset directory and for any local list that you have, make sure a file exists. these files can be empty. so for the example above, create /etc/ipset/localban and /etc/ipset/trusted_allowed

mkdir /etc/ipset
touch /etc/ipset/localban
touch /etc/ipset/trusted_allowed

The firewall script should be loaded before the network starts at boot time, to do this, create a symlink. On debian, create a symlink as follows:

ln -s /usr/local/sbin/firewall /etc/network/if-pre-up.d/firewall

On Centos/RHEL systems, the symlink should be

ln -s /usr/local/sbin/firewall /sbin/ifup-pre-local

Now, when you run the firewall script for the first time, it will download the defined blocklists and setup your firewall.