Intro

On having Docker Nodes without an external Firewall out in the wild your traditional iptables INPUT chain will have no effect. That means anyone out there can access your Docker Containers. And that is an issue.

So why is that? And how to get things save again?

The Problem

Pre Docker Installation

Before you had Docker installed on your node all you had to to do is adding your rules in the Chain INPUT of the Filter Table. Your Chain INPUT looked similar to this

iptables -L INPUT -n -v --line-numbers

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1    23673 9179K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
2      316 13083 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80
3      310 12640 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
4     256K   12M DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Post Docker Installation

After Docker is installed on your node it will make use of the Chain PREROUTING of the NAT Table and all your incoming requests to your Docker Containers will get forwarded to the Chain FORWARD. The first rule of the Chain FORWARD will forward the requests again to the DOCKER-USER Chain. Now we are coming closer.

Let’s take a look at the prerouting:

iptables -t nat -L DOCKER -n -v

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0
   83  4716 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:58080 to:172.17.0.2:8080
 4224  231K DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:172.17.0.2:443
 7164  368K DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:172.17.0.2:80
    5   224 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53306 to:172.17.0.5:3306

The Solution

Now we know why the Chain INPUT has no effect anymore. But how to solve this?

How not to solve it?

Stop Docker manipulating Iptables

A lot of tutorials recommend to stop Docker from manipulating Iptables like this:

/etc/systemd/system/docker.service.d/

ExecStart=/usr/bin/docker daemon -H fd:// --iptables=false

In my case that did not work out due to I ended up in adding those Docker Rules manually due to Production Issues. Docker put them in for a reason and if you are eg working with Docker subnets things get more and more complicated.

How to solve it?

Make use of an external Firewall

The best way to solve this issues is to put a Firewall in front of your nodes.

Add your rules to the Chain DOCKER-USER

Sometimes an external Firewall is not an option. So you have to marry your Firewall Rules with the ones from Docker. This could look like this:

iptables -L DOCKER-USER -n -v --line-numbers

Chain DOCKER-USER (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1      43M  380G RETURN     all  --  *      *       172.17.0.0/16        0.0.0.0/0
2    41505 3050K RETURN     all  --  *      *       1.2.3.0/24           0.0.0.0/0
3    2346M 3467G RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
4        0     0 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80
5        0     0 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
6     3499  153K DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0
7        0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Keep in mind that your changes are only in memory. So after a restart of your node all your Rules will be gone again. But the tool iptables-persistent comes to the rescue.

Configure the rules like this:

/etc/iptables/rules.v4

# Allow traffic from docker network
-A DOCKER-USER -s 10.0.41.0/24 -j RETURN

# Allow traffic that comes back as a response
-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN

# Allow everything send from an ip contained in "whitelist"
-A DOCKER-USER -m set --match-set whitelist src -j RETURN

# Allow HTTP access 
-A DOCKER-USER -p tcp -m tcp --dport 80 -j RETURN

# Allow HTTPS access 
-A DOCKER-USER -p tcp -m tcp --dport 443 -j RETURN

# Drop everything else
-A DOCKER-USER -j DROP

and engage those Rules like this:

netfilter-persistent reload
service restart docker-ce

Wait a minute. Restarting Docker in Production? Are you serious?

Yes we found the next issue. You can make use of a tool called ipset which allows you to create Iptables Rules which point to a list of IPs. Of course also this tool should survive a node restart. So you have to daemonize ipset like this:

[Service]
ExecStart=/sbin/ipset -exist -file /etc/ipset/ipset.conf restore
ExecStop=/sbin/ipset -file /etc/ipset/ipset.conf save

Afterwards you can manage your IPs in the file you configured in the Daemon:

/etc/ipset/ipset.conf

create whitelist hash:ip
add whitelist 1.2.3.4
add whitelist 1.2.3.5
add whitelist 1.2.3.6

The Conclusion

Manipulating the Chain DOCKER-USER with the help of iptables-persistent and ipset seems to work. But to be honest, I really recommend you to make use of an external Firewall in front of your nodes. That construct is not easy to understand or to maintain.

Useful stuff

Manipulate Iptables

Due the position in your Chain is crucial we need to know upfront where to put it.

iptables -L DOCKER-USER -n -v --line-numbers
iptables -I DOCKER-USER 42 -p tcp -m tcp --dport 443 -j RETURN

Log dropped messages

To get info about dropped traffic you can engage logging like this. Note that this line has to be inserted exactly one line before your DROP Rule.

iptables -L DOCKER-USER -n -v --line-numbers
iptables -I DOCKER-USER 42 -m limit --limit 2/min -j LOG --log-prefix "IPTables DOCKER-USER dropped: " --log-level 4

Afterwards, you can see dropped traffic like this:

tail /var/log/messages | grep "IPTables DOCKER-USER dropped: "

The tool nslookup

Reverse lookup for the IP behind a domain.

nslookup example.com

The tool whois

Getting further info about some domain.

whois example.com

The tool nmap

Get the list of open ports.

nmap example.com

An infinite while loop

Get some traffic on some domain every 2 seconds.

while true; do curl https://example.com; sleep 2; done

The tool ab

Benchmark some domain.

ab -n 50 -c 5 https://www.example.com/