On having Docker Nodes without an external Firewall out in the wild your traditional iptables INPUT chain will have no effect. That means anyone out there can access your Docker Containers. And that is an issue.
So why is that? And how to get things save again?
Pre Docker Installation
Before you had Docker installed on your node all you had to to do is adding your rules in the Chain INPUT of the Filter Table. Your Chain INPUT looked similar to this
iptables -L INPUT -n -v --line-numbers Chain INPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination 1 23673 9179K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 2 316 13083 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 3 310 12640 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 4 256K 12M DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Post Docker Installation
After Docker is installed on your node it will make use of the Chain PREROUTING of the NAT Table and all your incoming requests to your Docker Containers will get forwarded to the Chain FORWARD. The first rule of the Chain FORWARD will forward the requests again to the DOCKER-USER Chain. Now we are coming closer.
Let’s take a look at the prerouting:
iptables -t nat -L DOCKER -n -v Chain DOCKER (2 references) pkts bytes target prot opt in out source destination 0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0 83 4716 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:58080 to:172.17.0.2:8080 4224 231K DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 to:172.17.0.2:443 7164 368K DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:172.17.0.2:80 5 224 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53306 to:172.17.0.5:3306
Now we know why the Chain INPUT has no effect anymore. But how to solve this?
How not to solve it?
Stop Docker manipulating Iptables
A lot of tutorials recommend to stop Docker from manipulating Iptables like this:
ExecStart=/usr/bin/docker daemon -H fd:// --iptables=false
In my case that did not work out due to I ended up in adding those Docker Rules manually due to Production Issues. Docker put them in for a reason and if you are eg working with Docker subnets things get more and more complicated.
How to solve it?
Make use of an external Firewall
The best way to solve this issues is to put a Firewall in front of your nodes.
Add your rules to the Chain DOCKER-USER
Sometimes an external Firewall is not an option. So you have to marry your Firewall Rules with the ones from Docker. This could look like this:
iptables -L DOCKER-USER -n -v --line-numbers Chain DOCKER-USER (1 references) num pkts bytes target prot opt in out source destination 1 43M 380G RETURN all -- * * 172.17.0.0/16 0.0.0.0/0 2 41505 3050K RETURN all -- * * 126.96.36.199/24 0.0.0.0/0 3 2346M 3467G RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 4 0 0 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 5 0 0 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 6 3499 153K DROP all -- * * 0.0.0.0/0 0.0.0.0/0 7 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Keep in mind that your changes are only in memory. So after a restart of your node all your Rules will be gone again. But the tool
iptables-persistent comes to the rescue.
Configure the rules like this:
# Allow traffic from docker network -A DOCKER-USER -s 10.0.41.0/24 -j RETURN # Allow traffic that comes back as a response -A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN # Allow everything send from an ip contained in "whitelist" -A DOCKER-USER -m set --match-set whitelist src -j RETURN # Allow HTTP access -A DOCKER-USER -p tcp -m tcp --dport 80 -j RETURN # Allow HTTPS access -A DOCKER-USER -p tcp -m tcp --dport 443 -j RETURN # Drop everything else -A DOCKER-USER -j DROP
and engage those Rules like this:
netfilter-persistent reload service restart docker-ce
Wait a minute. Restarting Docker in Production? Are you serious?
Yes we found the next issue. You can make use of a tool called
ipset which allows you to create Iptables Rules which point to a list of IPs. Of course also this tool should survive a node restart. So you have to daemonize
ipset like this:
[Service] ExecStart=/sbin/ipset -exist -file /etc/ipset/ipset.conf restore ExecStop=/sbin/ipset -file /etc/ipset/ipset.conf save
Afterwards you can manage your IPs in the file you configured in the Daemon:
create whitelist hash:ip add whitelist 188.8.131.52 add whitelist 184.108.40.206 add whitelist 220.127.116.11
Manipulating the Chain DOCKER-USER with the help of
ipset seems to work. But to be honest, I really recommend you to make use of an external Firewall in front of your nodes. That construct is not easy to understand or to maintain.
Due the position in your Chain is crucial we need to know upfront where to put it.
iptables -L DOCKER-USER -n -v --line-numbers iptables -I DOCKER-USER 42 -p tcp -m tcp --dport 443 -j RETURN
Log dropped messages
To get info about dropped traffic you can engage logging like this. Note that this line has to be inserted exactly one line before your DROP Rule.
iptables -L DOCKER-USER -n -v --line-numbers iptables -I DOCKER-USER 42 -m limit --limit 2/min -j LOG --log-prefix "IPTables DOCKER-USER dropped: " --log-level 4
Afterwards, you can see dropped traffic like this:
tail /var/log/messages | grep "IPTables DOCKER-USER dropped: "
The tool nslookup
Reverse lookup for the IP behind a domain.
The tool whois
Getting further info about some domain.
The tool nmap
Get the list of open ports.
An infinite while loop
Get some traffic on some domain every 2 seconds.
while true; do curl https://example.com; sleep 2; done
The tool ab
Benchmark some domain.
ab -n 50 -c 5 https://www.example.com/