Pi-hole on a VPS

Wednesday February 28, 2018

I am using an iPhone. The mobile browsing experience is not so great since I could not do anything with ads that are always popping up uninvited, despite having Norton Ad Blocker configured correctly to work with Safari. I had been thinking to deploy a VPS with Pi-hole so that I wouldn’t be hugged to death by annoying ads anymore.

A little bit of my opinion what I do not want ads. First, they could be really annoying. Second, they could be really intrusive. Third, sometimes they are laced with scripts for bitcoin mining. Fourth, bad people could extract sensitive information from your browser with some nasty malicious script.

As a security-conscious person (well, I hope I am), preventing these scripts would be a next step towards achieving a secure browsing experience (well, sort of).

Today I decided to purchase a cheap OpenVZ VPS from RamNode ($8 for 3 months) so that I could install Pi-hole ad-blocker, a DNS-based ad-blocking solution. This is different from browser-based ad-blocking extension. The way it works is that Pi-hole searches against its database for domain-name and when it found a hit, it blocks request on the DNS level. Some browser extensions perform ad-blocking by hiding elements, meaning that the ads would still be downloaded to your computer but it will not appear on the web browser.

initial setup

I purchased OpenVZ-based VPS nearest to my current location on RamNode. This would minimize the latency although on human perception skill, the performance impact would be barely noticeable if I chose to host somewhere in Europe. Then, I did some initial warming-ups such as installing some important packages, adding new user with adduser, removing some packages, modifying sshd_config file, etc.

sudo apt update; sudo apt upgrade
sudo apt install git htop zsh tmux vim ncdu curl ufw iptraf nethogs

I am using my generic config for sshd_config with custom port and I explained here how to use iptraf and nethogs. UFW is a good firewall if iptables reads like Greek to you. There is a nice little trick with UFW that I will show you later. Do not enable UFW right after the installation.

Pi-hole uses lighttpd as the HTTP server. Ubuntu 16.04 LTS OpenVZ comes pre-installed with apache2, so that would spell trouble if both are running at the same time. apache2runs on port 80 upon boot thus lighttpd would fail to start, hence we couldn’t access Pi-hole’s web UI. Let’s purge apache2 from the system.

sudo apt purge apache2

installing Pi-hole

curl -sSL https://install.pi-hole.net | bash

A single command does the trick and we will be greeted with installation ncurses-based installation dialog. The installation dialog is pretty much self-explanatory. If our UFW is enabled at this time, the installation would fail. At the end of the installation, the installer would spit out the address of the web interface together with login password.

And here is where things get a little bit, erm, tricky.

By default, the web interface is served through HTTP so if you are trying to log in then someone could sniff your password. Plus, HTTP browsing session is inherently insecure. In theory, I could disable public access to lighttpd port 80 with UFW and reverse-proxy the web UI on a different port with caddy configured with automatic SSL (kindly provided by Let’s Encrypt), but I decided to keep my installation as lean as possible.

insecure HTTP form Firefox shames you when logging in over HTTP.

My current solution?

Let’s keep the port 80 access only within localhost (127.0.0.1) by adding a rule on UFW, then using SSH tunneling to access the web UI. Also, allow the port for DNS (a requirement for Pi-hole to work), which is port 53. Be sure to enable your custom ssh port if you changed it (I did).

sudo ufw allow 53
sudo ufw allow from 127.0.0.1 port 80

That should do it. Now, enable UFW.

sudo ufw enable
sudo ufw status

Then on the local machine (e.g. macOS or linux, it gets a little bit tricky on Windows), start tunneling with this command:

ssh -N -f -L 8888:localhost:80 [email protected]

This command tunnels port 80 on the remote server running Pi-hole to port 8888 on the local machine. In theory, we now should be able to access localhost:8888/admin on our browser to access the Pi-hole web UI.

If everything is working fine, time to change all devices pointing to the shiny Pi-hole DNS server.

how this could go absurdly wrong

First, exposing a good (and somewhat critical service) out in the open internet could bring bad guys to do bad stuff. The Pi-hole team has been asked about potential DNS amplification/reflection attack if a Pi-hole instance is accessible to the public. Port 53 (DNS port) could become a zombie in a DNS amplification/reflection attack, a scary kind of DDoS.

The DNS server that comes together with Pi-hole is dnsmasq, and by design, it is meant only for local networks.

shadowserver You have been visited by Shadowserver.

To demonstrate this point, less than 10 hours of deploying my Pi-hole instance, I was visited by Shadowserver DNS scan. Basically, the brief visit by Shadowserver was telling me that (which a huge WARNING sign, of course) that running an open resolver (i.e. my Pi-hole’s dnsmasq) is dangerous and I should be ashamed of myself and should quit what I am currently doing and start flipping burgers.

Okay just kidding. Let’s do something about this, or at least formulate an idea how we should mitigate this situation.

Here are a few strategies to “secure” an open resolver:

Only allow access to port 53 from a specific (set of) IP. One big flaw with this approach is that residential broadband users are assigned with dynamic IPs. If I could access port 53 on my Pi-hole today, there is no guarantee I could access it tomorrow. One way to circumvent this is to use VPN. Specifically, I can instruct the UFW firewalling Pi-hole’s port 53 to allow access from my private IPsec L2TP server. But my IPsec server is in Amsterdam and I don’t want to constitutively browse with VPN enabled. In theory, I could deploy IPsec L2TP together with my Pi-hole on the same server but this would make it a little bit complex.

Ban bad IPs. In theory, I could blacklist all IPs originated from China, Russia, etc from accessing my port 53. All the countries that are known for deploying bots and manipulating open resolvers based on previously-reported cases. This sounds feasible, but let’s explore other options first. Another reason is that bad bots do exist in the USA. A blanket ban is not always a good ban.

Configuring the dnsmasq itself. I’ve been reading for a while on how to secure a dnsmasq server by configuring itself, but man… tough luck. Even a thread on ServerFault tells you that running an open resolver is something anyone should avoid.

So, what happens now? To be honest, I have no idea. I would like to wait and see, vigilantly monitoring my Pi-hole logs. If I saw something fishy, I will respond with something like this. As far as I could tell, I’ve hardened my VPS enough with no-root no-password SSH and port 80 only accessible via tunneling.