Quad9 is a DNS platform that adds several layers of security. It does this via standard DNS queries/responses.Basically, if a machine on your network queries a known bad hostname, the Quad9 DNS server responds by stating that domain does not exist (NX DOMAIN or non-existent domain). Quad9 also allows you to use DNS over TLS. If you would like a bit more info on Quad9 including some speed benchmarks against other DNS services, I would suggest an earlier article, Quad9 – First Thoughts & Benchmarks.
This post is all about configuring, testing, and troubleshooting Quad9 on pfSense, although many of the same rules apply to nearly any firewall on the market. The DNS resolver on most pfSense distributions is unbound so this documentation was written as such.
29Nov2017 – Originally posted
26Feb2018 – Added unbound option “log-replies: yes” so source IP is recorded with NXDOMAIN response
10Mar2018 – Changed IPv6 secondary address based on feedback
6Apr2018 – Added DNS over TLS section
First things first, after logging into your firewall, go to System -> General Setup so you can change your primary and secondary DNS servers to those of Quad9. While you are there, make sure the “DNS Server Override” and “Disable DNS Forwarder” options are not checked (as shown below). Don’t forget to click ‘Save’ at the bottom.
Primary DNS: 188.8.131.52
Secondary DNS: 184.108.40.206
Primary DNS: 2620:fe::fe
Secondary DNS: 2620:fe::9 <- Quad9 states this is not operational at the moment
The secondary DNS for IPv6 is not permanent. I’ll try to update it when the change occurs, but keep an eye on the Quad 9 FAQ for the most up-to-date info.
Warning! Do not get cute and add in a tertiary DNS *or* think you know better by adding in OpenDNS or Google as a secondary DNS for improved redundancy. If you do this, you will get unexpected results. The DNS servers are *not* queried in order as you might expect. Instead, the servers are either asked simultaneously (dnsmasq) or they flip flop who is the “primary” DNS frequently. In my testing, the Google DNS was just a titch faster so it stayed primary more often than not. In a handful of tests, the 1st query was sent and it didn’t come back fast enough so Google DNS was queried second. Surprisingly, Google DNS still answered first in some instances despite its later start. As a result, the IP address resolved as it normally would and the second/later response of NX Domain was discarded. At any rate, any DNS configuration other than what I have stated above breaks the blocking features of Quad9, which is the main reason for using it IMO.
Next, go to the Services -> DNS Resolver and put a checkmark in the “DNS Query Forwarding” if it isn’t there already. You can leave the DNSSEC box checked too. Click ‘Save’ and then click ‘Apply Changes’ at the top.
Before moving on, I would strongly suggest following the “Log Configuration” section toward the bottom of this post. While it is not 100% necessary, it will make your life easier in the future and it is highly recommended!
Below are some various ways to test your configuration whether in Linux/FreeBSD, Windows, or from the browser using the isitblocked.org domain.
Linux command line – Not working
# dig +short isitblocked.org 220.127.116.11
Linux command line –Working
# dig +short isitblocked.org
Nothing is returned above.
Windows command line – Not working
C:\Users\User>nslookup isitblocked.org Non-authoritative answer: Name: isitblocked.org Addresses: 2607:f1c0:100f:f000::2d1 18.104.22.168
Windows command line – Working
C:\Users\User>nslookup isitblocked.org *** can't find isitblocked.org: Server failed
Assuming your configuration is correct, from a browser the isitblocked.org domain should appear as though it is down (below). If you see anything else or your browser gets re-directed to http://www.isitblocked.org/default, then something isn’t working right.
More Troubleshooting – Present & Future
Either your configuration isn’t working or you realized that at some point, you’re going to need to troubleshoot a domain getting blocked and you wanted to read ahead. 😉 One way to test your new DNS configuration is via the web GUI by going to Diagnostics -> DNS Lookup. Once again, type in isitblocked.org and see the response. If you see a “could not be resolved” as shown in yellow above, congrats! The only issue? That doesn’t necessarily mean everything is working. The DNS lookup via the web GUI performs a sequential lookup of DNS servers. Remember the red warning above? Nonetheless, this can still be a useful step in your troubleshooting endeavors.
For those interested, this is what the “race” looks like during a packet capture despite 22.214.171.124 being “above” 126.96.36.199 in the DNS configuration screen and getting queried first.
IP MYEXTIP.53125 > 188.8.131.52.53 <-- FIRST QUERY (TO QUAD9) IP MYEXTIP.54417 > 184.108.40.206.53 <-- SECOND QUERY (TO GOOGLE) IP 220.127.116.11.53 > MYEXTIP.54417 <-- FIRST RESPONSE (FROM GOOGLE) IP 18.104.22.168.53 > MYEXTIP.53125 <-- SECOND RESPONSE (FROM QUAD9)
Log Config – NXDOMAIN
What’s the best way to troubleshoot? Get familiar with your logs. Better yet, configure them properly when you first set this up. Go to Services -> DNS Resolver -> Advanced Settings and switch the default log level of 1 to 2 (or higher). Don’t forget ‘Save’ and ‘Apply Changes’ once you are done.
Now, you can go to your system logs (Status -> System Logs -> System -> DNS Resolver) and see every DNS query/response. Keep in mind the logs will also show which domains are returning as NXDOMAIN.
Log Config – Add requesting/source IP to NXDOMAIN response
Because of how Quad9 responds for malicious domain queries, you can see first-hand if any devices on your network are trying to contact known bad guys on the internet. By default though, the originating/source IP address is not recorded with the DNS NXDOMAIN response. To correct this, add “log-replies: yes” to the “Custom options” section of the general settings page of your DNS Resolver configuration (Services -> DNS Resolver) as shown in the highlighted section below. If you have other custom options there (such as one added by pfBlockerNG), simply add the log-replies option on a new line. Don’t forget to hit Save then Apply after you’ve finished adding it. It’s also worth mentioning the unbound documentation warns when using either the “log-queries” or “log-replies” options. From the unbound documentation, “Note that it takes time to print these lines which makes the server (significantly) slower.” I enabled “log-replies” and I didn’t see a discernible increase in disk I/O, CPU, memory, etc. but your mileage may vary.
Here are what the logs (Status -> System Logs -> System -> DNS Resolver) look like when querying a random hostname. These are the same logs in your SIEM if you forward your pfSense logs as well.
Feb 26 14:53:08 unbound 91442:0 info: control cmd: stats_noreset Feb 26 14:53:08 unbound 91442:1 info: [my IP address] abbbbbbbbbaaaa22222233333333322244422.com. A IN NXDOMAIN 0.042276 0 132 Feb 26 14:53:08 unbound 91442:1 info: validate(nxdomain): sec_status_insecure Feb 26 14:53:08 unbound 91442:1 info: query response was NXDOMAIN ANSWER
Add Security To DNS with DNS over TLS
Another security enhancement Quad9 offers that many other DNS providers do not is DNS over TLS. Using this option forces your pfSense firewall (unbound resolver) to encrypt the DNS transaction as it traverses the internet, i.e. a man-in-the-middle on the internet (or a nosy upstream network provides) can’t see which hostnames you are querying and as important, no one can modify the DNS responses.
If you’ve followed the logging directions above, you’ll see the example below has the “log-replies” setting. If there are other settings in the custom options, you can safely place these additional options below them.
If you want to copy/paste everything including the log-replies settings, here they are! Make sure you save and apply your settings.
server: log-replies: yes forward-zone: name: "." forward-ssl-upstream: yes forward-addr: 22.214.171.124@853 forward-addr: 126.96.36.199@853
You can test this by performing a packet capture on your firewall. Go to Diagnostics -> Packet Capture to get started. Just make sure your interface is set to WAN and add 188.8.131.52 as your host address before you start your capture or you’ll get tons of other data. As you can see in the simple connection data below, your queries/responses are now over TCP port 853 (DNS over TLS) instead of the standard DNS query/response (UDP port 53).
23:01:41.410868 IP myip.46451 > 184.108.40.206.853: tcp 0 23:01:41.439209 IP 220.127.116.11.853 > myip.46451: tcp 0 23:01:41.439266 IP myip.46451 > 18.104.22.168.853: tcp 0 23:01:41.439420 IP myip.46451 > 22.214.171.124.853: tcp 307 23:01:41.467633 IP 126.96.36.199.853 > myip.46451: tcp 1448 23:01:41.467710 IP myip.46451 > 188.8.131.52.853: tcp 0 23:01:41.467715 IP 184.108.40.206.853 > myip.46451: tcp 1448 23:01:41.467732 IP myip.46451 > 220.127.116.11.853: tcp 0 23:01:41.467736 IP 18.104.22.168.853 > myip.46451: tcp 42 23:01:41.467784 IP myip.46451 > 22.214.171.124.853: tcp 0
Taking a peek at the saved packet capture in Wireshark, you’ll see more connection data including the TLS handshake. What you won’t see are hostnames, responses, or any other standard query/response data you would find in a normal, clear-text DNS transaction!