Quad9 is a DNS platform that adds several layers of security. It does this via standard DNS queries/responses.Basically, if a machine on your network queries a known bad hostname, the Quad9 DNS server responds by stating that domain does not exist (NX DOMAIN or non-existent domain). Quad9 also allows you to use DNS over TLS. If you would like a bit more info on Quad9 including some speed benchmarks against other DNS services, I would suggest an earlier article, Quad9 – First Thoughts & Benchmarks.
This post is all about configuring, testing, and troubleshooting Quad9 on pfSense, although many of the same rules apply to nearly any firewall on the market. The DNS resolver on most pfSense distributions is unbound so this documentation was written as such.
29Nov2017 – Originally posted
26Feb2018 – Added unbound option so source IP is recorded with NXDOMAIN response
10Mar2018 – Changed IPv6 secondary address based on feedback
6Apr2018 – Added DNS over TLS section
18July2018 – Clarification to custom unbound options for log-replies
25July2018 – Updated for pfSense upcoming release of version 2.4.4
First things first, after logging into your firewall, go to System -> General Setup so you can change your primary and secondary DNS servers to those of Quad9. While you are there, make sure the “DNS Server Override” and “Disable DNS Forwarder” options are not checked (as shown below). Don’t forget to click ‘Save’ at the bottom.
Primary DNS: 22.214.171.124
Secondary DNS: 126.96.36.199
Primary DNS: 2620:fe::fe
Secondary DNS: 2620:fe::9 <- Quad9 states this is not operational at the moment
The secondary DNS for IPv6 is not permanent. I’ll try to update it when the change occurs, but keep an eye on the Quad 9 FAQ for the most up-to-date info.
Warning! Do not get cute and add in a tertiary DNS *or* think you know better by adding in OpenDNS or Google as a secondary DNS for improved redundancy. If you do this, you will get unexpected results. The DNS servers are *not* queried in order as you might expect. Instead, the servers are either asked simultaneously (dnsmasq) or they flip flop who is the “primary” DNS frequently. In my testing, the Google DNS was just a titch faster so it stayed primary more often than not. In a handful of tests, the 1st query was sent and it didn’t come back fast enough so Google DNS was queried second. Surprisingly, Google DNS still answered first in some instances despite its later start. As a result, the IP address resolved as it normally would and the second/later response of NX Domain was discarded. At any rate, any DNS configuration other than what I have stated above breaks the blocking features of Quad9, which is the main reason for using it IMO.
Next, go to the Services -> DNS Resolver and put a checkmark in the “DNS Query Forwarding” if it isn’t there already (red box below). I would strongly suggest making sure both “DNSSEC” and “Use SSL/TLS for outgoing DNS Queries to Forwarding Servers” (found in the purple boxes below) are checked too. If you are still on a version less than 2.4.4, go to the bottom of this post to add the custom options for DNS over TLS instead. Click ‘Save’ and then click ‘Apply Changes’ at the top.
DNSSEC and DNS over TLS are security enhancements Quad9 offers that many other DNS providers do not. DNS over TLS, for example, forces your pfSense firewall (unbound resolver) to encrypt the DNS transaction as it traverses the internet; what that means is a man-in-the-middle on the internet (or a nosy upstream network provider) can’t see which hostnames you are querying and as important, no one can modify the DNS responses. If you want to double-check whether your DNS over TLS configuration is working, you can follow the “Testing DNS over TLS on pfSense” section below.
You might also notice the “Enable SSL/TLS Service” and “SSL/TLS Listen Port” on the configuration screen. These options are for internal devices/clients and not external, WAN queries. These might be something to look into at a later date to improve your internal network security as more operating systems support it natively. As of July 2018, however, most do not.
Before moving on, I would strongly suggest following the “Log Configuration” section toward the bottom of this post. While it is not 100% necessary, it will make your life easier in the future and it is highly recommended!
Below are some various ways to test your configuration whether in Linux/FreeBSD, Windows, or from the browser using the isitblocked.org domain.
Linux command line – Not working
# dig +short isitblocked.org 188.8.131.52
Linux command line –Working
# dig +short isitblocked.org
Nothing is returned above.
Windows command line – Not working
C:\Users\User>nslookup isitblocked.org Non-authoritative answer: Name: isitblocked.org Addresses: 2607:f1c0:100f:f000::2d1 184.108.40.206
Windows command line – Working
C:\Users\User>nslookup isitblocked.org *** can't find isitblocked.org: Server failed
Assuming your configuration is correct, from a browser the isitblocked.org domain should appear as though it is down (below). If you see anything else or your browser gets re-directed to http://www.isitblocked.org/default, then something isn’t working right.
More Troubleshooting – Present & Future
Either your configuration isn’t working or you realized that at some point, you’re going to need to troubleshoot a domain getting blocked and you wanted to read ahead. 😉 One way to test your new DNS configuration is via the web GUI by going to Diagnostics -> DNS Lookup. Once again, type in isitblocked.org and see the response. If you see a “could not be resolved” as shown in yellow above, congrats! The only issue? That doesn’t necessarily mean everything is working. The DNS lookup via the web GUI performs a sequential lookup of DNS servers. Remember the red warning above? Nonetheless, this can still be a useful step in your troubleshooting endeavors.
For those interested, this is what the “race” looks like during a packet capture despite 220.127.116.11 being “above” 18.104.22.168 in the DNS configuration screen and getting queried first.
IP MYEXTIP.53125 > 22.214.171.124.53 <-- FIRST QUERY (TO QUAD9) IP MYEXTIP.54417 > 126.96.36.199.53 <-- SECOND QUERY (TO GOOGLE) IP 188.8.131.52.53 > MYEXTIP.54417 <-- FIRST RESPONSE (FROM GOOGLE) IP 184.108.40.206.53 > MYEXTIP.53125 <-- SECOND RESPONSE (FROM QUAD9)
Log Config – NXDOMAIN
What’s the best way to troubleshoot? Get familiar with your logs. Better yet, configure them properly when you first set this up. Go to Services -> DNS Resolver -> Advanced Settings and switch the default log level of 1 to 2 (or higher). Don’t forget ‘Save’ and ‘Apply Changes’ once you are done.
Now, you can go to your system logs (Status -> System Logs -> System -> DNS Resolver) and see every DNS query/response. Keep in mind the logs will also show which domains are returning as NXDOMAIN.
Log Config – Add requesting/source IP to NXDOMAIN response
Because of how Quad9 responds to malicious domain queries, you can see first-hand if any devices on your network are trying to contact known bad guys on the internet. By default though, the originating/source IP address is not recorded with the DNS NXDOMAIN response which would make tracking down the offending client/IP a little difficult. To correct this, add “server:” and “log-replies: yes” to the “Custom options” section of the general settings page of your DNS Resolver configuration (Services -> DNS Resolver) as shown in the first image below (highlighted). You can also copy/paste the code section in the gray box. Note: If you have other custom options there (such as the one added by pfBlockerNG), then add the log-replies option on a new line below it as shown in the second image (highlighted). The “server:” directive only needs to be specified once. Thus, if “server:” is specified on a line with pfBlockerNG already, you don’t need to add it before the “log-replies” line. Don’t forget to hit Save then Apply after you’ve finished adding it. It’s also worth mentioning in unbound documentation, it states “Note that it takes time to print these lines which makes the server (significantly) slower.” I have enabled “log-replies” multiple times on various shapes and sizes of pfSense boxes and I’ve never seen a discernible increase in disk I/O, CPU, memory, etc. but your mileage may vary.
server: log-replies: yes
log-replies without an existing pfBlockerNG custom option
log-replies with existing pfBlockerNG custom option
Here are what the logs (Status -> System Logs -> System -> DNS Resolver) look like when querying a random hostname. These are the same logs in your SIEM if you forward your pfSense logs as well.
Feb 26 14:53:08 unbound 91442:0 info: control cmd: stats_noreset Feb 26 14:53:08 unbound 91442:1 info: [my IP address] abbbbbbbbbaaaa22222233333333322244422.com. A IN NXDOMAIN 0.042276 0 132 Feb 26 14:53:08 unbound 91442:1 info: validate(nxdomain): sec_status_insecure Feb 26 14:53:08 unbound 91442:1 info: query response was NXDOMAIN ANSWER
Testing DNS over TLS on pfSense
You can test DNS over TLS by performing a packet capture on your firewall. Go to Diagnostics -> Packet Capture to get started. Just make sure your interface is set to WAN and add 220.127.116.11 as your host address before you start your capture or you’ll get tons of other data. As you can see in the simple connection data below, your queries/responses are now over TCP port 853 (DNS over TLS) instead of the standard DNS query/response (UDP port 53).
23:01:41.410868 IP myip.46451 > 18.104.22.168.853: tcp 0 23:01:41.439209 IP 22.214.171.124.853 > myip.46451: tcp 0 23:01:41.439266 IP myip.46451 > 126.96.36.199.853: tcp 0 23:01:41.439420 IP myip.46451 > 188.8.131.52.853: tcp 307 23:01:41.467633 IP 184.108.40.206.853 > myip.46451: tcp 1448 23:01:41.467710 IP myip.46451 > 220.127.116.11.853: tcp 0 23:01:41.467715 IP 18.104.22.168.853 > myip.46451: tcp 1448 23:01:41.467732 IP myip.46451 > 22.214.171.124.853: tcp 0 23:01:41.467736 IP 126.96.36.199.853 > myip.46451: tcp 42 23:01:41.467784 IP myip.46451 > 188.8.131.52.853: tcp 0
Taking a peek at the saved packet capture in Wireshark, you’ll see more connection data including the TLS handshake. What you won’t see are hostnames, responses, or any other standard query/response data you would find in a normal, clear-text DNS transaction!
Add Security To DNS with DNS over TLS
— These are only necessary if your pfSense version is < 2.4.4 (set for release sometime around September 2018) —
If you are on version 2.4.4 or later, follow the instructions above for the web interface configuration instead!
You’ll see the example below has the “log-replies” setting discussed above. If there are other settings in the custom options, you can safely place these additional options below them.
If you want to copy/paste everything including the log-replies settings, here they are! Make sure you save and apply your settings.
server: log-replies: yes forward-zone: name: "." forward-ssl-upstream: yes forward-addr: 184.108.40.206@853 forward-addr: 220.127.116.11@853
Dallas Haselhorst has worked as an IT and information security consultant for over 20 years. During that time, he has owned his own businesses and worked with companies in numerous industries. Dallas holds several industry certifications and when not working or tinkering in tech, he may be found attempting to mold his daughters into card carrying nerds and organizing BSidesKC.