Configuring Quad9 on pfSense

Configuring Quad9 on pfSense

Quad9 is a DNS platform that adds several layers of security. It does this via standard DNS queries/responses.Basically, if a machine on your network queries a known bad hostname, the Quad9 DNS server responds by stating that domain does not exist (NX DOMAIN or non-existent domain). Quad9 also allows you to use DNS over TLS. If you would like a bit more info on Quad9 including some speed benchmarks against other DNS services, I would suggest an earlier article, Quad9 – First Thoughts & Benchmarks.

This post is all about configuring, testing, and troubleshooting Quad9 on pfSense, although many of the same rules apply to nearly any firewall on the market. The DNS resolver on most pfSense distributions is unbound so this documentation was written as such.

Changelog
29Nov2017 – Originally posted
26Feb2018 – Added unbound option so source IP is recorded with NXDOMAIN response
10Mar2018 – Changed IPv6 secondary address based on feedback
6Apr2018 – Added DNS over TLS section
18July2018 – Clarification to custom unbound options for log-replies
25July2018 – Updated for pfSense upcoming release of version 2.4.4

pfSense Configuration

First things first, after logging into your firewall, go to System -> General Setup so you can change your primary and secondary DNS servers to those of Quad9. While you are there, make sure the “DNS Server Override” and “Disable DNS Forwarder” options are not checked (as shown below). Don’t forget to click ‘Save’ at the bottom.

IPv4
Primary DNS: 9.9.9.9

Secondary DNS: 149.112.112.112
IPv6
Primary DNS: 2620:fe::fe
Secondary DNS: 2620:fe::9 <- Quad9 states this is not operational at the moment
The secondary DNS for IPv6 is not permanent. I’ll try to update it when the change occurs, but keep an eye on the Quad 9 FAQ for the most up-to-date info. 

Warning! Do not get cute and add in a tertiary DNS *or* think you know better by adding in OpenDNS or Google as a secondary DNS for improved redundancy. If you do this, you will get unexpected results. The DNS servers are *not* queried in order as you might expect. Instead, the servers are either asked simultaneously (dnsmasq) or they flip flop who is the “primary” DNS frequently. In my testing, the Google DNS was just a titch faster so it stayed primary more often than not. In a handful of tests, the 1st query was sent and it didn’t come back fast enough so Google DNS was queried second. Surprisingly, Google DNS still answered first in some instances despite its later start. As a result, the IP address resolved as it normally would and the second/later response of NX Domain was discarded. At any rate, any DNS configuration other than what I have stated above breaks the blocking features of Quad9, which is the main reason for using it IMO.

Change pfSense DNS servers to Quad9

Next, go to the Services -> DNS Resolver and put a checkmark in the “DNS Query Forwarding” if it isn’t there already (red box below). I would strongly suggest making sure both “DNSSEC” and “Use SSL/TLS for outgoing DNS Queries to Forwarding Servers” (found in the purple boxes below) are checked too. If you are still on a version less than 2.4.4, go to the bottom of this post to add the custom options for DNS over TLS instead. Click ‘Save’ and then click ‘Apply Changes’ at the top.

pfSense DNS resolver - DNS over TLS

DNSSEC and DNS over TLS are security enhancements Quad9 offers that many other DNS providers do not. DNS over TLS, for example, forces your pfSense firewall (unbound resolver) to encrypt the DNS transaction as it traverses the internet; what that means is a man-in-the-middle on the internet (or a nosy upstream network provider) can’t see which hostnames you are querying and as important, no one can modify the DNS responses. If you want to double-check whether your DNS over TLS configuration is working, you can follow the “Testing DNS over TLS on pfSense” section below.

You might also notice the “Enable SSL/TLS Service” and “SSL/TLS Listen Port” on the configuration screen. These options are for internal devices/clients and not external, WAN queries. These might be something to look into at a later date to improve your internal network security as more operating systems support it natively. As of July 2018, however, most do not.

Before moving on, I would strongly suggest following the “Log Configuration” section toward the bottom of this post. While it is not 100% necessary, it will make your life easier in the future and it is highly recommended!

Testing/Troubleshooting

Below are some various ways to test your configuration whether in Linux/FreeBSD, Windows, or from the browser using the isitblocked.org domain.

Linux command line – Not working

# dig +short isitblocked.org
74.208.236.124

Linux command line –Working

# dig +short isitblocked.org

Nothing is returned above.

Windows command line – Not working

C:\Users\User>nslookup isitblocked.org
Non-authoritative answer:
Name:    isitblocked.org
Addresses:  2607:f1c0:100f:f000::2d1
          74.208.236.124

Windows command line – Working

C:\Users\User>nslookup isitblocked.org
*** can't find isitblocked.org: Server failed

Browser

Assuming your configuration is correct, from a browser the isitblocked.org domain should appear as though it is down (below). If you see anything else or your browser gets re-directed to http://www.isitblocked.org/default, then something isn’t working right.

NX DOmain firefox

More Troubleshooting – Present & Future

Either your configuration isn’t working or you realized that at some point, you’re going to need to troubleshoot a domain getting blocked and you wanted to read ahead. 😉 One way to test your new DNS configuration is via the web GUI by going to Diagnostics -> DNS Lookup. Once again, type in isitblocked.org and see the response. If you see a “could not be resolved” as shown in yellow above, congrats! The only issue? That doesn’t necessarily mean everything is working. The DNS lookup via the web GUI performs a sequential lookup of DNS servers. Remember the red warning above? Nonetheless, this can still be a useful step in your troubleshooting endeavors.

pfSense DNS lookup via web gui

For those interested, this is what the “race” looks like during a packet capture despite 9.9.9.9 being “above” 8.8.8.8 in the DNS configuration screen and getting queried first.

IP MYEXTIP.53125 > 9.9.9.9.53 <-- FIRST QUERY (TO QUAD9)
IP MYEXTIP.54417 > 8.8.8.8.53 <-- SECOND QUERY (TO GOOGLE)
IP 8.8.8.8.53 > MYEXTIP.54417 <-- FIRST RESPONSE (FROM GOOGLE)
IP 9.9.9.9.53 > MYEXTIP.53125 <-- SECOND RESPONSE (FROM QUAD9)

Log Config – NXDOMAIN

What’s the best way to troubleshoot? Get familiar with your logs. Better yet, configure them properly when you first set this up. Go to Services -> DNS Resolver -> Advanced Settings and switch the default log level of 1 to 2 (or higher). Don’t forget ‘Save’ and ‘Apply Changes’ once you are done.

pfSense log level on DNS resolver

Now, you can go to your system logs (Status -> System Logs -> System -> DNS Resolver) and see every DNS query/response. Keep in mind the logs will also show which domains are returning as NXDOMAIN.

pfSense DNS Resolver NXDOMAIN

Log Config – Add requesting/source IP to NXDOMAIN response

Because of how Quad9 responds to malicious domain queries, you can see first-hand if any devices on your network are trying to contact known bad guys on the internet. By default though, the originating/source IP address is not recorded with the DNS NXDOMAIN response which would make tracking down the offending client/IP a little difficult. To correct this, add “server:” and “log-replies: yes” to the “Custom options” section of the general settings page of your DNS Resolver configuration (Services -> DNS Resolver) as shown in the first image below (highlighted). You can also copy/paste the code section in the gray box. Note: If you have other custom options there (such as the one added by pfBlockerNG), then add the log-replies option on a new line below it as shown in the second image (highlighted). The “server:” directive only needs to be specified once. Thus, if “server:” is specified on a line with pfBlockerNG already, you don’t need to add it before the “log-replies” line. Don’t forget to hit Save then Apply after you’ve finished adding it. It’s also worth mentioning in unbound documentation, it states “Note that it takes time to print these lines which makes the server (significantly) slower.” I have enabled “log-replies” multiple times on various shapes and sizes of pfSense boxes and I’ve never seen a discernible increase in disk I/O, CPU, memory, etc. but your mileage may vary.

server:
log-replies: yes

 

unbound_log_replies_without_pfblockerng

log-replies without an existing pfBlockerNG custom option

Unbound Log Replies Option pfSense

log-replies with existing pfBlockerNG custom option

Here are what the logs (Status -> System Logs -> System -> DNS Resolver) look like when querying a random hostname. These are the same logs in your SIEM if you forward your pfSense logs as well.

Feb 26 14:53:08 unbound 91442:0 info: control cmd: stats_noreset
Feb 26 14:53:08 unbound 91442:1 info: [my IP address] abbbbbbbbbaaaa22222233333333322244422.com. A IN NXDOMAIN 0.042276 0 132
Feb 26 14:53:08 unbound 91442:1 info: validate(nxdomain): sec_status_insecure
Feb 26 14:53:08 unbound 91442:1 info: query response was NXDOMAIN ANSWER


Unbound log-replies nxdomain


Testing DNS over TLS on pfSense

You can test DNS over TLS by performing a packet capture on your firewall. Go to Diagnostics -> Packet Capture to get started. Just make sure your interface is set to WAN and add 9.9.9.9 as your host address before you start your capture or you’ll get tons of other data. As you can see in the simple connection data below, your queries/responses are now over TCP port 853 (DNS over TLS) instead of the standard DNS query/response (UDP port 53).

23:01:41.410868 IP myip.46451 > 9.9.9.9.853: tcp 0
23:01:41.439209 IP 9.9.9.9.853 > myip.46451: tcp 0
23:01:41.439266 IP myip.46451 > 9.9.9.9.853: tcp 0
23:01:41.439420 IP myip.46451 > 9.9.9.9.853: tcp 307
23:01:41.467633 IP 9.9.9.9.853 > myip.46451: tcp 1448
23:01:41.467710 IP myip.46451 > 9.9.9.9.853: tcp 0
23:01:41.467715 IP 9.9.9.9.853 > myip.46451: tcp 1448
23:01:41.467732 IP myip.46451 > 9.9.9.9.853: tcp 0
23:01:41.467736 IP 9.9.9.9.853 > myip.46451: tcp 42
23:01:41.467784 IP myip.46451 > 9.9.9.9.853: tcp 0

Taking a peek at the saved packet capture in Wireshark, you’ll see more connection data including the TLS handshake. What you won’t see are hostnames, responses, or any other standard query/response data you would find in a normal, clear-text DNS transaction!

Wireshark - Quad9 DNS over TLS transaction

Add Security To DNS with DNS over TLS
— These are only necessary if your pfSense version is < 2.4.4 (set for release sometime around September 2018) —

If you are on version 2.4.4 or later, follow the instructions above for the web interface configuration instead!

You’ll see the example below has the “log-replies” setting discussed above. If there are other settings in the custom options, you can safely place these additional options below them.

DNS over TLS pfSense

If you want to copy/paste everything including the log-replies settings, here they are! Make sure you save and apply your settings.

server:
log-replies: yes
forward-zone:
name: "."
forward-ssl-upstream: yes
forward-addr: 9.9.9.9@853
forward-addr: 149.112.112.112@853

Dallas Haselhorst has worked as an IT and information security consultant for over 20 years. During that time, he has owned his own businesses and worked with companies in numerous industries. Dallas holds several industry certifications and when not working or tinkering in tech, he may be found attempting to mold his daughters into card carrying nerds and organizing BSidesKC.

16 thoughts on “Configuring Quad9 on pfSense

  1. If I get an NXDOMAIN answer, is there any way to see which of my home IPs generated it? Currently the logs dont seem to list the originating IP address ?

    1. Great question! In the past I tried using unbound options to do this, e.g. “log-replies: yes” and “log-queries: yes” since those are native options. Unfortunately, I could never get them to work whether I used the “custom options” field (Services -> DNS Resolver) or added it to existing config files. My workaround was simply creating a firewall rule for port 53 and enabling logging on that rule. This does require some event correlation so it’s not ideal. Now that unbound has been around a bit longer on pfSense, I’ll dig around a bit more.
      — UPDATE —
      After thinking about this a bit I figured it out. In the “custom options” you also need to include “server:” before it. Thus, adding “server:log-replies: yes” will provide the desired outcome — the source IP address and the NXDOMAIN response on a single line (example below). I’ll update the documentation above since I believe this is an extremely helpful suggestion. Also, heed the warning about printing this information makes unbound “significantly slower” according to their own documentation (link below). Please let me know if you have any issues!
      Feb 26 13:01:30 unbound 90310:1 info: [my internal IP address] aaaaaaaaaaa222233333333333312222222222222222.com. A IN NXDOMAIN 0.434214 0 132
      https://www.unbound.net/documentation/unbound.conf.html
      — UPDATE —
      I tried this again without the “server:” directive and it works now as well. Thus, “log-replies: yes” or “server:log-replies: yes” should work in the “Custom options” field. Make sure it is on a new line and click save and apply.

  2. How can you get DNS over TLS to work with pfblockerng dnsbl filtering? i want to filter using this, but then any queries my resolver makes that is legit, i would like it to use the dns over tls.

    1. Micheal, DNSBL and the DNS over TLS with perfectly together. In fact, this is exactly how I have it configured on all my installs. Just make sure you leave the DNSBL-related line in the unbound configuration. If you get any errors during the initial setup, cut the DNSBL line out of the config, save it, and then re-add it. Feel free to holler if you have any questions!

      1. Ok i was wondering bc in your help you said to turn on dns query forward, but in the dnsbl help document it says to not have it on. So which way am i supposed to have it for both to work properly?

        1. I tested both checked/unchecked on the “DNS Query Forwarding” option and Quad9 successfully blocked the test site, isitblocked.org, in both instances. That said, all my firewalls have “Enable Forwarding Mode” checked. I based this off the documentation below. If you find anything different in your setup, please let me know!

          Enable Forwarding Mode: Controls whether Unbound will query root servers directly (unchecked, disabled) or if queries will be forwarded to the upstream DNS servers defined under System > General or those obtained by DHCP/PPPoE/etc (checked, enabled). Forwarding mode may be enabled if the upstream DNS servers are trusted and also provide DNSSEC support.

  3. your articles on pfSense have been very helpful as I’ve been learning out pfsense in the past few months. Thank you.

    I think I discovered an issue however with your supplemental log configuration “log-replies: yes” however.

    When I was setting up a new test environment without any packages loaded, I got an error message when trying to add the the extra command. Then I noticed in your tutorial above that you had pfblocker loaded. so I added that in and enabled it. then I was able to save with the configuration in the article. later on I removed pfblocker as some feeds were not loading correctly with plan to reinstall. However, with pfblocker removed, I found that I couldn’t load any web pages although I could ping them OK. On a whim, I removed “log-replies: yes” and everything went back to working appropriately. Why does pfblocker need to be enabled to get the additional DNS logging details?

    1. Excellent! Thanks for the feedback and glad they’ve been helpful!

      I see what you are referring to. The “include: /var/unbound/pfb_dnsbl.*conf” isn’t necessary for configuring log replies, however, the “server:” preceding it is. If you take a peek at the “DNS over TLS” section below that part of the guide, I think you’ll see what I’m referring to. That said, I’ll get the image and copy/paste section updated and I appreciate you bringing that to my attention!

  4. Hi, I was following your guide on this however it seems that whenever I access any website the DNS request times out and I have to wait a couple of seconds before the the page reloads. At first I thought this was pfblockerNG but I disabled it and I was still getting the same results. Have you seen this issue before or know how to resolve it?

    C:\Users\Jon>nslookup isitblocked.org
    Server: UnKnown
    Address: 192.168.2.1

    DNS request timed out.
    timeout was 2 seconds.
    DNS request timed out.
    timeout was 2 seconds.
    *** Request to UnKnown timed-out

    *run the command again immediately and get the expected results below*

    C:\Users\Jon>nslookup isitblocked.org
    Server: UnKnown
    Address: 192.168.2.1

    *** UnKnown can’t find isitblocked.org: Server failed

    1. I can’t say I’ve seen any delays with DNS requests to Quad9. Here are a few things I would recommend to track down where the issue lies…
      1) Are you experiencing issues whether using DNS over TLS or not? You might have to test both encrypted and unencrypted. I personally use DNS over TLS for all queries to Quad9.
      2) See if your internet is having issues. I happen to be going through this exact issue as we speak and my ISP is regularly at 10% packet loss with long ping times to the gateway on their network. Take a look at Status -> Monitoring and switch to quality on the left axis. If you see some packet loss or delays, run some packet/traffic graphs to see if there is any correlation between the two. If you regularly max out upload or download you will see issues similar to what you are describing.
      3) The pfSense web interface wouldn’t be the greatest to perform a packet capture in this instance. Instead, I would use packet capture (tcpdump) from the command line that allows you to capture both sets of packets simultaneously in a single pcap and you could then view the entire conversation in Wireshark. You are specifically looking for delays or missing packets in the conversation. You’ll have to get creative on the filter because the internal DNS query will be UDP, port 53 on the LAN (or other internal network) and the external DNS query will be TCP, port 853 on the WAN if you are using the DNS over TLS.
      4) If pcaps aren’t your thing, you could have some monitoring solution such as Nagios, the Dude, or Ping Plotter periodically test the route/roundtrip to both anycast IP addresses.
      5) Even if the routes and ping times to the IPs are good, there’s always the possibility the Quad9 servers are slow to respond to DNS queries. You could use Nagios or even DNS Benchmark to test the DNS service itself. I discuss DNS Benchmark a bit in a different Quad9 post. https://www.linuxincluded.com/quad9-benchmark/
      If you determine the issue is with Quad9, I would also recommend reaching out to them to let them know. Although I haven’t had any issues with the service, they have been very responsive to all of my past inquiries. Please let me know what you figure out!

  5. Hi
    I previously had dns-over-tls setup on my 2.4.4 install using cloudfare’s 1.1.1.1 and 1.0.0.1 and Quad9. I incorrectly assumed there was an order in which these where processed. Now I know thanks to your wonderful post here. Running the dns lookup test shows the response times for quad9 was much better than cloudfare.

    So is it better to use only quad9 in this case or still add in cloudflare – which ever comes back first wins , right? What if quad9 goes down?

    1. Hey Victor! Quite a few people have discovered the same sequential DNS conundrum. I wouldn’t recommend mixing and matching DNS from different providers simply because while Quad9 may be faster today, it might be milliseconds slower tomorrow causing the order to shift on-the-fly. Redundancy is a concern, although it isn’t as big of a problem because of anycast addressing. I use the primary and secondary block addresses for Quad9 and that is also what I recommend.

  6. Dallas –
    Thanks for the great post here on setting up Quad9 with pfSense. I’m a big pfSense user myself, but have fallen behind on the latest releases with TLS – this is a good notice for me to get my systems in the latest revisions! I happen to work for the project, so let me throw a few items out there for your readers:

    An interesting note that might be useful: Quad9 replies back with an NXDOMAIN at this point for blocked domains. We actually tag those blocked replies with a special bit – we answer with the “AD” bit set to 1, so if you are looking closely at the replies you could determine that it was a blocked response rather than a natural NXDOMAIN. The “tshark” program (part of wireshark) can detect these.

    Recipe to show blocked hosts (note: “dns.flags.recavail” flag used due to bug in most versions of tshark – should be “dns.flags.authoritative” in non-bug version – do some trials) – note, rolls /tmp files after 100k packets to avoid disk full errors (note: change “ens32” to your local interface name)

    while true; do tshark -np -i ens32 -q -c 100000 -E separator=, -E quote=d -T fields -e frame.time_epoch -e dns.qry.name -e ip.src -e ipv6.src -e ip.dst -e ipv6.dst -Y “dns.flags.rcode eq 3 && dns.flags.recavail eq 0” “src net 9.9.9.0/24 or src net 149.112.112.0/24 or src net 149.112.149.0/24 or src net 2620:fe::0/48” 2>/dev/null; done

    In the future, we are considering a shift to a “splash page” which would reply back with an IP address which would lead to one of our anycast servers, answering only on port 80, which could contain a brief description of why the user was blocked. We’ll have an announcement on our blog well in advance of this happening, and we expect that there will be an alternate IP address (tentatively 9.9.9.12) that would still answer with the current model of NXDOMAIN.

    1. Hey John! Thanks for stopping by and I *really* appreciate your discussion on future changes as well as tips/tricks. I love the Quad9 project and what it stands for so thank you for being a part of a great cause!

      I’m guessing you’ve peeked at other posts on this site such as the ones regarding pfBlockerNG? I use pfBlockerNG in conjunction with Quad9 as a bit of defense-in-depth. As far as I’m concerned, Quad9 blocks the worst of the worst and would also be much quicker to respond in the case of an “outbreak.” While pfBlockerNG also does DNS blackholing, it gives a bit more granularity and allows for blocking gray areas Quad9 can’t… A local pfBlockerNG install also has a bit of lag (see previous comment) by the time feeds update, feeds are downloaded, etc. Of course, pfBlockerNG has IP blocking functionality, which is a whole different beast altogether.

      One of my gripes in using NXDOMAIN is the number of false positives. Whether using Security Onion for syslog or even just glancing through logs directly on a pfSense, there are often a ton of NXDOMAIN activity to weed through. At any rate, I love the tshark command example. In fact, I might add a similar check using tcpdump for that in my pfSense/Nagios monitoring solution (yet another post on this site). 😉 Either way, looking for clients communicating with a specific website/page will be *far* easier to identify block activity, i.e. this is a great move on the part of Quad9. On a side note, too bad the ‘AD’ bit couldn’t be added to syslog messages. Maybe you could discuss that with pfSense and other vendors?

Leave a Reply

Your email address will not be published.