Resolution from the inside: referrals, glue, positive and negative caching

Objective

Learn to read what a resolver does between the client's question and the final answer, so that you recognize at a glance a referral, an authoritative answer, an answer served from the cache or a negative answer. This is the basis of any serious DNS troubleshooting.

Key points

• The client sends a recursive question to its resolver; the resolver sends non-recursive questions to the authoritative servers and follows the referrals.
• A referral = no answer, no aa flag, NS records in the AUTHORITY section and often their addresses (glue) in ADDITIONAL.
• Glue is mandatory when the name server has a name inside the zone it serves (ns1.fes.masterlinux.lab for fes.masterlinux.lab).
• The cache also keeps negative answers (NXDOMAIN, NODATA) for the TTL of the returned SOA, capped by the last field of the SOA.
• A UDP answer that is too large is truncated (tc flag) and the client tries again over TCP: port 53/TCP must be open.

1. Two jobs, two kinds of questions

The "Networks and VPN" course introduced the resolver and the authoritative server. Now let us look at the packets they exchange.

With dig, you reproduce a resolver's question by turning recursion off:

dig @10.20.0.53 www.fes.masterlinux.lab +norec    # as a resolver would do
dig @10.20.0.10 www.fes.masterlinux.lab           # as a client would do (rd=1)

A well-configured authoritative server refuses recursion: it answers with status: REFUSED (or without the ra flag) when you ask it for a name that is not in its zones. A server that accepts recursion for the whole Internet is an open resolver, and therefore an attack amplifier.

2. The referral and the glue

The MasterLinux branch in Fès has its own subzone, fes.masterlinux.lab, managed by its servers ns1.fes and ns2.fes. The parent zone's server (ml-dns-01, 10.20.0.53) does not know www.fes.masterlinux.lab, but it knows whom to ask:

;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 3
;; AUTHORITY SECTION:
fes.masterlinux.lab.     86400 IN NS ns1.fes.masterlinux.lab.
fes.masterlinux.lab.     86400 IN NS ns2.fes.masterlinux.lab.
;; ADDITIONAL SECTION:
ns1.fes.masterlinux.lab. 86400 IN A  10.30.0.53
ns2.fes.masterlinux.lab. 86400 IN A  10.30.0.54

Three clues say "referral": ANSWER: 0, no aa flag, and NS records for a zone more specific than the server's own zone.

Without the two lines in ADDITIONAL, the resolver would be stuck: to reach ns1.fes.masterlinux.lab, it would have to ask… ns1.fes.masterlinux.lab. These addresses, written in the parent zone, are called the glue. They are only needed when the server's name is under the delegated zone; with a server such as ns.hebergeur.ma, the resolver resolves that name separately.

In the parent zone, the delegation is written like this:

; db.masterlinux.lab (parent zone)
fes        IN NS  ns1.fes.masterlinux.lab.
fes        IN NS  ns2.fes.masterlinux.lab.
ns1.fes    IN A   10.30.0.53      ; glue
ns2.fes    IN A   10.30.0.54      ; glue

The child zone must publish the same NS records in its own file. When the two lists differ, some resolvers follow the parent and others follow the child: you get intermittent failures that are hard to reproduce. Check with:

dig @10.20.0.53 fes.masterlinux.lab NS +norec   # what the parent says (referral)
dig @10.30.0.53 fes.masterlinux.lab NS +norec   # what the child says (aa)

3. The full path, with minimisation

On the Internet, the chain starts at the root servers. Recent resolvers (BIND 9.18+, Unbound) use query minimisation (QNAME minimisation): at the root, they only ask NS ma. and not the full name www.boutique.exemple.ma.; each server only sees what it needs.

resolver ──NS ma.?────────────► root           ── referral to the .ma servers
resolver ──NS exemple.ma.?────► .ma servers    ── referral to ns1.exemple.ma
resolver ──A www.exemple.ma.?─► ns1.exemple.ma ── answer, aa flag

Practical consequence: a badly coded authoritative server that answers incorrectly to questions about intermediate names (or about a type it does not know) breaks resolution only for modern resolvers. In BIND, the option qname-minimization relaxed; (the default value) falls back to the old behavior when there is a problem.

4. The positive cache: the TTL counts down

The resolver keeps each answer for the duration of its TTL. When it serves it again, it shows the remaining TTL:

;; flags: qr rd ra; ANSWER: 1
www.fes.masterlinux.lab. 2712 IN A 10.30.1.80

• No aa: the answer comes from a cache.
• The zone publishes a TTL of 3600; 2712 remain: the answer has been in the cache for 888 seconds.
• Run the command again: the number goes down. If it goes back to 3600, the resolver has just fetched it again.

Two resolver settings distort this calculation: max-cache-ttl (BIND) or cache-max-ttl (Unbound) cap TTLs that are too long, and cache-min-ttl (Unbound) raises them. Raising TTLs saves a little traffic, but then a migration no longer propagates at the speed the team expects: avoid it.

5. The negative cache: NXDOMAIN and NODATA

There are two negative answers, and you must not confuse them:

In both cases, the server adds the zone's SOA in AUTHORITY. The resolver keeps the negative answer for the TTL of this SOA, which never exceeds the last field of the SOA (the "minimum", whose meaning changed over the years to become the negative cache duration):

masterlinux.lab. 300 IN SOA ns1.masterlinux.lab. hostmaster.masterlinux.lab. 2026102001 3600 900 1209600 300
                 ^^^                                                                                  ^^^
          negative cache TTL                                                         negative cache cap

A real case: the team creates paiement.masterlinux.lab at 10:00. A developer had tested it at 9:58 and received NXDOMAIN. With the last SOA field set to 86400, their resolver answers "does not exist" until the next day. Keep this field between 300 and 3600 seconds, and flush the cache in an emergency (rndc flushname paiement.masterlinux.lab on BIND, unbound-control flush paiement.masterlinux.lab on Unbound).

6. SERVFAIL: the resolver gives up

SERVFAIL from a resolver means "I could not get a reliable answer": all the authoritative servers are unreachable, the delegation is broken, or DNSSEC validation failed. The resolver also caches this result for a few seconds (servfail-ttl in BIND, 1 s by default) so that it does not keep hammering.

To find out which one, query the authoritative servers directly with +norec, then try again with +cd (checking disabled): if +cd works, the cause is DNSSEC (lesson 3.2).

7. EDNS, answer size and TCP

"Historical" DNS limited UDP answers to 512 bytes. EDNS(0) lets a client announce a larger size. Since the 2020 DNS Flag Day, the recommended value is 1232 bytes: above that, packets risk being fragmented, and fragments are often lost or filtered.

dig @10.20.0.53 masterlinux.lab DNSKEY +dnssec +bufsize=1232
# ;; flags: qr aa tc ...      <- tc: truncated, dig retries over TCP on its own
dig @10.20.0.53 masterlinux.lab DNSKEY +dnssec +tcp     # force TCP

A firewall that only allows 53/UDP breaks large answers (DNSSEC, long TXT lists) and zone transfers: always open 53/UDP and 53/TCP.

8. The client's local resolver (stub)

On the client, the C library reads /etc/resolv.conf:

nameserver 10.20.0.10              # up to 3 servers, tried in order
nameserver 10.20.0.11
search fes.masterlinux.lab masterlinux.lab   # suffixes added to short names
options timeout:2 attempts:2 rotate

• timeout:2: wait 2 seconds per server (5 by default, which feels endless when the first one is down).
• attempts:2: number of full rounds through the list.
• rotate: spreads the questions across the servers instead of always starting with the first one.
• ndots:n: a name that contains fewer than n dots is first tried with the search suffixes. Kubernetes sets ndots:5: a call to api.exemple.ma then generates several useless questions before the right one. Writing the name with a final dot (api.exemple.ma.) avoids this.

On Ubuntu and on systems with systemd-resolved, /etc/resolv.conf points to 127.0.0.53: you read the real servers with resolvectl status. On Rocky, NetworkManager rewrites this file from the connection settings (nmcli con mod … ipv4.dns).

By version

Common pitfalls

• Reading a resolver's TTL as the zone's TTL: it is a remaining TTL. Query the authoritative server (+norec) to read the real value.
• Confusing NXDOMAIN and NODATA: "no AAAA" is not "the name does not exist". An A record that works with no AAAA is normal.
• Last SOA field too large: a name created "too late" stays unreachable for hours.
• 53/TCP closed: everything works… except DNSSEC, large answers and transfers.

Lab

You receive dig captures taken on the MasterLinux network. You will recognize a referral and its glue, calculate how long an answer has been in the cache, classify negative answers and write the resolv.conf for the servers of the Fès branch.

Quiz

1. Which three clues let you recognize a referral in the output of dig?
2. When is glue essential?
3. What is the difference between NXDOMAIN and NODATA?
4. How long does a negative answer stay in the cache?
5. Why must you open 53/TCP on a firewall?