tao.
DNS 18 min read 阅读中文版

How to Fix Slow DNS Lookup: From curl and dig to Kubernetes

Learn how to diagnose and fix slow DNS lookups without blindly flushing caches or switching to 8.8.8.8. Isolate latency with curl, compare OS resolver paths, debug UDP truncation, IPv6, DNSSEC, and Kubernetes ndots/CoreDNS.

Mttao

How to Fix Slow DNS Lookup: From curl and dig to Kubernetes
On this page

A website hangs for several seconds on the first load, then snaps open on refresh. A backend service calling an external API times out sporadically. The same hostname resolves instantly on your laptop, yet crawls inside a Kubernetes Pod.

When that happens, most people reach for two reflexive fixes: flush the DNS cache or switch to 8.8.8.8. Sometimes you get lucky. More often, you mask the symptom or break something else—on a corporate network or VPN, pointing at a public resolver breaks internal hostnames and leaks private domain queries to a third party.

Worse, a sluggish page load or API timeout is not always DNS. TCP packet loss, an IPv6 routing black hole, or a stalled TLS handshake can all look identical from the outside. Effective troubleshooting comes down to one rule: prove where the request is actually waiting, see which resolver received the query, and inspect the exact response before changing anything.

Use this table to jump straight to the section matching your symptom:

SymptomWhere to look firstSection
Not sure if DNS is the bottleneckSplit lookup, TCP, TLS, and server wait with curlStep 1: Prove it’s DNS
dig is fast, browser or app is still slowCompare OS resolver path, browser DoH, and VPN split DNSOS & app resolver paths
Many unrelated domains are slowBenchmark your current recursive resolver against candidatesCompare recursive resolvers
Usually fast, occasional 2–5s stallsCheck UDP packet loss, TC=1 truncation, and TCP 53 fallbackPacket loss & TCP fallback
Slow on dual-stack networksSeparate IPv6 DNS transport from IPv6 target connectivitySeparate IPv6 issues
Only one domain fails across resolversCheck parent delegation, authoritative servers, and DNSSECAuthoritative & DNSSEC
Host is fast, only Kubernetes Pods are slowInspect ndots, search domains, CoreDNS, and NodeLocalKubernetes Pod DNS
Network queries are fast, app calls still lagCheck thread pools, connection pooling, and app cachingApplication-layer waits

Replace example.com and 192.0.2.53 in the commands below with your actual domain and resolver IP. Keep private corporate hostnames pointed strictly at authorized internal resolvers.

Step 1: Prove DNS is actually the bottleneck with curl

Before touching any DNS setting, measure where an HTTPS request actually spends its time. Run a short sampling loop against a small, repeatable endpoint from the affected machine and network (on Windows PowerShell, call curl.exe explicitly so you don’t hit the Invoke-WebRequest alias):

# Direct connection, HTTP/1.1, no redirects, 20 fresh processes
url='https://example.com/'
fmt='ip=%{remote_ip} http=%{http_version} new_conn=%{num_connects} code=%{http_code} lookup=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ready=%{time_pretransfer} first_byte=%{time_starttransfer} total=%{time_total}'

curl --version
for ((i=1; i<=20; i++)); do
  printf 'run=%s ' "$i"
  if curl -q --noproxy '*' --http1.1 -sS -o /dev/null \
      --connect-timeout 5 --max-time 15 -w "$fmt" "$url"; then
    rc=0
  else
    rc=$?
  fi
  printf ' curl_rc=%s\n' "$rc"
  sleep 1
done

Why these flags matter:

  • -q must be the very first flag; it skips your .curlrc so hidden defaults don’t skew the test.
  • --noproxy '*' forces a direct connection, ignoring proxy environment variables. If your production traffic must go through a proxy, run a separate proxied sample set rather than mixing the two.
  • Spawning a fresh curl process per iteration avoids reusing TCP connections or curl’s in-process DNS cache (though OS and upstream recursive caches remain warm).

Read curl’s cumulative timings correctly

A common mistake is adding curl -w timing fields together. Per the curl manual, every timing variable is cumulative from the start of the transfer, so you need to subtract:

PhaseFormulaWhat it points to
Name resolutiontime_namelookupIf dominant, investigate DNS or local resolution
TCP connecttime_connect - time_namelookupIf dominant, check routing, packet loss, or IPv6 reachability—not DNS
TLS handshaketime_appconnect - time_connectIf dominant, check packet loss, cert chain, or server handshake
Request & server waittime_starttransfer - time_pretransferIf dominant, check network RTT or backend latency (time_starttransfer itself is full TTFB)

Three caveats when reading the numbers:

  1. Check the exit code (curl_rc) first: 6 means target host resolution failed, 5 means proxy host resolution failed, and 28 is a timeout (which can happen during lookup, connect, or TLS). Unreached stages print 0.000 on failed runs—don’t mistake 0 for an instant lookup.
  2. HTTP/3 and redirects change the timeline: HTTP/3 runs over QUIC, so the TCP-then-TLS split doesn’t apply; test it separately with --http3-only. Following redirects (-L) combines multiple hosts into one line, so check the first hop without -L first.
  3. Proxies shift where DNS happens: socks5 resolves locally while socks5h resolves on the proxy, and time_connect may measure the connection to the proxy itself.

Run a pinned-IP A/B check with --resolve

To confirm that DNS resolution is what’s slowing things down, alternate normal requests with --resolve. Unlike editing /etc/hosts or hitting an IP URL with a Host header, --resolve keeps the domain in the URL so HTTP Host, TLS SNI, and certificate verification all work normally—no -k needed.

host='example.com'
url="https://$host/"
base=(-q --noproxy '*' --http1.1 --ipv4 -sS -o /dev/null
      --connect-timeout 5 --max-time 15)
fmt='ip=%{remote_ip} lookup=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} first_byte=%{time_starttransfer} total=%{time_total}'

if ! ip=$(curl "${base[@]}" -w '%{remote_ip}' "$url"); then
  printf 'Baseline request failed; cannot pin IP\n' >&2
  exit 1
fi
: "${ip:?No remote IP captured}"

for ((i=1; i<=10; i++)); do
  modes=(normal pinned)
  if ((i % 2 == 0)); then modes=(pinned normal); fi
  for mode in "${modes[@]}"; do
    extra=()
    if [[ "$mode" == pinned ]]; then extra=(--resolve "$host:443:$ip"); fi
    printf 'run=%s mode=%s ' "$i" "$mode"
    if curl "${base[@]}" "${extra[@]}" -w "$fmt" "$url"; then
      rc=0
    else
      rc=$?
    fi
    printf ' curl_rc=%s\n' "$rc"
  done
  sleep 1
done

If pinned runs stay steady while normal runs stall in lookup, you have strong evidence of a resolution bottleneck. Compare remote_ip across both groups—if normal DNS steers you to a different CDN edge node, end-to-end differences will include edge latency too. Never turn a pinned CDN IP into a permanent fix; providers rotate IPs and rely on DNS for failover.

dig is fast, but the app is slow: check resolver path differences

Few things waste more time than running dig example.com, seeing 10 msec, and concluding DNS is fine while the browser or application keeps stalling.

dig only tests the raw DNS wire path. Applications traverse the full host resolution stack: in-process cache → hosts file → OS name service (glibc NSS / systemd-resolved on Linux, DNS Client on Windows, mDNSResponder on macOS) → recursive resolver.

  • On Linux, dig reads /etc/resolv.conf directly. Even when that points to 127.0.0.53, dig skips glibc’s NSS pipeline (/etc/nsswitch.conf), and dig @8.8.8.8 bypasses the local stub entirely.
  • On Windows, Microsoft’s DNS client troubleshooting docs explicitly note that nslookup bypasses the client DNS cache. It is completely normal for nslookup to return a fresh IP while apps stay stuck on a stale negative cache entry.
  • Browsers like Chrome maintain their own HostCache, built-in async DNS client, and independent DNS-over-HTTPS (DoH) settings.

Compare OS getaddrinfo() against dig

Calling Python’s socket.getaddrinfo() exercises the same system API most applications use:

python3 - example.com <<'PY'
import socket
import sys
import time

host = sys.argv[1]
for i in range(1, 11):
    started = time.perf_counter()
    try:
        rows = socket.getaddrinfo(host, 443, type=socket.SOCK_STREAM)
        elapsed = time.perf_counter() - started
        addresses = sorted({row[4][0] for row in rows})
        print(i, "OK", f"{elapsed:.6f}s", ",".join(addresses))
    except socket.gaierror as error:
        elapsed = time.perf_counter() - started
        print(i, "ERROR", f"{elapsed:.6f}s", repr(error))
    time.sleep(1)
PY

This immediately narrows the scope:

  • getaddrinfo() is slow, dig is fast: focus on the OS resolver service, search domains, and per-domain routing.
  • getaddrinfo() is fast, only the browser is slow: focus on browser DoH settings and enterprise policies.

Windows: interface DNS isn’t the whole story

Inspect the full effective configuration in PowerShell:

ipconfig /all
Get-DnsClientServerAddress
Get-DnsClientNrptPolicy -Effective
nslookup example.com.
ipconfig /displaydns

On a VPN, pay special attention to Get-DnsClientNrptPolicy -Effective. The Name Resolution Policy Table (NRPT) routes specific domain namespaces to dedicated DNS servers—the first DNS server listed on your network adapter is often not the server handling internal corporate names.

macOS: look past the first nameserver

scutil --dns

Alongside default resolvers, macOS uses scoped and supplemental domain-matched configurations. Corporate VPN domains typically rely on supplemental entries, so a healthy default resolver for public sites does not prove internal routing is working.

Linux: compare systemd-resolved, glibc, and dig

On hosts running systemd-resolved, inspect all three paths side by side:

systemctl is-active systemd-resolved
resolvectl status
resolvectl statistics
ls -l /etc/resolv.conf
readlink -f /etc/resolv.conf
grep '^hosts:' /etc/nsswitch.conf

resolvectl query example.com
getent ahosts example.com
dig example.com. A

When /etc/resolv.conf points to 127.0.0.53, that is only the local stub listener; actual upstreams and per-link routing live in resolvectl status. Comparing resolvectl query (resolved path), getent ahosts (glibc NSS path), and dig (wire path) quickly exposes which layer is stalling.

Don’t overwrite /etc/resolv.conf or lock it with chattr +i. It is managed by NetworkManager, systemd-networkd, or your VPN client; bypassing it breaks per-interface split DNS. Make changes through the owning network manager instead.

Browsers and VPNs: verify who actually receives the query

Chromium’s host resolution stack can use either the OS getaddrinfo() or its built-in async DNS client with its own HostCache. If you manually configure a custom DoH provider in the browser, browser queries bypass your OS DNS and corporate resolvers completely. For a deeper look at what DoH encrypts and what it doesn’t, see Why Encrypt DNS Queries?.

On VPNs, keep two mechanisms distinct: split tunneling routes IP traffic, while split DNS routes DNS queries by domain name. In systemd-resolved, a tilde-prefixed domain like ~corp.example is a routing-only domain (routes queries without expanding short names), a plain entry is a search domain (expands wiki to wiki.corp.example), and ~. acts as the catch-all route.

Test both the FQDN and the short name inside the VPN session. If the FQDN works and the short name lags, check search suffixes. If CLI tools work and only the browser fails, check the browser’s DoH provider and managed policies.

Many unrelated domains are slow: compare recursive resolvers

When lots of unrelated websites stall during lookup, your local router forwarder, ISP recursive resolver, or uplink is the prime suspect. Conversely, if multiple resolvers all fail on the same single domain, the problem is almost certainly authoritative or delegation-side—changing your local DNS won’t fix it.

Benchmark your current resolver against public resolvers (1.1.1.1, 8.8.8.8) without modifying system settings. You can also cross-check public answers quickly with our online DNS Lookup tool.

# Linux / Bash 4+; requires BIND dig and GNU shuf. Replace CURRENT with your actual resolver IP.
CURRENT='192.0.2.53'
NAME='example.com.'
logdir=$(mktemp -d)
TIMEFORMAT='process_wall_s=%3R'
printf 'round\tserver\ttype\tdig_rc\tfile\n' > "$logdir/attempts.tsv"
dig -v > "$logdir/dig-version.txt"

for ((round=1; round<=10; round++)); do
  mapfile -t order < <(printf '%s\n' "$CURRENT" 1.1.1.1 8.8.8.8 | shuf)
  types=(A AAAA)
  if ((round % 2 == 0)); then types=(AAAA A); fi
  for server in "${order[@]}"; do
    for type in "${types[@]}"; do
      file="$logdir/${round}-${server}-${type}.txt"
      if { time dig -r "@$server" -q "$NAME" -t "$type" \
          +nosearch +timeout=2 +tries=1; } > "$file" 2>&1; then
        rc=0
      else
        rc=$?
      fi
      printf '%s\t%s\t%s\t%s\t%s\n' \
        "$round" "$server" "$type" "$rc" "$file" >> "$logdir/attempts.tsv"
    done
  done
  sleep 1
done
printf 'Logs saved to: %s\n' "$logdir"

Four pitfalls when interpreting resolver benchmarks

  1. Query time is not total user wait: In TCP mode, BIND dig records its start timestamp after the TCP handshake finishes, so dig +tcp excludes TCP connect time, and a final successful reply may hide earlier timeout retries. Compare Query time against the script’s process_wall_s.
  2. Bucket by status code before computing averages:
OutcomeWhat it means
No response / dig_rc != 0Timeout or transport error—keep exit code and stderr, don’t treat as NXDOMAIN
SERVFAILResolution failure (upstream timeout, bad delegation, or DNSSEC failure)
NXDOMAINDomain does not exist; dig still exits 0, so don’t count it as a healthy IP lookup
NOERROR with empty AnswerNormal NODATA (e.g., querying AAAA on an IPv4-only domain)
  1. Round 1 is not a “cold cache” on public DNS: Flushing your local cache doesn’t flush a shared public resolver’s cache. Querying random subdomains doesn’t simulate a cold lookup either, because random names hit NXDOMAIN or wildcard paths.
  2. Faster DNS doesn’t guarantee a faster website: Public resolvers use Anycast and may not land in the geographically closest PoP. More importantly, CDNs steer traffic using DNS—Google Public DNS sends EDNS Client Subnet (ECS) prefixes to supporting authoritatives, whereas Cloudflare does not by default. Saving 10 ms on DNS while landing on a suboptimal CDN edge that adds 200 ms is a net loss. See Cloudflare DNS vs Google Public DNS for a full comparison.

Also remember that configuring a primary and secondary DNS server rarely means parallel racing—most stacks wait for the primary to time out before trying the secondary. Never mix a corporate internal DNS with 8.8.8.8 as primary/secondary, or a brief primary hiccup will return NXDOMAIN for internal hosts.

Occasional multi-second stalls: packet loss, truncation, and TCP fallback

Steady high latency and sporadic 2–5 second freezes have different causes. Sporadic stalls usually point to UDP packet loss or broken TCP fallback after UDP truncation.

DNS does not use TCP only for zone transfers. RFC 7766 requires general-purpose DNS implementations to support both UDP and TCP. When a reply exceeds the UDP payload limit, the server sets TC=1 (Truncated), signaling the client to retry over TCP 53. If a firewall allows UDP 53 but blocks TCP 53, small queries succeed while large responses hang and time out.

Note: When a UDP packet is simply dropped in transit, there is no TC=1 flag—the client waits for its UDP retry timeout rather than switching straight to TCP.

Test UDP, TCP, and fallback behavior explicitly:

R='192.0.2.53'
Q='example.com.'

# 1. Pure UDP: +ignore stops dig from auto-retrying over TCP on TC=1
dig -r "@$R" "$Q" A +notcp +ignore +bufsize=1232 +timeout=2 +tries=1

# 2. Pure TCP
dig -r "@$R" "$Q" A +tcp +timeout=2 +tries=1

# 3. Normal UDP-with-TCP-fallback
dig -r "@$R" "$Q" A +notcp +noignore +bufsize=1232 +timeout=2 +tries=1

Watch the subtle flag rule: +notcp alone does not disable TCP fallback on truncation—you must pair it with +ignore.

  • UDP times out, TCP succeeds: investigate UDP packet loss, NAT state timeouts, or large-packet drops.
  • UDP returns tc, TCP connect fails: a firewall is almost certainly blocking TCP 53.
  • TCP connects, no DNS answer arrives: the issue is in server processing or upstream forwarding, not a closed port.

Large EDNS responses and IP fragmentation

DNSSEC signatures and DNSKEY records can push UDP responses past the path MTU. If a client advertises a large EDNS buffer (such as 4096 bytes) and an intermediate firewall drops IP fragments, ordinary small lookups fly while signed zones intermittently time out.

1232 bytes is the standard conservative baseline (IPv6 minimum MTU 1280 minus 40-byte IPv6 header and 8-byte UDP header), while RFC 9715 recommends a maximum UDP payload of 1400 bytes (allowing smaller values). To test for fragmentation drops, query a signed zone’s DNSKEY record:

Z='example.com.'

dig -r "@$R" "$Z" DNSKEY +dnssec +notcp +ignore +bufsize=4096 +timeout=2 +tries=1
dig -r "@$R" "$Z" DNSKEY +dnssec +notcp +ignore +bufsize=1232 +timeout=2 +tries=1
dig -r "@$R" "$Z" DNSKEY +dnssec +tcp +timeout=2 +tries=1

If 4096 times out, 1232 returns tc, and +tcp succeeds immediately, large UDP packets are being dropped on the path. Capture packets on Linux to confirm—don’t filter only on udp port 53, or you’ll miss trailing IP fragments (which lack a UDP header) and ICMP messages:

sudo tcpdump -ni any -s 0 -vv -c 200 \
  'host 192.0.2.53 or icmp or icmp6'

Separate IPv6 DNS transport from IPv6 website connectivity

On dual-stack networks, people routinely conflate “IPv6 DNS is slow” with “the website is unreachable over IPv6”—and respond by disabling IPv6 globally. They are two separate layers:

  • Record type: A returns IPv4 addresses; AAAA returns IPv6 addresses (see Understanding DNS Record Types).
  • Transport family: you can query AAAA over IPv4, and you can query A over IPv6.

First, test the resolver’s IPv4 vs IPv6 transport directly with dig -4 and dig -6:

R4='192.0.2.53'
R6='2001:db8::53'
Q='example.com.'

dig -r -4 "@$R4" "$Q" A +timeout=2 +tries=1
dig -r -6 "@$R6" "$Q" A +timeout=2 +tries=1

# Over IPv4 transport, request the IPv6 service address
dig -r -4 "@$R4" "$Q" AAAA +timeout=2 +tries=1

Then compare the actual HTTPS connection over IPv4 vs IPv6:

URL='https://example.com/'
FMT='ip=%{remote_ip} lookup=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total}\n'

curl -q --noproxy '*' -4 --http1.1 --max-time 15 -sS -o /dev/null -w "$FMT" "$URL"
curl -q --noproxy '*' -6 --http1.1 --max-time 15 -sS -o /dev/null -w "$FMT" "$URL"
  • dig -6 times out, dig -4 works: fix IPv6 routing or firewall rules to the resolver.
  • Both A and AAAA resolve fast, curl -6 hangs in connect or tls: DNS is healthy; fix IPv6 routing or path MTU to the destination.

Modern clients implement Happy Eyeballs (RFC 8305), racing IPv6 and IPv4 connection attempts to hide dead IPv6 routes, though it cannot rescue an IPv6 connection that succeeds and later hits an MTU black hole. Use -4/-6 to isolate the broken layer rather than disabling IPv6 across the board.

Only one domain fails everywhere: check delegation and DNSSEC

If 1.1.1.1, 8.8.8.8, and your ISP resolver all time out or return SERVFAIL for one specific domain, the problem lives in that domain’s delegation, authoritative servers, or DNSSEC chain.

Skip +short and inspect the full dig output. Modern resolvers often include Extended DNS Error (EDE) codes that tell you outright whether an authoritative server was unreachable, a signature expired, or a stale answer was served.

# Trace public delegation (runs iteratively from your machine, not inside a public resolver)
Q='www.example.com.'
dig -r +trace "$Q" A +timeout=2 +tries=1

# Query each authoritative IP directly with +norecurse
ZONE='example.com.'
AUTH_IP='192.0.2.54'

dig -r "@$AUTH_IP" "$ZONE" SOA +norecurse +timeout=2 +tries=1
dig -r "@$AUTH_IP" "$Q" A +norecurse +timeout=2 +tries=1
dig -r "@$AUTH_IP" "$Q" A +norecurse +tcp +timeout=2 +tries=1

Check three things on every authoritative server:

  1. The aa (authoritative answer) flag is present and SOA serial matches across all servers.
  2. Both UDP 53 and TCP 53 respond cleanly without overly aggressive rate limits or ACLs blocking public resolvers.
  3. After a DNS provider migration, the parent zone NS and glue records at the registrar actually point to the new nameservers.

Spot broken DNSSEC with +cdflag

A stale DS record at the registrar or an expired RRSIG causes validating resolvers to return SERVFAIL. Compare two queries against the same validating resolver:

R='192.0.2.53'
Q='www.example.com.'

dig -r "@$R" "$Q" A +dnssec +timeout=2 +tries=1
dig -r "@$R" "$Q" A +dnssec +cdflag +timeout=2 +tries=1

+cdflag (Checking Disabled) skips validation for that single query. If the normal query returns SERVFAIL while +cdflag returns valid records, you have almost certainly found a broken DNSSEC chain. Verify the parent DS against the child DNSKEY (tools like DNSViz make this obvious) and fix the chain—don’t permanently disable DNSSEC as a “performance tweak.”

Why updated records still return old answers or errors

Three caching behaviors explain most “I fixed it, why is it still broken?” moments:

  • Unexpired TTL: Lowering TTL after an incident doesn’t evict records already cached upstream. Lower TTLs ahead of planned migrations and wait for the old TTL to drain.
  • Negative caching: Querying a nonexistent name (NXDOMAIN) or missing record type (NODATA) gets cached for min(SOA TTL, SOA.MINIMUM) per RFC 2308. Creating the record afterward won’t clear upstream negative caches—you have to wait them out.
  • Serve-stale: Under RFC 8767, resolvers may return expired data with a positive TTL when authoritative servers are unreachable, so a non-zero TTL does not guarantee fresh data.

Also audit CNAME chains: deep cross-zone CNAME hops add cold-lookup latency. Trim unnecessary aliases and fix loops, but keep provider-mandated CNAMEs intact instead of replacing them with a raw CDN IP.

Slow DNS in Kubernetes Pods: ndots, CoreDNS, and NodeLocal DNSCache

When host lookups are fast but Pods are slow, the usual suspects are ndots search amplification, node conntrack contention, or CoreDNS upstreams.

Under the default dnsPolicy: ClusterFirst, a Linux Pod’s /etc/resolv.conf typically looks like this:

nameserver 10.32.0.10
search payments.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

With glibc, any hostname containing fewer than ndots (5) dots is queried against every search suffix first, and only tried as an absolute name if all of them fail. Looking up api.example.com (2 dots) can expand into:

api.example.com.payments.svc.cluster.local.
api.example.com.svc.cluster.local.
api.example.com.cluster.local.
api.example.com.

Because glibc often issues A and AAAA queries in parallel, a single external lookup can multiply into several internal queries, adding tail latency and load on CoreDNS.

Inspect the affected Pod and CoreDNS

NS='payments'
POD='your-pod-name'
CONTAINER='your-container-name'

kubectl -n "$NS" get pod "$POD" -o wide
kubectl -n "$NS" exec "$POD" -c "$CONTAINER" -- cat /etc/resolv.conf

kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
kubectl -n kube-system get endpointslice -l kubernetes.io/service-name=kube-dns -o wide
kubectl -n kube-system get configmap coredns -o yaml
kubectl -n kube-system logs -l k8s-app=kube-dns --since=10m --tail=200

If dig is available inside the container, compare an absolute FQDN (trailing dot) against +search with ndots=5 and ndots=2:

DNS_IP='10.32.0.10' # Use the nameserver from this Pod's /etc/resolv.conf
TARGET='www.kubernetes.io'

# Absolute FQDN lookup
kubectl -n "$NS" exec "$POD" -c "$CONTAINER" -- \
  dig -r "@$DNS_IP" "$TARGET." A +timeout=2 +tries=1

# Compare search expansion at ndots=5 vs ndots=2
kubectl -n "$NS" exec "$POD" -c "$CONTAINER" -- \
  dig -r "@$DNS_IP" "$TARGET" A +search +showsearch +ndots=5 +timeout=2 +tries=1
kubectl -n "$NS" exec "$POD" -c "$CONTAINER" -- \
  dig -r "@$DNS_IP" "$TARGET" A +search +showsearch +ndots=2 +timeout=2 +tries=1

Tip: Avoid blindly adding a trailing dot to HTTPS URLs in application code (like https://api.example.com./) just to skip search domains. Trailing dots can break TLS SNI matching, HTTP Host routing, cookie scoping, and server redirects.

Canary ndots:2 per workload

If your workload mostly calls external multi-label hostnames (2+ dots, like api.example.com), test ndots:2 on a single Deployment first:

spec:
  template:
    spec:
      dnsPolicy: ClusterFirst
      dnsConfig:
        options:
          - name: ndots
            value: "2"

Three Kubernetes gotchas to avoid:

  • Apex domains like example.com have only 1 dot, so they still traverse search suffixes under ndots:2.
  • Don’t switch dnsPolicy to Default thinking it means “Kubernetes default”—Default makes the Pod inherit the node’s DNS config and lose cluster Service resolution.
  • Linux Pods with hostNetwork: true need ClusterFirstWithHostNet to keep cluster DNS (not supported on Windows Pods).

CoreDNS metrics and node-level bottlenecks

Fast CoreDNS server latency doesn’t rule out Pod-side waiting—packets can be dropped or delayed by conntrack before ever reaching CoreDNS. If you scrape CoreDNS with Prometheus, check these three queries (adjust job="coredns" to match your scrape labels):

# CoreDNS internal handling p95
histogram_quantile(
  0.95,
  sum by (le, type) (
    rate(coredns_dns_request_duration_seconds_bucket{job="coredns"}[5m])
  )
)

# Upstream forward p95 (replaces deprecated coredns_forward_request_duration_seconds)
histogram_quantile(
  0.95,
  sum by (le, to) (
    rate(coredns_proxy_request_duration_seconds_bucket{
      job="coredns", proxy_name="forward"
    }[5m])
  )
)

# Response rate by rcode
sum by (rcode) (
  rate(coredns_dns_responses_total{job="coredns"}[5m])
)

Key things to check in CoreDNS and on the node:

  • coredns_forward_max_concurrent_rejects_total climbing: CoreDNS hit its upstream concurrency cap and is rejecting queries with REFUSED.
  • cache 30 is a ceiling, not a fixed TTL: it caps max cache duration. Check minimum TTL and serve_stale if stale answers linger.
  • NodeLocal DNSCache: cuts cross-node hops and conntrack pressure and uses TCP to CoreDNS, but adds a node-local failure point. If only one node’s Pods are slow, check that node’s NodeLocal Pod for OOMKilled or crash loops before scaling central CoreDNS.
  • Forwarding loops and older base images: pointing CoreDNS upstream at a host’s systemd-resolved stub (127.0.0.53) can trigger a forwarding loop. Also, as noted in the Kubernetes DNS debugging docs, Alpine 3.17 and earlier (musl 1.2.3) lack DNS TCP fallback, causing large responses to time out.

Application-layer waits: thread pools, connection pools, and caching

If packet captures and dig are both fast while your app’s DNS calls stay slow, look inside the runtime:

  • Node.js dns.lookup() thread-pool contention: dns.lookup() calls the OS resolver on the libuv thread pool. Heavy filesystem I/O or crypto work can starve that pool and queue up DNS lookups. dns.resolve4() / dns.resolve6() bypass libuv via c-ares and support { ttl: true }, but they also skip /etc/hosts and OS split-DNS rules—verify private and cluster name behavior before switching.
  • Bounded, TTL-aware app caching: If you add in-process caching, enforce a size cap, honor record TTLs, give errors a short negative TTL, and coalesce concurrent lookups for the same name (singleflight). Never cache indefinitely or hardcode CDN IPs in /etc/hosts.
  • Connection pooling vs. DNS failover: HTTP keep-alive eliminates repeated lookups, which also means existing connections keep talking to old IPs after a DNS change. Services relying on DNS failover need a finite connection max lifetime.
  • DoH / DoT cold starts: Encrypted DNS over TLS or HTTPS pays a handshake cost on the first connection and amortizes it over reused sessions. Don’t equate a one-shot cold CLI test with steady-state browser latency.

When flushing DNS cache actually helps—and how to verify the fix

Flushing cache only fixes stale local state (when upstream records have recovered but your machine still holds an old answer or negative cache entry). It will not fix network RTT, packet loss, broken VPN split DNS, or authoritative errors.

For example, if nslookup on Windows already returns the new IP while ipconfig /displaydns still shows a cached “name does not exist” entry, a flush makes sense:

# Windows
ipconfig /flushdns

On Linux with systemd-resolved:

# Flushes only the local stub cache, not browser or upstream caches
sudo resolvectl flush-caches

Note that resolvectl reset-server-features resets EDNS/TCP feature probing rather than flushing cached records—don’t bundle cache flushes and service restarts into a cron job, as periodic flushing creates cold-lookup spikes and makes latency worse.

How to confirm the fix worked: Re-run your measurements from the original application and network instead of stopping at one fast dig:

  1. Confirm the lookup tail latency and timeouts are gone across a multi-sample curl run.
  2. Check remote_ip and total request time to make sure a resolver change didn’t degrade CDN routing.
  3. For VPN or Kubernetes changes, re-test internal FQDNs, short names, and cluster Services to verify nothing regressed.

FAQ

Will switching to 1.1.1.1 or 8.8.8.8 fix slow DNS lookups?

Only if your current recursive resolver or router forwarder is the bottleneck. If the real cause is VPN split DNS, UDP packet loss, an IPv6 routing black hole, or an authoritative/DNSSEC issue on the target domain, switching to a public resolver won’t help—and can break internal corporate hostnames or worsen CDN routing.

Why is dig fast while my browser still hangs on pages?

They take different paths. dig sends a raw DNS packet and skips the browser’s HostCache, built-in DNS client, and DoH provider—and it doesn’t perform TCP connect, TLS handshake, or page rendering. Use curl -w first to verify time is actually spent in time_namelookup, then compare OS getaddrinfo() with your browser’s Secure DNS settings.

What are the side effects of setting ndots:2 in Kubernetes?

External hostnames with 2 or more dots (such as api.example.com) are queried as absolute names first, skipping unnecessary cluster search suffixes. The trade-off is that internal relative names with 2 dots—such as my-svc.my-ns.svc, which relies on .cluster.local being appended—will trigger an external lookup returning NXDOMAIN before falling back to the search list. Roll out ndots:2 per Deployment and test both internal and external names.

Why do I still get old IPs or errors after updating DNS records?

Three common causes: the previous record’s TTL hasn’t expired yet; querying the name before it existed triggered upstream negative caching (min(SOA TTL, SOA.MINIMUM)); or a brief authoritative outage caused a recursive resolver to serve stale data. Flushing your local machine cannot force upstream recursive resolvers to refresh early.

References

About the author

Mttao

Exploring technology and life's wisdom

About GitHub

Related Posts

Comments