Forward DNS Measurements

Zone-Based

Introduction

On this page, we provide technical background information about our forward DNS (fDNS) measurement. This information is targeted at DNS operators and academic researchers.

Measurement Goal

The goal of the OpenINTEL project is to capture daily snapshots of the state of large parts of the global Domain Name System. Because the DNS plays a key role in almost all Internet services, recording this information allows us to track changes on the Internet, and thus its evolution, over longer periods of time. By performing active measurements we build consistent and reliable time series of the state of the DNS.

What do we measure?

Our fDNS measurement is seeded with domain names that we extract from zone files of specific TLDs or from list-based sources. The notable difference between these two measurement bases is that zone files are largely complete in terms of covering all registered (actively delegated) domain names under the respective TLD (e.g., .com), whereas lists can cover something other than a TLD (e.g., popular domains).

Zone-based Measurement

Our fDNS measurement covers a large number of generic Top-Level Domains (gTLDs) which we obtain from registries, either directly or through ICANN's Centralized Zone Data Service. We also measure a number of country-code TLDs, for either ccTLDs that are public or through a data sharing agreement with the respective registry. We download snapshots of zones (at least) once per day.

Coverage

Generic TLDs

Our measurement currently covers the following generic top-level domains (TLDs):

^(*) Our measurement covers almost all new gTLDs available through ICANN’s Centralized Zone Data Service, which is currently around 1200 new gTLDs.

^(**) The measurement only covers US Federal domain names included in the list obtained from the US Federal Government open access API.

Country-code TLDs (ccTLDs)

Our measurement currently covers the following country-code top-level domains (ccTLDs):

What should I do if the measurement impacts my infrastructure?

If you are a DNS operator and you feel our measurement is impacting your infrastructure, please go to our Problems page for information on how to contact us and how you can block our traffic if necessary.

Records and Measurement Frequency

For forward measurements, the measurement system uses a fixed set of DNS queries, which it sends once every 24 hours for every registered domain name under a TLD. We store the resulting measurement data in a partitioned output, where the zone (source), year, month and day make up keys in the partition.

The set of queries currently consists of the following queries:

• SOA
• NS ⁽³⁾
• A ⁽¹⁾
• AAAA ⁽¹⁾
• MX ^(3,4)
• TXT
• DNSKEY
• DS
• NSEC3 ⁽²⁾
• CAA
• CDS ⁽⁵⁾
• CDNSKEY ⁽⁵⁾
  
  

1. These queries are also sent for the www label (e.g. www.example.com).
2. We send a query for a non-existent domain name to record authenticated denial-of-existence data, but only for DNSSEC-signed domains.
3. We resolve the A and AAAA records for these records in a separate infrastructure measurement.
4. We resolve associated TLSA records for ports 25, 465 and 587 in a separate infrastructure measurement.
5. We only resolve these records for DNSSEC-signed domains for which at least a DNSKEY or DS record exists.

All response records, including full CNAME expansions and RRSIG signature records are stored.

The data dictionary describes the data format, including type and explanation of the fields.

Data Access & Terms

The data collected by the OpenINTEL platform has numerous applications in network and network security research. To support these efforts, we make our open data available on the download page or through the links provided in the coverage overview above, under the terms and conditions outlined on our terms page. As using our data may require specialized knowledge and specific analysis infrastructure, we encourage academic researchers to contact us to discuss your needs.

Closed data may be available upon request (more information here). If you have an interest in licencing access to our data for commercial purposes feel free to contact us.

Example using Docker and Spark

Below we provide instructions to use the data using Docker, Jupyter Lab, the AWS SDK and Apache Spark.

Step 1: Create a Dockerfile and build the image

Create and change to a new working directory:

mkdir openintel-fdns-notebook && cd openintel-fdns-notebook

Create a Dockerfile with the following content:

FROM quay.io/jupyter/pyspark-notebook:spark-3.5.3

USER root
RUN wget -q https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.375/aws-java-sdk-bundle-1.11.375.jar -P /usr/local/spark/jars/
RUN wget -q https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.2.0/hadoop-aws-3.2.0.jar -P /usr/local/spark/jars/

USER ${NB_UID}
RUN pip install boto3

Build the Docker container:

docker build --tag 'openintel-fdns-notebook:spark-3.5.3' .

Step 2: Run the container

Use the following command to start the Jupyter Lab docker container:

docker run -p 8888:8888 openintel-fdns-notebook:spark-3.5.3

Step 3: Open Jupyter Lab

The standard output from the previous command will display a web link with authentication token.

Open the link in your browser to access Jupyter Lab, or use this link: http://127.0.0.1:8888/lab and submit the token.

Step 4: Upload the openintel-fdns example notebook

Download the example .ipynb python notebook here.

Click the up arrow ("Upload Files") and upload .ipynb notebook.

A preview web version of the notebook can be found here.