Forward DNS Measurements

Top Lists

Introduction

On this page, we provide technical background information about our forward DNS (fDNS) measurement. This information is targeted at DNS operators and academic researchers.

Measurement Goal

The goal of the OpenINTEL project is to capture daily snapshots of the state of large parts of the global Domain Name System. Because the DNS plays a key role in almost all Internet services, recording this information allows us to track changes on the Internet, and thus its evolution, over longer periods of time. By performing active measurements we build consistent and reliable time series of the state of the DNS.

What do we measure?

Our fDNS measurement is seeded with domain names that we extract from zone files of specific TLDs or from list-based sources. The notable difference between these two measurement bases is that zone files are largely complete in terms of covering all registered (actively delegated) domain names under the respective TLD (e.g., .com), whereas lists can cover something other than a TLD (e.g., popular domains).

List-based Measurement

Our fDNS measurement covers a number of list-based sources. We measure several lists of Top-ranked domain names such as the Tranco Top 1 million and Cisco Umbrella. We download the latest versions of these lists at least once daily. We also measure ccTLD domain names extracted from Certificate Transparency (CT) logs, which provides partial coverage of TLDs for which we do not have the zone file (the SeeSeeTLD page offers more information on these lists). Finally, we resolve the names our fDNS measurement encounters in so-called infrastructure records (i.e., the A/AAAA records for name servers {NS} and mail exchangers {MX}) and we perform TLSA queries for infrastructure names.

Coverage

Top list measurements we are currently running:

What should I do if the measurement impacts my infrastructure?

If you are a DNS operator and you feel our measurement is impacting your infrastructure, please go to our Problems page for information on how to contact us and how you can block our traffic if necessary.

Records and Measurement Frequency

For forward measurements, the measurement system uses a fixed set of DNS queries, which it sends once every 24 hours for every registered domain name under a TLD.

The set of queries currently consists of the following queries:

• SOA
• NS ⁽³⁾
• A ⁽¹⁾
• AAAA ⁽¹⁾
• MX ^(3,4)
• TXT
• DNSKEY
• DS
• NSEC3 ⁽²⁾
• CAA
• CDS ⁽⁵⁾
• CDNSKEY ⁽⁵⁾

1. These queries are also sent for the www label (e.g. www.example.com).
2. We send a query for a non-existent domain name to record authenticated denial-of-existence data, but only for DNSSEC-signed domains.
3. We resolve the A and AAAA records for these records in a separate infrastructure measurement.
4. We resolve associated TLSA records for ports 25, 465 and 587 in a separate infrastructure measurement.
5. We only resolve these records for DNSSEC-signed domains for which at least a DNSKEY or DS record exists.

All response records, including full CNAME expansions and RRSIG signature records are stored.

The data dictionary describes the data format, including type and explanation of the fields.

Data Access & Terms

The data collected by the OpenINTEL platform has numerous applications in network and network security research. To support these efforts, we make our open data available on the download page or through the links provided in the coverage overview above, under the terms and conditions outlined on our terms page. As using our data may require specialized knowledge and specific analysis infrastructure, we encourage academic researchers to contact us to discuss your needs.

Closed data may be available upon request (more information here). If you have an interest in licencing access to our data for commercial purposes feel free to contact us.

Example using Docker and Spark

Below we provide instructions to use the data using Docker, Jupyter Lab, the AWS SDK and Apache Spark.

Step 1: Create a Dockerfile and build the image

Create and change to a new working directory:

mkdir openintel-fdns-notebook && cd openintel-fdns-notebook

Create a Dockerfile with the following content:

FROM quay.io/jupyter/pyspark-notebook:spark-3.5.3

USER root
RUN wget -q https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.375/aws-java-sdk-bundle-1.11.375.jar -P /usr/local/spark/jars/
RUN wget -q https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.2.0/hadoop-aws-3.2.0.jar -P /usr/local/spark/jars/

USER ${NB_UID}
RUN pip install boto3

Build the Docker container:

docker build --tag 'openintel-fdns-notebook:spark-3.5.3' .

Step 2: Run the container

Use the following command to start the Jupyter Lab docker container:

docker run -p 8888:8888 openintel-fdns-notebook:spark-3.5.3

Step 3: Open Jupyter Lab

The standard output from the previous command will display a web link with authentication token.

Open the link in your browser to access Jupyter Lab, or use this link: http://127.0.0.1:8888/lab and submit the token.

Step 4: Upload the openintel-fdns example notebook

Download the example .ipynb python notebook here.

Click the up arrow ("Upload Files") and upload .ipynb notebook.

A preview web version of the notebook can be found here.