The OpenINTEL project is happy to announce significant updates aimed at expanding access to our data and providing enhanced tools for academic research. These updates mark a major milestone in our ongoing mission to capture the daily state of the Domain Name System and empower the research community with actionable insights.
In a major step forward, OpenINTEL is now making available country code domain name lists extracted from Certificate Transparency (CT) logs. For years, our flagship DNS measurements have provided robust time-series data, but covering country-code Top-Level Domains (ccTLDs) has been challenging due to limited access to zone files. By leveraging publicly available CT data, we’re now able to expand our measurements to other ccTLDS, enabling us to investigate ccTLD activity with greater depth and accuracy.
Key Highlights:
We have started publishing current ccTLD lists on this website. Historical data will be made available at a later date. These lists aim to empower researchers to extend the coverage of their name-based analyses or explore new insights into hard-to-obtain ccTLD zones.
Find more information about the domain lists here, and about the measurement campaign here.
Another exciting addition to our toolkit is Zonestream, a cutting-edge solution for real-time monitoring of DNS zone file changes. Developed under the OpenINTEL initiative, Zonestream uses Kafka and WebSockets to stream DNS zone updates, blending ccTLD data from IXFR and newly registered domains from CT logs.
Zonestream Capabilities:
This tool is a useful for researchers, offering actionable real-time insights into the constantly evolving DNS ecosystem. Find more information on Zonestream here.
To support our growing suite of tools and datasets, we’ve overhauled our website to provide a better overview of our current measurements and data sets. You can now explore detailed technical background, methods, and dataset-specific details.
As part of the website upgrade, our existing index for downloading open data will soon be retired. It has been replaced by a new, more granular index. This new system enables us to provide easier access to open ccTLD data, now divided into per-TLD datasets instead of a single archive. Moreover, the new index shares our data using the parquet file format, instead of the previously used avro file format, further reducing archive sizes.
In an effort to make accessing and using our data easier, we now provide detailed step-by-step instructions. On the pages of the data sets, you will find a guide to build a simple analysis environment, as well as python notebooks demonstrating how to download and perform a simple analysis on the data. These example notebooks are a good starting point, and straightforward to extent to more complex analysis tasks.
As we are soon approaching our 10-year anniversary, the OpenINTEL project is proud to advance its mission to capture the evolution of the Internet’s DNS infrastructure through these developments. We are excited about the potential of the new data and tools and invite the academic and security research communities to explore our new data and tools.