In case of problems

If you are a Website hoster and you believe your server(s) may be experiencing problems due to our crawling efforts, please consult the information below on what you can expect to see from us, how you can contact us, and as a last resort how you can block our traffic.

We thank you in advance for working with us to resolve any issues, and apologise if we have inadvertently caused problems through our crawling efforts.

What can Website hosters expect?

Web Crawling

As hoster, you can expect that our crawler will send a limited number of HTTP GET requests for each Website that you host, provided the apex domain is on our to fetch list. We use a widely used, matured, open-source Web crawler that first attempts to fetch, interpret and respect robots.txt and then attempts to fetch the apex domain's landing page, following redirects if applicable. Requests to the same host are spread out and should stay (well) below the request-rate exhibited by regular users visiting the site.



If you are a Website hoster and you feel our crawling is impacting your infrastructure, please read on.

Note: our crawler traffic originates from within a IPv4 /26 prefix. The parent prefix has detailed contact information in the RIPE database.

What should I do if the crawling impacts my infrastructure?

As we believe in responsible network measurement practices, we have taken care in configuring our crawler such that it should not impact Webservers adversely. Nevertheless, it is not up to us to judge if our crawling impacts your infrastructure as hoster. Therefore, if you think our crawling is impacting your infrastructure adversely, please contact us so we can try to resolve the problem.

If you decide to contact us, it would help if you can provide us with the following information:

  • The name of your organisation, and a point of contact (e-mail, and if possible a phone number);
  • The IP address(es) of the affected server(s);
  • A description of the impact (e.g. number of HTTP GETs per second you see, effect this has, …);
  • A date and time on which you (first) experienced problems you think may be caused by our measurement;
  • The IP address(es) from which you are receiving problematic traffic (this will help us determine if the traffic is actually coming from us, and if so, if it is only coming from a single measurement host or from multiple systems).

Blocking our traffic

If you decide you need to block our traffic, we would appreciate it if you contact us about this. As mentioned above, our traffic always originates from within a IPv4 /26 prefix. If the traffic that is causing you problems does not follow this pattern it is most likely not coming from us. Upon request, we can disclose the specific hosts from which we crawl through private communication. We ask for your understanding for not publishing this information publicly.

Response time

We are in the Central European Timezone (UTC+1 in winter, UTC+2 in summer). We try to answer any request concerning problems within one business day. If you need our urgent attention, please include “[URGENT]” in the subject line of any e-mail you send us.