WhoisXML API Blog

We are excited to introduce First Watch Malicious Domains Data Feed, the newest addition to our predictive threat intelligence product line. This innovative solution is designed to enhance early threat detection and response by identifying substantially more malicious domains than traditional feeds, right at the point of registration.

First Watch Malicious Domains Data Feed offers several key advantages for cybersecurity teams, whether they are part of in-house Security Operations Centers (SOCs) or Managed Security Service Providers (MSSPs).

Authors:
Ed Gibbs, Field CTO, WHOIS API Inc.
Jeff Vogelpohl

Introduction

Growing up, I remember the vast array of candies and ice cream flavors while visiting quaint candy shops. Today, we’re overwhelmed by the plethora of technologies any imaginative person could want – thanks to the provocativeness of human ingenuity. As flavors were designed for these memories of delightful treats, this same ingenuity has brought technological advancements like AI to aid and improve all life whereas some provide just the opposite. Our adversaries continuously exploit and weaponize our ingenuity to degrade life. Life is worth protecting.

Abstract

This document outlines the setup of a PostgreSQL database on Ubuntu Linux to efficiently manage and query WHOISXMLAPI’s Premium DNS database. Designed to store and analyze billions of DNS records, this database will handle large-scale data ingestion, facilitate rapid data retrieval, and support extensive analytical operations. PostgreSQL’s robust performance, scalability, and support for advanced indexing make it ideal for managing DNS data, while its compatibility with open-source tools provides a flexible environment for future scaling and data processing.

Abstract

Apache Cassandra is a highly scalable, distributed NoSQL database designed for handling massive volumes of data across many commodity servers without a single point of failure. Its decentralized nature and robust architecture make it particularly well-suited for applications that require high availability, fault tolerance, and horizontal scalability. Cassandra is engineered to handle very large datasets, supporting billions of records with ease, making it an ideal choice for organizations dealing with large-scale, real-time applications such as time-series data, IoT data, and customer logs. Through its use of a partitioned architecture and the ability to add nodes seamlessly as data grows, Cassandra offers an efficient means of managing big data with low latency and high throughput.

The WhoisXML API research team analyzed more than 8.2 million domains registered between 1 and 31 October 2024 to identify the most popular registrars, top-level domain (TLD) extensions, and other global domain registration trends.

We also determined the top TLD extensions used by the more than 57.4 billion domains from our DNS database’s A record full file released in the same month.

Next, we studied the top TLDs of more than 1.0 million domains detected as indicators of compromise (IoCs) in October.

Finally, we summed up our findings and provided links to the threat reports produced using DNS, IP, and domain intelligence sources during the period.

We are excited to announce that the Standard and Premium DNS Database files from DNS Database Download are now enriched with two new columns, namely, wildcard and active. These additions allow you to determine if a DNS record is part of a wildcard entry and check if a domain name or subdomain is active based on its most recent resolution status.

We are excited to announce that Snowflake users can now request access to WHOIS Database Download’s quarterly gTLD and ccTLD files on the platform, available in three formats:

• Simple: The simple CSV file contains the domain name, registrar name and email address, WHOIS and name servers, creation and expiration dates, and registrant and administrative contact details.

• Regular: The regular CSV file contains all the fields in the Simple file, along with information about the billing, technical, and zone contacts.

• Full: This file format contains all fields in the Simple file, in addition to the raw text from the WHOIS registry and registrar.

Abstract

This white paper provides a comprehensive guide for importing First Watch Malicious Domain data from a CSV file into a MySQL database and indexing it on the `domainName` column. The steps outlined here cover the creation of a database, the table structure design, and data import using MySQL. By following this approach, users can efficiently handle domain-related datasets for querying and analysis purposes.