WhoisXML API Blog

Handling CSV Files with CR/LF Line Endings

Ed Gibbs, VP of Research, whoisxmlapi.com

Issue Overview

CSV files containing CR/LF (carriage return/line feed) line endings can cause parsing errors with certain CSV-to-JSON conversion tools. This commonly occurs when CSV files are created on Windows systems, which use CR/LF (\r\n) line endings, while many Unix-based tools expect only LF (\n) line endings.  This applies to various datafeeds WHOISXMLAPI provides such as WHOIS, Netblocks, and IP Geolocation.

Ingesting IP Netblocks with DynamoDB

Introduction

Overview

DynamoDB is Amazon's fully managed NoSQL database service that's designed for high-performance applications at scale. DynamoDB is known for its single-digit millisecond response times, automatic scaling, and built-in security features.

Python Script for Capturing Reverse NS API’s Full Output

WhoisXML API has recently created a Python script to help Reverse NS API users capture the entire query output without being restricted to 300 results through automatic pagination.

The script essentially removes the limitation of standard API pagination, automatically retrieving all subsequent pages of the query output instead of the user manually requesting each page. This capability enables users to fully automate processes and pipelines that rely on Reverse NS API data.

Python Script for Transforming Domain Names from First Watch Malicious Domains Data Feed into STIX 2.1

WhoisXML API recently created a Python script to help users of First Watch Malicious Domains Data Feed, also known as “First Watch,” transform predictive domain intelligence into a machine-readable format—STIX 2.1—for automated data processing.

The script reads domain names from First Watch files, converts them to STIX 2.1 Indicator objects with domain-name observable types, and generates a TAXII 2.1-compatible STIX bundle. The output is downloadable as a JSON file in TAXII format.

Managing a Passive DNS Database Using PostgreSQL

Abstract

This document outlines the setup of a PostgreSQL database on Ubuntu Linux to efficiently manage and query WHOISXMLAPI’s Premium DNS database. Designed to store and analyze billions of DNS records, this database will handle large-scale data ingestion, facilitate rapid data retrieval, and support extensive analytical operations. PostgreSQL's robust performance, scalability, and support for advanced indexing make it ideal for managing DNS data, while its compatibility with open-source tools provides a flexible environment for future scaling and data processing.

Managing a Passive DNS Database Using Apache Cassandra

Abstract

Apache Cassandra is a highly scalable, distributed NoSQL database designed for handling massive volumes of data across many commodity servers without a single point of failure. Its decentralized nature and robust architecture make it particularly well-suited for applications that require high availability, fault tolerance, and horizontal scalability. Cassandra is engineered to handle very large datasets, supporting billions of records with ease, making it an ideal choice for organizations dealing with large-scale, real-time applications such as time-series data, IoT data, and customer logs. Through its use of a partitioned architecture and the ability to add nodes seamlessly as data grows, Cassandra offers an efficient means of managing big data with low latency and high throughput.

Importing and Indexing First Watch Malicious Domains Data Feed into MySQL

Abstract

This white paper provides a comprehensive guide for importing First Watch Malicious Domain data from a CSV file into a MySQL database and indexing it on the `domainName` column. The steps outlined here cover the creation of a database, the table structure design, and data import using MySQL. By following this approach, users can efficiently handle domain-related datasets for querying and analysis purposes.

Introducing Server-to-Server OAuth to Secure API Integrations

We’re thrilled to announce the availability of Server-to-Server OAuth for all our API users. Using this authentication method helps developers build robust integrations without compromising security and efficiency.

Server-to-Server OAuth or two-legged OAuth eliminates user interaction for authentication, making it ideal for automated workflows. It simplifies API integration by allowing a server to directly request and manage access tokens. The authorization method is also designed to handle high-volume API requests, supporting application scalability.