Scripting for Threat Intelligence: IOC Extraction and Feed Parsing

What Are Indicators of Compromise?

An indicator of compromise (IOC) is a piece of observable evidence that suggests a system may have been attacked or infected. Security analysts collect IOCs from threat reports, vendor advisories, and incident post-mortems and use them to hunt for malicious activity across their environments. The three most common IOC types are:

• IP addresses — source or destination addresses linked to malicious infrastructure
• Domain names — hostnames used for command-and-control, phishing, or malware delivery
• File hashes — MD5 or SHA-256 fingerprints of known malicious files

Threat reports are often published as plain text or PDF exports, meaning analysts must parse unstructured prose to pull out usable IOCs. Python's built-in `re` module makes this practical and repeatable.

Regular Expressions for IOC Extraction

The `re` module lets you describe a pattern and search text for all matches. The key function here is `re.findall(pattern, text)`, which returns a list of every non-overlapping match.

IPv4 pattern: An IPv4 address is four groups of one to three digits separated by dots. A working pattern is:

```python

import re

ipv4_pattern = r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b'

```

The `\b` word-boundary anchors prevent matching digits that are part of longer strings. Note that this pattern matches syntactically valid-looking addresses; you may still see values like `999.0.0.1`. Filter those with a quick range check if needed.

Domain pattern: Domains end in a known top-level domain (TLD). A practical pattern for common TLDs is:

```python

domain_pattern = r'\b(?:[a-zA-Z0-9-]+\.)+(?:com|net|org|io|gov|edu|co|uk|ru|cn)\b'

```

Adjust the TLD list to match your threat landscape.

MD5 and SHA-256 hash patterns: MD5 hashes are exactly 32 hexadecimal characters; SHA-256 hashes are exactly 64.

```python

md5_pattern = r'\b[a-fA-F0-9]{32}\b'

sha256_pattern = r'\b[a-fA-F0-9]{64}\b'

```

Combine both into one list with `re.findall(md5_pattern, text) + re.findall(sha256_pattern, text)`.

Building the Extraction Script

Your script should accept a filename as a command-line argument (`sys.argv[1]`), read the file, run all three patterns, deduplicate each list with `list(set(...))`, then write a JSON file.

```python

import re, json, sys

def extract_iocs(text):

ips = list(set(re.findall(r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b', text)))

domains = list(set(re.findall(

r'\b(?:[a-zA-Z0-9-]+\.)+(?:com|net|org|io|gov|edu|co|uk|ru|cn)\b', text)))

hashes = list(set(

re.findall(r'\b[a-fA-F0-9]{64}\b', text) +

re.findall(r'\b[a-fA-F0-9]{32}\b', text)))

return {"ips": ips, "domains": domains, "hashes": hashes}

with open(sys.argv[1]) as f:

text = f.read()

result = extract_iocs(text)

with open("iocs.json", "w") as out:

json.dump(result, out, indent=2)

print("Done. IOCs written to iocs.json")

```

Deduplication matters because threat reports often repeat the same IP or domain multiple times. Wrap each list in `set()` before converting back to a list.

Enriching IOCs with AbuseIPDB

Once you have a list of IP addresses, you can query the AbuseIPDB API to retrieve an abuse confidence score (0–100) for each one. A score of 100 means the IP is widely reported as malicious.

The API returns HTTP 429 (Too Many Requests) when you exceed your rate limit. Handle this with a retry loop using `time.sleep()`:

```python

import requests, time

API_KEY = "YOUR_FREE_KEY_HERE"

HEADERS = {"Key": API_KEY, "Accept": "application/json"}

def enrich_ip(ip, retries=3):

url = "https://api.abuseipdb.com/api/v2/check"

params = {"ipAddress": ip, "maxAgeInDays": 90}

for attempt in range(retries):

resp = requests.get(url, headers=HEADERS, params=params)

if resp.status_code == 429:

print(f"Rate limited. Waiting before retry {attempt+1}...")

time.sleep(60)

continue

resp.raise_for_status()

data = resp.json()["data"]

return {"ip": ip, "abuseScore": data["abuseConfidenceScore"]}

return {"ip": ip, "abuseScore": None, "error": "rate_limit_exceeded"}

```

Return `None` for the score rather than crashing when retries are exhausted. This keeps your output file valid JSON regardless of API errors.

Safety and Responsible Use

Always run these scripts inside your designated Ubuntu lab VM, never on a production network or against live systems without authorization. Querying external APIs with real IOCs from classified or sensitive reports may expose that data to a third party — use only the sample files provided by your instructor. Store your API key in an environment variable, not hard-coded in source files that may be shared or submitted. Threat intelligence data can contain live malicious URLs; do not click or browse to any IOC directly from your workstation.

Connection to Exam Objectives

CompTIA CySA+ covers threat intelligence as a core domain. Expect exam questions on IOC types, threat-intelligence feeds, and how analysts operationalize intelligence. CompTIA Security+ also addresses threat data and indicators under its threat-management objectives. The ISC2 CC covers the concept of threat intelligence at a foundational level.