Log Sources, Data Collection, and Normalization

Why Log Sources and Collection Matter

A Security Information and Event Management (SIEM) platform is only as useful as the data it receives. If log sources are missing, misconfigured, or deliver data in inconsistent formats, analysts will miss threats or waste time chasing false alerts. This lesson walks through connecting real log sources, transporting that data reliably, and normalizing it so every record speaks the same language inside the SIEM.

Types of Log Sources

A mature SIEM deployment collects from at least four categories of sources:

• Endpoints (Windows and Linux hosts): Operating systems generate authentication events, process creation records, and file access logs. Windows logs flow through the Windows Event Log subsystem; Linux systems write to syslog or the systemd journal.
• Network security devices: Firewalls (such as pfSense) produce allow/deny records for every connection attempt. These logs are critical for detecting lateral movement and policy violations.
• Web servers: Apache and similar servers log every HTTP request, including the client IP, URI, response code, and bytes transferred. Web server logs are a primary source of evidence during web-application incident investigations.
• Authentication services: Active Directory domain controllers and RADIUS servers log logon successes, failures, and privilege use.

For the CompTIA CySA+ exam, you are expected to understand log source types and their role in threat detection.

Forwarding Methods

Agent-based forwarding places a small program—called a forwarder or agent—on each endpoint. The agent reads local logs and ships them to the SIEM collector. Examples include Elastic Beats (specifically Winlogbeat for Windows events and Filebeat for flat-file logs) and the Splunk Universal Forwarder. Agents handle buffering, compression, and encrypted transport, making them reliable over unstable links.

Syslog forwarding is the standard method for network devices that cannot run an agent. The device sends log messages—formatted under RFC 5424 or the older RFC 3164—over UDP port 514 (or TCP for reliable delivery) to a syslog receiver on the SIEM. pfSense and Apache can both be configured to send syslog messages this way.

API and file-based ingestion covers cloud services and applications that expose logs through an API or write to a shared file path. The SIEM polls or subscribes to retrieve records.

The Collection Pipeline

Data moves through several stages before an analyst can search it:

1. Generation – The source system writes a log entry.

2. Transport – An agent or syslog sender transmits the entry to the SIEM.

3. Ingestion – The SIEM receiver accepts the raw data and tags it with a source identifier and receipt timestamp.

4. Parsing – The SIEM applies parsing rules to break raw text into named fields (source IP, destination IP, action, timestamp, etc.).

5. Normalization – Field values are mapped to a common schema so a "deny" from a firewall and a "blocked" from a proxy appear as the same action value.

6. Indexing – Parsed, normalized records are written to the search index.

If parsing fails at step 4, critical fields remain unparsed and the record becomes nearly useless for correlation rules.

Normalization and Field Extraction

Raw log lines from different vendors look nothing alike. A firewall might write `2024-03-15T14:22:01Z` while an older appliance writes `Mar 15 14:22:01`. If the SIEM stores both versions as raw strings, time-based correlation across sources breaks.

Grok patterns (used by Logstash and similar tools) and regular expressions (regex) are the two main tools for field extraction. A grok pattern is a named regex shortcut. For example, the pattern `%{TIMESTAMP_ISO8601:event_timestamp}` extracts an ISO 8601 timestamp and stores it in a field called `event_timestamp`. After normalization, every source writes to the same field name in the same format, regardless of the original log style.

The Elastic Common Schema (ECS) and Splunk's Common Information Model (CIM) are two widely used normalization frameworks. Both define standard field names (e.g., `source.ip`, `destination.ip`, `event.action`) so dashboards and detection rules work across log types without rewriting.

Verification

After configuring a log source, always verify ingestion before declaring the source operational:

1. Generate a known event on the source (log in, trigger a firewall rule, make an HTTP request).

2. Run a search in the SIEM filtered to that host and a narrow time window.

3. Inspect the returned record: confirm all critical fields (source IP, destination IP, timestamp, action or event ID) are populated and correctly typed.

4. Check for `_raw` or unparsed fields that suggest the parser missed something.

Document each source in a configuration log: source name, forwarding method, receiving input on the SIEM, parser or index-time extraction used, and the verification search query. This record is your audit trail and helps the next analyst troubleshoot gaps.

Safety and Lab Environment Notes

All lab work in this course uses isolated virtual machines that are not connected to production networks or the public internet unless the lab explicitly requires it. Treat credentials created in lab VMs as lab-only; never reuse lab passwords on real systems. When decommissioning a lab VM that held log data, delete the virtual disk to prevent sensitive test data from persisting on shared hardware. If you are working on a shared physical host, confirm with your instructor that your virtual network is host-only or NAT-only before starting any services that listen on network ports.