📈 SIEM & Log Monitoring
SIEM platforms, log aggregation architectures, correlation rules, UEBA, and SOC KPIs — the technology backbone of security monitoring and detection.
Security Information & Event Management (SIEM) is the technology backbone of security monitoring. It collects, normalizes, and correlates log data from across the enterprise — endpoints, network devices, cloud workloads, identity platforms, and applications. SIEM enables real-time threat detection, compliance reporting, forensic investigation, and drives the SOC KPIs that measure organizational security effectiveness. Modern SIEM has evolved to include User & Entity Behavior Analytics (UEBA), cloud-native architectures, and AI-powered anomaly detection.
Key Concepts
Correlation Rules & Use Cases
Rule Types: Single-event rules (match on one log — e.g., admin login from unusual country), multi-event correlation (chain events — e.g., 5 failed logins followed by success within 10 minutes), threshold-based (alert when event count exceeds N in time window), and behavioral baselines (deviation from normal patterns). Common Use Cases: Brute force detection, impossible travel, lateral movement (multiple hosts accessed in sequence), privilege escalation (user added to admin group), data exfiltration (unusual outbound data volume), credential stuffing, and C2 beacon detection (periodic outbound connections). Detection Languages: Splunk SPL, Sentinel KQL, QRadar AQL, Chronicle YARA-L, SIGMA (vendor-agnostic detection rules that compile to platform-specific queries).
Dashboards & Visualization
Operational Dashboards: Real-time alert feed, event volume trends, top triggered rules, open incidents by severity, analyst workload distribution, and SLA timers. Executive Dashboards: Risk posture heatmap, MTTx trend lines, compliance scorecard, top threat categories, and month-over-month comparisons. Threat Hunting Views: IOC search results, MITRE ATT&CK heat map (coverage vs. gaps), timeline visualization for attack reconstruction, and entity relationship graphs. Best Practices: Avoid dashboard sprawl — create role-based views (analyst, manager, CISO). Use drill-down from summary to raw events. Alert on dashboard metrics (e.g., if alert volume drops 90%, collectors may be down).
🔗 Extended Detection & Response (XDR)
What it is: XDR breaks down security silos by providing integrated visibility across endpoints, network, email, cloud, and identity into a single correlated detection and response platform. Unlike SIEM, which aggregates logs requiring manual correlation, XDR natively integrates telemetry from multiple security layers and automatically correlates related events into unified incidents. SIEM vs XDR: SIEM collects logs from any source and requires analysts to write correlation rules. XDR ingests native telemetry from its own security stack and provides out-of-the-box cross-layer detection. Many organizations run both — SIEM for compliance/log management and XDR for detection/response. Key Capabilities: Automatic attack chain reconstruction across endpoints+network+email, unified investigation timeline, automated response actions across all layers, and reduced mean time to detect by correlating weak signals that individually would not trigger alerts. Leading Platforms: Palo Alto Cortex XDR, Microsoft Defender XDR (formerly 365 Defender), CrowdStrike Falcon XDR, Trend Micro Vision One, SentinelOne Singularity XDR, Cisco XDR. CISO Value: Reduces tool sprawl, lowers MTTD/MTTR, and provides a single pane of glass for the SOC — enabling smaller teams to defend larger attack surfaces.
Log Aggregation & Forwarding
Fluentd: CNCF-graduated log collector — plugins for 500+ data sources, memory-efficient buffering, and Kubernetes-native with Fluent Bit. Logstash: Part of ELK stack — input/filter/output pipeline architecture, Grok parsing for unstructured logs, and 200+ plugins. Cribl Stream: Observability pipeline — route, reduce, enrich, and transform log data before it reaches the SIEM. Reduces ingestion volume (and cost) by 40-60% through filtering and summarization. Syslog-ng: High-performance log collector with advanced filtering, pattern matching, and reliable delivery. Key Pipeline Concepts: Parsing, normalization (mapping fields to common schema like ECS or OCSF), enrichment (adding GeoIP, threat intel, asset context), filtering (dropping noise/debug logs), and routing (sending different logs to different destinations).
Log Retention & Compliance
Retention Tiers: Hot storage (0-30 days, fast search, expensive — SSD/high-performance), warm storage (30-90 days, moderate search speed), cold/archive (90 days-7 years, slow retrieval, cost-effective — S3/Glacier/Blob). Regulatory Requirements: PCI-DSS Req 10.7: minimum 1 year retention, 3 months immediately accessible. HIPAA: 6 years for audit logs. SOX: 7 years for financial system logs. GDPR: no specific minimum but logs containing PII must respect data subject rights. FFIEC: 5-7 years for banking audit trails. Best Practices: Define retention by log type (security vs. operational vs. debug). Implement immutable logging (WORM storage) for forensic integrity. Automate lifecycle policies for cost optimization. Ensure chain-of-custody for incident evidence.
Log Sources & Architecture
Common Log Sources: Firewalls (Palo Alto, Fortinet), endpoints (EDR/AV logs), Active Directory and LDAP, DNS/DHCP, web proxies, VPN gateways, cloud audit trails (AWS CloudTrail, Azure Activity Log, GCP Audit Logs), email gateways, databases, and application logs. Log Formats: Syslog (RFC 5424), Windows Event Log (EVTX), CEF (Common Event Format — ArcSight), LEEF (Log Event Extended Format — QRadar), JSON/structured logs, and flow data (NetFlow/sFlow/IPFIX). Architecture Patterns: Agent-based collection (Splunk UF, Elastic Agent, Sentinel AMA), agentless (syslog forwarding, API polling), and hybrid approaches. Multi-tier architecture with collection → aggregation → processing → storage → presentation layers.
SIEM Platforms
Splunk Enterprise / Cloud: Industry leader — SPL query language, 1,500+ apps on Splunkbase, real-time dashboards, and SOAR integration (Splunk SOAR/Phantom). Strengths: powerful search, massive scalability. Microsoft Sentinel: Cloud-native SIEM built on Azure — KQL query language, native M365/Azure integration, built-in UEBA, automated playbooks via Logic Apps. Cost model: pay-per-GB ingestion. IBM QRadar: Enterprise SIEM with Ariel Query Language (AQL), built-in flow analysis, offense management, and strong compliance reporting. Elastic SIEM (ELK Stack): Open-source foundation — Elasticsearch, Logstash, Kibana with Elastic Security detection rules. Great for custom deployments and cost-sensitive environments. Google Chronicle (SecOps): Google-scale infrastructure with 12-month default retention, YARA-L detection language, petabyte search in seconds. LogRhythm: Self-hosted SIEM with embedded SOAR, case management, and built-in network/endpoint monitoring.
SOC KPIs & Metrics
Detection Metrics: MTTD (Mean Time to Detect) — target: <1 hour for mature SOCs. False Positive Rate — target: <30%. True Positive Rate and alert-to-incident ratio. Detection coverage mapped to MITRE ATT&CK techniques. Response Metrics: MTTA (Mean Time to Acknowledge) — target: <15 minutes. MTTC (Mean Time to Contain) — target: <4 hours. MTTR (Mean Time to Remediate) — target: 24-72 hours by severity. Operational Metrics: Alert volume (daily/weekly trends), escalation rate, SLA compliance (% of incidents handled within SLA), analyst utilization, mean dwell time, and number of proactive hunts conducted. Reporting Cadence: Weekly — alert trends, open incidents, SLA adherence. Monthly — MTTx trends, MITRE coverage gaps, tool ROI, executive risk posture.
UEBA (User & Entity Behavior Analytics)
How It Works: UEBA establishes behavioral baselines for users and entities (hosts, applications, network segments) using machine learning. It detects anomalies that traditional rules miss — subtle deviations indicating insider threats, compromised accounts, or advanced persistent threats. Detection Scenarios: Unusual login times or locations, abnormal data access patterns, first-time access to sensitive resources, privilege escalation patterns, peer group deviation (user behaving differently from role peers), and data hoarding before resignation. Integration: Most enterprise SIEMs include UEBA modules — Sentinel UEBA, Splunk UBA, QRadar UBA, Exabeam, Securonix. Feeds into SOC alert pipeline with risk scores that prioritize analyst investigation.
SIEM Architecture
End-to-End SIEM Pipeline
From log source ingestion through parsing, enrichment, and correlation to actionable alerts, dashboards, and compliance reports
SIEM Platform Comparison
| Platform | Deployment | Query Language | Strengths | Best For |
|---|---|---|---|---|
| Splunk | On-prem / Cloud | SPL | Powerful search, 1500+ apps, massive scalability | Large enterprises, advanced analytics |
| Microsoft Sentinel | Cloud-native (Azure) | KQL | Native M365/Azure integration, built-in UEBA, pay-per-GB | Microsoft-centric environments |
| IBM QRadar | On-prem / SaaS | AQL | Flow analysis, offense management, strong compliance | Regulated industries, network-heavy |
| Elastic SIEM | Self-hosted / Cloud | EQL / Lucene | Open-source core, flexible, detection-as-code | Custom deployments, cost-sensitive |
| Google Chronicle | Cloud-native (GCP) | YARA-L | Google-scale infra, 12-month retention, petabyte search | High-volume ingestion, SecOps |
| LogRhythm | On-prem / Cloud | Proprietary | Embedded SOAR, case management, NDR | Mid-size enterprises, all-in-one |
Interview Preparation
What is XDR and how does it differ from SIEM? When would you deploy one versus the other?
XDR (Extended Detection and Response) and SIEM serve different but complementary purposes: SIEM is a log aggregation and analysis platform — it collects logs from any source (firewalls, endpoints, cloud, applications), normalizes them, and runs correlation rules to generate alerts. SIEMs like Splunk or Sentinel are source-agnostic and require analysts to write detection rules. XDR is a detection and response platform that natively integrates telemetry from its own security stack (endpoint + network + email + cloud + identity). Unlike SIEM, XDR automatically correlates weak signals across layers into unified incidents — for example, linking a suspicious email attachment → endpoint process execution → lateral movement → data exfiltration into one attack chain without manual rule creation. WHEN TO USE SIEM: Compliance requirements for log retention (PCI-DSS Req 10, HIPAA). Custom data sources not covered by XDR. Advanced hunting with ad-hoc queries. Centralized security data lake strategy. WHEN TO USE XDR: Smaller SOC team needing out-of-the-box detection. Focus on detection and response speed over log management. Desire to reduce tool sprawl and consolidate vendors. TOGETHER: Many mature organizations run both — SIEM for compliance, log management, and custom detection; XDR for high-fidelity detection, automated response, and unified investigation across the kill chain. Key vendors: Palo Alto Cortex XDR, Microsoft Defender XDR, CrowdStrike Falcon XDR, SentinelOne Singularity.
How does a SIEM work? Walk me through the log lifecycle.
A SIEM operates through a multi-stage pipeline:
1LOG COLLECTION — Agents (Splunk Universal Forwarder, Elastic Agent, Sentinel AMA) or agentless methods (syslog forwarding, API polling) collect logs from endpoints, network devices, cloud services, and applications.
2TRANSPORT — Logs are forwarded to the SIEM via TCP/UDP syslog, HTTP/S, or message queues (Kafka). Log pipeline tools like Cribl or Logstash can pre-filter and route data.
3PARSING & NORMALIZATION — Raw logs are parsed into structured fields and mapped to a common schema (Elastic Common Schema, OCSF). This enables cross-source correlation.
4ENRICHMENT — Fields are enriched with context: GeoIP lookup on source IPs, threat intelligence IOC matching, asset inventory data (is this a production server?).
5INDEXING & STORAGE — Parsed events are indexed for fast search and stored in hot/warm/cold tiers based on retention policies.
6CORRELATION & DETECTION — Rules engine evaluates events against detection logic: single-event rules, multi-event correlation, threshold alerts, and UEBA behavioral baselines. Generated alerts are prioritized by severity.
7ALERTING & RESPONSE — High-fidelity alerts are sent to SOC analysts or trigger automated SOAR playbooks for containment.
How do you reduce false positives in a SIEM?
False positive reduction requires a multi-layered approach:
1TUNE DETECTION RULES — Add exclusions for known-good activity (e.g., whitelist vulnerability scanner IPs, service accounts with expected behavior). Use threshold tuning — if a rule triggers on 3 failed logins, evaluate if 5 or 10 is more appropriate for the environment.
2CONTEXTUAL ENRICHMENT — Enrich alerts with asset criticality (is this a developer workstation vs. a domain controller?), user context (admin vs. standard user), and threat intel confidence scores.
3CORRELATION — Replace single-event rules with multi-stage correlation. Instead of alerting on any failed login, correlate failed logins followed by successful login from same source within a time window.
4BASELINE BEHAVIOR — Implement UEBA to establish normal patterns. Alert on deviations rather than static thresholds.
5ALERT SCORING — Assign risk scores that factor in asset value, user privilege level, threat intel match, and behavioral deviation. Only escalate alerts above a threshold score.
6FEEDBACK LOOP — Track analyst dispositions (true positive, false positive, benign true positive). Use this data to refine rules weekly. Target false positive rate below 30%.
What SOC KPIs would you track and present to leadership?
I organize SOC KPIs into three categories for leadership reporting:
1DETECTION EFFECTIVENESS — MTTD (Mean Time to Detect): average time from threat entry to detection, target under 1 hour for critical threats. Detection coverage: percentage of MITRE ATT&CK techniques covered by at least one detection rule. False positive rate: percentage of alerts that are false positives — target under 30%.
2RESPONSE EFFICIENCY — MTTA (Mean Time to Acknowledge): time from alert to analyst triage, target under 15 minutes. MTTC (Mean Time to Contain): time to isolate the threat, target under 4 hours for high severity. MTTR (Mean Time to Remediate): full resolution time, target 24-72 hours by severity. SLA compliance rate: percentage of incidents handled within defined SLAs.
3OPERATIONAL HEALTH — Alert volume trends (rising can indicate misconfiguration or emerging threats), escalation rate (Tier 1→2→3), analyst utilization, mean dwell time, and number of proactive threat hunts. For executive dashboards, I present monthly MTTx trends with sparklines, risk posture heatmaps, comparison against industry benchmarks, and ROI metrics showing how tool investments reduced response times.
Compare Splunk and Microsoft Sentinel. When would you choose one over the other?
The choice between Splunk and Sentinel depends on the environment and priorities: SPLUNK ADVANTAGES — Mature ecosystem with 1,500+ Splunkbase apps, unmatched search flexibility with SPL, supports any data source and deployment model, and strong in hybrid/multi-cloud environments. Better for organizations that need maximum customization and have dedicated Splunk admins. SENTINEL ADVANTAGES — Cloud-native on Azure with zero infrastructure management, native integration with M365, Azure AD, and Defender suite, built-in UEBA and entity pages, cost-effective pay-per-GB pricing model (with free ingestion for many Microsoft sources), and Logic Apps for SOAR playbooks. Better for Microsoft-centric organizations. WHEN TO CHOOSE SPLUNK — Multi-cloud or cloud-agnostic requirement, heavy on-prem infrastructure, need for advanced SPL analytics, existing Splunk investment and expertise, or very high customization needs. WHEN TO CHOOSE SENTINEL — Primarily Microsoft/Azure environment, want to minimize infrastructure overhead, budget-conscious (free M365/Azure log ingestion), small-to-mid security team that benefits from built-in content, or rapid deployment timeline. Many mature SOCs run both — Sentinel for M365/Azure-native detection and Splunk for everything else.
Framework Mapping
| Framework | Relevant Controls |
|---|---|
| NIST | SP 800-92 (Guide to Computer Security Log Management), CSF DE.CM (Continuous Monitoring), CSF DE.AE (Anomaly & Event Detection), SP 800-53 AU (Audit & Accountability) |
| MITRE | Data Sources: process, network, file, authentication logs. D3FEND: Network Traffic Analysis, Log Analysis, User Behavior Analysis |
| ISO | ISO 27001 A.12.4 (Logging & Monitoring), A.16.1 (Incident Management), A.12.4.1 (Event Logging), A.12.4.3 (Admin & Operator Logs) |
| PCI-DSS | Req 10 (Log & Monitor All Access), Req 10.7 (Retain audit trail 1 year, 3 months immediately available) |
