PROJECT 003 · CLOUD SECURITY

> cat cloudsec-pipeline.md

Building a Hybrid Cloud Security Operations Center.

Extending an existing on-premises Wazuh SIEM into AWS using native cloud services, centralized network telemetry, and behavioral analytics — so one SOC covers both environments instead of two that don't talk to each other.

mvp — live telemetry hybrid soc, one siem mitre att&ck aligned no per-gb siem licensing

View Repository Back to Portfolio

cat why-build-this.md

Why Build This?

Most organizations already run a SOC and a SIEM. The problem isn't a lack of security tooling — it's that cloud adoption outpaces it. Workloads move into AWS generating real network telemetry, while the SOC keeps watching on-prem traffic through the same tools it always has. That split is a visibility gap, and it's the gap this project closes.

Commercial cloud SIEMs — Microsoft Sentinel, Splunk Enterprise Security, IBM QRadar, Cortex XSIAM — are genuinely excellent at solving this. They're also expensive: licensing, per-GB ingestion, and storage costs that are hard to justify for a lab, a small team, or anyone still learning the space.

The engineering goal here was never to replace an enterprise SIEM. It was to explore how far AWS-native services can extend an existing SOC before that spend becomes necessary — and how much of the noise-reduction work can happen before anything reaches the SIEM at all.

Cloud Visibility

AWS workloads generate valuable network telemetry that traditionally stays isolated from on-prem security monitoring.

Cost

Enterprise cloud SIEM platforms bring licensing, ingestion, and storage costs that are hard to justify for a personal lab.

Signal vs. Noise

Forwarding every raw VPC Flow Log into a SIEM inflates storage and makes analyst investigations harder, not easier.

cat design-decisions.md

Architecture Story

Four decisions shaped this pipeline before a single log ever moved — the diagram below is what they add up to.

01VPC Flow Logs, not host agents

02CloudWatch as the managed transport layer

03An EC2 layer to normalize events

04Forward enriched events, never raw logs

path — vpc to soc dashboardclick to expand

6 hops · flow logs → behavioral events · on-prem soc dashboard

show ip flow-logs

Cloud Networking Focus

Every ENI in the VPC — on an EC2 instance, an ALB, a NAT Gateway — emits its own VPC Flow Log records. That's the mechanism: no agent to install, no host to touch. The network fabric itself is the sensor.

That matters because network telemetry can surface activity before an endpoint log ever would. A port scan against a host with minimal logging might never show up locally — but it's unmissable at the network layer, because the connections themselves are the evidence.

There's a real difference between a raw flow record and behavioral intelligence. A flow record is just src_ip, dst_ip, dst_port, action — a fact about one connection. Fourteen of those facts from the same source in sixty seconds is a port scan. The record doesn't know that; the processing layer does.

whatThe /aws/vpc/flowlogs CloudWatch log group, with one log stream per ENI.

whyThis is the backbone — a managed, durable transport layer that exists between "AWS generates telemetry" and "something processes it," with nothing to run or patch.

stageArchitecture · CloudWatch transport layer

whatRaw, unprocessed flow records — data.src_ip, data.dst_port, data.action — landing in Wazuh's archive index.

whyThis is what "before behavioral intelligence" actually looks like: 28,000+ individual facts with no story attached yet. Necessary raw material, not something an analyst should be reading directly.

stageNetworking · raw telemetry, pre-processing

show inventory

Technologies Used

AWS NetworkingAmazon VPCVPC Flow Logs CloudWatch LogsAmazon EC2AWS IAM Hybrid CloudWazuhOpenSearch SIEM IntegrationDetection EngineeringCloud Security

show investigation timeline

Follow an Attack

A real captured sequence, walked stage by stage — from first recon touch to a signal worth an analyst's attention.

Internet Attacker

An external host begins probing the environment. Nothing has fired yet — this is just traffic arriving at a public IP.

recon begins

Port Scan Detected

The same source IP touches an unusual spread of destination ports in a short window. The pipeline flags it as reconnaissance before anything downstream is touched.

rule: potential port scannetwork layer

Brute Force Against MySQL

The same source IP (119.94.164.8) shifts from scanning to repeated connection attempts against the database host on port 3306 — recon giving way to an actual attempt.

rule: potential brute forcedst_port: 3306

MariaDB Failed Login

On the database host itself, MariaDB's own audit log confirms it — repeated Access denied for user 'root'@'119.94.164.8' entries, the same source IP the network layer already flagged.

rule: mariadb failed loginhost layer

MariaDB Successful Login

Interspersed with the failures, successful logins start appearing from the same source — the same view, the same host, a very different outcome.

rule: mariadb successful loginroot account

Signal Worth Escalating

This pipeline also watches session and privilege-escalation activity on its own collector infrastructure — PAM login sessions and a sudo to root, captured the same way. Shown here as what that class of signal looks like when it fires, not a confirmed link to the database event above; that's exactly the kind of connection the correlation work below is for.

rule: pam / sudo to rootprivilege escalation signal

show correlation-engine

Cross-Source Correlation

The platform correlates three sources instead of trusting any one of them in isolation.

VPC Flow Logs

Ground truth for network activity — every connection attempt, accepted or rejected, regardless of what happens at the application layer.

Behavioral Detections

Pattern recognition over those flow logs — a port scan, a brute force shape — the recon and attempt stages of an investigation.

MariaDB Audit Logs

The application's own record of what actually happened — authentication success or failure, independent of what the network saw.

Individually, a brute-force alert is a maybe. A failed login is routine. Correlated — same source IP, same destination, adjacent timestamps, moving from network-layer recon into application-layer authentication — an analyst can tell whether reconnaissance actually progressed into a successful login, instead of treating three separate low-confidence signals as three separate non-events.

whatThe expanded document behind one MariaDB failed-login alert — rule level 10, fired 27 times, decoded automatically by Wazuh's built-in MariaDB decoder.

whyThis is the kind of structured detail correlation runs on — not just "a login failed," but a rule ID, a severity level, and a repeat count that can be matched against the network-layer alert for the same source and window.

stageCorrelation · application-layer evidence

show detections

What Gets Detected

Instead of forwarding every raw network record into Wazuh, the processing layer normalizes AWS telemetry and emits structured behavioral events — built so new detections can be added later without redesigning the pipeline.

example structured event · syntheticmedium

{
  "event_type": "port_scan_detected",
  "source_ip": "203.0.113.42",
  "destination_ports_touched": 14,
  "window_seconds": 60,
  "mitre_technique": "T1595 - Active Scanning",
  "severity": "medium"
}

illustrative event shape — not a captured production alert

show engineering-decisions

Engineering Decisions

Every choice here traded something for something — worth being honest about both sides.

Why VPC Flow Logs?

Network-layer visibility with zero agents to install, update, or lose track of across instances.

trade-offNo process-level context — you see the connection, not what made it.

Why CloudWatch?

Managed, durable transport with nothing to provision or keep patched between generation and processing.

trade-offCost scales with volume, so raw logs can't just sit there indefinitely.

Why EC2, not Lambda?

Port-scan and brute-force detection both need state — tracking unique ports or attempts per source over a rolling window. That's awkward across short-lived, stateless invocations.

trade-offAn always-on instance to patch and secure, instead of Lambda's zero-idle-cost model.

Why Wazuh?

Already the on-prem SOC's SIEM, with a mature decoder and rule ecosystem that already understands MariaDB, PAM, and sudo logs out of the box — no second platform to learn.

trade-offTies the design to Wazuh's rule/decoder model rather than a cloud-native alternative.

Why Preprocess Before Ingestion?

Cuts storage and alert fatigue — the SOC sees "port scan," not fourteen raw connection records to piece together by hand.

trade-offA bug in detection logic happens before Wazuh ever sees the event. Early builds ran with temporary synthetic test events specifically to validate that logic before trusting it against real traffic.

cat business-value.md

Business Value

From an engineering manager's read, not a feature list.

Centralized AWS Visibility

One SOC sees both environments instead of running cloud security blind between two disconnected tools.

Reduced Alert Fatigue

Analysts see behavioral events, not thousands of raw ACCEPT/REJECT flow records to sift through by hand.

Hybrid SOC Architecture

Extends what already exists instead of standing up parallel tooling that has to be maintained twice.

Scalable Telemetry Pipeline

New detections plug into the same processing layer without a pipeline redesign each time.

Lower Operational Cost

No second SIEM license, no per-GB cloud ingestion fees for a lab-scale or small-team environment.

Reusable Detection Framework

Today it's port scans and brute force — the same shape extends to CloudTrail, GuardDuty, and beyond.

show soc-dashboard

Operational Security Operations Center

The platform is no longer just an architecture demonstration—it is an operational hybrid SOC ingesting AWS telemetry, running custom behavioral detections, and presenting correlated security events inside Wazuh.

Threat Hunting Dashboard

Provides analysts with a single pane of glass for cloud telemetry, authentication events, behavioral detections, and rule trends.

Highlights~7,000 collected events, behavioral detections, authentication statistics and rule distribution.

Behavioral Detection Engine

Instead of forwarding raw flow logs, the EC2 processing engine emits structured security events with confidence scores, detector names and attack metadata.

DetectorBruteForceDetector

Confidence100

BehaviorSSH brute force

Vulnerability Management

Wazuh continuously evaluates managed hosts for CVEs and vulnerable packages, extending the platform beyond network telemetry.

Managed Infrastructure

The SOC monitors Windows endpoints, Windows Server, public cloud workloads and the AWS Log Collector from one centralized manager.

show maturity-model

Enterprise Roadmap

Framed as maturity phases rather than a flat feature backlog — each phase builds on the last.

PHASE 0

MVP current

Production telemetry ingestion, modular behavioral detection, live Wazuh integration, MITRE-ready alerting, OpenSearch dashboards.

PHASE 1

CloudTrail / GuardDuty / Security Hub next

Broaden beyond network flow into API-level activity and AWS's own managed detections.

PHASE 2

Threat Intelligence & GeoIP planned

Enrich source IPs with reputation and geolocation context before they reach an analyst.

PHASE 3

AWS Organizations & Central Logging Account planned

Consolidate telemetry from multiple AWS accounts into one dedicated logging account.

PHASE 4

Terraform & CI/CD planned

Move from manually configured infrastructure to versioned, repeatable deployments.

PHASE 5

Multi-Account AWS Environment planned

Extend the same pattern across a full multi-account landing zone.

show notes

Lessons Learned

#Designing a hybrid telemetry pipeline end to end

#Reducing signal-to-noise before SIEM ingestion

#Structuring network events for behavioral analysis

#Extending on-prem SOC tooling into the cloud

#Correlating network detections with auth logs

#Counting connection attempts, not just log records

#Least-privilege IAM for telemetry pipelines

#Designing for modular growth, not a rebuild