The Cloud SIEM Egress Trap
As organizations transition to autonomous security operations, the volume of security telemetry has exploded by 10x. Many teams attempt to build an aws security data lake alternative by dumping logs into S3 and using AWS Lambda or ECS Fargate to run AI threat-hunting agents. This is a fatal FinOps mistake.
Cloud providers penalize you for querying your own data. Every time an agent routes logs out of a private subnet to an external API (like Claude or OpenAI), you are hit with a $0.045/GB NAT Gateway processing fee. Furthermore, AWS Lambda imposes a strict 15-minute hard timeout—killing deep, historical threat hunts mid-execution.
To truly scale an AI SOC, you must deploy open source threat hunting AI locally. In this technical tutorial, we will build a Dual-Output Security Data Lake and deploy a local LLM agent (Ollama) on iRexta Bare Metal—bypassing cloud API costs, hypervisor latency, and egress taxes entirely.
Step 1: Architecting the Dual-Output Data Lake
The most common mistake engineers make when building a data lake is sending 100% of their raw logs directly into a hot indexing engine like OpenSearch. This instantly bloats provisioned IOPS and storage costs.
Here is the production-ready fluent-bit.conf configuration to execute this split routing natively. Notice we are using the modern opensearch plugin instead of the legacy es plugin, and explicitly parsing JSON telemetry:
[SERVICE] Flush 1 Daemon Off Log_Level info Parsers_File parsers.conf
[INPUT] # SRE FIX: AI Telemetry requires JSON parsing, not raw Syslog text Name tail Path /var/log/ai_telemetry/*.json Parser json Tag ai_security.logs
[FILTER] # SRE Best Practice: Drop useless debug noise BEFORE it hits the network Name grep Match * Exclude level debug
# ----------------------------------------------------
# SRE FIX: Dual Output Data Lake Architecture
# ----------------------------------------------------
[OUTPUT] # Output 1: Send ALL logs to MinIO (Cold Lake) for cheap, immutable retention Name s3 Match * Bucket threat-telemetry-archive Endpoint http://minio.irexta.internal:9000 Store_Dir /tmp/fluent-bit/s3 # Note: Requires iRexta Bare Metal unmetered private networking
[OUTPUT] # Output 2: Send ONLY critical events to OpenSearch (Hot Index) # SECURITY FIX: Use 'opensearch' plugin, not the legacy 'es' plugin Name opensearch Match ai_security.logs Host opensearch.irexta.internal Port 9200 Index hot-threat-telemetry Type _docStep 2: Enrich First, Think Second
Before we write the AI Agent, we must establish a core rule of AI operations: Do not let the LLM do the heavy lifting of raw data retrieval.
Step 3: Deploying the Local AI SOC Agent
To build AI SOC agent infrastructure securely, we will deploy Ollama on our Bare Metal server. This allows us to run the massive Llama 3 model entirely offline, meaning our sensitive production logs never leave our physical hardware.
Below is the advanced Python Agent utilizing the Self-Critique Loop. This forces the model to generate multiple hypotheses and double-check its own logic before classifying an alert:
import ollama
import json
import sys
def analyze_security_event(structured_context_json): """ Executes a Local LLM Triage utilizing Hypothesis-Driven Investigation and a mandatory Self-Critique loop. """ # The SRE Prompt Architecture system_prompt = """ You are an autonomous Tier-2 Security Analyst Agent. Phase 1 (Hypotheses): Generate 2 plausible hypotheses for the provided event: 1. A benign administrative explanation. 2. A malicious attack vector explanation. Phase 2 (Evaluation): Evaluate the provided structured JSON context against both hypotheses. Phase 3 (Self-Critique): CRITICAL - You must critique your own initial conclusion. Ask yourself: "What evidence might I have missed? Is my confidence justified?" Phase 4 (Output): Respond strictly in valid JSON format: { "classification": "BENIGN | SUSPICIOUS | MALICIOUS", "confidence_pct": 85, "reasoning": "Short explanation", "next_questions": ["What should the human analyst check next?"] } """ print("[*] Dispatching enriched context to Local Llama 3 for autonomous triage...") try: response = ollama.chat( model='llama3', messages=[ {'role': 'system', 'content': system_prompt}, {'role': 'user', 'content': f"Enriched Event Context:\n{structured_context_json}"} ] ) # Output the parsed AI Triage Report print(response['message']['content']) except Exception as e: print(f"[FATAL] Local inference engine failed: {e}") sys.exit(1)
# Example Execution
if __name__ == "__main__": # In production, this JSON is generated deterministically from OpenSearch/MinIO mock_context = json.dumps({ "event_type": "Multiple failed SSH logins followed by successful key-based login", "source_ip": "203.0.113.42", "target_user": "root", "historical_pattern": "IP has never accessed this subnet before." }) analyze_security_event(mock_context)Step 4: Security Advisory & Approval Gates
When you deploy a local LLM for security logs, you introduce new attack vectors to your own SOC. As an SRE, you must implement the following non-negotiable safeguards.
The SRE Solution: Why AI SOC Agents Demand Bare Metal
We have engineered a robust, privacy-first AI Threat Hunter. The final architectural decision dictates its financial viability: where do we run it?
Running continuous Log Ingestion, OpenSearch Indexing, and Heavy LLM Inference on AWS EC2 or Fargate is a massive financial leak. The IOPS required to write terabytes of logs to AWS EBS volumes triggers exorbitant Provisioned IOPS (io2) fees. When your agent queries cross-AZ or hits external threat-intel APIs via NAT, the Egress taxes multiply your monthly bill.
To scale your Security Data Lake profitably, Elite SOC teams deploy on iRexta Dedicated Bare Metal Servers. By utilizing direct-attached Enterprise PCIe NVMe drives, your OpenSearch nodes process millions of write IOPS at zero extra cost. By running local models (like Llama 3) directly on our dedicated hardware, you eliminate the 15-minute Lambda timeouts, achieve 0% Cloud Egress fees, and maintain absolute data sovereignty over your critical security telemetry.