Pruning false positives when hunting for lateral movement inside Kubernetes pods.

When an attacker gets a foothold in a cluster, the next move is usually to hop around pods, namespaces, or even nodes. Catching that hop is a staple of any Kubernetes security program, but the noise can drown you out. A single benign kubectl exec or a normal service‑mesh sidecar can set off the same alerts that a malicious container‑to‑container exfiltration would trigger. Below is a practical, repeatable walk‑through to cut the noise while still keeping the detection surface wide enough to catch the bad guys.

1. Define the Threat Model

Before you start hunting, nail down what “lateral movement” actually means for your stack.

Context	Typical Indicator	Example Attack
Pod‑to‑Pod	Traffic between pods that normally don’t talk	A compromised web‑app pod reaches a database pod
Namespace‑to‑Namespace	Cross‑namespace API calls or `kubectl` usage	A malicious pod in `dev` calls the API to list resources in `prod`
Node‑to‑Node	SSH or kubelet API access from a pod	A container uses `ssh` to a node to copy binaries

Writing those scenarios down helps you decide which telemetry to collect and what a “normal” baseline looks like.

2. Instrumentation Strategy

2.1. Network Flow Collection

Cilium eBPF – grabs fine‑grained flow data per pod: source/destination IP, port, protocol. Export to Prometheus or Loki for correlation.
Istio Telemetry – if you already run Istio, the Envoy sidecar logs hold request metadata that can flag cross‑namespace calls.

2.2. Process & File System Monitoring

Falco – eBPF‑based syscall watcher. Configure rules to flag execve of binaries not in the pod image or open of files outside the pod’s mount namespace.
Sysdig – offers a richer query language for process trees. sysdig -c top is handy for spotting unexpected binaries.

2.3. Kubernetes API Auditing

Enable Audit Logging with a policy that records GET, LIST, CREATE, DELETE verbs on critical resources. Store logs in an immutable backend (S3 with versioning, for example).

3. Baseline Creation

A noisy environment usually means you’ve never built a baseline. Do it in two steps:

3.1. Normal Traffic Profile

Run cilium monitor for 48 h during normal ops.
Aggregate flows by pod pair and protocol.
Export to CSV and calculate mean & standard deviation for bytes per second.

cilium monitor --output json | jq -r '. | [.timestamp, .source.pod.name, .destination.pod.name, .bytes] | @tsv' > flows.tsv

Then a quick Python helper:

import pandas as pd
df = pd.read_csv('flows.tsv', sep='\t', names=['ts','src','dst','bytes'])
thresholds = df.groupby(['src','dst'])['bytes'].agg(['mean','std'])
thresholds.to_csv('thresholds.csv')

3.2. Process Baseline

Run Falco for a week, capture all execve events, and filter by image name and command line. Store the list of allowed binaries per pod.

falco -o json | jq -r '. | [.pod_name, .proc_name, .proc_args] | @tsv' > allowed.tsv

4. Detection Rules

With baselines in place, craft rules that fire only when anomalies exceed thresholds.

4.1. Network Anomaly Rule (Cilium)

apiVersion: cilium.io/v1alpha1
kind: CiliumNetworkPolicy
metadata:
  name: lateral-movement-detection
spec:
  endpointSelector:
    matchLabels:
      app: any
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: any
    toPorts:
    - ports:
      - port: "3306"
        protocol: TCP
    rules:
      - alert: "Suspicious DB access from non‑DB pod"
        condition: "bytes > mean + 3*std"

The rule fires when a pod sends more than three standard deviations of traffic to a MySQL port – a pattern that most workloads never hit.

4.2. Process Anomaly Rule (Falco)

- rule: Unexpected Exec
  desc: Detect execution of binaries not in the allowed list
  condition: proc.name in (not allowed_binaries) and proc.namespace == "prod"
  output: "Unexpected exec of %proc.name in %pod.name"
  priority: WARNING

allowed_binaries is a dynamic list you keep updated from the baseline script.

4.3. API Call Rule (Audit Log)

A simple Splunk (or Loki) query that flags cross‑namespace GET calls:

index=apiaudit sourcetype=k8s_audit
| where verb="GET" and namespace!="prod"
| stats count by user, namespace, target_namespace
| where count > 10

5. Practical Hunting Workflow

Collect Data – run instrumentation for a short burst (15 min) during a scheduled deployment or a known benign activity.
Run Baseline Scripts – update thresholds and allowed binaries.
Apply Rules – feed the data into your SIEM or alerting system.
Investigate Alerts:
- Network alerts → drill into flow details in Cilium UI or cilium monitor.
- Process alerts → check the pod’s container logs and the Falco event payload.
- API alerts → cross‑reference the user’s IAM role and recent activity.
Validate – if an alert is a false positive, add the observed behavior to the baseline. If it’s a true positive, follow your incident response playbook.

6. Common False‑Positive Triggers & Mitigations

Trigger	Why It Happens	Mitigation
Sidecar Proxies	Envoy opens many connections to other pods	Exclude sidecar labels from baseline or add a whitelist rule
Helm Hooks	`helm upgrade` may exec binaries not in the image	Tag hook pods and add to allowed list
CI/CD Pipelines	`kubectl exec` used for debugging	Restrict `kubectl` usage to a dedicated namespace and monitor that namespace separately
Service Mesh Traffic	mTLS traffic appears as high‑volume flows	Use Istio’s `request_count` metric instead of raw bytes

7. Troubleshooting Common Issues

7.1. Too Many Alerts

Check Thresholds – lower the standard‑deviation multiplier if the baseline is too noisy.
Label Accuracy – ensure pod labels match the selectors in your policies.
Data Quality – verify that eBPF hooks aren’t dropping packets due to kernel limits.

7.2. Missing Alerts

Instrumentation Gaps – some containers run privileged or use --net=host. They bypass eBPF filters. Add cgroup‑based monitoring for those.
API Log Rotation – if audit logs rotate too quickly, you may miss short‑lived API calls. Increase retention or stream logs to a persistent store.

7.3. Performance Impact

eBPF Overhead – monitor CPU usage on nodes. If spikes occur, reduce the number of active probes or switch to a sampling mode.
SIEM Load – use incremental ingestion and keep rule complexity low. Offload heavy correlation to a dedicated analytics cluster.

8. Trade‑offs to Consider

Trade‑off	Benefit	Risk
Strict Baselines	Fewer false positives	Higher chance of missing novel attacks
Broad Baselines	Detects more anomalies	More noise, higher alert fatigue
Real‑time Alerts	Immediate response	Requires powerful infrastructure
Batch Analysis	Lower cost	Delayed detection

Balance depends on your organization’s tolerance for alert fatigue versus the criticality of the workloads.

9. Automation & Continuous Improvement

GitOps for Rules – store rule definitions in a Git repo, use CI to validate syntax, and deploy via ArgoCD.
Feedback Loop – after each incident, update baselines and rule thresholds. Automate this with a script that parses Falco logs and updates the allowed binaries list.
Metrics Dashboard – build a Grafana panel that shows alert counts, false‑positive rates, and baseline drift over time.

10. Real‑World Example

During a recent audit, a spike in traffic from a frontend pod to a db pod on port 3306 triggered the network anomaly rule. Investigation revealed the frontend pod had a sidecar that performed health checks to the database. By adding a label sidecar=true to the sidecar container and updating the Cilium policy to ignore traffic from that label, the alert rate dropped by 85 % without compromising detection of genuine lateral movement.

11. Staying Current

Cilium v1.13 introduced a new bytes metric that aggregates per‑flow, reducing noise. Upgrade if you’re on an older version.
Falco 1.7 added a proc.allow rule type that simplifies whitelist maintenance.
Kubernetes 1.30 now supports native audit policy templates that reduce configuration overhead.

Keep your tooling up to date and review the changelogs for any new features that can help prune false positives.