Audit logging

Every mutation in nanosync emits an audit.Event carrying the actor, action, resource reference, decision, outcome, and request context. Events flow through an asynchronous bus to one or more sinks.

The bus is non-blocking: a slow sink can never backpressure the data path. Overflow drops the oldest queued event, increments a Prometheus counter, and emits a rate-limited warning log. Durability is a sink-level concern (the file sink fsyncs after every batch).

Architecture

   handler (Create/Update/Delete/Start/Stop)
        │
        ▼
   audit.Recorder.Record(event)        ← non-blocking
        │
        ▼
   bounded channel (queue_size=8192)
        │
        ▼
   dispatcher goroutine                ← micro-batches (≤100 events or ≤200ms)
        │
        ├──▶ file sink   (NDJSON segments, rotating)
        ├──▶ otlp sink   (OTel log SDK → SIEM)
        └──▶ objstore    (v1.1; S3 / GCS)

Each sink runs in its own goroutine via errgroup. One sink failing or slowing down does not affect the others.

Default behaviour

Out of the box, nanosync writes audit events to a local NDJSON file under <data_dir>/audit/:

tail -f /var/lib/nanosync/audit/audit-2026-06-21T15-30-04.ndjson

{"id":"…","timestamp":"2026-06-21T15:30:42.123Z","actor":{"type":"user","id":"…","email":"alice@acme.com"},"action":"pipeline.create","resource":{"type":"pipeline","id":"checkout","scope":{"type":"project","id":"…"}},"decision":"allow","outcome":"success","request_id":"…"}
{"id":"…","timestamp":"2026-06-21T15:31:05.001Z","actor":{"type":"user","id":"…","email":"bob@acme.com"},"action":"pipeline.delete","resource":{"type":"pipeline","id":"orders","scope":{"type":"project","id":"…"}},"decision":"deny","outcome":"failure","error":"rbac: forbidden","request_id":"…"}

Segments rotate at 64 MiB (configurable). Permissions are 0600 on files and 0700 on the parent directory.

Configuration

The full sink config lives under audit: in config.yaml:

audit:
  queue_size:        8192      # bounded channel capacity
  batch_max_events:  100       # flush when this many queued
  batch_max_latency: 200ms     # or when this much wall time has passed
  sinks:
    - type: file
      dir: /var/lib/nanosync/audit
      max_segment_bytes: 67108864    # 64 MiB; rotate to a new file at this size

    - type: otlp
      endpoint: https://otlp.your-siem.example.com:4318
      headers:
        DD-API-KEY: ${env:DD_API_KEY}
        Authorization: ${env:SIEM_BEARER}

    # Object storage (S3/GCS) lands in v1.1
    # - type: objstore
    #   bucket: my-audit-archive
    #   prefix: nanosync/

If sinks: is omitted entirely, a single file sink under <data_dir>/audit/ is registered automatically.

SIEM integrations

Ship via the OTel HTTP exporter. Splunk Enterprise 9+ accepts OTel logs natively via the Splunk OTel Collector. Point nanosync at the collector:

audit:
  sinks:
    - type: otlp
      endpoint: https://splunk-otel-collector.internal:4318
      headers:
        X-Splunk-Token: ${env:SPLUNK_HEC_TOKEN}

Each event arrives as a single log record with:

severity = INFO (outcome=success), WARN (decision=deny), or ERROR (outcome=failure)
body = the full JSON event
attributes = actor.id, action, resource.type, decision, outcome, request.id (indexed by Splunk for fast pivots)

Datadog accepts OTLP logs at https://http-intake.logs.datadoghq.com/api/v2/logs (region-specific subdomain). Use the DD-API-KEY header:

audit:
  sinks:
    - type: otlp
      endpoint: https://http-intake.logs.datadoghq.com/api/v2/logs
      headers:
        DD-API-KEY: ${env:DD_API_KEY}
        DD-EVP-ORIGIN: nanosync
        Content-Type: application/json

In Datadog, filter by service:nanosync source:nanosync to isolate audit events. The actor.id and resource.type attributes are first-class searchable fields.

Any OTLP-compatible backend works. For Honeycomb:

audit:
  sinks:
    - type: otlp
      endpoint: https://api.honeycomb.io
      headers:
        x-honeycomb-team: ${env:HONEYCOMB_API_KEY}
        x-honeycomb-dataset: nanosync-audit

Run both — local NDJSON for forensic backup, OTLP for live SIEM:

audit:
  sinks:
    - type: file
      dir: /var/lib/nanosync/audit
    - type: otlp
      endpoint: https://otlp.your-siem.example.com:4318
      headers:
        Authorization: Bearer ${env:SIEM_TOKEN}

If the OTLP backend is unreachable, the file sink keeps catching events with zero data loss.

Event schema

Every event is a JSON object with this shape:

type AuditEvent = {
  id:         string;                          // UUID, generated at Record-time
  timestamp:  string;                          // RFC3339Nano, server clock
  request_id: string;                          // correlates with HTTP X-Request-ID
  actor: {
    type:  'user' | 'service_account' | 'token' | 'system';
    id:    string;                             // user_id or token_id
    email: string;                             // empty for token / system
  };
  action:   string;                            // dotted, e.g. 'pipeline.create'
  resource: {
    type:  string;                             // 'pipeline' | 'connection' | …
    id:    string;
    scope: { type: 'org' | 'workspace' | 'project'; id: string };
  };
  decision:    'allow' | 'deny' | 'n/a';       // RBAC decision (n/a for non-gated events)
  outcome:     'success' | 'failure';
  error:       string;                         // populated when outcome=failure
  before:      object | null;                  // pre-mutation snapshot (writes only)
  after:       object | null;                  // post-mutation snapshot
  attributes:  Record<string,string>;          // free-form labels
  remote_addr: string;
};

Tuning knobs

Field	Default	When to change
`queue_size`	8192	Raise for very bursty workloads where drops would be tolerable but visible (each drop increments `nanosync_audit_dropped_total`).
`batch_max_events`	100	Raise to reduce sink call overhead on high-throughput deployments; lower for tighter SIEM latency.
`batch_max_latency`	200ms	Lower for tighter latency (more sink calls); raise for less network chatter.
`max_segment_bytes` (file)	64 MiB	Lower for easier log shipping; raise to reduce inode churn.

Backpressure & metrics

Audit MUST never block the data path. The bus enforces this:

Channel full → drop the oldest event, increment nanosync_audit_dropped_total{sink="_bus_"}, emit a slog.Warn rate-limited to once per 30s.
Sink write failure → log + nanosync_audit_dropped_total{sink="<sink-name>"} — the other sinks still get the event.
Close(ctx) drains the channel within the supplied deadline.

Prometheus metrics:

Metric	Type	Purpose
`nanosync_audit_events_total{sink,outcome}`	Counter	Per-sink throughput
`nanosync_audit_dropped_total{sink}`	Counter	Drops per sink (`"_bus_"` = queue overflow)
`nanosync_audit_queue_depth`	Gauge	Current channel depth — alert if growing
`nanosync_audit_sink_latency_seconds{sink}`	Histogram	Per-sink write latency

Recommended Prometheus alert:

- alert: NanosyncAuditDrops
  expr:  rate(nanosync_audit_dropped_total[5m]) > 0
  for:   5m
  annotations:
    summary: nanosync audit bus dropped events on sink {{ $labels.sink }}
    runbook: https://nanosync.example.com/docs/security/audit#backpressure--metrics

Verifying events flow

# Live tail
tail -f /var/lib/nanosync/audit/audit-*.ndjson

# Confirm Prometheus counter increments after a mutation
curl -fsSL http://localhost:9090/metrics | grep nanosync_audit_events_total

# Smoke test: create + delete a pipeline; expect 2 events
ns pipeline create -f /tmp/test.yaml
ns pipeline delete test
grep test /var/lib/nanosync/audit/audit-*.ndjson | tail -2

What’s next

RBAC roles & bindings — audit:read is the permission that lets a user see audit events in the UI.
SSO setup — every login becomes an auth.login audit event with the right actor.