---
title: "Pipeline"
description: "What happens to an audit event from fire() until the destination acknowledges it."
---

> Documentation Index
> Fetch the complete documentation index at: https://logbun-docs.abshahin.workers.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipeline

```text
fire / fireAsync
  → redact, cap, UUIDv7, optional integrity seal
  → durable: ReliabilityAdapter.appendJournal
  journal append failure → writeDlq (single log)
  journal + DLQ failure → admission fails (fireAsync rejects; fire drops)
  → RAM tenant queue / short batching timer
  new tenant key beyond maxActiveTenants:
    durable persistent → writeDlq (no journal)
    volatile → drop
  queue already at cap after journal append → fair-share dump or
    single-log DLQ + journal ack
  → destination bulkInsert
  success → acknowledgeJournal
  failure → writeDlq → acknowledgeJournal
  destination + DLQ failure → leave journal unacknowledged
```

The destination adapter must be idempotent on `LogbunLog.id`. Recovery can re-deliver a batch after a crash between a successful destination write and its journal ack.

## Admission vs delivery

- **Admission** is “the library accepted the event.” In durable mode, `fireAsync` resolves after the journal (or DLQ fallback) commit.
- **Delivery** is “the destination `bulkInsert` succeeded.” That happens later, on a batching timer, `flush()`, or `runMaintenance()`.

`fire()` never throws. Failures become `onEvent` (`drop`, `wal_fail`, …). On Workers, pass `context.waitUntil` so the isolate stays alive for the admission task.

`fireAsync()` waits for admission and may reject. It does **not** wait for destination insert unless you also `flush()`.

## Batching

RAM queues flush when they hit `batching.maxSize` (default 100) or `batching.flushInterval` (default 5_000 ms), whichever comes first. The interval is a short batching timer, not DLQ retry scheduling.

Queues snapshot before asynchronous insertion. Concurrent enqueues receive a fresh queue and cannot be erased by a DLQ spill.

## Backpressure

| Cap | Default | When exceeded |
|-----|--------:|---------------|
| Per-tenant `batching.maxQueueSize` | 1_000 | Fair-share dump of the largest queues, or drop if `onQueueFull: 'drop'` (volatile only) |
| `maxActiveTenants` | 10_000 | New tenant key: durable → DLQ (`dlq` / `max_active_tenants`); volatile → drop |
| `maxTotalQueued` | 50_000 | Global sum of queue lengths + reservations |
| Journal / DLQ size | adapter-specific | `wal_full` / `dlq_full` |

`onQueueFull: 'drop'` is rejected when `mode: 'durable'`.

## Bootstrap order

1. Initialize reliability and acquire exclusive ownership when applicable (`CloudflareReliabilityAdapter.init` also recovers DLQ orphans and requests a pending-work alarm).
2. Initialize the destination (`adapter.init`).
3. Read a bounded wave of unacknowledged journal records into queues (`maxRecoveryBatch`, floored at 1).
4. Recover orphaned DLQ `processing` claims.
5. Construct the retry engine (no timers — hosts call `runMaintenance()`).

Logs accepted before `ready` sit in a volatile pre-ready buffer. Bootstrap drains that buffer through the real enqueue path (including journal) before `ready` resolves.

Source: https://logbun-docs.abshahin.workers.dev/concepts/pipeline/index.mdx