Tracing Model

What a span is, how they form a tree, and how that tree becomes an execution record.

What is a span?

A span is a single unit of work with a name, a start time, a duration, and an outcome. Every step a request takes — from arriving at the gateway to the last database write — produces a span.

Spans are hierarchical: each span has a parent. The root span is always the gateway receiving the request. Everything that happens downstream is a child or grandchild of that root. This tree of spans is called the span tree, and it is the core of every execution record.

root
└─ gateway                       ← root span (request arrived)
   └─ create_user                ← function execution span
      ├─ db.insert(users)        ← database span
      ├─ stripe.charge           ← outbound HTTP span
      └─ queue.enqueue(email)    ← async hand-off span

Span types

Flux emits five categories of spans automatically — no instrumentation required.

1. Gateway spans

Emitted by the API gateway when a request arrives. Contains: auth result, tenant resolution, rate limit check outcome, and the injected request_id. Every other span in the tree descends from this one.

gateway  2ms
  auth_check    0.4ms  ✔ API key valid
  rate_limit    0.1ms  ✔ 18/100 req/min used
  route_match   0.3ms  → create_user

2. Runtime spans

Emitted by the Deno runtime for each function invocation. Captures start, end, return value shape, and any uncaught exception with stack trace. The runtime span wraps everything the function does.

create_user  81ms
  [input]   { email: "a@b.com", name: "Alice" }
  [output]  { id: 42 }  201

3. Database spans

Emitted by the Data Engine for every ctx.db.query() call. Contains: the compiled SQL, parameters, execution time, rows affected, and — for INSERT/UPDATE/DELETE — the before and after row values.

db.insert(users)  4ms
  [sql]    INSERT INTO users (email, name) VALUES ($1, $2) RETURNING *
  [rows]   1 inserted  id=42

This is the source of the mutation log. Every DB span with a write operation automatically produces a state_mutations record linked to the request_id. See Mutation logging.

4. Outbound HTTP spans

Emitted whenever your function calls an external service via fetch() or a built-in provider like ctx.stripe, ctx.email, or ctx.slack. The span records the outbound URL, method, response status, and latency.

stripe.charge  180ms  ✗ timeout
  [url]      https://api.stripe.com/v1/charges
  [method]   POST
  [status]   — (timed out after 10s)

During incident replay, outbound HTTP spans are stubbed — the runtime returns the recorded response instead of making a real request, so external services are never contacted.

5. Async job hand-off spans

Emitted when ctx.queue.enqueue() is called. Records the job type, the payload, and whether the enqueue committed or was rolled back. When the job runs, its own execution record starts a new span tree with the job span tree linked to its parent via a parent_request_id.

queue.enqueue(send_welcome_email)  1ms
  [job]     send_welcome_email
  [payload] { userId: 42, email: "a@b.com" }
  [status]  committed  → job req:b3c4d5e6

How the span tree is assembled

Each component — gateway, runtime, Data Engine — emits spans independently and writes them to the execution_spans table keyed by request_id. At query time (e.g. flux trace) the Data Engine reads all spans for that request_id and assembles them into the tree using parent span IDs.

Client
  → Gateway          emits: root span + sub-spans
  → Runtime          emits: function span + tool-call spans
  → Data Engine      emits: db query spans + mutation records
  → Response         span tree assembled and stored atomically

request_id: 550e8400
  All spans → execution_spans WHERE request_id = '550e8400'
  All mutations → state_mutations WHERE request_id = '550e8400'

The complete assembled record is what powers every flux debugging command:

Command	What it reads
`flux trace <id>`	Full span tree from `execution_spans`
`flux trace debug <id>`	Spans + input/output at each step
`flux trace diff <a> <b>`	Two span trees compared structurally
`flux why <id>`	First error span + upstream mutations
`flux state history`	`state_mutations` for a row across all records
`flux state blame`	Last writing span per column
`flux incident replay`	`request_input` + recorded outbound responses

Custom spans

You can emit custom spans from your function code to annotate specific operations:

export default async function handler(req, ctx) {
  // Wrap a block in a named span
  const prices = await ctx.trace.span('fetch_prices', async () => {
    return await fetch('https://prices.internal/list').then(r => r.json())
  })

  return new Response(JSON.stringify(prices))
}

Custom spans appear in the trace tree alongside automatic spans and are queryable with the same commands.

← Execution Record · Database →