# Chapter 15: Observability

## Table of Contents

- [15.1 Debug Server](#151-debug-server)
- [15.2 Debug Endpoints](#152-debug-endpoints)
- [15.3 Health Check Endpoints](#153-health-check-endpoints)
- [15.4 Prometheus Metrics](#154-prometheus-metrics)
- [15.5 Prometheus Integration](#155-prometheus-integration)
- [15.6 Distributed Tracing with Tempo](#156-distributed-tracing-with-tempo)
- [15.7 Tracing Architecture](#157-tracing-architecture)
- [15.8 Tempo Setup with Docker](#158-tempo-setup-with-docker)
- [15.9 pprof Profiling](#159-pprof-profiling)
- [15.10 Statsviz Real-Time Monitoring](#1510-statsviz-real-time-monitoring)
- [15.11 Logging](#1511-logging)
- [15.12 Configuration Reference](#1512-configuration-reference)

---



Kronk provides comprehensive observability through distributed tracing,
Prometheus metrics, pprof profiling, and real-time visualizations.

### 15.1 Debug Server

Kronk runs a separate debug server for observability endpoints, isolated
from the main API for security.

**Default Ports:**

- Main API: `localhost:11435`
- Debug server: `localhost:11445`

**Configure Debug Host:**

```shell
kronk server start --debug-host localhost:9090
```

Or via environment variable:

```shell
export KRONK_WEB_DEBUG_HOST=localhost:9090
kronk server start
```

### 15.2 Debug Endpoints

The debug server exposes these endpoints:

**Prometheus Metrics:**

```
http://localhost:11445/metrics
```

**pprof Profiling:**

- `http://localhost:11445/debug/pprof/` - Index page
- `http://localhost:11445/debug/pprof/profile` - CPU profile
- `http://localhost:11445/debug/pprof/heap` - Heap profile
- `http://localhost:11445/debug/pprof/goroutine` - Goroutine stacks
- `http://localhost:11445/debug/pprof/trace` - Execution trace

**Statsviz (Real-time Visualizations):**

```
http://localhost:11445/debug/statsviz
```

Provides live charts for memory, goroutines, GC, and more.

### 15.3 Health Check Endpoints

Available on the main API port (no authentication required):

**Liveness Check:**

```shell
curl http://localhost:11435/v1/liveness
```

Response:

```json
{
  "status": "up",
  "build": "v1.0.0",
  "host": "hostname",
  "GOMAXPROCS": 8
}
```

**Readiness Check:**

```shell
curl http://localhost:11435/v1/readiness
```

Returns 200 OK when the server is ready to accept requests.

### 15.4 Prometheus Metrics

Kronk exposes detailed inference metrics in Prometheus format.

**Fetch Metrics:**

```shell
curl http://localhost:11445/metrics
```

**Available Metrics:**

System metrics:

- `goroutines` - Current goroutine count
- `requests` - Total request count
- `errors` - Total error count
- `panics` - Total panic count

All timing distributions are exposed as Prometheus **histograms** (suffix
`_seconds`), which emit `_bucket`, `_sum`, and `_count` series. Compute
averages with `rate(X_sum[5m]) / rate(X_count[5m])` and percentiles with
`histogram_quantile(...)`.

Model loading histograms (seconds):

- `model_load_seconds` — model file load time
- `model_load_proj_seconds` — multimodal proj file load time

Inference timing histograms (seconds):

- `model_prompt_creation_seconds`
- `model_prefill_seconds` (labeled by `kind="text|media|imc-decode"`)
- `model_prefill_ttft_seconds` (prefill-start to first sampled token)
- `model_request_ttft_seconds` (end-to-end request to first sampled token)

Token usage:

- `usage_tokens_total` — counter, labeled by `kind="prompt|reasoning|completion"`. Use `rate(...)` for fleet throughput. Output and total are linear combinations and can be derived in PromQL.
- `usage_tokens_per_second` — histogram of per-request decode rate (computed after TTFT).

Request lifecycle:

- `chat_requests_total{model_id, status="ok|error|cancel"}` — counter of chat completion outcomes.
- `chat_errors_total{model_id, class}` — counter of chat errors by class (`pre-batch`, `fail-job`, `context-cancelled`, …).
- `chat_request_duration_seconds` — histogram of end-to-end chat request duration.
- `chat_queue_wait_seconds` — histogram of time spent waiting in the batch engine queue.

Pool (sdk/pool):

- `pool_acquire_total{result="hit|miss|dedup|busy|error"}` — counter of acquire outcomes.
- `pool_acquire_duration_seconds{cache="hit|miss"}` — histogram of acquire latency.
- `pool_singleflight_wait_seconds` — histogram of duplicate-load wait time.
- `pool_evictions_total{reason, selection}` — counter labelled by reason (`ttl|cap|budget|replacement|invalidation|unknown`) and selection (`smallest-fit|coldest-idle|n/a`).
- `pool_evict_before_load_total` — counter of pre-admission evictions.
- `pool_evict_wait_seconds` — histogram of time waiting for an eviction callback to release its reservation.
- `pool_unload_duration_seconds{model_id}` — histogram of unload duration.
- `pool_load_failures_total{stage="plan|reserve|evict|load"}` — counter of load failures by stage.
- `pool_items_in_cache`, `pool_max_items_in_cache` — gauges for current/configured cache occupancy.
- `pool_active_streams{model_id}` — gauge of streaming requests per model.
- `pool_inflight_loads` — gauge of loads currently holding a reservation but not yet visible in the cache.

Resource manager (sdk/pool/resman):

- `resman_budget_percent`, `resman_headroom_bytes`, `resman_unified_memory` — configuration gauges.
- `resman_reservations` — gauge of active reservations.
- `resman_ram_total_bytes`, `resman_ram_budget_bytes`, `resman_ram_used_bytes`, `resman_ram_free_bytes` — RAM accounting.
- `resman_device_total_bytes{device,type}`, `resman_device_budget_bytes`, `resman_device_used_bytes`, `resman_device_free_bytes` — per-GPU accounting.
- `resman_reservation_bytes{model_id, kind="ram|vram", device}` — per-reservation memory commitment.
- `resman_reserve_rejections_total{reason="no_capacity|unknown_device|invalid_plan|duplicate_key|no_gpus|other"}` — counter of `Reserve` rejections.

### 15.5 Prometheus Integration

**Example Prometheus Configuration:**

```yaml
# prometheus.yml
scrape_configs:
  - job_name: "kronk"
    static_configs:
      - targets: ["localhost:11445"]
    scrape_interval: 15s
```

**Grafana Dashboard Query Examples:**

Average end-to-end time to first token:

```promql
rate(model_request_ttft_seconds_sum[5m])
  / rate(model_request_ttft_seconds_count[5m])
```

P99 end-to-end time to first token:

```promql
histogram_quantile(0.99,
  sum by (le, model_id) (rate(model_request_ttft_seconds_bucket[5m])))
```

Fleet token throughput by kind (tokens/second):

```promql
sum by (model_id, kind) (rate(usage_tokens_total[5m]))
```

Average per-request tokens-per-second:

```promql
rate(usage_tokens_per_second_sum[5m])
  / rate(usage_tokens_per_second_count[5m])
```

Request rate:

```promql
rate(requests[5m])
```

Error rate:

```promql
rate(errors[5m]) / rate(requests[5m])
```

### 15.6 Distributed Tracing with Tempo

Kronk supports OpenTelemetry tracing with Grafana Tempo integration.

**Enable Tracing:**

```shell
kronk server start \
  --tempo-host localhost:4317 \
  --tempo-service-name kronk \
  --tempo-probability 0.25
```

Or via environment variables:

```shell
export KRONK_TEMPO_HOST=localhost:4317
export KRONK_TEMPO_SERVICE_NAME=kronk
export KRONK_TEMPO_PROBABILITY=0.25
kronk server start
```

**Configuration Options:**

- `--tempo-host` - Tempo collector address (OTLP gRPC endpoint)
- `--tempo-service-name` - Service name in traces (default: `kronk`)
- `--tempo-probability` - Sampling probability 0.0-1.0 (default: `0.25`)

**Sampling Probability:**

- `1.0` - Trace every request (development only)
- `0.25` - Trace 25% of requests (recommended for production)
- `0.05` - Trace 5% of requests (high-traffic production)

**Excluded Routes:**

Health check endpoints are automatically excluded from tracing:

- `/v1/liveness`
- `/v1/readiness`

### 15.7 Tracing Architecture

**Request Flow with Tracing:**

```
Client Request
      │
      ▼
┌─────────────────────────────┐
│  Kronk Server               │
│  ┌───────────────────────┐  │
│  │ Inject Trace Context  │  │
│  │ (trace_id, span_id)   │  │
│  └───────────┬───────────┘  │
│              ▼              │
│  ┌───────────────────────┐  │
│  │ Handler Span          │  │
│  │ (chat, embed, etc.)   │  │
│  └───────────┬───────────┘  │
│              ▼              │
│  ┌───────────────────────┐  │
│  │ Inference Span        │  │
│  │ (model operations)    │  │
│  └───────────────────────┘  │
└─────────────────────────────┘
      │
      ▼
   Tempo Collector (OTLP gRPC)
      │
      ▼
   Grafana (Visualization)
```

**What Gets Traced:**

- HTTP request handling
- Model acquisition from pool
- Prefill and generation phases
- Token streaming

### 15.8 Tempo Setup with Docker

**Run Tempo Locally:**

```shell
docker run -d --name tempo \
  -p 3200:3200 \
  -p 4317:4317 \
  grafana/tempo:latest \
  -config.file=/etc/tempo/tempo.yaml
```

**Run Grafana:**

```shell
docker run -d --name grafana \
  -p 3000:3000 \
  grafana/grafana:latest
```

**Configure Grafana:**

1. Open http://localhost:3000 (admin/admin)
2. Add data source → Tempo
3. Set URL: `http://tempo:3200`
4. Save and explore traces

### 15.9 pprof Profiling

Use Go's pprof tools for performance analysis.

**Capture CPU Profile (30 seconds):**

```shell
go tool pprof http://localhost:11445/debug/pprof/profile?seconds=30
```

**Capture Heap Profile:**

```shell
go tool pprof http://localhost:11445/debug/pprof/heap
```

**View Goroutine Stacks:**

```shell
curl http://localhost:11445/debug/pprof/goroutine?debug=2
```

**Generate Flame Graph:**

```shell
go tool pprof -http=:8081 \
  http://localhost:11445/debug/pprof/profile?seconds=30
```

Opens interactive web UI with flame graph visualization.

### 15.10 Statsviz Real-Time Monitoring

Statsviz provides live runtime visualizations in your browser.

**Access Statsviz:**

```
http://localhost:11445/debug/statsviz
```

**Available Charts:**

- Heap size and allocations
- Goroutine count
- GC pause times
- CPU scheduler latency
- Memory by size class

Useful for real-time monitoring during load testing or debugging
memory issues.

### 15.11 Logging

Kronk logs structured JSON to stdout by default.

**Log Levels:**

Logs include context like trace IDs, request details, and timing.

**Insecure Logging:**

For debugging, enable verbose logging that includes message content:

```shell
kronk server start --insecure-logging
```

**Warning:** Insecure logging exposes user prompts and model responses.
Never enable in production.

**Environment Variable:**

```shell
export KRONK_INSECURE_LOGGING=true
```

### 15.12 Configuration Reference

**Debug Server:**

- `--debug-host` - Debug server address (env: `KRONK_WEB_DEBUG_HOST`,
  default: `0.0.0.0:11445`)

**Tracing:**

- `--tempo-host` - Tempo collector address (env: `KRONK_TEMPO_HOST`,
  default: `localhost:4317`)
- `--tempo-service-name` - Service name (env: `KRONK_TEMPO_SERVICE_NAME`,
  default: `kronk`)
- `--tempo-probability` - Sampling rate 0.0-1.0
  (env: `KRONK_TEMPO_PROBABILITY`, default: `0.25`)

**Logging:**

- `--insecure-logging` - Log message content
  (env: `KRONK_INSECURE_LOGGING`, default: `false`)
- `--llama-log` - llama.cpp log level, 0=off, 1=on
  (env: `KRONK_LLAMA_LOG`, default: `1`)

---

_Next: [Chapter 16: MCP Service](#chapter-16-mcp-service)_
