Observability

Each Versola service (auth, central, edge) exposes health probes and Prometheus metrics on its diagnostics port, writes JSON logs to standard output, and can export traces over OTLP. This page lists what is there and which signals are worth alerting on. Versola ships no monitoring stack; you bring Prometheus (or a compatible scraper) and a log pipeline.

Health endpoints

The diagnostics port (DPORT) serves three GET endpoints and nothing else. It is a separate listener from the application, and none of the application’s security headers or authentication apply to it, so keep it off the public internet.

The diagnostics server starts before the service builds its dependencies. While auth or edge waits for its first configuration sync from central (up to two minutes), /liveness already answers 200 and /readiness answers 503, so a liveness probe does not kill a service that is only waiting. If central stays unreachable, startup fails and the process exits so the supervisor restarts it.

Default diagnostics ports: auth 8081, central 8091, edge 8096 on the vps target; the chart uses the same numbers. Check a service from the host:

curl -sf http://127.0.0.1:8091/readiness && echo central ready

On Kubernetes the chart probes /liveness (initial delay 10 s, period 10 s) and /readiness (5 s, 5 s) on the diagnostics port. versola up waits on the same readiness endpoints before it starts the next service.

Metrics

Scrape /metrics on every replica of every service. Series come from three places.

HTTP

status_class is 2xx, 4xx, 5xx and so on. route is the route pattern, never the raw path, so cardinality stays bounded. The http_client_* series cover the calls a service makes to others, for example auth to central. http_client_active_requests counts outbound requests in flight, queued ones included, and is the first series to climb when the outbound pool is too small.

JVM and process

The JVM runtime metrics are published with Prometheus-convention names (for example jvm_memory_used_bytes), covering memory pools, garbage collection, threads, class loading and process CPU. They are sampled on a schedule, not per scrape.

Database

The WAL and slot series are read by each service from its own database once a minute (pg_stat_wal, pg_stat_checkpointer, pg_settings, pg_replication_slots), so no postgres_exporter is needed and the database role needs no extra privilege; a view it cannot read is logged once and left out. Every replica of a service reports the same numbers, and where the three services share one database all three report them: read these with max by (...), not sum. Cumulative counters keep their meaning through a statistics reset or a database restart (the reset shows as a counter restart, not a negative rate). | db_notification_listener_connected | gauge | 1 while the LISTEN connection is proven to deliver, not merely open. | | db_notifications_received_total | counter | Notifications received. | | db_notification_listener_reconnects_total | counter | Reconnects of the LISTEN connection. | | db_notification_listener_silent_total | counter | Times the connection was open but stopped delivering. | | db_notification_listener_queue_overflow_total | counter | Notifications dropped because a subscriber fell behind. |

The pool series are off by default; set postgres.pool-metrics-interval (for example 10 seconds) to publish them. See Configuration.

Application

Password hashing runs behind a semaphore, so CPU stops following demand once all permits are held. argon2_hashes_waiting and the wait histogram are better scaling signals than CPU utilisation. central and edge register the Argon2 series too but never hash, so they report zeros.

Only metrics found in source are listed. Labels of the application counters are not documented here.

What to alert on

Scraping

A plain Prometheus job for the vps target, where the diagnostics ports are on loopback (run Prometheus on the same host or reach them through a tunnel):

scrape_configs:
  - job_name: versola
    metrics_path: /metrics
    static_configs:
      - targets: ["127.0.0.1:8081"]
        labels: { service: auth }
      - targets: ["127.0.0.1:8091"]
        labels: { service: central }
      - targets: ["127.0.0.1:8096"]
        labels: { service: edge }

Replicas added with versola replica listen on ports shifted by 100 per slot (slot 2 of auth on 8181), so add each replica as a target.

On Kubernetes the chart annotates every pod when metrics.prometheusAnnotations is true (the default):

prometheus.io/scrape: "true"
prometheus.io/port: "8081"       # the service's diagnostics port
prometheus.io/path: /metrics

Use pod discovery with these annotations and not the ClusterIP Service names, which would hide which pod a series came from. See Kubernetes for the reasoning and a working vmagent configuration.

Grafana board

The product repository holds a generated Grafana board for the whole system, dashboards/grafana/versola.json (uid versola). Import it in Grafana (Dashboards, New, Import), or provision it from a ConfigMap with the label your Grafana sidecar watches. It has an Overview section (two lines of four tiles per service: pods, requests, 5xx share, p99, two service-specific tiles, database pool, heap), then a section per service with traffic, errors and latency per endpoint and the service’s own signals (auth: sign-in funnel and token refresh; edge: resources, web, mobile /native and token revocation; central: configuration syncs, admin writes and change notifications). The database and runtime rows are collapsed. It also shows the outbound pool, connection timeouts, the WAL section, DPoP edge assertions, Argon2 admission and cleanup.

The only variables are the Prometheus data source and the namespace. The board selects series by namespace and app_kubernetes_io_component (auth, central or edge), the pod label the chart sets. All three services are scraped under one job, so your scrape configuration must copy pod labels onto the series (labelmap, as in k8s/monitoring/vmagent-values.yaml). Some panels stay empty until their source exists: the pool panels need postgres.pool-metrics-interval, and the failure panels are empty while nothing fails.

The JSON is generated: edit dashboards/grafana/generate.py and run python3 dashboards/grafana/generate.py rather than editing the file. The same directory has loadgen.json, a board for the project’s own load generator, which you do not need.

Logs

Services write one JSON object per line to standard output. On the vps target read them with docker logs:

docker logs --tail=200 versola-auth 2>&1 | jq -r '"\(.timestamp) \(.level) \(.message)"'

Each line has these fields, in this order:

An illustrative line (values are placeholders; fields that are empty are omitted):

{"timestamp":"2026-10-01T10:15:30.123Z","system":"auth","env":"prod","level":"INFO","thread":"zio-fiber-123","message":"Set temporary bootstrap password for admin user 'admin'","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736"}

Authorization headers are masked (Bearer ***). versola-proxy’s access log is plain nginx format, not JSON.

Tracing

Set otel-exporter at the top level of the service’s configuration to an OTLP gRPC endpoint to export traces:

otel-exporter = "http://otel-collector:4317"

The service name in traces is <service>-<env>, for example auth-prod. Context is propagated with the W3C traceparent header. Without the key the exporter is a no-op and nothing leaves the process. The generated files carry the key commented out. See Configuration.

Diagnostics

The images set JVM flags (see JVM defaults) that leave something to investigate after a crash. /diagnostics is a directory in each container, not an HTTP endpoint:

• heap.hprof: a heap dump written when the JVM runs out of memory. The path is a fixed file name, so each crash overwrites the previous dump. The JVM then exits, and the supervisor (Docker’s restart policy, or Kubernetes) restarts the container.
• gc.log: a GC log, rotated over 5 files of 20 MB.

Both deployments mount a volume there: a named volume per service on vps (versola-auth-diagnostics-vps, versola-central-diagnostics-vps, versola-edge-diagnostics-vps) and an emptyDir in the chart, sized by diagnosticsSizeLimit (1Gi). A dump is at most about the heap size. Copy it out:

docker cp versola-auth:/diagnostics/heap.hprof .

kubectl cp <namespace>/<pod>:/diagnostics/heap.hprof ./heap.hprof

A heap dump contains whatever was in memory, including keys and tokens. Treat it as a secret.

jcmd is in the image. The JVM runs with -XX:NativeMemoryTracking=summary, so you can attribute container memory to heap, direct buffers, metaspace, code cache and thread stacks:

docker exec versola-auth jcmd 1 VM.native_memory summary

Next steps

• Configuration: otel-exporter, postgres.pool-metrics-interval and the JVM defaults.
• Kubernetes: scraping with the chart’s annotations.
• CLI reference: versola status for the container view of a vps deployment.