Compromises

What we did

PostgreSQL writes every change to the write-ahead log (WAL) before it applies it, so that nothing is lost in a crash. Versola turns this off for tables whose rows live for minutes at most. These tables are UNLOGGED:

The login-state tables became UNLOGGED in 0.7.0. The replay-guard tables have been UNLOGGED since they were created. Each of them is a ring of 12 partitions, and every partition is UNLOGGED on its own.

Sessions, refresh tokens, users, consents and configuration are ordinary logged tables. No credential, session, token or consent is stored in an UNLOGGED table.

Why

Logging data that expires in a minute buys durability for nothing, and it costs a lot. In one of our load-testing campaigns, a step-load of one million users, the login-state tables produced about 63% of the WAL bytes and about 40% of the statements that write WAL. The auth database was limited by commits waiting for the WAL, not by statement time (versola#598). The replay-guard tables are written on every token request (in auth) and every proxied request (in edge), for rows that are useless a minute later.

What it costs

After an emptying event:

• A user in the middle of a login starts again from /authorize.
• An authorization code that was not exchanged yet, or a pushed request that was not used yet, is rejected as invalid, and the client retries.
• An authorization code that was already exchanged has done its work: the session and the refresh token are in logged tables. Replaying it after the crash is rejected as an unknown code instead of also revoking the refresh-token family the first exchange issued.
• Replay protection for DPoP proofs and client assertions restarts empty. A proof or assertion that was already used can be accepted once more until its window passes: about the iat leeway (dpop.iat-leeway, 60 seconds by default) for a DPoP proof, and up to the assertion’s lifetime for a client assertion. A restart of a service carries the same exposure.

How to turn logging back on

Do it only if you accept the cost: the WAL load comes back, and with it the limit described above. This is manual, outside Versola’s migrations and tests, and at your own risk.

ALTER TABLE … SET LOGGED takes an ACCESS EXCLUSIVE lock and rewrites the table and its indexes, so every statement on the table waits for it. The tables are small, so this is short, but run it at low traffic. The rewrite is written to the WAL once, as a full copy of what the table holds at that moment.

Login-state tables, in the schema where the auth service lives (auth in the default setup):

ALTER TABLE auth.auth_conversations SET LOGGED;
ALTER TABLE auth.authorization_codes SET LOGGED;
ALTER TABLE auth.pushed_authorization_requests SET LOGGED;

The replay-guard tables are partitioned, and PostgreSQL refuses SET LOGGED on a partitioned parent (This operation is not supported for partitioned tables). Change the 12 partitions of each, one by one:

DO $$
BEGIN
  FOR i IN 0..11 LOOP
    EXECUTE format('ALTER TABLE auth.dpop_proofs_%s SET LOGGED', i);
    EXECUTE format('ALTER TABLE auth.client_assertions_%s SET LOGGED', i);
    EXECUTE format('ALTER TABLE edge.edge_dpop_proofs_%s SET LOGGED', i);
  END LOOP;
END $$;

Replace auth and edge with the schemas of your deployment. If edge uses a database of its own, run the edge.edge_dpop_proofs_* part there. Check the result: p means logged, u means UNLOGGED.

SELECT n.nspname, c.relname, c.relpersistence
FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind = 'r'
  AND c.relname ~ '^(auth_conversations|authorization_codes|pushed_authorization_requests|dpop_proofs_[0-9]+|client_assertions_[0-9]+|edge_dpop_proofs_[0-9]+)$'
ORDER BY 1, 2;

Two things to keep in mind:

• Persistence belongs to the table, not to Flyway. Versola does not check it at runtime. A table that a later release creates anew is created UNLOGGED again, so repeat the check after an upgrade. Versola does not guarantee compatibility before 1.0.0.
• A partition added later without the keyword is logged. Versola’s tests guard against that for the replay-guard tables, but they do not know about your manual change.

To go back, run the same statements with SET UNLOGGED.

What we did

After a browser sign-in, edge sets the EDGE_SESSION cookie. Its value is <presetId>:<accessToken>: the access token itself, a JWT. The cookie is HttpOnly, Secure and SameSite=Strict, with the domain and path of the authorization preset. The refresh token never goes to the browser: edge keeps it encrypted in its own edge_sessions table and uses it to renew an expired access token, which rotates the cookie.

Why

The cookie is a self-contained credential. edge authenticates a proxied request by checking the token in the cookie in memory (signature, expiry, binding and the revocation list), without a database read for every request. JavaScript on the page cannot read the cookie.

What it costs

Whoever copies the cookie value can call edge as that user until the access token expires (one hour by default) or is revoked. The attributes make a copy hard to get from the browser, but they do not make the cookie sender-constrained: unlike a token in an Authorization header, it is not tied to a DPoP key held by the caller.

What you can change

There is no switch to move the token out of the cookie. To limit the exposure:

• Shorten the client’s access token lifetime on the client page.
• Narrow the cookie with the preset’s cookie domain and path.
• Sign the user out: edge rejects revoked tokens (see revocation).

What we did

A refresh token’s lifetime is fixed when it is issued. It is the client’s refreshTokenTtl, 180 days by default, counted from the sign-in. Using the refresh token does not extend it, and a token that replaces a rotated one keeps the original expiry. Before 0.7.0 the lifetime was 90 days and was extended on use.

Why

A token that is extended every time it is used never expires for a user who stays active, so a single leaked refresh token stays valid as long as its owner keeps using the application. A fixed lifetime puts an upper bound on it.

It also saves database writes, and so WAL. Refreshing a token that is bound to a DPoP key or a certificate, without changing its scope, is now a read-only check; before, every refresh updated the expiry in the database. Refresh is the most frequent token request, so one write less per refresh matters at volume.

What it costs

A user signs in again once the refresh token reaches its lifetime, even if the user was active the day before. The client must handle invalid_grant on refresh and send the user through the authorization flow again.

What you can change

Set refreshTokenTtl per client (in days on the client page, Refresh Token TTL). There is no option to make it slide.

What we did

central is the source of truth for configuration: clients, resources, scopes, roles, permissions, presets and signing keys. auth and edge keep a copy in memory and reload it every configuration-cache-refresh-interval.

Why

Requests are answered from memory and do not wait for a round trip to central.

What it costs

A change made in the admin console or through the Central API reaches auth and edge after up to one interval. The generated default is 5 minutes for the vps and Kubernetes setups and 1 minute locally. In that time auth and edge may still apply the previous configuration, for example for a newly registered client or a permission you just changed.

What you can change

Set a shorter configuration-cache-refresh-interval in the configuration of auth and edge. A shorter interval means more requests to central. See Configuration.

What we did

Every edge replica keeps the list of revoked tokens in memory and checks each request against it. When a token is revoked, the replicas learn about it through PostgreSQL LISTEN/NOTIFY. As a backstop, each replica reconciles its list with the database every revocation.reload-interval, 600 seconds by default, with jitter.

Why

Checking revocation must not cost a database read for every proxied request.

What it costs

• A replica that misses a notification, for example because its listening connection dropped, accepts a revoked token until its next reconciliation: up to the reload interval.
• LISTEN does not work through a transaction-mode connection pooler. edge needs a direct connection for notifications (postgres.notifications-url) and does not start without one when it uses a pooler.

What you can change

Set a shorter revocation.reload-interval to shrink the window, at the price of more database reads. See Edge configuration.