Field Glossary
Most log lines carry structured fields alongside their human-readable message. Because message wording can change between Service versions, filter on these fields rather than the message text. The table below covers the fields on most lines, but a line can carry many more (especially with metadata expanded), and each source connector adds its own. Treat fields not listed here as additional context, and see the Detailed Event Reference for fields specific to individual events.| Field | Meaning |
|---|---|
level | Log severity: info, warn, error, or fatal. |
timestamp | ISO-8601 UTC time the line was emitted. |
rid | Request ID. The same rid appears on both the start and complete line of a session, so you can correlate them. The prefix marks the endpoint type: s/ for the WebSocket sync stream, h/ for HTTP API requests. |
user_id | The authenticated user ID extracted from the client’s JWT. |
client_id | The client device ID, assigned per install by the SDK. One user can have many clients, so this differs from user_id. |
user_agent | The SDK and platform the client reported, for example powersync-js/1.43.1 powersync-react-native react-native/0.77 ios/26.5. |
app_metadata | Custom metadata the client attached via connect({ params }). See Custom Metadata in Sync Logs. |
client_params | The parameter values the client passed to connect(). Large values may be truncated. |
pod | The Kubernetes pod that emitted the line. Distinguishes pods in a multi-pod deployment. |
duration_ms | Wall-clock duration of the request in milliseconds. On streaming access log lines this is the full session length. |
stream_ms | How long a sync session stayed open, in milliseconds. Same value as the duration_ms on the session’s streaming access log line. |
operations_synced | Total number of operations sent to the client during the session. |
operation_counts | Per-type breakdown of operations: { "put": N, "remove": N, "move": N, "clear": N }. |
data_synced_bytes | Uncompressed (plaintext) bytes synced to the client. |
data_sent_bytes | Bytes actually sent over the wire. Smaller than data_synced_bytes when compression is enabled. |
close_reason | Why a sync session ended. See Close Reasons. |
status | HTTP response status code. 200 for streaming sync (it streams as 200 even if it later errors), 401 for auth failures, 429 for queue overflow. |
error_code / errorData | The structured error on a failed line. errorData is { code, name, description, details?, severity, status }, where code is the PSYNC error code and details carries the underlying-driver message. |
lsn | The source database position a checkpoint or flush corresponds to: a Postgres LSN, a MongoDB resume token, or an MSSQL LSN. |
replication_lag / replication_lag_seconds | Seconds between when a change was committed on the source database and when the replicator flushed it to storage. Zero or one second is healthy. See Replication Lag. |
checkpoint | The internal checkpoint op_id a sync session has reached. |
Severity and Recovery
A line’s severity (level) does not necessarily tell you whether you need to act. Most error and warn lines describe transient events that the Service recovers from on its own:
- Per-session errors stay isolated to the affected client.
Sync stream errorandStreaming sync request failedtear down a single stream, and that client reconnects on its own. Service-wide failures such as a pod restart are the exception. They interrupt all connected clients until the Service recovers. - The replicator retries source-database operations automatically, so transient connection drops, query timeouts, and keep-alive failures are caught, logged, and retried.
- Some
errorlines, such asReplication failed., trigger a pod restart that the supervisor handles automatically. Operator action is needed only when the cause persists across restarts.
| Recovery | What it means |
|---|---|
| Auto | The Service handles it. No client or operator action required. |
| Client retries | A single client is affected. Its SDK reconnects. |
| Pod restart | The pod restarts automatically. Persistent occurrences usually point to a configuration or source-database problem. |
| Investigate | The line alone breaks nothing, but a burst of them is worth looking into. |
| Operator action | The Service cannot recover without intervention. |
MongoDB is the storage backend for all PowerSync Cloud instances, which some lines reflect (for example
Successfully activated storage: mongodb. or checksum errors like PSYNC_S2403). If you self-host on Postgres-backed storage, those lines look different.Sync & API Logs
Produced by the PowerSync Service API pods. These cover client sync sessions (WebSocket and HTTP), HTTP API requests, checkpoint calculations, and write checkpoint endpoints.Sync Session Lifecycle
These events trace a client’s sync session from start to finish. Searching for arid returns both ends.
| Event | Level | Recovery | When it fires |
|---|---|---|---|
Sync stream started | info | n/a | A client opened a sync connection and authenticated. Both the HTTP streaming endpoint and the WebSocket endpoint emit this. |
Sync stream complete | info | n/a | A sync session ended. The fields on this line summarize what the session did (operations_synced, data_synced_bytes, close_reason). |
Sync stream error | error | Client retries | The WebSocket sync stream threw an exception and is being torn down. The SDK reconnects on its 5s retry. Other clients are unaffected. |
Streaming sync request failed | error | Client retries | The HTTP streaming sync request threw an exception and is being torn down. Same impact as Sync stream error, but on the HTTP endpoint. |
stream_ms paired with operations_synced: 0 usually means the connection was opened and torn down before any sync line was sent. This is normal when many clients reconnect at once, or when a client cancels quickly.
For the full field breakdown of each line, see Sync session field details below.
Close Reasons
Theclose_reason field on Sync stream complete takes one of these values:
| Value | Recovery | Meaning |
|---|---|---|
client closing stream | n/a | The client called disconnect() or its TCP connection closed cleanly. Normal for a session that ends because the user closed the app. |
service closing stream | n/a | The server closed the stream for non-error reasons, for example because the Sync Config changed and existing streams need to reconnect. |
stream error | Client retries | An exception was thrown during the stream. See the preceding Sync stream error or Streaming sync request failed for details. |
process shutdown | n/a | The API pod is shutting down (deploy, scale-down, restart). |
unknown | Investigate | The session ended without a recognized cause. Treat as a likely bug if it appears frequently. |
Checkpoint Events
The PowerSync Service computes checkpoints to decide whether each connected client needs new data. In steady state you mostly seeUpdated checkpoint lines with empty updated/removed arrays, which are rechecks where nothing changed for that user.
| Event | Level | Recovery | When it fires |
|---|---|---|---|
New checkpoint: ... | info | n/a | The first checkpoint sent to a session, or after a parameter or sync-stream membership change forces a full checkpoint. |
Updated checkpoint: ... | info | n/a | An incremental checkpoint (diff). |
Common Errors in Sync & API Logs
| Code | What it means | Recovery |
|---|---|---|
PSYNC_S2001 | Generic internal server error wrapper. The real cause is in the nested error. | Depends on cause |
PSYNC_S2103 | JWT has expired. The client should be refreshing credentials. | Client retries |
PSYNC_S2305 | Too many buckets for the user. Sync for this user is blocked until either the data or the Sync Config changes. See Troubleshooting: Too Many Buckets. | Operator action (per user) |
PSYNC_S2401 | Checksum mismatch. The client deletes the affected bucket and re-fetches. No data loss. | Auto |
PSYNC_S2403 | Query timed out while reading checksums (MongoDB storage). The Mongo aggregate that builds the checkpoint exceeded its timeout (50s). | Client retries |
Replicator Logs
Produced by the replication pod that pulls changes from your source database (Postgres, MongoDB, MySQL, or MSSQL) and writes them into the PowerSync storage layer.Most replicator lines are prefixed with the Sync Config identifier in brackets, for example
[powersync_69b93ff7358aa0646ff0dbca_1_e6dc]. The format is [powersync_<instance_id>_<group_id>_<hash>]. The group_id increments when the Sync Config changes, so a new version means a new group. Use this prefix to scope log queries to a particular Sync Config iteration.Streaming Replication (Steady State)
Once initial replication is done, the replicator streams changes. TheFlushed line is the most important steady-state line and the bulk of normal replicator logs.
| Component | Meaning |
|---|---|
N + M + K updates | N = bucket data rows written. M = bucket parameter rows written. K = current_data records updated (the source-row mirror used for parameter recalculation). |
Xkb | Uncompressed size of the bucket data written in this flush. |
Yms | Wall-clock time the flush took. Sustained values above ~500ms indicate the storage layer is under load. |
Last op_id: Z | The highest internal op_id assigned in this flush. Increases monotonically across all flushes for the iteration. |
Replication lag: Ls | Seconds between when the oldest uncommitted change in this flush was made on the source and when it was flushed to storage. Zero or one second is healthy. Sustained values above 60s warrant investigation. See Replication Lag. |
flushed object on the line, easier to filter on than the message text:
Replication lag is absent, the flush had no source-side change timestamps to derive it from. This happens only on snapshot flushes, never steady-state.
Replication Health
These lines report problems in the replication loop. Almost all are non-fatal, and the replicator handles them on its own.| Event | Source DBs | Level | Recovery | What it indicates |
|---|---|---|---|---|
Failed to ping connection | Postgres | warn | Auto | A pooled connection failed its health check. Dropped and replaced. |
KeepAlive failed, unable to post to WAL | Postgres | warn | Auto | The replicator could not write a keep-alive heartbeat to the source WAL. Usually transient (source briefly read-only, brief network blip). Sustained occurrences indicate credentials or permissions. |
KeepAlive failed, unable to write an update to the <table> | MSSQL | warn | Auto | Same as above, for MSSQL. |
Could not check RLS access for <schema>.<table> | Postgres | warn | Investigate | The replicator could not verify whether its user bypasses Row-Level Security. If RLS is in fact enforced, you will see missing data downstream. |
Could not query max_slot_wal_keep_size | Postgres | warn | Investigate | The Postgres user lacks permission to read pg_settings. Cosmetic, but indicates a permission gap. |
Query error, retrying.. | MSSQL | warn | Auto | A CDC query failed and is being retried. |
Replication error | All | warn | Auto | A non-fatal error in the replication loop. The replicator retries. |
Replication failed. | All | error | Pod restart | Top-level error from the replication job loop. The pod is being restarted by the supervisor. Persistent occurrences usually carry a PSYNC code and indicate the underlying cause needs intervention. |
Fatal replication error | All | error | Pod restart | Same as above. |
Common Errors in Replicator Logs
| Code | What it means | Recovery |
|---|---|---|
PSYNC_S1003 | Replication stream is locked by another process. Normal for up to 1 minute after a deploy or after a process was killed; the lock has a 60s TTL. | Auto (within 1 minute) |
PSYNC_S1004 | JSON nested object depth exceeds 20. Affects replication of that row. | Operator action (data fix) |
PSYNC_S1146 | Postgres replication slot was invalidated (source dropped it or exceeded max_slot_wal_keep_size). Replication restarts from snapshot. | Auto (re-snapshot) |
PSYNC_S1344 | MongoDB change stream invalidated. Replication restarts with a new change stream. | Auto |
PSYNC_S1345 | MongoDB change stream timed out reading from the source. | Auto (retry) |
Detailed Event Reference
The sections above cover the most common lines. The events below are lower-frequency or verbose, collapsed here to keep the page scannable but documented in full.Sync & API Logs
Sync session field details
Sync session field details
Fields on Sync stream started
| Field | Meaning |
|---|---|
user_id, client_id, rid, user_agent, app_metadata | See Field Glossary. |
client_params | Parameter values the client passed to connect(). Subject to size limits, so very large values may be truncated. |
Fields on Sync stream complete
| Field | Meaning |
|---|---|
user_id, client_id, rid, user_agent, app_metadata | See Field Glossary. |
stream_ms | Duration the session was open, in milliseconds. Same value as duration_ms on the corresponding STREAM /sync/stream access line. |
operations_synced | Total operations sent to this client in the session. |
operation_counts | Per-type breakdown: { "put": N, "remove": N, "move": N, "clear": N }. |
data_synced_bytes | Plaintext (uncompressed) bytes synced to the client. |
data_sent_bytes | Bytes actually sent on the wire. With compression enabled, this is smaller than data_synced_bytes. |
encoding | Wire encoding used for the session, for example permessage-deflate (WebSocket compression). |
large_buckets | Buckets that contributed an unusually large number of operations in this session, as an object mapping <bucket descriptor> -> <operation count>. Useful for spotting outlier buckets that dominate a session’s payload. |
close_reason | Why the session ended. See Close Reasons. |
Fields on Sync stream error and Streaming sync request failed
| Field | Meaning |
|---|---|
user_id, client_id, rid, user_agent | See Field Glossary. |
errorData | Structured error: { code, name, description, details?, severity, status }. The code is the PSYNC code, and details carries the underlying-driver message (for example, "Connection to <mongo host>:<port> interrupted due to server monitor timeout"). |
is_service_error | true when the error came from a recognized Service code path; false for unexpected exceptions. |
stack | Stack trace from the throw site. Useful for cross-referencing against the Service source. |
Checkpoint payload anatomy
Checkpoint payload anatomy
Example payloads:
Structured fields on the same line include
| Component | Meaning |
|---|---|
1152792849 | The internal checkpoint op_id this session is at. |
write: 1585 | The latest write checkpoint applicable to this user, if any. null if there are no pending writes. See Write Checkpoints in the PowerSync protocol. |
buckets: 2 | The total number of buckets visible to this user given current parameters. The default cap is 1,000 (see PSYNC_S2305). |
param_results: 2 | The number of distinct parameter results evaluated for this user. Each parameter result corresponds to a row returned by a parameter query. |
[<bucket descriptors>] | The list of bucket descriptors, truncated to 20. Format is <priority>#<stream_name>_<version>[<parameter_values>]. |
updated: [...] (Updated only) | Buckets whose data changed since the previous checkpoint. |
removed: [...] (Updated only) | Buckets that left the user’s set since the previous checkpoint. |
checkpoint (the op_id), buckets (count), parameter_query_results (count), and on Updated checkpoint only, updated and removed counts as integers. These are easier to filter on than parsing the message body.HTTP endpoint and write checkpoint events
HTTP endpoint and write checkpoint events
| Event | Level | Recovery | When it fires |
|---|---|---|---|
STREAM /sync/stream | info | n/a | RSocket access log line for a WebSocket sync subscription. Emitted on stream close, so the duration_ms is the full session length. Carries client_id, rid, duration_ms, user_id, user_agent. |
GET /write-checkpoint2.json?client_id=<uuid> | info | n/a | A client requested a write checkpoint number to wait for. See Write Checkpoints in the PowerSync protocol. Carries method, path, route, status, rid, duration_ms. |
Write checkpoint for <user>/<client>: <num> | <lsn> | info | n/a | Result of a write checkpoint computation. <lsn> is the source database LSN this checkpoint corresponds to (Postgres LSN, MongoDB resume token, MSSQL LSN). |
Currently have N active WebSocket connection(s) | info | n/a | Periodic count of open WebSocket connections on this pod. Per-pod, not per-instance. |
<METHOD> <path> with status: 429, queue_overflow: true | warn | Client retries | The request queue is full and the request is being rejected with HTTP 429. The client retries on its own. |
| Field | Meaning |
|---|---|
method | HTTP method, for example GET, POST, STREAM. |
path | Full request path, including query string. |
route | The matched route template (with placeholders), useful for grouping. |
status | HTTP response status code. 200 for streaming sync (it streams as a 200 even if it later errors), 401 for auth failures, 429 for queue overflow. |
duration_ms | Wall-clock duration in milliseconds. For streaming endpoints this is the full session length. |
rid | Request ID. h/ prefix for HTTP, s/ for sync stream. |
Concurrency and queueing
Concurrency and queueing
These appear when the API pod is near its configured concurrent-sync limit.
Frequent
| Event | Level | Recovery | When it fires |
|---|---|---|---|
Sync concurrency limit reached, waiting for lock | info | Auto | A new sync session is queued behind currently-active sessions. It will start when a slot frees up. |
Got sync lock. Slots available: N | info | n/a | The queued session acquired a slot. |
Releasing sync lock | info | n/a | The session released its slot, either because it finished or because it errored. |
Sync concurrency limit reached typically means traffic exceeds what the pod can handle, or the per-pod cap is set too low. See Production Readiness Guide.Replication propagation (observed from the API pod)
Replication propagation (observed from the API pod)
| Event | Level | Recovery | When it fires |
|---|---|---|---|
Waiting for LSN checkpoint: <lsn> | info | n/a | The write checkpoint endpoint is waiting for the replicator to catch up to <lsn>. |
Got write checkpoint: <lsn> : <checkpoint> | info | n/a | The wait completed; the checkpoint number is returned to the client. |
Replicator Logs
Startup and lifecycle
Startup and lifecycle
Fired when a replication pod starts, stops, or restarts. Outside of a deploy, these usually mean the pod was killed and restarted by the supervisor.
| Event | Level | Recovery | When it fires |
|---|---|---|---|
Starting system | info | n/a | First line after the pod starts. |
Starting Storage Engine... / Successfully started Storage Engine. | info | n/a | The bucket storage backend is being connected. The Successfully activated storage: <type> line in between names the backend (typically mongodb for Cloud). |
Starting Replication Engine... / Successfully started Replication Engine. | info | n/a | The replication subsystem is initializing. |
Starting Replicator: <id> | info | n/a | One per source database connection (for example postgresql-postgres). |
Starting Router Engine... / Running on port 8080 / Successfully started Router Engine. | info | n/a | The replicator’s HTTP health and metrics endpoints are up. The replicator does not serve sync traffic. |
Loaded sync config (or Loaded sync rules) | info | n/a | The current Sync Config was loaded from storage. Both wordings exist depending on Service version. |
No sync streams or rules configured - configure via API | info | n/a | No Sync Config is deployed yet. |
Locked replication stream for processing | info | n/a | This pod has acquired the replication lock for a Sync Config iteration. Only one pod can replicate a given iteration at a time. |
Sync rules N has been locked for replication with lock ID <hex> | info | n/a | Companion line emitted after Locked replication stream for processing. The lock ID is the 8-byte hex identifier this pod is holding. Useful for confirming which pod owns the active lock. |
[PSYNC_S1003] Sync rules N have been locked by another process for replication, expiring at <ts>. | info | Auto | Pod tried to acquire the lock but another pod (or a prior incarnation of this pod) still holds it. The lock has a 60s TTL, so this typically clears within a minute. Logged at info (not warn) precisely because it is expected behavior. |
Successfully registered Replicator <id> with ReplicationEngine | info | n/a | Per-connector registration event during startup. |
System started | info | n/a | All subsystems are up. The pod is actively replicating. |
Sync Config changes
Sync Config changes
| Event | Level | Recovery | When it fires |
|---|---|---|---|
Failed to update sync config | error | Pod restart | Sync Config update validation failed. The pod exits and the supervisor restarts it. |
Stopping replication job for sync rule iteration: N | info | n/a | An older Sync Config iteration is being stopped, usually because a new iteration replaced it. |
Shutdown sequence
Shutdown sequence
Triggered by SIGTERM (deploys, scale-down) or by a fatal error. A clean shutdown ends with If you see
Exiting. Lines you will see in order:Replication failed. or Fatal replication error before the shutdown lines, the pod is being restarted by the supervisor after a crash.Initial Replication
Initial Replication
The first time a Sync Config iteration is set up, every table covered by it is copied over. The exact wording differs by source database.
| Event | Source DBs | Level | Recovery | When it fires |
|---|---|---|---|---|
To replicate: <schema>.<table> <count>/<total> | Postgres, MongoDB | info | n/a | Logged once per table before the snapshot starts. The fraction is replicated rows over estimated total. |
Replicating <schema>.<table> <count>/<total> - resumable | Postgres | info | n/a | Table snapshot is starting or resuming. resumable means the snapshot is chunked on the table’s single-column primary key, so it can pick up after an interruption. |
Replicating <schema>.<table> <count>/<total> - not resumable | Postgres | info | n/a | The table has a composite primary key or no usable primary key, so an interrupted snapshot has to restart from the beginning. |
Replicating <schema>.<table> <count>/<total> - resuming from <pk_col> > <value> | Postgres | info | n/a | An interrupted resumable snapshot is picking up from a stored cursor. |
Replicating <schema>.<table> <count>/<total> | Postgres, MongoDB | info | n/a | Progress update during a snapshot, logged per chunk flush. |
Snapshotting <schema>.<table> <count>/<total> | MSSQL | info | n/a | MSSQL CDC equivalent of Replicating. |
Skipping <table> - snapshot already done (or Skipping table [<table>] - snapshot already done.) | All connectors | info | n/a | Table has already been fully snapshotted for the current iteration. Logged on startup when resuming. |
Initial replication already done | All connectors | info | n/a | All tables in the iteration are already snapshotted. The replicator goes straight to streaming. |
Snapshot completion and replication slot management
Snapshot completion and replication slot management
| Event | Source DBs | Level | Recovery | When it fires |
|---|---|---|---|---|
Created replication slot <slot_name> | Postgres | info | n/a | A new logical replication slot was created. Happens on first replication or after the previous slot was dropped (Sync Config changes, slot health check failure, pg_drop_replication_slot, pg_upgrade). See Postgres Maintenance. |
Cleaning up Postgres replication slot: <slot_name>... | Postgres | info | n/a | A slot is being dropped, usually after Sync Config changes. The next replication run creates a fresh slot. |
WAL budget: no limit configured (max_slot_wal_keep_size is unlimited). | Postgres | info | n/a | Source Postgres has no slot retention limit. Normal. |
WAL budget: ... budget-low | Postgres | warn | Investigate | max_slot_wal_keep_size is set and the slot is approaching it. If the slot is invalidated, replication has to restart from snapshot. See PSYNC_S1146. |
Marking snapshot at <lsn> | MongoDB, MSSQL | info | n/a | The snapshot transaction captured all tables at <lsn>. Streaming will resume from this point. |
Resuming snapshot at <lsn> | MongoDB, MSSQL | info | n/a | A previously interrupted snapshot is resuming from a stored LSN. |
Flushed snapshot at <op_id> | MongoDB | info | n/a | The MongoDB snapshotter finished writing a collection up to the resume token <op_id>. |
Streaming replication variants
Streaming replication variants
Beyond the
The
Flushed line, the streaming phase emits these connector-specific lines.| Event | Source DBs | Level | Recovery | When it fires |
|---|---|---|---|---|
[<sync_rule>] Replicating op N <lsn> | Postgres | info | n/a | Periodic progress log during initial sync streaming, every 100 operations. |
[<sync_rule>] Processed batch of N changes / X bytes in Yms | MongoDB | info | n/a | A batch of MongoDB change-stream events was consumed from the source. Carries structured fields count, bytes, duration, useful for filtering on slow batches. |
Resume streaming at <token> / <lsn> | Token age: Ns | MongoDB | info | Investigate if Token age is large | The change stream resumed from a stored token. Token age approaching MongoDB’s oplog retention window means the slot is at risk. |
New collection: <schema>.<name> | MongoDB | info | n/a | A new matching collection was discovered after replication started. It will be snapshotted before streaming resumes. |
Enabled postImages on <db>.<collection> | MongoDB | info | n/a | The replicator enabled changeStreamPreAndPostImages on a collection that needs it for replica identity. Done once per collection. |
Flushed line’s structured flushed object maps to the message components as follows:| Sub-field | Maps to message component |
|---|---|
bucket_data_count | N |
parameter_data_count | M |
current_data_count | K |
duration | Yms |
size | Bytes (the Xkb value is size / 1024, rounded). |
replication_lag_seconds | Ls |
Schema changes
Schema changes
| Event | Source DBs | Level | Recovery | When it fires |
|---|---|---|---|---|
Processing schema change <type> for table [<schema>.<table>] | MySQL | info | n/a | A DDL event from the source binlog is being applied to the replicator’s schema view. |
Successfully processed N schema change(s). | MySQL | info | n/a | All DDL events from one binlog chunk were applied. |
Table <schema>.<table> has been dropped. Handling schema change... | MSSQL | info | n/a | The MSSQL CDC poller noticed a tracked table was dropped or its capture instance was removed. Handled by halting replication for that table or re-snapshotting, depending on config. |
Unable to retrieve schema changes for capture instance: [<name>]. | MSSQL | warn | Auto | Schema delta query failed. Usually a transient permissions or connection issue. |
CDC polling (MSSQL only)
CDC polling (MSSQL only)
The MSSQL connector polls CDC capture tables rather than streaming a logical replication slot.
| Event | Level | Recovery | When it fires |
|---|---|---|---|
CDC polling started with interval of Nms... | info | n/a | Replication is starting. N is the polling interval in ms (default 1000). |
Polling a maximum of N transactions per polling cycle. | info | n/a | The per-cycle batch limit. |
Polling bounds are <startLSN> -> <endLSN> spanning N transaction(s). | info | n/a | One polling cycle’s range. |
CDC polling stopped... | info | n/a | The polling loop is shutting down. |