Data Synchronization Architecture

This page describes how data flows from your WordPress site to the StorePilot Platform.

Architecture Overview

WordPress Site (Browser)
        │
        ├── JS Errors ──────────────────┐
        ├── PHP Errors (server-side) ───┤
        ├── Lead capture form data ─────┤──► WordPress PHP Proxy
        ├── Pageview events ────────────┤        (WP REST API endpoint)
        └── Visitor identification ─────┘              │
                                                       │  HTTP POST
                                                       ▼
                                             NestJS Ingest Endpoints
                                              api.store-pilot.net
                                                       │
        ┌──────────────────────────────────────────────┤
        │                                              │
        ▼                                              ▼
  BullMQ Queue                              Direct Insert (Prisma → DB)
  (errors only)              pageviews / visitors / recordings / leads
        │
        ▼
  Worker Process
  (upsert + notify)

---

Why a WordPress Proxy?

Most of what the browser collects goes straight to the Platform (/sdk/v1/*, identified by the site's public key and the page's Origin): errors, interactions, session recordings, time-on-page, and — for an anonymous visitor — the pageview itself. Only the calls whose value IS what WordPress can add still pass through a WordPress REST route (/wp-json/storepilot/v1/*) that forwards to the Platform:

1. Leads and forms — PHP attaches the live WooCommerce cart, WooCommerce's own session id and the logged-in user id before forwarding
2. Surveys — kept same-origin so a strict Content-Security-Policy needs no exception
3. Pageviews of a logged-in visitor — the user id is read from the WordPress cookie on the request, never printed into the page

For an anonymous pageview the page itself states which product it is, signed by the shop's server with a key derived from the site API key; the Platform verifies that signature and attributes the view to the product without WordPress handling the request. See WordPress Proxy.

---

Ingest Endpoints

Endpoint	Auth	Method	Description
POST /ingest/errors	X-StorePilot-Key	Async (BullMQ)	Submit JS/PHP error
POST /ingest/pageviews	X-StorePilot-Key	Direct	Record a pageview
POST /ingest/recordings/start	X-StorePilot-Key	Direct	Start recording session
POST /ingest/recordings/segment	X-StorePilot-Key	Direct	Push a recording segment
POST /ingest/recordings/end	X-StorePilot-Key	Direct	End recording session
POST /ingest/recordings/interactions	X-StorePilot-Key	Direct	Store rage/dead clicks
POST /ingest/leads	X-StorePilot-Key	Direct	Submit lead capture
POST /ingest/visitors	X-StorePilot-Key	Direct	Register/update visitor

---

Asynchronous Error Processing

Error ingest is the only endpoint that uses async processing via BullMQ:

1. The request arrives at POST /ingest/errors
2. The controller immediately adds a job to the errors-ingest Redis queue and returns { "ok": true } to the client
3. A BullMQ worker (concurrency: 5) picks up the job and:
   • Checks if this error fingerprint already exists
   • Creates or updates the error record (incrementing occurrenceCount)
   • Sends notifications if this is a new error or a re-open
   • Retries up to 3 times with exponential backoff if the database is temporarily unavailable

Benefits:

• The WordPress proxy gets a fast response even during database load spikes
• Retries handle transient database failures transparently

---

Real-Time Dashboard Updates

When a new error or lead is processed by the BullMQ worker, the Platform emits a Socket.IO event to all connected dashboard clients:

new_error   → broadcasts to all users watching the same siteId
new_lead    → broadcasts to all users watching the same siteId

The dashboard receives these events and updates the UI in real time without polling.

---

Multi-Instance (PM2 Cluster)

The Platform runs in PM2 cluster mode (2 instances by default). Because the stats cache is stored in Redis (not in-process memory), all instances share the same cache state. BullMQ also uses Redis and distributes jobs across all worker instances automatically.