Data Synchronization Architecture

This page describes how data flows from your WordPress site to the StorePilot Platform.

Architecture Overview

WordPress Site (Browser)
        │
        ├── JS Errors ──────────────────┐
        ├── PHP Errors (server-side) ───┤
        ├── Lead capture form data ─────┤──► WordPress PHP Proxy
        ├── Pageview events ────────────┤        (WP REST API endpoint)
        └── Visitor identification ─────┘              │
                                                       │  HTTP POST
                                                       ▼
                                             NestJS Ingest Endpoints
                                              api.store-pilot.net
                                                       │
        ┌──────────────────────────────────────────────┤
        │                                              │
        ▼                                              ▼
  BullMQ Queue                              Direct Insert (Prisma → DB)
  (errors only)              pageviews / visitors / recordings / leads
        │
        ▼
  Worker Process
  (upsert + notify)

---

Why a WordPress Proxy?

The browser tracker does not call the NestJS API directly. Instead, it calls a WordPress REST API endpoint that acts as a proxy:

POST /wp-json/wpel/v1/ingest

Reasons for this architecture:

1. Server-side enrichment — PHP can add wp_user_id, WooCommerce cart data, and other server-side context before forwarding
2. CORS avoidance — The browser calls the same domain it is already on
3. IP resolution — The Platform sees the visitor's real IP from within the proxy request, enabling geolocation
4. Nonce validation — WordPress nonce prevents cross-site request forgery for lead submissions

---

Ingest Endpoints

Endpoint	Auth	Method	Description
POST /ingest/errors	X-StorePilot-Key	Async (BullMQ)	Submit JS/PHP error
POST /ingest/pageviews	X-StorePilot-Key	Direct	Record a pageview
POST /ingest/recordings/start	X-StorePilot-Key	Direct	Start recording session
POST /ingest/recordings/segment	X-StorePilot-Key	Direct	Push a recording segment
POST /ingest/recordings/end	X-StorePilot-Key	Direct	End recording session
POST /ingest/recordings/interactions	X-StorePilot-Key	Direct	Store rage/dead clicks
POST /ingest/leads	X-StorePilot-Key	Direct	Submit lead capture
POST /ingest/visitors	X-StorePilot-Key	Direct	Register/update visitor

---

Asynchronous Error Processing

Error ingest is the only endpoint that uses async processing via BullMQ:

1. The request arrives at POST /ingest/errors
2. The controller immediately adds a job to the errors-ingest Redis queue and returns { "ok": true } to the client
3. A BullMQ worker (concurrency: 5) picks up the job and:
   • Checks if this error fingerprint already exists
   • Creates or updates the error record (incrementing occurrenceCount)
   • Sends notifications if this is a new error or a re-open
   • Retries up to 3 times with exponential backoff if the database is temporarily unavailable

Benefits:

• The WordPress proxy gets a fast response even during database load spikes
• Retries handle transient database failures transparently

---

Real-Time Dashboard Updates

When a new error or lead is processed by the BullMQ worker, the Platform emits a Socket.IO event to all connected dashboard clients:

new_error   → broadcasts to all users watching the same siteId
new_lead    → broadcasts to all users watching the same siteId

The dashboard receives these events and updates the UI in real time without polling.

---

Multi-Instance (PM2 Cluster)

The Platform runs in PM2 cluster mode (2 instances by default). Because the stats cache is stored in Redis (not in-process memory), all instances share the same cache state. BullMQ also uses Redis and distributes jobs across all worker instances automatically.