Back to Projects
System RedesignStripe • BullMQ • MongoDB • Race Conditions

From one mover to N: scaling payouts.

A walkthrough of how a moving & storage platform refactored its synchronous, single-employee Stripe payout into an idempotent, queue-backed multi-employee payout — and the duplicate-charge race condition that forced the rewrite.

01 — Single-mover origin

One booking, one mover, one synchronous transfer.

Single-employee payout · synchronousBlocking

BO UI

Backend

GQL proxy

Supplier Payments

awaits Stripe

sync

Stripe API

write

Main DB

bookings · transfers

write

Payments DB

audit log

Worked because

One booking → one mover. A single payout per booking meant a single Stripe transfer, a single DB row, and a single UI request. Synchronous was simple and easy to reason about.

Stopped working when

We supported multiple movers per booking. Now the same UI could fire several transfer requests for the same booking — and the backend had no defense against two of them landing at the same moment.

02 — The race condition

Two requests, no atomicity, $100 orphaned in Stripe.

Request A

user double-clicks Pay

  1. Check existing transfer

    transfer = null

  2. Create Stripe transfer

    +$100 to mover_1

  3. Save to DB

    transfer_A persisted

  4. Return success

    request A done

t=0ms

Request B

2ms later

  1. Check existing transfer

    transfer = null

  2. Create Stripe transfer

    +$100 to mover_1

  3. Save to DB

    transfer_B overwrites A

  4. Return success

    both look fine

t=2ms

03 — Async redesign

Enqueue, return, let the workers chew.

Async path

  1. UI POSTs createBulkTransfers

    { bookingId, userIds? }

  2. Backend validates & upserts pending rows🔒 idempotent

    unique (bookingId, userId)

  3. Enqueue N transfer jobs⚙ bullmq

    one per mover, dedup key = userId

  4. Backend returns processing immediately⚡ async

    no waiting on Stripe

  5. Workers pull jobs in parallel

    transfer_worker × N

  6. Each worker hits Stripe & updates status

    pending → processing → success

  7. UI polls or refreshes for resolution

    partial success/failure handled

BullMQ workers

N transfer jobs. N workers pull in parallel. Each worker is idempotent — if it crashes, the job retries against the same dedup key.

W1
W2
W3
W4

transfers (Payments DB)

unique (bookingId, userId)
mover_1
mover_2
mover_3

The unique (bookingId, userId) index turns the race into a no-op: the second insert fails fast, the second job exits without touching Stripe. Workers process in parallel, the UI never blocks on the slowest mover, and partial failures isolate to the single transfer that broke instead of poisoning the whole batch.

04 — Phased rollout

Four phases, two services, zero customer-facing breakage.

  1. PHASE 01

    Database & infrastructure

    supplier-paymentsschema · redis · queue
    • add userId, amount, version to transfer schema
    • compound unique index (bookingId, userId)
    • add processing status for optimistic UI updates
    • wire redis module for connection pooling
    • stand up BullMQ queue + worker module
  2. PHASE 02

    API surface

    supplier-paymentsgraphql · stripe service · migration
    • createBulkTransfers(bookingId, userIds?) — userIds undefined = "Pay All"
    • getTransfersForBooking(bookingId) for N transfers per booking
    • retryTransfer(transferId) for failed-row recovery
    • atomic upsert on stripe-db service
    • migration to backfill userId on existing transfer rows
  3. PHASE 03

    Backend integration

    us-nest-backend-apigql proxy · job service · types
    • GQL proxy layer exposes the new bulk + retry endpoints
    • FlexSuppliersJobService updated for transfers array
    • PaymentSummary type for UI rollups + backwards compat
    • dependent services migrated off the boolean transfer signature
  4. PHASE 04

    UI changes

    us-nest-backend-api · clientmodal · status chips · jobs table
    • per-employee status component — grey · amber · emerald · red
    • PaymentModal: "Pay All" → "Pay Rest" with already-paid disabled
    • jobs table indicator + click-to-open detail modal
    • GQL schema additions for transfers array + PaymentSummary
    • partial success/failure display with retry per-row

05 — Payoff

Pay All. In parallel. With per-mover status.

Booking #BK-19284

Pay crew for completed move

Total payout

$1,800.00

SC

Sarah Chen

Lead

$520Pending
MR

Marcus Rivera

Mover

$410Pending
DP

David Park

Mover

$410Pending
LW

Lisa Wong

Driver

$460Pending

Tap Pay All to enqueue transfers for the unpaid crew.

Status legend

pendingPending
processingProcessing
successPaid
failed!Failed

What changed

Pay All enqueues N jobs and returns immediately. Each row reflects its own worker's progress. Already-paid movers are disabled; failures isolate to a single row with a retry option.