Building a high-performance payment system — a step-by-step guide

Let’s build a payment system from scratch.

We will not start with a wall of boxes labeled “event-driven microservices” and pretend that is design. We will start with a simple Node.js API that can take a payment, and we will keep asking one question:

What can go wrong with this design?

Every time we find a real failure — a double charge, a lost email job, a provider timeout, a traffic spike — we will add the smallest architectural change that fixes that failure. By the end, the system will look “complex,” but you will know why each piece is there. You will have Node.js, PostgreSQL, database transactions, idempotency, a transactional outbox, Amazon SQS, AWS Lambda, webhooks, retries, dead letter queues, horizontal scaling, rate limiting, circuit breakers, and observability — earned one problem at a time.

Part 1: Let’s build the simplest payment system

Imagine we are building an e-commerce application. A customer picks a product that costs ₹1,000 and taps Pay. The frontend sends a plain HTTP request to our backend:

POST /payments
Content-Type: application/json

{
  "orderId": "order_123",
  "amount": 1000,
  "currency": "INR"
}

Our first architecture is almost embarrassingly simple. The browser talks to one Node.js process. That process writes a row to PostgreSQL and then calls an external payment provider. There is no queue, no second service, no fancy fan-out. For a weekend prototype or a tiny store, this is often where people honestly begin.

In code, the happy path might look like this. We create a payment in PENDING, ask the provider to charge the card, update our row with the provider’s status, and return that status to the client.

app.post("/payments", async (req, res) => {
  const { orderId, amount } = req.body;

  const payment = await db.payments.create({
    orderId,
    amount,
    status: "PENDING",
  });

  const result = await paymentProvider.charge({
    amount,
    orderId,
  });

  await db.payments.update(payment.id, {
    status: result.status,
  });

  res.status(201).json({
    paymentId: payment.id,
    status: result.status,
  });
});

On a good day, with one user and a healthy network, this feels complete. The money moves, the database records it, the UI shows success. The trouble is that real users do not live on good days. So we start looking for cracks.

Part 2: What happens if the user clicks Pay twice?

Picture the customer on a flaky mobile connection. They tap Pay ₹1,000. The browser fires the first request. Our server reaches the payment provider and the charge goes through. Somewhere between the provider and the phone, the connection dies. The browser only sees a network error. From the customer’s point of view, nothing happened — or worse, it looks like the payment failed.

So they tap Pay again. A second request arrives. Our simple API has no memory that the first attempt already succeeded. It creates another payment and calls the provider again. The customer is charged ₹2,000 for a ₹1,000 order. Support tickets follow. Trust evaporates. In a payment system, this is not an edge case you can shrug off — it is a product failure.

The fix is idempotency: a way for the client to say “these two HTTP requests are the same logical payment.” The browser (or our checkout page) sends a stable key with every attempt for that checkout action:

POST /payments
Idempotency-Key: 8f72c1-payment

When the first request arrives, we look up the key, find nothing, process the payment, and store the result against that key. When the second request arrives with the same key, we find the stored result and return it. We do not call the provider again. The customer sees the original success (or the original failure) without a second charge.

This gives us our first hard reliability rule for money:

The same logical request should not produce multiple financial side effects.

A unique index on the idempotency key makes that rule enforceable even when two requests race at the same moment:

CREATE UNIQUE INDEX payments_idempotency_key_idx
ON payments(idempotency_key);

We have not made the system “distributed” yet. We have only made it honest about retries — which humans and browsers will always do.

Part 3: Our API is becoming slow

Assume the charge now succeeds reliably. Product people immediately ask for more: generate a PDF invoice, email the receipt, write analytics events, notify the warehouse, update loyalty points. The natural instinct in a small codebase is to do all of that before returning the response.

app.post("/payments", async (req, res) => {
  const payment = await createPayment();
  await chargePayment();
  await generateInvoice();
  await sendEmail();
  await generateReceipt();
  await updateAnalytics();
  await sendNotification();
  res.json(payment);
});

Add the latencies in your head. The charge might take a few hundred milliseconds. Invoice generation might take longer. Email providers are rarely instant. Analytics can stall. Notifications depend on another team’s API. Suddenly the customer is staring at a spinner for several seconds after they already paid.

Ask the human question, not the engineering one: does the customer need the invoice email to finish before they deserve to see “Payment successful”? They do not. They need to know whether money moved. Everything else is important — but it is not what should block the checkout UI.

So we draw a line. The synchronous path keeps only the critical work: validate, create the payment record, charge, return status. Invoice, email, analytics, and notifications move off that path into background processing. The customer gets a fast answer. The business still gets its paperwork — just a moment later.

Part 4: Where should background work go?

The first idea many teams reach for is a queue inside the Node process itself — an array of jobs pushed after the payment succeeds:

const jobs = [];

jobs.push({
  type: "GENERATE_INVOICE",
  paymentId: "pay_123",
});

On a laptop demo, it works. A background loop drains the array and generates invoices. The problem is where that array lives: in the memory of one Node.js process. If the process restarts for a deploy, runs out of memory, or simply crashes, every job that had not finished is gone. The payment is still in the database. The invoice never starts. Nobody gets an email. You have a silent hole in the business process.

Scaling makes it worse. Put three API servers behind a load balancer and each server gets its own in-memory queue. Jobs are stranded on whichever machine accepted the original request. There is no shared backlog, no shared retry story, and no way for a healthy machine to help a dying one. We need a queue that exists outside any single Node.js process — something durable that all producers and consumers can share.

Part 5: Introducing Amazon SQS

This is where a managed queue like Amazon SQS earns its place. Instead of pushing into process memory, the API publishes a small message describing the work that still needs to happen:

{
  "type": "GENERATE_INVOICE",
  "paymentId": "pay_123"
}

That message now lives in SQS, not inside our Node heap. If the API process dies a second later, the message is still there. If a sale floods us with a hundred thousand invoice jobs, we do not have to process them all before the next HTTP response. We accept the work into the queue and let workers catch up at a sustainable pace.

That separation is the point. The checkout API can absorb a spike of payments. The invoice workers can run cooler and steadier. Producer and consumer no longer have to move at the same speed — which is exactly what you want when human traffic is bursty and PDF generation is not.

The producer and consumer don’t have to operate at exactly the same speed.

Part 6: But who reads from SQS?

Publishing to SQS solves durability. It does not generate invoices by itself. Something has to read the queue and do the work. That “something” is a worker. You can run workers on EC2, ECS, Kubernetes, a long-lived Node process, or a serverless function. For this walkthrough we choose AWS Lambda, because it pairs naturally with SQS and scales concurrency without us babysitting a permanent fleet of invoice servers.

The shape becomes: the Node.js API accepts the payment and publishes a job; SQS holds the job; Lambda pulls the job and runs the side effects. A handler might look like this:

export const handler = async (event) => {
  for (const record of event.Records) {
    const job = JSON.parse(record.body);

    if (job.type === "GENERATE_INVOICE") {
      await generateInvoice(job.paymentId);
    }
  }
};

The customer-facing request no longer waits for PDF rendering. The API’s job is to make the money correct and hand off the rest. The worker’s job is to finish the rest reliably.

Part 7: Why Lambda?

On a normal Tuesday you might see a hundred invoice jobs a minute. On a sale day you might see a hundred thousand. Permanently renting enough worker machines for the sale is expensive. Permanently renting only enough for Tuesday means the sale backlog grows forever. SQS plus Lambda is a practical answer to that mismatch: as the queue deepens, more Lambda executions can start (within your account and concurrency limits). When the queue drains, you stop paying for idle workers.

We are not choosing Lambda because serverless is fashionable. We are choosing it because we have a concrete problem: asynchronous work that spikes hard, must not live in API memory, and should not force us to operate a large always-on worker fleet. If your team already runs workers well on ECS or Kubernetes, the same lesson applies — the queue is the important idea; Lambda is one way to consume it.

We need scalable compute for asynchronous work without maintaining a fleet of worker servers ourselves.

Part 8: What happens if Lambda crashes?

Reliability is not only about the happy path. Suppose SQS delivers a GENERATE_INVOICE job for pay_123. Lambda starts building the PDF. Midway through, the function crashes — out of memory, a bad deploy, a killed container. What happens to the message?

If the message simply vanished, we would be back to silent business failure. SQS avoids that with a visibility timeout. When a worker receives a message, SQS temporarily hides it from other workers. If that worker finishes cleanly and deletes the message, the job is done. If the worker dies without deleting it, the hide period eventually expires, the message becomes visible again, and another worker can pick it up. The invoice gets a second chance instead of disappearing into the void.

That is progress. It also plants the seed of our next problem: retries mean the same job can run more than once.

Part 9: But now we have a new problem

Imagine a nastier timing. Lambda receives the job, performs a financial side effect successfully, and then crashes before it can tell SQS “delete this message.” From SQS’s point of view, the worker never finished. After the visibility timeout, the same message is delivered again. A naive worker will happily run the charge (or another irreversible step) a second time.

This is the heart of at-least-once delivery. Queues in the real world prefer “maybe twice” over “maybe never,” because losing money or losing an order update is usually worse than doing careful duplicate detection. So we stop pretending messages arrive exactly once. We assume they can arrive more than once, and we design the consumer accordingly.

A message can be delivered more than once.

Part 10: Making the consumer idempotent

Every payment already has an identity — say pay_123. Before a worker performs anything irreversible, it asks a boring question: have we already processed pay_123 for this kind of work? If yes, return the stored outcome and stop. If no, do the work and record that it completed.

That check has to be safe under concurrency. Two Lambdas can receive duplicates close together. A unique constraint in the database is a blunt but effective last line of defence:

CREATE UNIQUE INDEX
ON processed_payments(payment_id);

If two workers race, only one insert wins. The loser treats that as “already handled” instead of charging again. Our operating model is no longer “exactly once, please.” It is “at least once on the wire, exactly once in the ledger,” which is what money actually needs.

Part 11: Another serious problem — database and SQS

Return to the API for a moment. The payment write to PostgreSQL succeeds. A moment later we try to publish PAYMENT_CREATED to SQS and the publish fails — network blip, throttling, misconfigured credentials. Now the database believes the payment exists, but no message exists for workers to consume. Invoices never start. Notifications never fire. The system is inconsistent in a way that is hard to spot unless someone complains.

This is the classic dual-write problem. Two systems (database and queue) cannot be updated as one atomic unit if you talk to them separately. We need a pattern that ties the business fact and the “please continue” signal together.

Part 12: Transactional outbox

The transactional outbox solves that dual write by refusing to dual-write in the first place. Inside a single database transaction we insert the payment and we insert an outbox row that describes the event. Either both commit, or neither does.

BEGIN;

INSERT INTO payments (id, order_id, status)
VALUES ('pay_123', 'order_123', 'PENDING');

INSERT INTO outbox (event_id, event_type, payload, processed)
VALUES (
  'evt_123',
  'PAYMENT_CREATED',
  '{"paymentId":"pay_123"}',
  false
);

COMMIT;

After commit, a separate publisher process (or a scheduled job) reads unprocessed outbox rows and pushes them to SQS. If SQS is down, the rows stay in PostgreSQL. Nothing is lost. When SQS recovers, the publisher catches up. Workers still do the heavy lifting; the outbox is only the reliable bridge between “we recorded the business fact” and “someone should continue the workflow.”

In plain language: we stopped trying to make Postgres and SQS agree in one breath. We made Postgres the source of truth for “this payment happened,” and we made publishing a retryable follow-up.

Part 13: But the outbox has a problem too

The outbox is not magic. Suppose the publisher reads an event, successfully sends it to SQS, and then crashes before it can mark the outbox row as processed. On restart, the row still looks pending, so the publisher sends it again. SQS now has duplicates.

That does not mean the outbox failed. It means we have to be precise about what it guarantees. The outbox prevents lost events after a successful database commit. It does not invent exactly-once delivery on the queue. Duplicate detection still belongs in the consumer — the same discipline we already needed for Lambda crashes. The two pieces work as a pair: outbox for durability of intent, idempotent consumers for safety under replay.

Part 14: External payment APIs can fail too

Sooner or later a worker (or the API) must call the payment provider. That call is a trip across someone else’s network. You will see every flavour of answer: 200 OK, 400 for bad input, 401 for auth mistakes, 429 when you are too eager, 500 and 503 when their side is unhappy, timeouts when nobody answers, and plain network failures when packets vanish.

Hoping for perpetual 200s is not a strategy. We put a hard timeout on every provider call so a hung socket cannot hold a worker forever. We retry only the failures that look temporary — not permanent validation errors that will fail the same way on attempt fifty. Retries wait longer each time (one second, then two, then four, then eight) and add a little randomness (jitter) so a thousand workers do not wake up in lockstep and stampede the provider again.

Even with that discipline, one scenario remains uniquely dangerous for payments. We turn to it next.

Part 15: The payment timeout problem

Lambda calls the provider. On the provider’s side, the charge succeeds. On the way back, the response is lost. Our side only sees a timeout. We are now in Schrödinger’s payment: money may or may not have moved, and our logs are uncertain.

If we “recover” by creating a brand-new payment request with a new identity, we risk charging the customer again. The safe move is to retry with the same idempotency key the provider already saw. A well-behaved provider recognizes pay_123, refuses to create a second charge, and returns the original result. Our worker can then update state with confidence.

Request #1
Idempotency-Key: pay_123
  → payment succeeds on provider
  → response lost on the wire

Retry
Idempotency-Key: pay_123
  → provider recognizes the request
  → returns the existing result

Idempotency is not only a gift we give our own API. It is a contract we need with the payment provider, because the most expensive failures hide in the gap between “they did it” and “we heard about it.”

Part 16: How does the payment provider tell us the final result?

Many providers do not finish in the first HTTP round trip. They accept the payment, return something like PENDING, and finish authentication, risk checks, or bank confirmation later. Polling forever from our side is clumsy. The usual contract is a webhook: when the provider knows the final state, it calls us.

{
  "paymentId": "pay_123",
  "status": "PENDING"
}

Later, their servers send something like:

POST /webhooks/payment

{
  "eventId": "evt_123",
  "eventType": "payment.completed",
  "paymentId": "pay_123"
}

Our Node.js API receives that call and updates the payment. From the customer’s perspective, checkout may have already shown “processing,” and a moment later the order flips to paid. Architecturally, we have accepted that the truth can arrive on a second channel, asynchronously, after the original request ended.

Part 17: Can we trust the webhook?

We should not. POST /webhooks/payment is a public URL. Anyone who discovers it can POST a JSON body that claims payment.completed for an order they never paid. If we trust the body blindly, we mark orders as paid for free.

Providers solve this by signing the payload with a shared secret. We recompute the signature (often an HMAC over the raw body and a timestamp) and compare it to the header they sent. If it matches, we continue. If it does not, we reject the request and log the attempt. Signature verification is not ceremony. It is the difference between “the bank said this” and “a stranger said this.”

Part 18: What if the webhook arrives three times?

Even a legitimate provider will retry. If our server was slow, returned 500, or never answered, they will send evt_123 again — and again. Exactly-once delivery is still a fantasy. So the webhook handler needs the same mindset as the queue consumer: verify the signature, look up the event id, and if we already processed it, return success without applying the side effect twice.

A duplicate webhook is usually not an application error. Treating it as one only creates noise. Keep the handler fast: verify, dedupe, enqueue any heavy follow-up work, and return 200 quickly. If you generate invoices inside the webhook request itself, you invite timeouts — which invite more duplicates. The pattern should feel familiar by now, because it is the same lesson wearing a different coat.

Part 19: Now traffic gets huge

The system works for ordinary days. Then marketing launches a sale. Steady traffic that was a thousand requests per second becomes fifty thousand. One Node.js process will melt: CPU climbs, event loop latency spikes, health checks fail, and customers see errors on the worst possible day.

So we put a load balancer in front and run several Node instances — three at first, then ten, then fifty if we must. That is horizontal scaling: more copies of a mostly identical service, not one giant machine. For this to work, the API has to stay mostly stateless. If you stash the “current payment” in a process variable, the next request may hit a different server and find nothing. Session-ish or critical state belongs in PostgreSQL or another shared store that every instance can see.

Part 20: API scaling vs worker scaling

Here is a subtle win from introducing the queue earlier. The number of HTTP servers you need and the number of background workers you need are not the same number. Checkout might need twenty Node instances to accept payments quickly. Invoice generation during the same hour might need two hundred concurrent Lambda executions. Without a queue, those pressures fight inside the same process. With SQS, they scale on separate knobs. You grow the API for click latency. You grow workers for backlog depth. That independence is one of the main reasons the architecture got a queue in the first place.

Part 21: What if the payment provider can’t handle our traffic?

Your platform might accept fifty thousand requests per second. Your payment provider might only allow five thousand. If every accepted checkout immediately becomes a provider call, you will collect 429 responses and timeouts. Aggressive retries then amplify the damage: the provider stays overloaded, your workers stay busy failing, and the queue fills with poison that keeps coming back.

The queue is again an ally if you use it with restraint. Let SQS absorb the burst. Cap how many workers may call the provider at once. Prefer controlled concurrency over “scale Lambda to the moon.” Protecting a downstream dependency is part of being a good citizen in a payment ecosystem — and part of protecting your own success rate.

Part 22: What if the payment provider is completely down?

Now imagine the provider is not slow — it is gone for ten minutes. Without defence, every request fails, retries, fails, retries. Thousands of workers do this in parallel. You build a retry storm that helps nobody and may delay recovery when the provider returns.

A circuit breaker changes the behaviour. While the dependency looks healthy, calls flow. After repeated failures, the circuit opens: we stop calling for a while, fail fast or queue work deliberately, and periodically probe with a small test. If the probe succeeds, we close the circuit and resume. If it fails, we stay open. The goal is simple: stop hammering a dead dependency so both sides can recover cleanly.

Part 23: What if a job keeps failing?

Some messages will never succeed. The payload is invalid. There is a bug that throws on a certain shape of event. The provider permanently rejects the payment method. Endless retries burn money and hide the real issue under a sea of identical errors.

So we set a maximum number of attempts. After that, SQS (or our worker framework) moves the message to a dead letter queue. The DLQ is not a trash can you ignore. It is a waiting room for engineers: inspect the payload, fix the bug or the data, and replay deliberately. Production systems need a place for “this needs a human,” not only a place for “try again.”

Part 24: The database eventually becomes a bottleneck

You can add Node instances all afternoon. They still share one primary database for payment truth. At high throughput, the database becomes the conversation everyone should have been having earlier: are the hot queries indexed? Are connections pooled so we are not opening a new TCP session per request? Are transactions short, or are we holding locks while we call the internet? Is lock contention showing up on the payments table during the sale?

Read replicas can help for read-heavy screens. They are dangerous for fresh payment status. If a payment just flipped from PENDING to SUCCESS on the primary, a replica that is a second behind may still tell the customer they have not paid. For financial reads where correctness matters more than a few milliseconds, prefer the primary or a consistency strategy you can explain. Stale money state is not a performance win.

Part 25: What about Redis?

Not every read is a payment status check. Product details, feature flags, and other slowly changing data can live happily in Redis. The API checks the cache first; on a miss it loads from PostgreSQL and fills the cache. Checkout feels snappier, and the database spends less time answering the same question.

Be deliberate about what you refuse to cache. Payment status, capture state, refund state — these are places where a stale cache entry can lie to a customer or to your own fulfillment logic. In those paths, a slower correct query is better than a fast wrong one.

Stale data can be more dangerous than a slower query.

Part 26: Protecting the API with rate limiting

Even with a beautiful interior architecture, POST /payments should not accept infinite traffic from one client. Bots, buggy mobile apps, and hostile actors will find the endpoint. Rate limiting returns 429 when a caller exceeds a budget, and lets ordinary customers through.

With multiple Node instances, that budget cannot live only in one process’s memory. Otherwise instance A thinks the client already made ten requests while instance B thinks they made zero — and the client routes around your limit by sheer luck of the load balancer. Keep counters in a shared store such as Redis so every instance tells the same story.

Part 27: Observability

By now a single payment can travel a long road: browser, load balancer, Node.js, PostgreSQL, outbox publisher, SQS, Lambda, payment provider, webhook back into Node.js, and another database update. When a customer writes “my payment is stuck,” you cannot SSH into one box and “just know.” You need identifiers that follow the journey — requestId, orderId, paymentId, eventId, idempotencyKey — and you need metrics that show where the road is clogged.

Watch the API for traffic, latency, and error rates. Watch payments for success, failure, pending age, and retries. Watch SQS for depth and how long messages sit. Watch Lambda for errors and throttling. Watch the database for slow queries, connection saturation, and lock contention. Observability is not a dashboard you buy after the first outage. It is how a distributed payment system remains operable by humans.

Part 28: Our architecture has now evolved

We began with a straight line: client, Node.js, database, payment provider. We ended with a system that can accept payments quickly, finish side work asynchronously, survive crashes and retries, talk safely to a provider, accept asynchronous truth over webhooks, and scale the HTTP tier separately from the worker tier.

The diagram looks busy if you drop into the middle of it. It looks inevitable if you walked here from Part 1. We did not add boxes because they sounded like a conference talk. We added them because something concrete went wrong, and the smaller fix was this component rather than a prayer.

Part 29: Why does each component exist?

If you ever have to defend this architecture in a design review, do not recite brand names. Recite the failure each piece prevents. The table below is the short map of that argument.

Part 30: The mental model

The finished picture looks complicated. The thinking that produced it is not. Whenever you design a system that moves money — or anything else that cannot be wrong quietly — keep returning to four questions in ordinary language.

First: what happens if this step fails halfway through? Second: can the same step run twice because of a retry, a double click, or a redelivered message? Third: if it runs twice, is that safe, or did we just invent a second charge? Fourth: what happens when traffic is ten times today — does the critical path stay short, and do the buffers absorb the shock?

For payments those questions are not academic. Networks fail. Processes crash. Providers timeout after succeeding. Customers tap twice. Sales multiply traffic. The architecture we built is simply the trail of answers to those failures.

Good system architecture is often the result of repeatedly asking “what can go wrong?” and then designing the system so that failure does not produce an incorrect result.

If you are building checkout, wallets, or marketplace settlement and want a path that survives spikes without inventing double charges, talk to us. We start with the simple system too — then we keep breaking it on purpose until the money stays correct.