Resilient Transaction API

Problem

Transactional operations must handle concurrency, ambiguous external responses, and dependencies that can fail. This project explores those properties in a payment API, with public code, tests, and measurements; it is not a financial system running in production.

Architecture and decisions

The main flow is HTTP → service-to-service authentication → rate limiting → validation → idempotency claim → provider resilience → persistence → response.

• PostgreSQL is the source of truth. The Idempotency-Key is claimed atomically in the database. Results distinguish a new claim, completed replay, fingerprint conflict, and an operation in progress. The database transaction is not held open during the external provider call.
• Redis is degradable. It supports cache-aside and distributed rate limiting; it is not the source of truth. During an outage, reads continue through PostgreSQL, with timeout cost and reduced rate-limiting protection.
• The provider can fail. The system uses timeouts, bounded retries, exponential backoff, jitter, and a circuit breaker, while classifying definitive, transient, and ambiguous failures. Retries can add latency and cost; an ambiguous failure does not prove that the external operation did not happen.
• Operational visibility. The API records X-Request-Id, redacted JSON logs, Prometheus metrics at /metrics, liveness, and readiness. Shutdown handles SIGTERM and drains in-flight requests.

The local Compose stack runs PostgreSQL, Redis, migrations, and a fake provider. The Docker image is multi-stage, runs Bun as PID 1 and a non-root user, with a read-only filesystem, dropped capabilities, and no-new-privileges.

Local benchmark

The baseline ran in containers on the same Linux x86_64 host, with 4 vCPUs and approximately 3.8 GiB RAM; Bun 1.4.2, PostgreSQL 15.19, Redis 7.2.16, Docker, and a fake provider. Each main scenario had three 300-request runs at concurrency 25, after warmup. These are local measurements, not a production or AWS throughput forecast.

Additional controlled runs: a provider with 100 ms artificial latency had a 109.281 ms p50 and 149.119 ms p95; a 503 → success retry completed in 626.263 ms, with one retry recorded. With 20 concurrent POSTs sharing a key, there was one creation, four completed replays, and 15 responses indicating processing. With Redis unavailable, observed requests returned 200; three runs had p50 values between 192.934 ms and 753.890 ms, a spread that does not support summarizing fallback with one typical latency. The rate-limit test observed 429 with Retry-After after exceeding the configured limit.

Temporary AWS validation lab

Terraform provisioned 35 resources for a real temporary lab in us-east-1. The exercised architecture was HTTPS/ACM → ALB → ECS/Fargate → Bun API → private RDS PostgreSQL / private ElastiCache Redis, with Secrets Manager, CloudWatch Logs, ECR, and IAM as supporting services. The API and deterministic fake provider ran as containers in the same task; a one-off Fargate task ran the migrations.

To reduce lab cost, the tasks ran in public subnets with public IPs, but accepted inbound traffic only from the ALB. RDS and Redis remained private. This was a temporary lab choice, not the production-preferred topology: that keeps ECS tasks in private subnets with deliberate NAT Gateway and/or VPC endpoint egress.

Deployment surfaced two required adjustments: RDS rejected a seven-day backup retention period in the account used, so it was set to one day; and PostgreSQL on RDS required TLS, so sslmode=require was added to DATABASE_URL.

The cycle was closed after testing: ECR was emptied, terraform destroy removed all 35 resources, and Terraform state was empty. The final audit found no ECS, RDS, ElastiCache, ALB, ECR, or CloudWatch Logs resources; the public CNAME was removed. The ALB and all other Terraform lab resources were destroyed. The public ACM certificate was intentionally retained for possible reuse; the secrets were scheduled for deletion.

This was a temporary validation lab and does not represent a production environment. The service ran with desired=1: high availability and rate-limiter sharing across multiple replicas were not demonstrated.

Local validation and limitations

The local container validation report records 164 tests, 662 assertions, zero failures — including 20 PostgreSQL and 11 Redis integration tests — and a real SIGTERM shutdown during a request: the request completed, the process exited without SIGKILL, and shutdown took about 2.31 s.

• The benchmark is local, specific to the hardware described, uses a fake provider, and runs client and server on the same host; it does not represent production, AWS, or multiple replicas. The AWS lab was not used for benchmarking.
• There is no exactly-once guarantee. The external outcome still depends on the provider honoring the same idempotency key.
• Redis fail-open preserves part of service availability but reduces rate limiting protection during an outage; timeouts can significantly increase latency.
• The circuit breaker is process-local and is not coordinated across replicas.
• The lab’s deterministic fake provider does not validate integration with an external payment provider.
• A single-task lab does not demonstrate high availability, multi-replica behavior, or shared rate limiting between replicas.
• The temporary setup used tasks in public subnets. It does not validate the production-preferred topology with private ECS tasks.

Evidence

• Repository: resilient-transaction-api on GitHub
• AWS lab report: aws-lab-real.md
• Sanitized AWS lab artifacts: artifacts/aws-lab
• ADRs: architecture decisions
• Container validation: phase-7v-progress.md
• Benchmark report: phase-8-baseline.md
• Machine-readable local Docker benchmark artifacts: artifacts/benchmarks