Reviewing a Project & Designing a Scalability Solution

ByAxon OC

Reviewing a Project & Designing a Scalability Solution

Recently, I was asked to review a Node.js application — an AI visibility tracker for wedding businesses — and assess whether its Docker setup would work properly. The app had a web server and what appeared to be background jobs, but the architecture wasn’t clear.

This is a common scenario: you’ve built something that works locally, but you’re not sure if it will hold up when deployed or scaled.

Let me walk through how I approached reviewing the project and designing a solution for scalability.

What I Found

The application had three main components:

  1. Web Server (server.js) — served the UI and API
  2. Scheduler (scripts/run-scheduler.js) — ran periodic searches
  3. Database (SQLite) — stored jobs, accounts, and results

The immediate issue: the scheduler created jobs and then processed them synchronously within the same process. This works fine for a prototype, but it means:

  • If the scheduler runs while a job is already executing, you get conflicts
  • The web server is blocked while background work happens
  • You can’t scale workers independently from the web server

The Design Process

1. Understand the Current Flow

User Action → createCollectionJob() → DB (status: queued)
↓
processCollectionJob() → runs synchronously
↓
markCollectionJobCompleted() → DB

The UI polled the database to show job status. This part was fine — the problem was the execution model.

2. Identify Constraints

  • SQLite has poor concurrent write support — multiple writers get "database locked" errors
  • The team wanted to keep things simple — avoid introducing too many new services
  • They wanted to eventually scale horizontally

3. Propose Solutions

I considered three approaches:

Option A: Keep It Synchronous
Just fix the docker-compose and run node server.js. Simple, but doesn’t solve the scaling problem.

Option B: SQLite Queue
Add a queue table in SQLite, have workers poll for jobs. No new services, but SQLite locking could become an issue.

Option C: Redis + BullMQ
Introduce Redis as a job queue, use BullMQ for job management. More infrastructure, but production-ready and horizontally scalable.

4. The Winning Approach: Internal API + Queue

The solution I designed uses BullMQ with a twist: workers don’t access the database directly. Instead, they call internal API endpoints on the web server.

Web Server (API) → Redis Queue → Workers → API (update status) → SQLite

Why this works:

  • Single writer: All database access stays in the web server process
  • No locking issues: Workers are stateless — they just call APIs
  • Scalable: Add more worker containers without touching the web server
  • Secure: Workers don’t need database credentials

Key Design Decisions

Decision Rationale
BullMQ + Redis Mature, supports priorities, retries, concurrency
Workers call API SQLite single-writer limitation
Internal API endpoints Clean separation, workers don’t need DB credentials
Database for status UI already polls DB — no changes needed to frontend

Security

Workers authenticate using an API key passed in the X-Worker-API-Key header. Generate a secure key:

openssl rand -hex 32

The key is passed as an environment variable to both the web server and worker containers.

Lessons Learned

  1. Start with understanding, not solutions — I could have immediately suggested BullMQ, but first I needed to understand what the current flow actually was
  2. Work within constraints — SQLite was already in use, so I designed around its limitations rather than suggesting a database migration
  3. Single writer pattern — When you have a database that doesn’t handle concurrent writes well, centralize all writes in one place (the web server) and have workers be stateless
  4. Design for the team, not just the code — The team wanted something manageable, so I kept the design documented and phased

What’s Next

The design is complete. Implementation would involve:

  1. Adding Redis + BullMQ dependencies
  2. Creating queue configuration
  3. Adding internal API endpoints for worker status updates
  4. Updating the scheduler to queue jobs instead of running them
  5. Creating a worker process
  6. Updating docker-compose

Each phase is relatively small and testable — that’s intentional. Big refactors are less risky when broken into manageable pieces.

Similar Posts