How I Overcame LLM Rate Limits to Scale the Sight Pro Dashboard?

2026-09-02

When we won the MLH Hack Days Kanpur with a Google Gemini-powered application, the architecture was held together by caffeine and synchronous API calls. It worked perfectly for a 3-minute live judge demonstration, but when I decided to evolve that prototype into Sight Pro—a full-fledged enterprise AI document parsing SaaS—the original architecture immediately hit a wall.

Here is how I re-architected the Python backend and Next.js frontend to handle heavy unstructured data without dropping client requests.

The Problem: Synchronous Bottlenecks

In the initial prototype, when a user uploaded a document, the Next.js frontend would await a response from our Python backend, which in turn waited for the Google Gemini API to parse the data.

# The Prototype Way (Synchronous & Brittle)
@app.post("/parse")
def parse_document(file: UploadFile):
    text = extract_text(file)
    response = gemini_client.generate_content(text) # Blocks the thread!
    return {"data": response.text}

As soon as multiple users uploaded large PDFs simultaneously, two things happened:

1. We hit the Gemini API rate limits (Too Many Requests).
2. The Vercel-hosted frontend timed out waiting for the Python server, leaving users staring at an infinite loading spinner.

The Architecture Shift: Sync vs Async

To visualize the bottleneck, here is how the architecture changed.

Old Synchronous Flow (Failed at scale):

[Next.js Client] --(Upload PDF)--> [FastAPI Server] --(Wait 30s)--> [Gemini API]
[Next.js Client] <--(Timeout!)---- [FastAPI Server] <--(Response)-- [Gemini API]

New Asynchronous Flow (Sight Pro Production):

[Next.js Client] --(Upload PDF)--> [FastAPI Server] ---> [Redis/Celery Queue]
[Next.js Client] <--(Job ID 123)-  [FastAPI Server]
...
[Celery Worker]  --(Process PDF)-> [Gemini API]
[Next.js Client] --(Poll ID 123)-> [FastAPI Server] <--- [Redis DB]
[Next.js Client] <--(Final JSON)-- [FastAPI Server]

The Implementation: Asynchronous Job Queues

To make Sight Pro enterprise-ready, I had to decouple the data ingestion from the LLM processing. I transitioned the architecture to an asynchronous job queue model.

Instead of waiting for the LLM, the backend now immediately returns a job_id. The Next.js frontend then polls a status endpoint (or uses WebSockets) to check if the job is complete.

1. Redis & Celery/RQ: I introduced a background worker layer. The main FastAPI server simply dumps the parsing task into a queue.
2. Rate Limit Throttling: The background workers are configured to respect the Google Gemini token-per-minute (TPM) limits, intentionally pacing themselves rather than crashing.

Tradeoffs and Failures

It wasn't an immediate success. Initially, I polled the status endpoint from the React frontend every 500 milliseconds. This essentially DDOS'd my own server.

The Fix: I implemented exponential backoff in the React frontend. It checks at 1 second, then 2 seconds, then 4 seconds. This drastically reduced server load while keeping the UI responsive.

Key Lessons Learned

1. Never block the main thread with an LLM call: Generative AI is inherently slow and unpredictable. Treat every LLM request as a background job.
2. UI/UX matters during long waits: Users don't mind waiting 30 seconds for an AI response if you give them engaging visual feedback. We implemented skeleton loaders and GSAP-powered status updates on the dashboard.

If you are a business struggling with scaling your internal GenAI tools or facing API bottlenecks, this is exactly the kind of architecture I build. Feel free to check out my AI integration services or contact me directly to discuss your infrastructure.

---

Frequently Asked Questions (FAQ)

Q: Why use Python for the backend instead of Next.js API routes? While Next.js Edge functions are great, Python remains the undisputed king of the AI ecosystem. Libraries for data ingestion, PDF parsing, and native LLM SDKs are much more robust in Python. Sight Pro uses Next.js for the UI and Python/FastAPI for the heavy lifting.

Q: How do you handle LLM timeouts? By using background job queues. The user gets a tracking ID immediately, and the backend worker retries the LLM call with exponential backoff if the API times out, preventing data loss.