Opus – Polling App

Millions of records, sub-second queries The web scraper pulls roughly 10,000 articles daily. With millions of historical records in the database, users still expect instant search results and complex filtering. The database alone couldn't deliver that performance

Complex visualizations over massive datasets Corporate innovation teams need charts showing technology trends, patent landscapes, and competitive intelligence. Each visualization aggregates thousands or millions of data points while staying interactive

The key was a multi-layer caching strategy. Redis stores pre-aggregated statistics, frequently accessed datasets, and query results. When users search or filter, the system checks Redis first before hitting the main database. Query times dropped from seconds to milliseconds

I also implemented background workers using queues to process complex aggregations asynchronously, ensuring the main thread is never blocked

Frontend: Next.js, TailwindCSS, TypeScript

Backend: Node.js, Express

Database: PostgreSQL, Redis

Deployment: Docker, AWS, Nginx