Virtual Internships · 2025
Thumbnail Pipeline
Intern profiles link out to portfolios and certifications, and the browse page showed a card for each. A card only had an image if the intern had uploaded one — and most never did, so the page was mostly empty boxes. The fix was to stop waiting for an upload and generate the thumbnail from the link itself.
Two tiers
The pipeline is two-tier because most of the web already solves this problem for you: sites publish og:image. Extracting it with cheerio costs one HTTP request and no browser, and it covers the majority of links on its own.
Puppeteer handles the remainder — the pages with no og:image to read. It is the fallback rather than the default, which is what keeps a browser launch off the critical path of an API call.
Never blocking
Everything else is about never blocking. Redis with a 10-minute TTL, then a database lookup on url_hash — a generated BINARY(32) column holding SHA2(url, 256) with a unique index, so lookups are O(1) and duplicate URLs across profiles generate once. On a miss the API writes a PENDING row, enqueues a BullMQ job and returns a placeholder immediately.
Promise.allSettledon the batch enqueue so one failure doesn’t take the others with it. Stale PENDING rows re-queue after five minutes, because jobs do get lost.
Two things worth pointing at
Fetching arbitrary user-supplied URLs server-side is an SSRF hole, so urlValidator does IP-range and DNS-level validation behind safe HTTP agents — 125 lines of implementation, 434 lines of tests.
And BrowserPool keeps a launchPromises map to deduplicate concurrent launches for the same slot: a race closed before it happened rather than after.