Virtual Internships · 2025
Thumbnail Pipeline
Intern profiles link out to portfolios and certifications, and the browse page rendered a card for each one. A card only had an image if the intern had uploaded one, and most never did — so the page was mostly identical placeholders. I proposed and built a service that stops waiting for an upload and derives the preview from the link itself.
What the API was actually returning
The candidates endpoint returned two fields per row: image_url when one existed, and link. The frontend rendered the image if it was there and a static placeholder if it wasn’t.
Nothing about that was broken, and that was the problem — it worked exactly as written, and the result was a browse page where most cards looked the same. The information needed to do better was already in the response. Nobody was using the link.
The same card, before and after
One intern row, dragged. The two paperclip placeholders on the left are what every card with an un-uploaded image looked like; on the right, the same row after the pipeline resolved both links — one via og:image, one screenshotted.
The constraint that shaped the design
I set one constraint in the proposal before choosing an approach: the frontend must not change. It keeps rendering whatever image_url the API hands it, and generation is entirely the backend’s problem.
That is what rules out the obvious alternatives. Generating client-side would have meant shipping the work to every browser on every render and exposing arbitrary outbound fetches from the user’s session. Adding a second endpoint would have meant a waterfall on a page that renders a dozen cards. Filling the existing field is the only version where the browse page gets better without a single component being touched.
The whole flow, end to end
The beat worth watching is the one where the response leaves before the work happens — the API returns a placeholder and queues the job in the same instant, so nothing blocks on a browser launch.
og:image first, because the web already solved this
Most sites publish an Open Graph image so their links look right when shared. That image is chosen and sized by the site owner specifically to represent the page — it is better than anything I would generate, and it costs one HTTP request to find.
So tier one fetches the HTML and parses og:image out of the head with cheerio, falling back to twitter:image and then a plain meta name="image". No browser, no rendering, no JavaScript execution. In production this resolves ~55% of URLs in about a second.
It is not universal, which is the whole reason there is a second tier: plenty of sites publish no tags at all, and the URL they do publish is sometimes stale or already broken. So the extracted URL gets validated rather than trusted.
A screenshot only when there is nothing to read
Tier two renders the page in headless Puppeteer and captures the viewport at 1200×630 — the same dimensions Open Graph uses, so both tiers produce interchangeable output. It works for effectively any reachable URL, which is exactly what a fallback needs to do.
It is also the expensive path: two to five seconds per capture, real CPU and memory per instance, and it fails in ways the first tier cannot — auth walls, bot detection, pages that never settle. Every one of those properties argues for it being the fallback rather than the default. Making it tier two is what keeps a browser launch off the critical path of an API call, and it is why the fast path runs at roughly an eighth of the fallback’s latency.
When both tiers fail the response still carries the original placeholder. A missing thumbnail is a worse card; it is never a failed request.
Never on the critical path
A request checks Redis first with a 10-minute TTL, then the database on url_hash — a generated BINARY(32) column holding SHA2(url, 256) under a unique index. Lookups are O(1), and because the hash is of the URL rather than the profile, the same link shared across two interns generates once.
On a miss the API writes a PENDING row, enqueues a BullMQ job and returns the placeholder immediately. Nothing waits. The batch enqueue runs through Promise.allSettled so one bad URL cannot take its eleven neighbours down with it, and PENDING rows still sitting there after five minutes get re-queued, because jobs do get lost and a row stuck in PENDING is a card that never recovers.
Fetching URLs a stranger supplied
The uncomfortable part of this design is that it makes a server fetch arbitrary user-supplied URLs, which is a textbook SSRF hole — point it at 169.254.169.254 and it will happily read cloud metadata for you. urlValidator does IP-range and DNS-level checks behind pinned safe HTTP agents: 125 lines of implementation against 434 lines of tests, which is the ratio a security boundary earns.
BrowserPool keeps a launchPromises map so concurrent requests for the same slot await one launch instead of racing to start several. That was closed before it happened rather than after, which is rarer than it should be.
Where it landed
95% success in production, roughly 55% of URLs served by the cheap tier in about a second, and 8× lower latency on that path than the browser fallback. The browse page went from mostly placeholders to mostly real previews.
The frontend diff was empty, which was the point. I would revisit one thing: the service started as a module inside the candidates API because that was the fastest route to shipping, and browser instances are the kind of workload that eventually wants its own process and its own scaling. The seam is drawn for it; it has not needed crossing yet.