ContentsWho this is for: people who want long-running data collection on GitHub Actions without renting a server; people designing a crawl → dedupe → render → LLM pipeline; people wondering how to spread one analysis across several days on a free quota.
Bottom line first: the hard part of InterviewAtlas was never writing crawlers. It was three things — deciding what is new before sending the request, making the writes idempotent, and turning model calls into a budgeted, cacheable two-stage pipeline. What actually held it back was not anti-bot defences either: it was two silent failures — a collector that swallowed exceptions, and an analysis step blocked at compile time by a Python 3.11 f-string restriction.
Reading time: ~28 minutes · Level: advanced
Interview reports have a short half-life. A report from the autumn 2026 hiring season is down to its process skeleton by spring; the actual questions and the interviewers' style have turned over. They are also scattered across systems that do not talk to each other: Nowcoder's forums, Juejin's columns, Mianshiya's question banks — each with its own search, its own login wall, and site search that can only answer "contains this keyword", never "what did this company ask for this role recently".
InterviewAtlas is an attempt at that aggregation problem: crawl three sites daily, normalise company and role, dedupe and merge by URL, render a README plus per-company-per-role Markdown, then use a model to summarise each group into four sections — interview process, high-frequency topics, representative questions, preparation advice. Everything runs on GitHub Actions: no server, no database, and the model calls go through GitHub Models' free quota.
At the time of writing the repository holds 41,771 records, 157 companies and 1,634 company-role files; data/interviews contains 158 .json.gz files (35.9 MB) and interviews/ holds 1,791 Markdown files (92.3 MB). The source is 22 Python files, with exactly four runtime dependencies: requests, bs4, lxml, PyYAML. What follows is not a feature list — it is the design reasoning and the holes I actually fell into.
src/main.py contains no crawling logic; it only orchestrates. It calls each source in configured order, hands results to normalisation, storage and rendering, then prints how many records were added and how long it took. The real code sits in two layers:
src/main.py orchestration and timing
├─ sources/nowcoder.py Nowcoder: search + detail
├─ sources/mianshiya.py Mianshiya: bank list → questions
├─ sources/juejin.py Juejin: deep pagination + body
├─ normalize.py company/role detection, cleaning, relevance
├─ storage.py merge, shard by company, write
└─ render.py README and per-group Markdown
src/analyze.py separate process: reads stored data, calls GitHub Models
Analysis is a separate process rather than a function call after collection, for three reasons. Isolation: rate limits, exhausted quota or empty responses must not rot data that has already been collected in memory. Runtime: collection takes thirty minutes, and analysis on a free quota may not finish the same day — it must be resumable tomorrow. Cache: analysis output is written to analysis/ and committed, so it is repository state like the data itself, not something to hold in one run's memory.
Skipping the database comes from the same reasoning. Actions runners are ephemeral, so the only durable state is git; and 41,771 records read fully into memory are nowhere near needing an index. The price of that choice is the write amplification discussed below.
The orchestration layer also does something easy to overlook: it prints each source's duration separately. The daily log shows record counts and seconds for Nowcoder, Mianshiya and Juejin. That is performance telemetry and a failure signal at once. Mianshiya finishing in one second next to the other two taking a dozen minutes is jarring, and the contrast is itself the anomaly; with only a total runtime, a missing ten minutes hides inside thirty.
A report is one JSON object. Few fields, but each has a job:
| Field | Meaning | Used for |
|---|---|---|
id | URL digest, or source plus external id | Stable identity |
source | nowcoder, mianshiya, juejin | Source statistics |
external_id | The site's internal id | Cross-bank merge in Mianshiya |
title / url | Title and original link | Display, dedupe key |
company / position | Normalised company and role | Sharding, grouping |
content | Cleaned body text | Dedupe, model input |
published_at | Publication date | Ranking the "newest half" |
kind | report, question, bank | Content shape |
first_seen / last_seen | First and latest sighting | Retention pruning |
Normalisation has three layers. First, company aliases: COMPANY_ALIASES matches longest string first, so "阿里云" (Alibaba Cloud) never loses to "阿里" (Alibaba). Second, role detection in a fixed order — backend, frontend, algorithms, testing, product — falling back to "other role". Third, the fallback layer: unknown companies and roles both land in "other", and those are excluded from LLM analysis because they cannot support a meaningful summary. The final 157 companies and 1,634 company-role pairs are what survives those three layers.
The alias table looks dumb and is the precondition for the dataset being usable at all. One company appears as "字节", "字节跳动" and "bytedance" across three sites; without normalisation you get three thin sets of statistics. Yet "阿里云" and "阿里巴巴" must stay separate, or the cloud unit's roles get diluted by group-level reports. Longest-match-first solves exactly this containment problem: try the long alias, stop on a hit, and let short aliases fire only when nothing else matched.
The three sites expose completely different interfaces, so the strategies differ too:
| Source | Entry point | Pagination ceiling | Body text from |
|---|---|---|---|
| Nowcoder | POST gw-c.nowcoder.com/api/sparta/pc/search | 3 pages × 20 per keyword | Search truncates; detail completes |
| Juejin | search_api cursor deep pagination | Pages until there is no next cursor | web_html_content in search results |
| Mianshiya | Bank list → question list | Hard limit of 5 pages per query | Question stem + bank description |
Nowcoder's coverage comes from a keyword matrix: ten generic terms plus 157 companies times two templates ("company interview report", "company interview experience") — well over three hundred queries, each capped at three pages. This necessarily produces duplicates, but the search API offers no stable full-scan cursor, so cross-coverage through keywords is the only option. Search returns truncated bodies, so every unseen record gets one more detail request before storage; details run on six threads with a 0.2-second per-thread interval, which keeps total runtime down without hammering the site.
Juejin supports cursor pagination, so paging is not the bottleneck — body formatting is. The search API's web_html_content is an escaped string: quotes, newlines and backslashes each carry an extra layer, and standard JSON decoding does not yield usable HTML. The code contains a hand-written unescape routine that restores the string segment by segment, then hands it to the shared cleaner. No library helps here, because the payload's escaping sits on top of JSON's own escaping — decoding twice eats legitimate backslashes in the body. It is code you only dare write once you know exactly what the platform emits.
Mianshiya is a third shape: walk the bank list, then expand each bank into questions. The question endpoint has a hard limit of five pages per query. To get more data, the collector runs three variants per bank — no filter, difficulty:1, difficulty:3 — turning "five pages per query" into "three groups of five pages". The cost is triple the requests, with overlap between the groups that needs local deduplication by question id. And it only relieves the ceiling; it does not remove it.
Anti-bot handling is uniform: a Chrome/124 User-Agent, a per-site Referer, Retry(total=3, backoff_factor=1.0, status_forcelist=[429, 5xx]), and a global 0.5-second request interval. No browser automation, no login handling — so only publicly visible content is covered. That is the capability boundary, and the reason the site does not block us.
Re-crawling all three sites daily is impossible: three hundred keywords on Nowcoder, a dozen deep pages on Juejin, a thousand banks on Mianshiya — the request count alone blows the budget. The pipeline's first act is load_records(), which reads all history back and builds two sets: known_urls and known_content (bodies of 300 characters or more only).
When the collector gets a search result it first computes the normalised URL. A hit in known_urls means skip — not even a detail request. A URL miss whose body exactly matches something in history is also skipped. What this saves is the expensive detail request, not the search request: searches still page through, but detail calls only land on new or incomplete content.
Merging is by URL, and merge_records() behaves as "old record wins, new record fills blanks": title, source, company and role are overwritten only when missing, the body is replaced only when the new one is longer, and last_seen is refreshed every thirty days rather than daily — otherwise every shard gets rewritten every day because of one timestamp field. The returned new_count and new_records feed the README's "this run" block. Retention is retention_days: 365; anything unseen for a year is dropped.
Deduplication happens in three layers. First, URL normalisation: strip the # fragment, the query string and the trailing slash, and lowercase — so a shared link and a tracked link to the same post collapse into one key. Second, Mianshiya's cross-bank merge: one question can be indexed by several banks, merged by external_id, preferring the bank that lets us identify a company. Third, before the model sees anything, _dedupe() filters again on "title + first 120 characters of body", because different sources repost the same report — different URL, near-identical content, invisible to the first two layers.
def record_key(record: dict) -> str:
url = (record.get("url") or "").split("#")[0].split("?")[0].rstrip("/").lower()
if url:
return url
source = record.get("source") or ""
external_id = record.get("external_id") or ""
return f"{source}:{external_id}" if (source or external_id) else (record.get("id") or "")
The payoff is direct: daily runs only handle increments, and detail requests are spent only on genuinely new posts. The cost is that every run decompresses, deserialises and indexes 35.9 MB of compressed data — a fixed startup cost. One order of magnitude more data and this would need lazy per-company loading.
One detail decides whether "incremental" is trustworthy: the content set only accepts bodies of 300 characters or more. A very short body may be a truncation artefact, and using it as evidence of "already crawled" would permanently lock an incomplete record in place — later visits to the same URL would never fill it in. The threshold keeps incomplete records out of the known set, leaving room for completion.
Storage is sharded by company — one .json.gz each, not one big JSON. The reason is practical: GitHub caps a single file at 100 MB, and 41,771 records in one file would hit it eventually; sharded, every file stays small and _meta.json holds only totals. load_records() reads both the legacy single-file format and the new directory format, preferring .json.gz, so a company is never read twice during migration.
Writes are idempotent in two layers. First, _write_if_changed(): identical content is never written, so no spurious git change. Second, gzip.compress(raw, mtime=0): the timestamp in the gzip header is pinned, so the same content produces the same bytes — otherwise every compression differs and git reports all shards modified every day.
raw = (json.dumps(payload, ensure_ascii=False, separators=(",", ":")) + "\n").encode("utf-8")
data = gzip.compress(raw, mtime=0) # same content → same bytes, no meaningless diff
if path.exists() and path.read_bytes() == data:
return
Rendering produces two things: the root README and interviews/<company>/<role>.md. The README is an index — company list, role list, totals, per-source statistics and "this run" — and the split files are the human-readable bodies. The catch is that the README is fully re-sorted daily: one new record can change company ordering, role counts and the update time, so the whole page is rewritten. Measured daily README diff: about 3,404 lines, far more than the data increment itself.
Write amplification is the most expensive hidden cost of this design. A daily commit is 53,614 insertions and 41,513 deletions; the weekly run with --full reaches 541,773 insertions and 372,746 deletions; the initial commit was 2.03 million lines. The repository is 348 MB while the current snapshot is about 128 MB — the difference is mostly blobs from whole-file rewrites throughout history. Git compresses, but it does not forget.
It is worth stressing how little new information those diffs contain. A few hundred new reports per day may be a few hundred lines; the remaining fifty-something thousand lines are whole sections moving because a sort position, a count or a timestamp changed. Git's diff algorithm records that as deletions plus insertions, and history grows. Idempotency can guarantee "do not write when nothing changed"; it cannot prevent "one thing changed, the page reshuffled".
Analysis has a threshold, min_records: 20: a company-role pair needs at least twenty reports to be worth summarising, otherwise the model can only invent. Groups that qualify get a group-level report first, then group reports roll up into a role-level report — textbook map-reduce.
I deliberately did not take the easy route of stuffing every report into one prompt. 41,771 bodies far exceed any context window, and even if they fit, cost and latency would be uncontrollable. Map-reduce splits the problem into "eight reports per company-role" and "sixteen companies per role"; both inputs stay budgetable. The cost is information loss across two truncation layers — anything the group report omitted is gone at role level.
The group prompt's input is selective: company, role, total count, sample time span, and at most eight samples. Half the samples are the longest bodies (high information density), half the newest (closest to current process); each is clipped to 1,200 characters and the whole block to 16,000. The prompt mandates four headings and asks the model to write "insufficient sample" rather than invent process steps, salaries or rounds.
Requirements:
1. Use only the material provided. Do not invent company process, salary, interview rounds, or questions absent from the material. If the material is insufficient, write "insufficient sample".
2. Use level-2 headings (##) and output exactly these four sections: process and rounds, high-frequency topics, representative questions, difficulty and preparation advice.
3. "Representative questions" must be a bulleted list of 5-10 items, preserving the original wording where possible.
Call parameters are temperature: 0.3 and max_tokens: 1600. Low temperature keeps two runs over the same material close together, which also makes cache decisions easier; 1600 tokens is enough for four sections of 400-800 characters and discourages drift. The clipper _clip() cuts on paragraph boundaries — newline, full stop, exclamation, question mark — and appends "(truncated)" so a sentence is never cut in half.
There are two quality guards. _missing_sections() checks that all four headings exist; if one is missing it retries once with an explicit "these four headings are mandatory" instruction. If it is still missing, no further retry: a note is prepended to the body saying which sections are absent, so readers know this is a degraded artifact rather than a complete report.
Role level is the second stage: collect each company's group reports for a role, clip each to 700 characters, allow at most sixteen companies, and skip the role entirely below three. It never looks at raw reports, only at summaries already produced, which bounds the input and avoids reading the same originals twice.
Sample selection has one more fallback: after taking "longest half + newest half", if there are still fewer than eight, keep adding from the remaining longest records up to the cap. Small groups therefore do not end up with two or three samples just because the newest half is all short posts, and length still prioritises information density. The three caps — eight samples, 1,200 characters, 16,000 characters — are not arbitrary: together they fix the per-prompt token budget, which fixes how many groups 100 calls can cover.
Analysis is not recomputed every run. analysis/manifest.json records PROMPT_VERSION, the model name, and per-group counts and generation time. PROMPT_VERSION is currently 2026-10-29.1; if either the prompt version or the model name changes, the whole cache is invalidated and everything is recomputed. That is blunt but correct: once the prompt changes, old reports no longer represent the current standard, and partial reuse would leave two standards in the repository.
A single group is recomputed after refresh_after_days: 30, or once its count grows by max(5, 10%). The first prevents stale material, the second ensures a re-call only happens when enough new samples justify it; for one or two new reports the benefit is below the cost.
Budget is the other hard constraint. The free tier is about 15 RPM, so the config is conservatively rpm: 12, workers: 2, max_calls_per_run: 100. A hundred calls means one day covers only part of the groups, and the rest rolls to tomorrow. Hence the budget split: reserve = min(roles, max_calls // 6) calls are held for role-level summaries before company groups get anything, so groups cannot starve the rollups forever.
Failures are routed by status code. On 429, read x-ratelimit-remaining: zero means the daily quota is gone, so throw QuotaExceeded and stop the whole round instead of retrying pointlessly; non-zero means read Retry-After and retry after at most 60 seconds. A 403 whose body mentions rate limiting is treated the same as quota exhaustion; other 401/403 cases are permission errors pointing at models: read. 5xx gets exponential backoff, three attempts. The point of routing is to separate "waiting will help" from "do not try again today".
Cache decisions happen before the call, in this order: read the manifest, compare prompt version and model name; then per group compare count and generation date; only when both pass is the group skipped. That order ensures a changed version cannot accidentally reuse an old report just because the count did not move. The manifest doubles as an observability surface: which groups were analysed, with which model, when — no need to dig through Actions logs.
Two workflows, one daily and one weekly. daily.yml runs on cron 0 1 * * * — 09:00 Beijing time — with timeout-minutes: 60; weekly-full.yml runs Sunday 10:00 with --full --limit 150 and a 180-minute timeout. --full enables more pagination and more company query templates; --limit caps detail completion per source so the weekly run does not drain the quota in one go.
Permissions are only two: contents: write for commits and models: read for GitHub Models. Credentials come from the repository's own GITHUB_TOKEN, with no extra secret, so forks and pull requests never get write access. Concurrency uses group: interview-crawl with cancel-in-progress: false: when daily and weekly collide they queue rather than cancel each other — collection is idempotent, but a mid-flight cancellation leaves half a dataset.
permissions:
contents: write
models: read
concurrency:
group: interview-crawl
cancel-in-progress: false
The commit step first runs git status --porcelain -- README.md data/ interviews/ analysis/ and exits 0 when there is nothing; only when there are changes does it git add those four paths and commit. Note the absence of git add -A, which keeps temp files, logs and local config out of the repository. The analysis step is marked continue-on-error: true, on the theory that quota or permission trouble should not block committing collected data. That decision was reasonable in itself, and it later amplified the second incident.
Step order matters too: collect, then analyse, then commit once. That way a failed analysis still commits collected data; the reverse could end the run early and lose both. The price is that analysis shares the job with collection and its runtime stacks behind it, inside the same 60-minute timeout. Daily runs currently sit just over 30 minutes, enough headroom — but not a structure that can absorb unlimited new sources.
One day's Actions run (41792215767) logged exactly two lines for Mianshiya: Mianshiya: 0 banks, expanding questions and Mianshiya done: 0 banks, 0 records. From commit 116976f8 onward those two lines appeared every day, for days. More confusingly, the whole pipeline was green: no failed step, no alert, the README updated as usual, and Mianshiya still showed 1,962 records in the source statistics.
The investigation started from "this does not look right": a full bank traversal cannot finish in one second, and the daily run was still thirty minutes — so the other two sources were fine and only Mianshiya was spinning idle. Scrolling back through logs showed the same two lines repeating after a specific commit; following that commit hash landed inside the Mianshiya changes. Reading the code path, the chain was complete — request, exception, catch, break, return empty list — and nothing in it could ever fail the job.
The immediate cause was the bank-list endpoint returning a 567 Server Error. MianshiyaSource.fetch() wraps that request in try/except Exception, logs a warning and breaks; the function continues, banks is empty, and fetch() returns an empty list. Upstream, main.py treats "returned an empty list" as a normal result and proceeds to merge, render and commit. Historical data is never deleted, so the README keeps reporting the old 1,962 records and nothing on the page suggests the source stopped updating.
The real problem is error semantics: the collector uses "empty result" to mean two completely different states — "there genuinely is nothing new today" and "this source is broken". The first should succeed, the second should fail, and the code collapses them into one return value. On top of that, except Exception swallows HTTP errors, parse errors and structural changes alike, leaving one warning line, and no assertion anywhere checks the most basic health condition: banks > 0.
As recorded, the API side has not recovered and the silent break is still there. So the state is: Mianshiya returns zero every day, historical data stays, and the README numbers look unchanged. This is not "fixed" — it is a documented, unfixed defect.
Three changes should land. First, stop swallowing collector exceptions: a failure on the first page of the bank list raises SourceError, which main.py catches and marks the source failed instead of returning an empty list. Second, add a minimum-output assertion per source — banks > 0, questions > 0 — and record a violation; "nothing new today" should be "request succeeded, zero new", never "request failed, so zero". Third, surface per-source last-success time and current status in the README, so a dead source is visible on the page rather than buried in logs.
Data missed while the source was down does not come back by itself; only a --full rerun recovers it, at far higher request volume and runtime than a daily run. The more expensive cost is trust: a pipeline that is green every day with unchanging numbers convinces both maintainer and reader that everything is fine. The later such a silent failure is found, the higher the backfill cost — and every statistic produced in between carries a known gap.
analysis/README.md showed "analysed 0/237" for a long time and analysis/ held no company reports. The Actions step named "Analyze with GitHub Models" finished in 0 seconds as a success every time. 237 is the number of groups meeting min_records: 20, so the data side was fine; the analysis side had never actually started.
The investigation ran in reverse order: first the model, then the quota, and only then whether the code ran at all. The log had no model request, no 401, no 429, no timeout — not even a "cache skipped" line, which matches no known call failure. Zero seconds also contradicts "walk 237 groups and check each cache". Running the same code locally with python -m src.analyze produced the real error: a SyntaxError in the f-string inside _dedupe(). The error is at import time, not runtime, which is why no runtime log line ever appeared.
The root cause is in _dedupe():
key = f"{title}|{re.sub(r'\s+', '', (record.get('content') or '')[:120])}"
This line is legal from Python 3.12 and illegal on 3.11: a 3.11 f-string expression may not contain a backslash, so r'\s+' makes the whole module fail at compile time with SyntaxError. The workflow pins python-version: "3.11", so python -m src.analyze exited before the first line of business code ran.
Why nobody noticed: the analysis step had continue-on-error: true, so a compile failure counted as an ignorable analysis exception and the job stayed green; and a 0-second duration reads like "nothing needed analysis, returned fast". Locally, a newer Python runs the same line fine, so the environment difference hid the error even deeper. Three factors stacked into one compile-time error disguised as "analysis ran normally, produced nothing".
Still unfixed: the _dedupe() line remains, the workflow is still 3.11, and the step still has continue-on-error: true. Either fix works — move the workflow to 3.12, or hoist the re.sub out of the f-string.
normalized = re.sub(r"\s+", "", (record.get("content") or "")[:120])
key = f"{title}|{normalized}"
But that one line is not enough. The intent of continue-on-error — do not lose data when the model is down — should stay; what must change is that failure cannot disappear as "0 seconds, success". A sturdier shape is to split analysis into its own job, commit collected data regardless, and mark the job red with the reason in the summary; or read steps.analyze.outcome in the same job and emit an annotation on failure. The rule: failure is allowed, invisibility is not.
Not a single call for 237 groups was sent, so no quota was consumed — but the core "LLM summary" feature effectively did not exist, and the README showing 0 made readers think the data did not meet the threshold. The subtler cost is debugging time: zero seconds, no logs, no artifacts all point at "there was no work to do", while the truth was one layer down — the code never loaded. When a step is wrapped in continue-on-error, the failure signal decays in exactly the direction you least want.
Symptom: some reports land with only one or two hundred characters, ending in an ellipsis; some posts vanish from results entirely. Cause: Nowcoder's search API returns abstracts, not full text; only the detail endpoint has the body. Normalisation has _is_truncated(), which treats "under 300 characters and ending in an ellipsis" as a crawl failure and drops it. That heuristic blocks most broken records, and also misfires on genuinely short posts that happen to end with an ellipsis.
Fix: for unseen records, make one detail request and decide after fetching the full body; for records already seen, the content set skips them, so no repeat request. This mechanism is live: the daily run makes 2,900+ detail requests in 18 minutes 46 seconds, and that is where most of the time goes. The cost is that "short but complete" content may still be dropped, with no separate counter — an invisible loss.
Symptom: one report appears several times, company-role counts inflate, and the model sees duplicated samples. Three causes: the keyword matrix overlaps by design; share links carry #, query parameters and trailing slashes; and Mianshiya indexes one question in several banks. The fix is the three-layer dedupe above — URL normalisation, cross-bank merge by external_id, and a title-plus-120-characters filter before the model. The cost is that over-aggressive normalisation can merge pages whose query parameters genuinely distinguish content, and the title-plus-opening heuristic can merge two reports that share an opening but diverge later.
Symptom: bodies contain , zero-width spaces and runs of blank lines; sentences are glued together by <br>. Cause: all three sites return HTML fragments or rich text, mixing tags, entities and platform-injected invisible characters. The fix is a single clean_content: convert <br> and </p> into newlines, run html.unescape, strip \u200b and \xa0, then collapse whitespace. The cost is that cleaning order is sensitive — unescape before stripping tags turns text that was meant to be visible into real tags and deletes it. Cleaning also happens before dedupe, so changing the rules changes both content hashes and dedupe results.
Symptom: early on all records lived in one JSON, and as counts grew the file approached GitHub's 100 MB per-file hard limit — past which pushes are rejected outright. The cause was not only current size but git history: every whole-file rewrite leaves a new blob, so the repository grew far faster than the data. The fix was sharding by company into 158 .json.gz files totalling 35.9 MB, reading the legacy single file and deleting shards for companies that no longer exist. The cost: 158 files opened per run, slug collisions need a digest suffix, and the shard add/remove logic is itself a new failure surface.
Symptom: analysis stops halfway and later groups never get results; occasionally 429s repeat and retries do nothing. Cause: GitHub Models' free tier is roughly 15 RPM, and 237 groups far exceed a single day's budget. The fix: max_calls_per_run: 100, rpm: 12, workers: 2, plus 429 handling via Retry-After and x-ratelimit-remaining, a 403 rate-limit response raising QuotaExceeded to stop the round, and reserve = min(roles, max_calls // 6) holding quota for rollups. The cost: a complete analysis spans several days, and the "company groups first, rollups later" order means new roles lag forever; the split prevents starvation, not unfairness.
Symptom: PracHub is mostly foreign companies while the config has exclude_foreign_companies: true, so most of what it returned was filtered out — poor value for the request cost. Cause: the source no longer matched the goal. The fix: prachub: enabled: false plus exclude_sources: [prachub], and the next run drops its history per the dropped list. The cost is irreversibility: once deleted, re-enabling means crawling from scratch, and old links may already be dead. It is a straight trade between data assets and running cost, with no both-ways option.
First, there is no robots.txt check. Collection covers only public content and is rate-limited, but "are we allowed to crawl this" was never verified in code — an explicit compliance gap. Second, Juejin requests carry no signature parameters; they work today, but the day the API requires signing they will fail again — and that failure will very likely present as "0 records", re-enacting Mianshiya's silent failure. Third, the README derives its last-update time from file mtime, and Actions always checks out fresh, so unchanged files all get the same timestamp — the "updated at" readers see is not real. All three are open items, not decorations on a to-do list.
A daily run measures 30 minutes 08 seconds: Nowcoder 18:46 with 12,945 records and 2,900+ details; Mianshiya 1 second (the zero described above); Juejin 10:46 with 3,924 search results and 3,333 bodies. The weekly run takes 24:48, shorter than daily, mainly because --limit 150 caps detail completion. Timeouts are 60 and 180 minutes, so both have more than double the headroom they need; the real risk is API change and quota, not duration.
The time breakdown shows where the bottleneck is: Nowcoder accounts for over 60% of runtime, almost all of it those 2,900+ detail requests; Juejin fetched 3,333 bodies in just over ten minutes because its search API returns full text, so far fewer detail calls are needed. That is exactly why the incremental design must prioritise saving detail requests: search paging is a fixed cost, details scale linearly with new records.
Three judgements. First, the architecture holds under its constraints — zero server, free quota, one run a day: incremental checks eliminate most detail requests, idempotent writes avoid meaningless commits, and the two-stage analysis turns an unfinishable job into a resumable one. Second, its weakest point is not anti-bot defence but the way failure is expressed: a dead source or a compile failure can both leave the pipeline green. Third, the repository will keep growing — the full README re-sort and the history of whole-file rewrites are the main sources — so either accept it or make rendering incremental.
If work continues, the priority is: fix the f-string and the workflow version so analysis actually runs; add health assertions and last-success times per source; only then robots.txt, Juejin signing and the mtime correctness issues. Collection volume is no longer the bottleneck — visibility is.
InterviewAtlas is a long-running data pipeline held up by four dependencies, two workflows and one git repository. Its design compresses into four sentences: decide what is new before sending the request; dedupe in three layers at three different stages; pin the gzip timestamp to buy idempotency; version the manifest and budget the calls to buy control.
The holes it fell into are worth more than the features. Mianshiya's zero shows that "empty result" cannot mean both success and failure. The 0/237 shows a compile-time error can be dressed up as normal operation by continue-on-error. Truncation, duplicates and dirty characters show every cleaning heuristic has collateral damage. Write amplification shows the storage cost of a git repository lives in history, not in the snapshot.
Neither of the two worst problems is fixed: Mianshiya's API has not recovered and the silent break remains; the f-string-and-3.11 combination remains, and 0/237 will not become 237/237 on its own. Writing the fix down plainly is more honest than claiming it is solved.
If I started again, I would write "failure is visible" before writing collection: every source needs three distinct states — success, failure, zero output — and every background step's exit code must reach the summary instead of being flattened by continue-on-error. Most pipeline incidents are not "we could not fetch"; they are "we could not fetch and nobody knew".