ResearchForge

ResearchForge reads an academic PDF and produces three things: a structured summary covering the research problem, methodology, key findings and conclusion; a research-gap analysis where each gap is shown alongside the wording in the paper that supports calling it a gap; and a literature review of the prior work the paper itself discusses.

It is deployed on its own subdomain as a single Vercel project running two services behind one origin, with email and Google sign-in and a research library private to each account.

The failure mode of an AI summarising tool is not that it produces nothing. It is that it produces something plausible for a section the source never covered: an invented methodology for a position paper, a confidently stated gap with nothing behind it.

For a tool meant to help with research, that failure is worse than no answer at all, because it is indistinguishable from a correct one unless the reader already knows the paper.

ResearchForge was developed as part of BIT4543 Artificial Intelligence at the University of Cyberjaya. It is a university project, but it was not built as a classroom prototype: it is deployed on its own subdomain, has accounts and stored user data, and the reported results come from running the deployed system rather than from estimates.

It began as a stateless tool with no accounts, which kept the first version defensible while there was nothing stored to protect. Adding a library meant that stopped being true, so authentication, per-user ownership and database-level access control went in together rather than being retrofitted around a feature that had already shipped.

I took this from an empty repository to a deployed system that is still maintained, and the order matters: the first commit is the project structure, the working rules, a project plan with milestones, risks and numbered decisions, the environment contract and the documentation skeleton. Features came after that, not before it.

Planning and specification. Milestones were tracked and closed in the repository (M1 complete, risk R1 resolved), and technical choices were recorded as numbered decisions rather than made silently, including one that was locked, revisited and re-locked when a better option was found. Requirements were derived from the objectives and refined during development as defects exposed expectations nobody had written down.

Architecture and design. Two services in one Vercel project behind a single origin, a provider interface that keeps vendor SDKs out of the analysis code, a storage-independent repository, the database schema and its migrations, and the decision to enforce ownership in PostgreSQL rather than in application code.

Implementation, both sides. The Python and FastAPI backend, its API endpoints and error semantics, PDF ingestion and text normalisation, the three-pass analysis pipeline, prompt and structured-output handling, schema validation, the content-hash cache, authentication and owner configuration. On the frontend, the Next.js application: the landing page, sign-in and account flows, the analysis workspace, the research library, paper detail and cross-paper review, themes, and the responsive layout.

Then the parts that only exist once something is real: production configuration and environment management, deployment, debugging live failures, refactoring, the test suite, and maintenance after the system was already working.

A signed-in user drops a PDF on the dashboard. The browser checks type, emptiness and size, then posts it to POST /api/analyze. The backend validates it again, extracts and cleans the text, checks whether that exact text has been analysed before, and if not makes three structured model calls: summary, research gaps, literature review. The result comes back as one validated JSON response and renders in tabs.

Nothing is stored until the user chooses Save to library. Saved papers can then be selected together in the Workspace to produce one literature review across several of them. That second step reads the stored analyses rather than re-reading the PDFs.

A new analysis takes one to three minutes; the interface says so and lists the steps the request performs, without pretending to know which one is running, because the backend does not report progress.

The founding principle is that every claim must be grounded in the uploaded paper, and it is enforced in four layers rather than requested once in a prompt. The prompt requires it, and the prompts live in version-controlled files rather than scattered through the code. The response schema carries explicit insufficient-evidence fields, giving the model a way to decline that is as easy as complying. Every reply is validated against that schema on return. And output that fails validation is discarded rather than repaired, which is the rule that makes the other three mean anything: a partially valid analysis that the system patched up would be an invented analysis.

Structured output is the mechanism, not a convenience. Each response model is converted to a JSON Schema and handed to the model as the required output format, with additional properties forbidden. Every reply is validated on return, and a truncated or malformed answer is refused outright rather than partially rendered.

The three analyses run as three separate model calls. They are different tasks with different evidence rules, so separating them means a failure in one does not corrupt the others, and each can be improved on its own. They run sequentially on purpose: running them in parallel would multiply the peak rate-limit burden for a latency win that does not matter on a single upload.

One Vercel project runs two services. A Next.js frontend serves everything except the API, and a FastAPI backend serves /health and /api/*, with routing declared in the project configuration. Because both share one origin, the frontend calls the API with a relative path, which is what makes the custom domain, the .vercel.app domain and every preview URL work from the same build.

Generation sits behind a provider interface. The analysis service depends on that interface and never on a vendor SDK, each vendor's SDK is imported only inside its own provider module, and the concrete provider is built by a factory with a local import so adding one never forces every caller to import every SDK. An earlier version used Google Gemini; it is a historical provider only and produces none of the current analyses.

Vendor errors are wrapped in project-owned exception types, with a missing API key separated out from the rest because it is a deployment problem rather than a user's fault and maps to a different status code. Status codes are chosen so the frontend can tell the cases apart without parsing message text: too large, unusable PDF, unusable model reply, no credentials configured. Nothing expected returns a 500.

The owner picks which of the two providers is primary, and the other automatically becomes the fallback. Groq running qwen3.6-27b is the configured primary and Anthropic claude-opus-5 the fallback, as the live owner settings showed on 25 September 2026. There is no per-user model picker: the choice is an operational one, made once, and the interface does not pretend otherwise.

Availability is enforced when the router is constructed rather than checked at call time, so a provider that has been switched off is never built and no code path can reach it.

Fallback is deliberately narrow. It fires once per analysis, and only for a rate limit or a temporary provider failure, checked against a whitelist so a new error type does not become retryable by default. It does not fire for a malformed PDF, a schema validation failure or a missing key, because those fail identically on either vendor and retrying them just spends a second vendor's quota to produce the same error more slowly.

One switch per analysis, not per call. An analysis makes at least three calls, and allowing each to fail over independently would let different sections be written by different models, which makes the recorded model identity meaningless.

Every stored analysis records which provider and model actually produced it, whether the fallback was used, and how long the call took. Without that, a result whose quality looks off has no explanation attached to it, and 'which model wrote this' becomes unanswerable a week later.

The screenshots below show that working rather than asserted. Both demonstration papers were analysed with Groq as primary, and both records say Claude produced them with the fallback used. That is the behaviour the evaluation predicted, since the primary's free tier is smaller than one call over a whole paper.

Authentication is Supabase Auth: email and password with sign-up, sign-in, forgot-password and reset flows, plus Google sign-in completing at a dedicated callback route. Passwords never reach the ResearchForge database.

Every paper, analysis and review belongs to exactly one account. Application-level ownership checks alone proved insufficient for reliable isolation in production, so the guarantee was moved into the database: PostgreSQL Row Level Security enforces per-user access at the data layer.

The mechanism is which credential the backend uses to reach the database. Requests are made as the signed-in user, so PostgreSQL resolves the authenticated identity and applies every policy automatically. A forgotten ownership filter then returns nothing rather than everything, which is the opposite of how that mistake usually fails. Ownership columns default to the authenticated identity, so a row cannot be inserted without an owner even if the application code omits it.

Records created before authentication existed remain unowned. They were deliberately neither deleted nor assigned to an owner that could not be established, and they are unreachable because a null owner never matches an authenticated identity.

Re-uploading a paper that has already been analysed reuses the stored analysis instead of paying for it again. Identity is a content hash of the extracted text, not the filename, so the same paper saved under a different name still matches.

A saved paper keeps its analysis, and the library can be searched by title or filename, filtered by status and sorted. Opening one shows the same four tabs as a fresh analysis, plus the document facts that were measured rather than generated: pages, characters extracted, file size, whether it was analysed whole, and the model recorded for it.

Selecting two or more saved papers in the Workspace builds one literature review across them. It deliberately reads each paper's stored summary, findings, limitations and themes rather than re-extracting the PDFs, which keeps the combined prompt inside the context window and avoids paying for the same reading twice. The trade-off is stated in the code: the cross-paper review cannot surface anything the original analyses missed.

Every paper in a review must still exist and carry a stored analysis, or the request is refused: 404 for a paper that is missing or belongs to someone else, 422 naming any paper with no stored analysis. A review that quietly covered four of the five papers the user picked would be worse than an error, because the interface would still say five.

The frontend is Next.js 16 with React 19, TypeScript in strict mode and plain CSS rather than a component library. Light and dark themes follow a stored preference or the operating system, the navigation collapses to a menu on narrow screens, and empty and error states are written to say what happened, for example distinguishing an empty library from one that is not connected.

An upload is validated twice. The browser checks type, emptiness and size; the backend re-checks the real byte count and verifies the file actually begins with a PDF signature, so a renamed file cannot get through on its content type alone.

Encrypted files get an empty-password decrypt attempt, which covers the common printing-restricted case, and are rejected otherwise rather than guessed at. A file whose extraction yields almost no text is a scanned image with no text layer; it is rejected with an explanation instead of returning an empty result.

Extracted text is cleaned conservatively: ligatures normalised, words rejoined across hyphenated line breaks, runs of spaces collapsed but never across newlines, because paragraph structure is a real signal about where sections begin. Nothing in the cleaning removes content.

Chunking is deliberately conditional. A paper that fits is sent whole, because cross-section reasoning is exactly what gap analysis depends on and chunking would destroy it. Only a genuinely oversized document takes the map step, and its boundaries are chosen at paragraph breaks where possible, falling back to sentence ends and only then to a hard offset.

538 passing tests: 522 backend with pytest and 16 frontend with Vitest, plus 5 live-provider tests skipped by design. No test in the suite calls a paid interface. Every provider is replaced by a fake through FastAPI's dependency-override mechanism, which is the practical reason the endpoint takes its provider as a dependency rather than constructing one.

The distribution is deliberate rather than even. Authentication is the largest module at 61 tests, on the judgement that a silent failure there costs most. Then the LLM providers at 49, every database access path at 48, rate-limit recognition and retry discipline at 42, and the provider router at 36.

Automated tests were not sufficient on their own. The isolation defect described above passed every one of them while being wrong in production, because the ownership tests modelled Row Level Security with a stub. Verifying it needed live scripts running a scenario as two different people against the deployed system. That is the argument for keeping unit, integration and live testing as separate levels: each can pass while another fails.

The model tier was chosen against a platform constraint rather than a benchmark. A Vercel function has a 300-second ceiling, and three sequential schema-constrained calls have to complete inside it. A deeper-reasoning model is one environment variable away with no code change.

pypdf was chosen over faster alternatives on licensing. The fastest option ships native binaries and is AGPL, which is incompatible with an MIT repository intended to be read publicly. Only one function touches the library, so swapping it later is a one-function change.

Provenance columns degrade per migration rather than all at once: a database that has not yet run a later migration loses only the fields that migration added, instead of the whole write failing. A schema change should not be able to take the feature down while it is rolling out.

This did not stop at localhost. One Vercel project runs both services behind a single origin: a Next.js frontend serving everything except the API, and a FastAPI backend serving /health and /api/*, with the routing declared in the project configuration and a 300-second function ceiling the analysis has to fit inside.

Single-origin is the decision that makes the rest work. Because both services answer on the same origin, the frontend calls the API with a relative path, so the custom domain, the .vercel.app domain and every preview URL all work from one build with no per-environment base URL to get wrong.

Production configuration is its own body of work: environment and secret management across the application, the database and two model providers, a guard that refuses a publishable key in a slot that requires a private one rather than silently showing an empty library, and a health endpoint that reports whether auth and the library are actually wired rather than just that the process is up.

The most instructive defect was in data isolation, and it is instructive precisely because nothing caught it. An earlier version of the backend connected to the database with a privileged key, the kind designed to bypass Row Level Security. Every policy was present. Every policy was correctly written. Every policy was inert. The application worked, every feature behaved correctly, and the whole automated suite passed, because at the unit level nothing was wrong.

It was found by signing in as a second real account and checking whether the first account's papers were visible. The fix was to connect as the signed-in user so PostgreSQL resolves the caller identity itself. That is the episode that turned isolation from something every query has to remember into something the database enforces.

A content delivery network hid the real errors. The custom domain is served through a CDN that replaces an origin error body with a short generic message, so every failed analysis looked identical and carried no explanation. Addressing the application origin directly returned the real provider message, which is how the free-tier token limit behind the whole evaluation was identified. Without that step the central finding would have stayed invisible.

A schema change broke every library read. Queries were selecting new provenance and cache columns before the matching migration had been applied, and the database rejected the entire request rather than the missing columns, so all reads failed at once. The fix retries and degrades one migration level at a time, returning the columns that do exist instead of failing completely.

Rate limiting arrived disguised as generic server errors. Handling was rewritten to be vendor-neutral: rate limits are recognised as such, retry information is passed through, an exhausted quota is distinguished from temporary throttling, and the hidden retries inside client libraries were switched off because they multiply cost without telling the caller.

Sign-out, sign-in and same-origin routing each needed their own fix: pinning the PKCE flow and deriving every redirect from the running origin, making logout take effect promptly, and calling the backend same-origin so the custom domain and every preview URL work from one build.

Work did not stop at the first working version. After the analysis path was live, the system gained authentication and private libraries, Google sign-in, a second provider with owner-controlled availability, account settings, analysis reuse, and provenance recording, each of which meant revisiting code that already worked.

Some of that was correcting earlier decisions rather than adding to them. The provider architecture was rebuilt when the original vendor's rate limits proved unworkable, the frontend was rebuilt as a research dashboard, and provenance columns were changed to degrade per migration after the schema-skew failure rather than being left to fail as a unit.

A caching subtlety is recorded rather than quietly fixed: because the cache is keyed by content rather than by user or provider, a cached result can be returned where a fresh analysis was expected. That affected measurement twice during evaluation. Neither was a defect in the cache, which behaved as designed; both were defects in measurement, resolved by clearing the relevant rows before measuring. It is worth stating because a correctness-preserving optimisation invalidating an experiment silently is a general risk, not a one-off.

A working, deployed application with accounts, a private research library and recorded provenance, that does what it says and states what it does not do. The analysis and library paths are complete and in production; retrieval over a stored corpus is written only as far as the embedding request construction, and both the interface and the documentation say so rather than implying otherwise.

Measured on the deployed system: a new analysis of a full paper takes a median of 98.8 seconds, ranging from 72.6 to 115.2 across the evaluation corpus, because three separate passes are made over the whole document. A cache hit returns in 2.7 to 3.2 seconds.

The evaluation also produced a negative result worth stating. The intended comparison between the two providers could not be completed: with the fallback removed, the configured primary refused every request, because its free tier allows 7,000 input tokens per minute and every corpus paper needed more than that in a single call. The architecture supports two providers and the fallback is verified, but the system is effectively single-provider until a primary that can accept a whole paper is configured.