Important
AISysRev is a Minimum Viable Product (MVP) with core functionality working. Some features are missing and there might be bugs. You can also checkout command line alternative AISysRevCmdLine
- AISysRev - LLM-based Tool for Title-abstract Screening
The AISysRev web-application offers research-backed and AI-based support for Systematic Literature Reviews. Currently, the main supported step is title–abstract screening (see the feature table). Although the application runs in a web browser, all data is stored locally on your machine. LLMs are accessed through OpenRouter, OpenAI or through a local provider (OpenAI SDK). Data for screening can be imported from two sourced as a CSV from: 1) Scopus and from 2) Github. For Github data see our tool Github Query Tool that queries and collects github repository data.
Scopus data screening works like normal title-abstract screening for academic papers. For GitHub we apply the same logic. Title from GitHub project becomes the title and abstract is the project description + README.md contents. GitHub screening can be useful if one is interested in finding software tools that satisfy inclusion and exclusion rules. Our paper on tool discovery with LLMs.
AISysRev focuses on study selection. The table below shows which steps of the systematic review process the application supports.
| Systematic review step | Status | Notes |
|---|---|---|
| SR protocol definition | ❌ Not supported | Inclusion/exclusion criteria are defined per screening task, but there is no protocol editor. |
| Automated search | ❌ Not supported | Import search results as CSV from Scopus or from our Github Query Tool. |
| Study selection (title-abstract) | ✅ Supported | LLM-based and manual screening. See Title-abstract screening. |
| Study selection (full-text) | ✅ Supported | LLM screening using the passages of an uploaded PDF that best match each criterion. See Full-text screening and pdf-screening.md. |
| Quality assessment | ❌ Not supported | |
| Data extraction | ❌ Not supported | |
| Meta-analysis | ❌ Not supported | |
| Final report | ❌ Not supported | Screening results can be exported to CSV. |
In the future, we plan to expand the feature set to cover more steps of the systematic review process.
The application is based on our research papers on this topic. Please consider citing if you use the application 1–3.
The application allows you to:
- Import a CSV file with paper titles and abstracts. You can also use our Demo CSV file
- Specify include/exclude criteria for paper screening
- Evaluate papers against the criteria using multiple LLMs, zero-shot or few-shot, with built-in or your own prompts
- Receive LLM evaluations as binary decisions (include/exclude), ordinal ratings (1-7), or inclusion probabilities (0–1)
- Perform manual evaluation of titles and abstracts alongside LLM evaluations
- Export evaluation results to CSV for further analysis in Microsoft Excel, Google Sheets, R, Python, etc., or view them as an HTML report All of the above steps are also done for Github repositories.
AISysRev supports full-text screening of PDFs in addition to title-abstract screening. For each criterion, the most relevant passage of the PDF is retrieved and given to the LLM. Full-text screening requires the OpenAI or OpenRouter provider, because the passages are found with embeddings.
PDFs can be attached to papers in two ways:
- Manual - on a paper's card, click Upload full text and select a PDF file to upload.
- Bulk upload with Zotero/EndNote XML
- Click Download papers missing full text to export a RIS file containing all papers in the project that don't have a PDF attached.
- Import the RIS file into Zotero and use Find Full Text to retrieve full-text PDFs.
- Export the Zotero collection with the automatically retrieved PDFs in EndNote XML format with Export notes and Export files checked.
- In AISysRev, click Import full text (Zotero Export Folder) and select the exported folder. PDFs are attached to papers based on DOI.
When creating a screening task, choose which screening mode to use:
- Abstract - the paper's title and abstract are given to the LLM.
- PDF - excerpts from the paper's full-text PDF are given to the LLM.
- Automatic - uses PDF screening mode for papers with a PDF attached and Abstract for papers without a PDF attached.
See pdf-screening.md for details on how PDF screening mode works, including its limitations.
A screening task runs one model over all papers of a project. Click New task on the project page, then choose:
- Screening method
- LLM screening - a language model judges each criterion and explains its decision.
- Jev screening - TypeSafe Jev returns a calibrated probability and an include/exclude decision per criterion, without written reasoning. Jev screens zero-shot only and is called through OpenRouter with your OpenRouter API key.
- Model - the provider (OpenRouter, OpenAI or a local model) and the model. Under Advanced you can set the temperature, top_p and the number of parallel calls (20 by default, at most 50). For OpenRouter you can require Zero Data Retention (ZDR) for a task, or force it for all tasks in Settings > LLM Settings.
- Options (under Customize)
- Screening mode - see Screening modes.
- Evaluation mode - All criteria together evaluates all criteria in one model call. One call per criterion evaluates each criterion in its own call and combines the answers with the project's per-criteria logic (see below).
- Prompting strategy - Zero-shot gives the model only the criteria. Few-shot adds papers you have already labelled as examples: you pick the included and excluded seed papers when you start the task.
- Prompt - the prompt the task sends to the model, see Custom prompts.
The summary next to the form shows an estimate of the input and output tokens the task will use before you start it. Tasks run in the background, and can be cancelled or deleted from the task's menu.
When creating a project, you can set how per-criterion answers are combined into a decision, as an expression of the criterion IDs, e.g. IC1 AND IC2 for inclusion and EC1 OR EC2 for exclusion. By default, all inclusion criteria must be met (AND), and meeting any exclusion criterion excludes the paper (OR). The logic is used by tasks run with One call per criterion.
The Prompts page lists the prompts used for screening. They are shared by all of your projects, and you pick one when you create a screening task.
- Built-in prompts - the prompts AISysRev screens with by default, one for each prompting strategy (zero-shot, few-shot, per criterion) and screening target (papers, GitHub repositories). They can't be changed; a new prompt starts from the built-in text of its strategy and target. Papers: zero-shot is the prompt published in our ESEM '25 paper 2.
- My prompts - prompts you have written. Only you can see them.
A prompt has a system prompt, which sets the model's role, and a user prompt, which is a template with variables such as {{title}}, {{abstract}}, {{criteria}} (or {{inclusion_criteria}} and {{exclusion_criteria}}), {{additional_instructions}} and, for few-shot prompts, {{few_shot_examples}}. Variables are filled in for each paper when the task runs. Every save is kept as a new version, and each task records the exact prompt version it ran with.
- Task details - open a screening task to see each paper's decisions, the model's reasoning per criterion, and the prompts that were sent to the model.
- Manual evaluation - Start manual evaluation shows one paper at a time with every model's suggestions and the average likelihood of each criterion. Decide with the buttons or the keyboard shortcuts (Y/I include, N/E exclude, U unsure).
- List of papers - all papers with their average inclusion probability over all tasks, sortable and with evaluated papers hidden if you want. Expand a paper to read its abstract, upload its full text and make a decision.
- Per-criteria agreement - with at least two completed per-criterion tasks, the project's menu shows agreement statistics (percent agreement and Gwet's AC1) between the tasks for each criterion.
- Export - download the results as CSV, or open them as an HTML report.

Figure 1: Project view shows the LLM screening tasks of a project.

Figure 2: Creating a screening task: pick the screening method and the model, and customize the options, such as the prompt.

Figure 3: Manual evaluation view, with LLM evaluations (binary, ordinal, probability) alongside manual review, and the average likelihood of each criterion.

Figure 4: List of papers, sorted by inclusion probability according to all executed LLMs, with manual decisions.

Figure 5: Prompts view: the built-in prompts, including the one published in ESEM '25, and your own versioned prompts.
The tool has been developed and tested with CSV data exported from Scopus and Github with data exported by our custom Github Query Tool. Support for Web of Science can be achieved by editing the columns headers to match the ones from Scopus. The minimum required fields are: Document title, DOI, Abstract, Authors, and Source title.
The application is integrated with OpenRouter, which supports multiple LLMs ranging from very affordable to top-tier models like OpenAI’s ChatGPT, Google’s Gemini, Anthropic’s Claude, Meta's LLama, and Mistral. To use the models, you need to provide an OpenRouter key. You can set spending limits for each key directly on the OpenRouter website.
LLM calls are parallelized, and you should achieve a screening speed exceeding 100 papers per minute when using OpenRouter. The screening speed depends on the model used.
AISysRev runs entirely in Docker containers: the frontend, backend, database, Redis and the Celery workers. Docker is the only software you need to install to run it; there is nothing to install or configure on the host.
- Docker, with Compose and buildx plugins installed.
- Enough RAM (At least 8GB recommended)
- Enough disk space (Around 8GB)
- Network connection
Tip
See https://docs.docker.com/desktop/ for Docker installation instructions. Docker Desktop includes Docker Compose, Docker Buildx, Docker Engine and the Docker CLI.
Note
If Docker Desktop did not include the Buildx plugin, see: https://github.com/docker/buildx
- Run
docker infoto verify you have Docker installed- Docker
26.0.0has been tested as working. For MacOS computers with Colima, Docker version28.5.1confirmed to be working.
- Docker
- Run
docker buildx versionto verify you have Docker Buildx installed. For MacOS computers, Buildx plugin version0.29.1confirmed to be working. - Run
docker compose versionto verify you have Compose installed. For MacOS computers, Compose plugin version2.40.3confirmed to be working.- Version
2.33.1has been tested as working, newer versions should also work. - Note: Older versions of Compose use
docker-composeas the compose command. We don't provide support for legacy Compose versions.
- Version
First, clone the repository to your local computer.
git clone https://github.com/EvoTestOps/AISysRev.gitmove to correct directory
cd AISysRevThe default way to run AISysRev on your own machine is local mode: the production build of the app, running on your machine. It needs no login or account: the app runs as a single local user, and all data stays on your machine.
make startIf you do not have Windows Subsystem for Linux (WSL), run:
./start.batNote
The startup of the app may take a while due to the download of corresponding Docker images & services, application dependencies and building of the application.
The app opens in your browser at https://localhost:3000 as soon as it is reachable, and shows a startup page until it is ready (set NO_BROWSER=1 to skip opening it). The Caddy server's root CA is by default untrusted, so you can bypass the browser warning.
Stopping the app keeps your data: it is stored in Docker volumes, which are only removed if you remove them yourself (e.g. docker compose -p local down -v).
make start-prod(./start-prod.baton Windows) runs the same production build with login (OIDC), as deployed on a server, also at https://localhost:3000.make start-dev(./start-dev.baton Windows) runs the development stack, see Running in development mode.
The AISysRev tool is built on bleeding front-end and back-end technology, with industry-demonstrated stability. We follow the best known coding conventions and toolkit.
TypeScript, React, Tailwind CSS, Vite, Wouter, Zod, Redux
Python, FastAPI, PostgreSQL, SQLAlchemy, Alembic
See architecture.md
Development also happens in containers: make start-dev runs the whole stack in Docker and syncs your edits into it, and the tests and migrations run in containers too. The host tools below are only needed for linting, type checking and generating the API client.
- Node.js v24 LTS
- Python 3.14
- Docker, with Compose plugin installed
- UV v0.12.17 or later: https://docs.astral.sh/uv/getting-started/installation/
- Updating uv:
uv self update
- Updating uv:
make start-dev
To skip the login, as in local mode: LOCAL_ENV=true make start-dev
./start-dev.bat
Local mode is the environment variable LOCAL_ENV=true. make start sets it on the production stack, and it can be added to the development stack too. The backend then skips the login: every request acts as a hardcoded mock user (local-user, with a fixed UUID) who has accepted the terms and privacy policy, so no OIDC credentials are needed. Logging out and deleting the account are disabled, and the local OpenAI-compatible model provider is available, as the app runs on the user's own machine. The backend refuses to start with LOCAL_ENV=true if APP_ENV is staging.
Open up the client: https://localhost:3001
Note
/api is internally proxied to the backend container, e.g. http://localhost:3001/api/v1/health will be proxied to http://localhost:8080/api/v1/health.
API: https://localhost:3001/api/v1
API docs: https://localhost:3001/docs
Adminer GUI: http://localhost:8081/?pgsql=postgres&username=your_username&db=your_database_dev&ns= password: your_password
Generate TypeScript types and Zod schemas from the backend's OpenAPI spec by running, in the repository root:
make openapiThis doesn't need the server or containers running. It imports the FastAPI app locally (with uv), writes app.openapi() to client/openapi.json, and then runs npm run generate:api in client/ to generate client/src/services/api/ (client, types, fetcher) from that spec. The app's startup hooks aren't run, so nothing connects to the database or Redis; the connection URLs are set to placeholder values only because the settings require them. The spec is exported with APP_ENV=dev, so it matches the routes of the dev server.
Alternatively, with make start-dev running, you can run the steps separately in client/:
npm run get:openapi-spec— fetches the spec fromhttps://localhost:3001/openapi.jsonintoclient/openapi.json.npm run generate:api— generatesclient/src/services/api/from that spec.
client/src/services/api/ and the openapi.json are committed to git. Re-run make openapi whenever backend endpoints or schemas change.
A smoke test is in place, which instructs you to update openapi.json then the spec changes.
- docs/README.md: index of all documentation
- CONTRIBUTING.md: development workflow and documentation rules
- docs/architecture.md: system design and ports
- docs/pdf-screening.md: how PDF screening works
- docs/local-models.md: using local LLMs
- fuzzing/README.md: API fuzzing tools
- docs/maintenance.md: upgrading dependencies and Docker images
Mock data is located in data/mock -folder.
Run in client/:
npm testfor unit and component testsnpm run test:e2efor e2e tests
Run in repository root:
make backend-unitfor backend unit tests onlymake backend-testfor all backend testsmake backend-test-htmlfor all backend tests and an HTML coverage report
Windows (non-WSL) users need to run the equivalent docker compose commands from the Makefile manually, since only start.bat, start-dev.bat and start-prod.bat are provided.
The project includes a Makefile for common development and database operations:
| Command | Description |
|---|---|
make start |
Run the production build locally without login (local mode, the default setup), at https://localhost:3000 |
make start-dev |
Start dev containers with live reloading and build on startup (add LOCAL_ENV=true to skip the login) |
make start-dev-debug |
Same as start-dev, with the backend debugger enabled |
make start-test |
Start the e2e test stack: the production images with APP_ENV=test, at https://localhost:3002 |
make start-test-debug |
Start the dev stack with APP_ENV=test and the backend debugger, at https://localhost:3002 |
make start-prod |
Start production container and rebuild images |
make backend-unit |
Run backend unit tests |
make backend-test |
Run all backend tests |
make backend-test-html |
Run all backend tests and create an HTML coverage report |
make extract-caddy-ca |
Copy the production Caddy root CA to ./caddy-local-root.crt so you can trust it locally |
Note: Run all commands from the project root.
Containers are isolated by environment using the Docker Compose-pflag.
| Command | Description |
|---|---|
make m-create m="Message" |
Create a new migration with an autogenerated diff (replace Message) |
make m-up |
Apply all pending migrations (upgrade to latest) |
make m-hist |
Show the full migration history with details |
make m-current |
Display the current migration version in the database |
Each container started with Docker Compose has a CPU and memory limit (deploy.resources.limits in docker-compose-dev.yml, docker-compose.yml and docker-compose-e2e.yml), so a local stack doesn't take over the whole machine. They are roughly sized to the 4-CPU GitHub runner the e2e tests run on. The Kubernetes deployments, defined in a separate deployment repository, set their own resources.
| Service | CPU | Memory |
|---|---|---|
| Frontend (dev, Vite) | 1 | 1 GiB |
| Frontend (prod and test, Caddy) | 0.5 | 128 MiB |
| Backend | 1 | 512 MiB |
| Celery | 1 | 1 GiB |
| PostgreSQL | 0.5 | 1 GiB |
| Redis | 0.25 | 256 MiB |
| Flower | 0.25 | 256 MiB |
| Adminer | 0.25 | 128 MiB |
Celery runs 4 worker processes (CELERY_CONCURRENCY) instead of one per CPU, to stay within its memory limit. The e2e test stack has no Flower or Adminer.
Currently, we support models provided via Openrouter, OpenAI or via a local provider (OpenAI SDK). The list of supported LLMs is continuously evolving and changing, which is why we don't provide a comprehensive list.
Please use the following BibTeX citation to cite our work:
@inproceedings{huotalaAISysRevLLMbasedTool2026,
address={Concordia University Montreal QC Canada},
title={AISysRev - LLM-based Tool for Title-abstract Screening},
ISBN={9798400726361},
url={https://dl.acm.org/doi/10.1145/3803437.3806408},
DOI={10.1145/3803437.3806408},
booktitle={Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering},
publisher={ACM},
author={Huotala, Aleksi and Kuutila, Miikka and Turtio, Olli-Pekka and Sipilä, Simo and Mäntylä, Mika},
year={2026},
month=july,
pages={142–146},
language={en} }
Parts of AISysRev may have been developed with AI-assisted coding tools, including "vibe coding". Where AI assistance was used, the correctness and code style of the resulting code were reviewed.
MIT
[1] Huotala, A., Kuutila, M., Ralph, P., & Mäntylä, M. (2024). The promise and challenges of using llms to accelerate the screening process of systematic reviews. Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, 262–271. https://doi.org/10.1145/3661167.3661172
[2] Huotala A, Kuutila M, Mäntylä M. SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews. In Proceedings of the The 19th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) 2025 Oct (pp. 1-12) https://arxiv.org/abs/2507.19027
[3] Huotala A, Kuutila M, Turtio Olli-Pekka, Sipilä S, Mäntylä M. AISysRev: LLM-based Tool for Title-abstract Screening. In FSE Companion '26: Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering 2026 Jul (pp. 142-146) https://doi.org/10.1145/3803437.3806408