Skip to content
EvoTestOpsPublic

About

LLM-based Tool for Title-abstract Screening

Topics

Resources

Contributing

Stars

22 stars

Watchers

0 watching

Forks

Latest commit

 

History

1,261 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

AISysRev - LLM-based Tool for Title-abstract Screening

GitHub Actions Workflow Status GitHub commit activity GitHub License GitHub Repo stars GitHub contributors

Claude ChatGPT Google Gemini MistralAI Ollama

Important

AISysRev is a Minimum Viable Product (MVP) with core functionality working. Some features are missing and there might be bugs. You can also checkout command line alternative AISysRevCmdLine

Table of contents

Introduction

The AISysRev web-application offers research-backed and AI-based support for Systematic Literature Reviews. Currently, the main supported step is title–abstract screening (see the feature table). Although the application runs in a web browser, all data is stored locally on your machine. LLMs are accessed through OpenRouter, OpenAI or through a local provider (OpenAI SDK). Data for screening can be imported from two sourced as a CSV from: 1) Scopus and from 2) Github. For Github data see our tool Github Query Tool that queries and collects github repository data.

Scopus data screening works like normal title-abstract screening for academic papers. For GitHub we apply the same logic. Title from GitHub project becomes the title and abstract is the project description + README.md contents. GitHub screening can be useful if one is interested in finding software tools that satisfy inclusion and exclusion rules. Our paper on tool discovery with LLMs.

Features

AISysRev focuses on study selection. The table below shows which steps of the systematic review process the application supports.

Systematic review step Status Notes
SR protocol definition ❌ Not supported Inclusion/exclusion criteria are defined per screening task, but there is no protocol editor.
Automated search ❌ Not supported Import search results as CSV from Scopus or from our Github Query Tool.
Study selection (title-abstract) ✅ Supported LLM-based and manual screening. See Title-abstract screening.
Study selection (full-text) ✅ Supported LLM screening using the passages of an uploaded PDF that best match each criterion. See Full-text screening and pdf-screening.md.
Quality assessment ❌ Not supported
Data extraction ❌ Not supported
Meta-analysis ❌ Not supported
Final report ❌ Not supported Screening results can be exported to CSV.

In the future, we plan to expand the feature set to cover more steps of the systematic review process.

The application is based on our research papers on this topic. Please consider citing if you use the application 1–3.

Title-abstract screening

The application allows you to:

  • Import a CSV file with paper titles and abstracts. You can also use our Demo CSV file
  • Specify include/exclude criteria for paper screening
  • Evaluate papers against the criteria using multiple LLMs, zero-shot or few-shot, with built-in or your own prompts
  • Receive LLM evaluations as binary decisions (include/exclude), ordinal ratings (1-7), or inclusion probabilities (0–1)
  • Perform manual evaluation of titles and abstracts alongside LLM evaluations
  • Export evaluation results to CSV for further analysis in Microsoft Excel, Google Sheets, R, Python, etc., or view them as an HTML report All of the above steps are also done for Github repositories.

Full-text screening

AISysRev supports full-text screening of PDFs in addition to title-abstract screening. For each criterion, the most relevant passage of the PDF is retrieved and given to the LLM. Full-text screening requires the OpenAI or OpenRouter provider, because the passages are found with embeddings.

Uploading PDFs

PDFs can be attached to papers in two ways:

  • Manual - on a paper's card, click Upload full text and select a PDF file to upload.
  • Bulk upload with Zotero/EndNote XML
    1. Click Download papers missing full text to export a RIS file containing all papers in the project that don't have a PDF attached.
    2. Import the RIS file into Zotero and use Find Full Text to retrieve full-text PDFs.
    3. Export the Zotero collection with the automatically retrieved PDFs in EndNote XML format with Export notes and Export files checked.
    4. In AISysRev, click Import full text (Zotero Export Folder) and select the exported folder. PDFs are attached to papers based on DOI.

Screening modes

When creating a screening task, choose which screening mode to use:

  • Abstract - the paper's title and abstract are given to the LLM.
  • PDF - excerpts from the paper's full-text PDF are given to the LLM.
  • Automatic - uses PDF screening mode for papers with a PDF attached and Abstract for papers without a PDF attached.

See pdf-screening.md for details on how PDF screening mode works, including its limitations.

Screening tasks

A screening task runs one model over all papers of a project. Click New task on the project page, then choose:

  • Screening method
    • LLM screening - a language model judges each criterion and explains its decision.
    • Jev screening - TypeSafe Jev returns a calibrated probability and an include/exclude decision per criterion, without written reasoning. Jev screens zero-shot only and is called through OpenRouter with your OpenRouter API key.
  • Model - the provider (OpenRouter, OpenAI or a local model) and the model. Under Advanced you can set the temperature, top_p and the number of parallel calls (20 by default, at most 50). For OpenRouter you can require Zero Data Retention (ZDR) for a task, or force it for all tasks in Settings > LLM Settings.
  • Options (under Customize)
    • Screening mode - see Screening modes.
    • Evaluation mode - All criteria together evaluates all criteria in one model call. One call per criterion evaluates each criterion in its own call and combines the answers with the project's per-criteria logic (see below).
    • Prompting strategy - Zero-shot gives the model only the criteria. Few-shot adds papers you have already labelled as examples: you pick the included and excluded seed papers when you start the task.
    • Prompt - the prompt the task sends to the model, see Custom prompts.

The summary next to the form shows an estimate of the input and output tokens the task will use before you start it. Tasks run in the background, and can be cancelled or deleted from the task's menu.

Per-criteria logic

When creating a project, you can set how per-criterion answers are combined into a decision, as an expression of the criterion IDs, e.g. IC1 AND IC2 for inclusion and EC1 OR EC2 for exclusion. By default, all inclusion criteria must be met (AND), and meeting any exclusion criterion excludes the paper (OR). The logic is used by tasks run with One call per criterion.

Custom prompts

The Prompts page lists the prompts used for screening. They are shared by all of your projects, and you pick one when you create a screening task.

  • Built-in prompts - the prompts AISysRev screens with by default, one for each prompting strategy (zero-shot, few-shot, per criterion) and screening target (papers, GitHub repositories). They can't be changed; a new prompt starts from the built-in text of its strategy and target. Papers: zero-shot is the prompt published in our ESEM '25 paper 2.
  • My prompts - prompts you have written. Only you can see them.

A prompt has a system prompt, which sets the model's role, and a user prompt, which is a template with variables such as {{title}}, {{abstract}}, {{criteria}} (or {{inclusion_criteria}} and {{exclusion_criteria}}), {{additional_instructions}} and, for few-shot prompts, {{few_shot_examples}}. Variables are filled in for each paper when the task runs. Every save is kept as a new version, and each task records the exact prompt version it ran with.

Reviewing results

  • Task details - open a screening task to see each paper's decisions, the model's reasoning per criterion, and the prompts that were sent to the model.
  • Manual evaluation - Start manual evaluation shows one paper at a time with every model's suggestions and the average likelihood of each criterion. Decide with the buttons or the keyboard shortcuts (Y/I include, N/E exclude, U unsure).
  • List of papers - all papers with their average inclusion probability over all tasks, sortable and with evaluated papers hidden if you want. Expand a paper to read its abstract, upload its full text and make a decision.
  • Per-criteria agreement - with at least two completed per-criterion tasks, the project's menu shows agreement statistics (percent agreement and Gwet's AC1) between the tasks for each criterion.
  • Export - download the results as CSV, or open them as an HTML report.

Showcase


Figure 1: Project view shows the LLM screening tasks of a project.


Figure 2: Creating a screening task: pick the screening method and the model, and customize the options, such as the prompt.


Figure 3: Manual evaluation view, with LLM evaluations (binary, ordinal, probability) alongside manual review, and the average likelihood of each criterion.


Figure 4: List of papers, sorted by inclusion probability according to all executed LLMs, with manual decisions.


Figure 5: Prompts view: the built-in prompts, including the one published in ESEM '25, and your own versioned prompts.

Getting started

Data

The tool has been developed and tested with CSV data exported from Scopus and Github with data exported by our custom Github Query Tool. Support for Web of Science can be achieved by editing the columns headers to match the ones from Scopus. The minimum required fields are: Document title, DOI, Abstract, Authors, and Source title.

image

LLMs Access

The application is integrated with OpenRouter, which supports multiple LLMs ranging from very affordable to top-tier models like OpenAI’s ChatGPT, Google’s Gemini, Anthropic’s Claude, Meta's LLama, and Mistral. To use the models, you need to provide an OpenRouter key. You can set spending limits for each key directly on the OpenRouter website.

{585DBE92-5A2F-412E-BEF1-A727015EE872}

LLM screening speed

LLM calls are parallelized, and you should achieve a screening speed exceeding 100 papers per minute when using OpenRouter. The screening speed depends on the model used.

System and software requirements

AISysRev runs entirely in Docker containers: the frontend, backend, database, Redis and the Celery workers. Docker is the only software you need to install to run it; there is nothing to install or configure on the host.

  • Docker, with Compose and buildx plugins installed.
  • Enough RAM (At least 8GB recommended)
  • Enough disk space (Around 8GB)
  • Network connection

Tip

See https://docs.docker.com/desktop/ for Docker installation instructions. Docker Desktop includes Docker Compose, Docker Buildx, Docker Engine and the Docker CLI.

Note

If Docker Desktop did not include the Buildx plugin, see: https://github.com/docker/buildx

Verifying Docker setup and environment

  1. Run docker info to verify you have Docker installed
    • Docker 26.0.0 has been tested as working. For MacOS computers with Colima, Docker version 28.5.1 confirmed to be working.
  2. Run docker buildx version to verify you have Docker Buildx installed. For MacOS computers, Buildx plugin version 0.29.1 confirmed to be working.
  3. Run docker compose version to verify you have Compose installed. For MacOS computers, Compose plugin version 2.40.3 confirmed to be working.
    • Version 2.33.1 has been tested as working, newer versions should also work.
    • Note: Older versions of Compose use docker-compose as the compose command. We don't provide support for legacy Compose versions.

Running the AISysRev application

First, clone the repository to your local computer.

git clone https://github.com/EvoTestOps/AISysRev.git

move to correct directory

cd AISysRev

The default way to run AISysRev on your own machine is local mode: the production build of the app, running on your machine. It needs no login or account: the app runs as a single local user, and all data stays on your machine.

MacOS, Linux and Windows (WSL)

make start

Windows (non-WSL)

If you do not have Windows Subsystem for Linux (WSL), run:

./start.bat

Note

The startup of the app may take a while due to the download of corresponding Docker images & services, application dependencies and building of the application.

The app opens in your browser at https://localhost:3000 as soon as it is reachable, and shows a startup page until it is ready (set NO_BROWSER=1 to skip opening it). The Caddy server's root CA is by default untrusted, so you can bypass the browser warning.

Stopping the app keeps your data: it is stored in Docker volumes, which are only removed if you remove them yourself (e.g. docker compose -p local down -v).

Other modes

  • make start-prod (./start-prod.bat on Windows) runs the same production build with login (OIDC), as deployed on a server, also at https://localhost:3000.
  • make start-dev (./start-dev.bat on Windows) runs the development stack, see Running in development mode.

Technology

The AISysRev tool is built on bleeding front-end and back-end technology, with industry-demonstrated stability. We follow the best known coding conventions and toolkit.

Front-end

TypeScript, React, Tailwind CSS, Vite, Wouter, Zod, Redux

Back-end

Python, FastAPI, PostgreSQL, SQLAlchemy, Alembic

System design

See architecture.md

Development requirements

Development also happens in containers: make start-dev runs the whole stack in Docker and syncs your edits into it, and the tests and migrations run in containers too. The host tools below are only needed for linting, type checking and generating the API client.

Running in development mode

MacOS, Linux and Windows (WSL)

make start-dev

To skip the login, as in local mode: LOCAL_ENV=true make start-dev

Windows (non-WSL)

./start-dev.bat

Local mode

Local mode is the environment variable LOCAL_ENV=true. make start sets it on the production stack, and it can be added to the development stack too. The backend then skips the login: every request acts as a hardcoded mock user (local-user, with a fixed UUID) who has accepted the terms and privacy policy, so no OIDC credentials are needed. Logging out and deleting the account are disabled, and the local OpenAI-compatible model provider is available, as the app runs on the user's own machine. The backend refuses to start with LOCAL_ENV=true if APP_ENV is staging.

Getting started with development

Open up the client: https://localhost:3001

Note

/api is internally proxied to the backend container, e.g. http://localhost:3001/api/v1/health will be proxied to http://localhost:8080/api/v1/health.

API: https://localhost:3001/api/v1

API docs: https://localhost:3001/docs

Adminer GUI: http://localhost:8081/?pgsql=postgres&username=your_username&db=your_database_dev&ns= password: your_password

Generating API types

Generate TypeScript types and Zod schemas from the backend's OpenAPI spec by running, in the repository root:

make openapi

This doesn't need the server or containers running. It imports the FastAPI app locally (with uv), writes app.openapi() to client/openapi.json, and then runs npm run generate:api in client/ to generate client/src/services/api/ (client, types, fetcher) from that spec. The app's startup hooks aren't run, so nothing connects to the database or Redis; the connection URLs are set to placeholder values only because the settings require them. The spec is exported with APP_ENV=dev, so it matches the routes of the dev server.

Alternatively, with make start-dev running, you can run the steps separately in client/:

  • npm run get:openapi-spec — fetches the spec from https://localhost:3001/openapi.json into client/openapi.json.
  • npm run generate:api — generates client/src/services/api/ from that spec.

client/src/services/api/ and the openapi.json are committed to git. Re-run make openapi whenever backend endpoints or schemas change.

A smoke test is in place, which instructs you to update openapi.json then the spec changes.

Other documentation

Mock data

Mock data is located in data/mock -folder.

Tests

Client

Run in client/:

  • npm test for unit and component tests
  • npm run test:e2e for e2e tests

Server

Run in repository root:

  • make backend-unit for backend unit tests only
  • make backend-test for all backend tests
  • make backend-test-html for all backend tests and an HTML coverage report

Windows (non-WSL) users need to run the equivalent docker compose commands from the Makefile manually, since only start.bat, start-dev.bat and start-prod.bat are provided.

Makefile Commands

The project includes a Makefile for common development and database operations:

Development

Command Description
make start Run the production build locally without login (local mode, the default setup), at https://localhost:3000
make start-dev Start dev containers with live reloading and build on startup (add LOCAL_ENV=true to skip the login)
make start-dev-debug Same as start-dev, with the backend debugger enabled
make start-test Start the e2e test stack: the production images with APP_ENV=test, at https://localhost:3002
make start-test-debug Start the dev stack with APP_ENV=test and the backend debugger, at https://localhost:3002
make start-prod Start production container and rebuild images
make backend-unit Run backend unit tests
make backend-test Run all backend tests
make backend-test-html Run all backend tests and create an HTML coverage report
make extract-caddy-ca Copy the production Caddy root CA to ./caddy-local-root.crt so you can trust it locally

Note: Run all commands from the project root.
Containers are isolated by environment using the Docker Compose -p flag.

Database Migrations (Alembic)

Command Description
make m-create m="Message" Create a new migration with an autogenerated diff (replace Message)
make m-up Apply all pending migrations (upgrade to latest)
make m-hist Show the full migration history with details
make m-current Display the current migration version in the database

Resource limits

Each container started with Docker Compose has a CPU and memory limit (deploy.resources.limits in docker-compose-dev.yml, docker-compose.yml and docker-compose-e2e.yml), so a local stack doesn't take over the whole machine. They are roughly sized to the 4-CPU GitHub runner the e2e tests run on. The Kubernetes deployments, defined in a separate deployment repository, set their own resources.

Service CPU Memory
Frontend (dev, Vite) 1 1 GiB
Frontend (prod and test, Caddy) 0.5 128 MiB
Backend 1 512 MiB
Celery 1 1 GiB
PostgreSQL 0.5 1 GiB
Redis 0.25 256 MiB
Flower 0.25 256 MiB
Adminer 0.25 128 MiB

Celery runs 4 worker processes (CELERY_CONCURRENCY) instead of one per CPU, to stay within its memory limit. The e2e test stack has no Flower or Adminer.

Supported LLMs

Currently, we support models provided via Openrouter, OpenAI or via a local provider (OpenAI SDK). The list of supported LLMs is continuously evolving and changing, which is why we don't provide a comprehensive list.

BibTeX Citation

Please use the following BibTeX citation to cite our work:

@inproceedings{huotalaAISysRevLLMbasedTool2026, 
  address={Concordia University Montreal QC Canada}, 
  title={AISysRev - LLM-based Tool for Title-abstract Screening}, 
  ISBN={9798400726361}, 
  url={https://dl.acm.org/doi/10.1145/3803437.3806408}, 
  DOI={10.1145/3803437.3806408}, 
  booktitle={Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering}, 
  publisher={ACM}, 
  author={Huotala, Aleksi and Kuutila, Miikka and Turtio, Olli-Pekka and Sipilä, Simo and Mäntylä, Mika}, 
  year={2026},
  month=july, 
  pages={142–146}, 
  language={en} }

AI disclaimer

Parts of AISysRev may have been developed with AI-assisted coding tools, including "vibe coding". Where AI assistance was used, the correctness and code style of the resulting code were reviewed.

License

MIT

References

[1] Huotala, A., Kuutila, M., Ralph, P., & Mäntylä, M. (2024). The promise and challenges of using llms to accelerate the screening process of systematic reviews. Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, 262–271. https://doi.org/10.1145/3661167.3661172

[2] Huotala A, Kuutila M, Mäntylä M. SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews. In Proceedings of the The 19th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) 2025 Oct (pp. 1-12) https://arxiv.org/abs/2507.19027

[3] Huotala A, Kuutila M, Turtio Olli-Pekka, Sipilä S, Mäntylä M. AISysRev: LLM-based Tool for Title-abstract Screening. In FSE Companion '26: Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering 2026 Jul (pp. 142-146) https://doi.org/10.1145/3803437.3806408

About

LLM-based Tool for Title-abstract Screening

Topics

Resources

Contributing

Stars

22 stars

Watchers

0 watching

Forks

Releases

Used by

Contributors

Languages