[DMP 2026]: IDinsight Experiment Analytics & Bandits Engine Enhancements

[markbotterill]

Ticket Contents

Description

Evidential is an open-source experimentation platform designed for nonprofits to run rigorous, low-cost experiments (A/B tests, Bayesian methods, and bandits) on digital and hybrid programs. This ticket focuses on building and improving the experimentation analytics and adaptive allocation engine.

Goals & Mid-Point Milestone

Goals

• Continue to build out the multi-armed bandit engine (e.g. Thompson Sampling / UCB)
• Add contextual bandits support using user-level features
• Improve experiment results unified dashboard
• Improve existing integrations with platforms like Turn.io

Setup/Installation

Quickstart guide is here

Expected Outcome

1-2 months collaboration with the team culminating in multiple feature Pull Requests into the Agency Fund / IDinsight repository. Features that are most appealing and productive for the collaboration can be clarified in subsequent early-stage meetings.

Acceptance Criteria

Knowledge of Python, basic statistics (we are willing to mentor the RL aspects of the project) and (ideally) some broader deployment/ infrastructure knowledge (integrating with APIs, deploying via Railway, GHA etc.)

Implementation Details

Backend:

• Python (FastAPI)
• SQLAlchemy / Postgres
• Frontend in React

Experimentation:

• Frequentist methods:
  • t-tests / proportion tests
• Bayesian methods:
  • Beta-Bernoulli models
• Bandits:
  • Thompson Sampling
  • UCB
  • Contextual bandits (Bayesian linear regression)

Data:

• Event data + warehouse integration
• Metric computation layer

APIs:

• Experiment creation
• Result retrieval
• Allocation updates

Mockups/Wireframes

No response

Product Name

Evidential

Organisation Name

IDinsight

Domain

Open Source Library

Tech Skills Needed

Python, SQL, Docker, CI/CD, React

Mentor(s)

@markbotterill @poornimaramesh

Category

Data Science

[Somsubhra-Nandi]

Hi @markbotterill @poornimaramesh,

I'm very interested in tackling Issue #766 for DMP 2026. I have strong experience building out complex Python backends (FastAPI) and automated data pipelines, and I'd love to help build out the experimentation analytics engine.

I noticed the quickstart guide link and located the main codebase over at agency-fund/evidential-be and have started reviewing the current FastAPI routes and SQLAlchemy models.

Before I draft a formal implementation plan for the Multi-Armed Bandit (Thompson Sampling / UCB) logic, I had a quick architectural question: Are we planning to calculate the metric computation layer purely on the fly within the FastAPI backend, or should the logic rely heavily on the Postgres warehouse integration for pre-aggregating the event data?

I will drop by the Discord soon to discuss further, but I'm excited to start working on this!

[santhosh-7777]

Hi @markbotterill @poornimaramesh,
This is a really interesting problem space, especially building a robust experimentation platform with adaptive allocation using bandits.
I’ve been working on projects involving data-driven decision systems and ML-backed pipelines, including:

• Building data processing pipelines for extracting and structuring real-time inputs for downstream decision logic
• Designing backend systems where model outputs dynamically influence system behavior (inference → decision → response)
• Working with Python-based APIs and integrating statistical/ML components into production-style workflows
  These experiences have given me a strong foundation in handling event-driven data, metric computation, and designing systems where decisions evolve based on incoming data — which aligns closely with experimentation engines and bandit-based allocation.
  I’ve started exploring the Evidential backend to understand how experiment data flows across ingestion, metric computation, and allocation. I’m particularly interested in contributing to:
• Thompson Sampling / UCB implementation
• Designing scalable allocation update mechanisms
• Extending toward contextual bandits using user-level features
  One aspect I’m thinking about is how to design the exploration–exploitation balance in a way that remains stable under sparse or noisy data, especially in early stages of experiments.
  I’ll continue exploring the repository and share a concrete implementation plan soon.
  Looking forward to contributing!

[Sarfaraz604]

@markbotterill @poornimaramesh

I'm Sarfaraz, a 4th semester CS student at IIIT Agartala. I've been going through the Evidential codebase and this ticket genuinely excites me — the intersection of adaptive experimentation and nonprofit impact is a problem space I want to invest real time in.

From exploring the backend, I can see the allocation layer currently handles Thompson Sampling at a per-experiment level. A few things I'm already thinking about:

• For contextual bandits, covariate shift will happen for user-level features through time, so I'm interested in implementing an online posterior Bayesian linear regression that would enable us to perform stable allocation decisions without batch retraining.
• For UCB algorithms, it's important to know whether bounds are calculated each time for allocation decisions or are they cached? This might become crucial in production when running multiple tests simultaneously.
• For the dashboard, it is crucial to display per-arm posteriors as well in order to provide more insightful information about the results of experiments for our nonprofit customers who don't necessarily have expertise in statistics.

I feel comfortable on the full stack: from Python/FastAPI backend, passing by SQLAlchemy database layer and React frontend to Docker containerization.

Having experience in working with data driven decision systems in combination with statistical models gives me some additional advantage regarding this task.

I will propose my approach to implementing this module next week. I can discuss implementation via call or even use Discord if that would help moving things faster.

This issue seems very interesting to me, so I'd really like to work on it.

[Diyapareta]

Hi @markbotterill @poornimaramesh

I’ve been going through Issue #766 and exploring the Evidential backend, and I’d really like to contribute to this.

My background is primarily in Python and backend development using FastAPI, where I’ve worked on building APIs, handling data flow, and structuring systems that process and respond to incoming data. While I’m relatively new to bandit algorithms, I’m comfortable with core statistics and I’m actively learning concepts like Thompson Sampling and UCB as I go through the codebase.

From my initial exploration, I’m particularly interested in contributing to the allocation layer and understanding how experiment data flows from event ingestion to metric computation and finally to decision-making. I’d like to start by working on a clear and simple implementation of the bandit logic (Thompson Sampling/UCB), and then gradually explore improvements or extensions once I have a solid understanding of the current system.

One thing I’m trying to understand better is how tightly coupled the allocation logic is with the metric computation layer — whether decisions are expected to rely on real-time computed metrics or pre-aggregated data from the warehouse.

I’ll continue digging into the repository and share a more concrete approach soon. Looking forward to contributing and learning through this project!

Thanks!