[chores] Investigate and brainstorm ways to speed up CI

[pandafy]

The CI pipeline is currently running over 1000 tests, which can result in prolonged build and feedback times. This issue is for brainstorming and tracking ideas to optimize, speed up, or parallelize the CI process.

Potential discussion points:

• Identifying the slowest parts of the CI
• Opportunities for test parallelism or sharding
• Caching dependencies or test results
• Removing unnecessary or redundant tests
• Leveraging incremental or selective test runs for PRs
• CI configuration or workflow improvements

Contributions and suggestions from the community are welcome!

[k-sumayya]

Hi, I would like to contribute to this discussion.

I noticed that the CI runs a large number of tests (1000+), which likely increases execution time significantly. Based on this, I have a few suggestions:

1. Test Parallelization
   If not already implemented, splitting the test suite across multiple jobs could reduce total runtime.

2. Caching Improvements
   It may help to cache Python dependencies (pip) and possibly Docker layers to avoid repeated installations.

3. Identify Slow Tests
   Running tests with profiling (e.g., pytest --durations) could help identify bottlenecks in the slowest tests.

4. Selective Test Execution
   For certain PRs, it might be possible to run only relevant tests based on changed modules.

I would be interested in helping analyze the current CI workflow and identifying specific slow points or improvements.

Please let me know if I can proceed with profiling or investigating the CI setup further.

Thank you!

[nemesifier]

@k-sumayya tests already run in parallel AFAIK and selenium tests are run only once, so I don't think there's much room for 1 and 4 but feel free to explore if you think you can find anything positive.

2 and 3 are probably the most relevant points. However, please keep in mind that letting LLM tools handle this without understanding what is really going on will not be considered a valid contribution, will be flagged as spam and closed. We need real effort, not LLM prompting.

[k-sumayya]

Thanks for the clarification!

That makes sense about parallelization and Selenium tests. I’ll focus on caching and understanding how tests are currently executed.

I’m going to go through the CI workflow files and also try running the test suite locally to see where most of the time is spent (using pytest durations).

Once I have some concrete observations, I’ll share specific findings instead of assumptions.

Appreciate the guidance!

[asmodehn]

I would personally focus on :

• Identifying the slowest parts of the CI
• Removing unnecessary or redundant tests
• CI configuration or workflow improvements

over the other points, as I think they carry a risk to make the test setup more complex and fragile than it currently is.

Also the number of test isn't a problem in itself, but the time they take versus the amount of confidence they provide the developer might not feel worth it sometimes...
=> we should look for ways to make tests more meaningful and useful.

For improvements I would look at :

• Give detailed feedback to the developer earlier in the workflow, before tests, via type hints (I already started doing this in the extended version I m working on)
• Reduce the matrix of tests. We probably don't need all django / python version tests all the time on all branches (I'm also doing this, focusing on packaged versions in debian releases). Not sure if it will actually speedup the CI though, depending on how github allocates resources for a project... Might be worth a try, if it wasn't done before ?
• Simplify the test setup if possible and make tool usage consistent across all repos (django test / selenium / pytest / coverage / --parallel / etc.)
• Simplify the test setup by (gradually) making tests/openwisp2/settings.py similar in all repos.
  The aim is to force the developer to embed all necessary, specific, settings for a test with the test code (via override_settings or modify_settings as context manager), which shall in turn give us more isolated/modular sets of tests.
  Hopefully these tests can, in time, be run in various ways with more confidence, and shall eventually prove more reliable and useful. Maybe we also will, along the way, find a better way to organize tests...
  Full disclosure: The lack of isolation in tests is the biggest time sink for me currently, as it forces me to rerun them multiple times in various contexts to find subtle issues (or just a missing setting)...

And finally, we can investigate potential simplifications/optimizations of the whole software stack, which we can start only after identifying the actual bottlenecks.

I can help with some of these, let me know what you think.