Usage with vitest browser breaks when using codspeed-runner

[JCQuintas]

Use Case

I know this is probably not a supported use-case

We want to benchmark the test in a real-browser environment to test closer to what the user will actually experience. Also, some of our planned tests would not work at all in a jsdom environment.

We looked at codspeed to help us detect performance changes between pull request, as to not slow down the MUI X Charts library unexpectedly.

Issue

Tests break when running them with codspeed-runner using vitest in browser mode with playwright

They break in the CodSpeedHQ/action, and when installing and running codspeed-runner manually. But they don't break if we run vitest bench

reproducible repo: https://github.com/JCQuintas/codspeed-mui-example
failed actions: https://github.com/JCQuintas/codspeed-mui-example/actions/runs/10074486926

[adriencaccia]

Hey @JCQuintas, thanks for the issue.

It seems that there will be a bit of work to support browser testing within CodSpeed, too bad that the error is not really helpful 😕.
I will need to investigate.

In the meantime, to still be able to leverage CodSpeed and detect performance changes, you can still set up CodSpeed using jsdom for tests where it is possible. I expect that performance changes will still be picked up in this simulated environment, even if the relative values will be different.

[bernardobelchior]

Hey, @adriencaccia! Is there any plan and timeline on adding support for this feature? It would be helpful to us because our library targets browsers, so testing in JSDOM doesn't accurately measure the performance impact of certain changes, namely manipulating DOM elements.

[adriencaccia]

Hey @bernardobelchior,

Thanks for updating the issue.
The bad news is that our usual instrumentation approach won't work in browsers - we rely on valgrind under the hood, which just can't run there.

We could potentially do walltime measurements using our macro runners. Real time would be measured, and it would be possible to actually run in real browsers and catch performance changes.

We don't support macro runners for JS yet, but if this sounds useful to you guys, it's definitely something we could look into adding. Would that approach work for what you're trying to do?

EDIT: fixed the link

[bernardobelchior]

Hey, @adriencaccia,

Thanks for the answer. The link you provided leads to a not found page.

If by wall time measurement you mean something like this example from the docs:

Then I guess it would already help a little.

I assume we'd lose access to the flame graph, but if the graph is constructed from the JSDOM data then it isn't as useful. We'll always need to measure it manually using a browser to debug a performance regression, so I think it's fine if we don't have it.

[adriencaccia]

Thanks for pointing out that the docs link was not working, I fixed it.

I see that this could work.
Regarding flame graphs in walltime, we are actively working to have them for the current supported languages, using profiling (I do not know yet how if that will play nice with browsers though).

Regarding prioritisation of this feature, it's not high on our roadmap at the moment, since we prioritize paying customers feature requests first. If that interests you, we can set up a shared private Slack channel to elaborate on this 🙂

[bernardobelchior]

I see that this could work. Regarding flame graphs in walltime, we are actively working to have them for the current supported languages, using profiling (I do not know yet how if that will play nice with browsers though).

Nice, thanks.

Regarding prioritisation of this feature, it's not high on our roadmap at the moment, since we prioritize paying customers feature requests first.

Yeah, makes sense.

If that interests you, we can set up a shared private Slack channel to elaborate on this 🙂

Yes, that would be nice.

[adriencaccia]

Great! @bernardobelchior I sent you an invite to join the Slack channel