Skip to content

Remove volumes when removing container.#2378

Open
jbehley wants to merge 1 commit into
codalab:developfrom
jbehley:fix_remove_volumes
Open

Remove volumes when removing container.#2378
jbehley wants to merge 1 commit into
codalab:developfrom
jbehley:fix_remove_volumes

Conversation

@jbehley
Copy link
Copy Markdown

@jbehley jbehley commented May 18, 2026

This change resolves an issue, which occured with our codabench_worker using Podman. (Not sure if the same happens with Docker at some point)

The problem

Our worker stopped working after couple of weeks and it took some time to understand the root cause of the problem, since the worker got always stuck when creating the container. We only recognized after we investigated several options that podman volume ls | wc -l resulted in 2047 entries, which was too close to a "threshold". As it turns out, the create_container was failing as podman could not create additional volumes and therefore our process was always running into an execution timeout.

We could resolve the problem by removing all dangling volumes that have been created (podman volumes rm --all). After that "cleaning", the worker is now operational again.

The fix

Therefore, we propose the change to have remove_container also remove the volumes via the option v=True. In our tests, this helps to avoid the problem described above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant