Skip to content

sdk: Add REQUIRE_COUCHDB_INTEGRATION_TESTS to fail CouchDB tests - #653

Open
paul-gerber-svg wants to merge 2 commits into
eclipse-basyx:developfrom
rwth-iat:improve/couchdb-require-integration-tests
Open

paul-gerber-svg wants to merge 2 commits into
eclipse-basyx:developfrom
rwth-iat:improve/couchdb-require-integration-tests

Conversation

@paul-gerber-svg

Copy link
Copy Markdown
Contributor

CouchDB tests in test/backend/test_couchdb.py are skipped whenever
no CouchDB is reachable. In CI, the sdk-test job always starts a
CouchDB service container, so a broken setup there just skipped the
tests instead of failing the build.

This adds a REQUIRE_COUCHDB_INTEGRATION_TESTS environment variable,
same idea as REQUIRE_SERVER_INTEGRATION_TESTS for the server's
Docker integration tests: set it to fail instead of skip when no
CouchDB is reachable. CI now sets it for the sdk-test job.

Also fixes the setup_testdb.py docstring, which said CI passes
--failsafe to avoid failing on an unreachable CouchDB, which was false.

@zrgt
zrgt self-requested a review October 6, 2026 13:22

@zrgt zrgt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Since #639 is merged, this now conflicts in test/_helper/test_helpers.py and in the unittest step of pr.yml. Please rebase onto develop and keep both flags (REQUIRE_SCHEMA_TESTS and REQUIRE_COUCHDB_INTEGRATION_TESTS). Ideally both should share one small helper to parse the env var instead of duplicating .lower() in {"1", "true", "yes"}.
  • test/examples/test_tutorials.py also uses skipUnless(COUCHDB_OKAY) for the CouchDB tutorial test, so it can still skip silently in CI. Could you cover it with the same flag?

@zrgt

zrgt commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

ping @paul-gerber-svg

@paul-gerber-svg
paul-gerber-svg force-pushed the improve/couchdb-require-integration-tests branch from ef11093 to dfe1177 Compare November 19, 2026 13:22
@paul-gerber-svg

Copy link
Copy Markdown
Contributor Author

Rebased onto develop and kept both flags. Both are now parsed by a shared env_flag() helper in test_helpers.py, and the CouchDB tutorial test uses the same flag. Note
that the tutorial test itself is still a no-op (#634), so the flag only makes it fail when no CouchDB is reachable.

@paul-gerber-svg
paul-gerber-svg requested a review from zrgt November 19, 2026 14:50

@zrgt zrgt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, failing instead of silently skipping is a good change. A few points inline. The first four were already wrong before this PR, but they sit in code it touches, so this seems like a good moment to fix them.

),
)
@unittest.skipUnless(COUCHDB_OKAY or REQUIRE_COUCHDB_INTEGRATION_TESTS, COUCHDB_UNREACHABLE_MESSAGE)
def test_tutorial_backend_couchdb(self):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_tutorial_backend_couchdb still has only pass as its body. It never imports basyx.aas.examples.tutorial_backend_couchdb, so with REQUIRE_COUCHDB_INTEGRATION_TESTS=1 it passes even if the tutorial is broken. Requiring CouchDB here adds no coverage. The import was probably removed by ruff's unused-import fix. Could you import the tutorial inside the test body (with a # noqa: F401 if needed)? The other tutorial tests in this file are empty the same way.

@@ -53,3 +62,7 @@
except urllib.error.URLError as e:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that CI relies on this probe, it should be more robust:

  • urlopen(request) on L59 has no timeout. A CouchDB that accepts the connection but never answers stalls the job until the CI timeout.
  • Only URLError is caught. ConnectionResetError, http.client.RemoteDisconnected and TimeoutError are not URLError subclasses. They escape at import time and break every test module that imports test_helpers, including the schema tests, instead of producing the clear message.

Suggestion: urllib.request.urlopen(request, timeout=5) and except OSError as e: (URLError is an OSError).


If no CouchDB server at the configured URL, the script will exit with exit code 1. To avoid the error exit code (for use
in CI), provide the ``--failsafe`` option.
If no CouchDB server is reachable at the configured URL, or the test database already exists, the script exits with

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docstring presents --failsafe as safe for idempotent re-runs, but it has two side effects that should be mentioned:

  • It also exits 0 when the server is unreachable.
  • If the database already exists, the script exits before it creates the test user and the _security entry. If an earlier run failed after creating the DB, re-running with --failsafe leaves the user missing, the probe gets a 401, and the tests skip or fail.

COUCHDB_OKAY,
COUCHDB_UNREACHABLE_MESSAGE,
REQUIRE_COUCHDB_INTEGRATION_TESTS,
TEST_CONFIG,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.lstrip("http://") strips the character set {h, t, p, :, /}, not the prefix. With http://test-couchdb:5984 the result is couchdb://est-couchdb:5984/.... source_core is not used anywhere, so I'd delete it (or use removeprefix if it's needed).

TEST_CONFIG["couchdb"]["url"], TEST_CONFIG["couchdb"]["database"], COUCHDB_ERROR
),
)
@unittest.skipUnless(COUCHDB_OKAY or REQUIRE_COUCHDB_INTEGRATION_TESTS, COUCHDB_UNREACHABLE_MESSAGE)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same condition is checked twice here: once in skipUnless(COUCHDB_OKAY or REQUIRE_...) and again in setUpClass. test_tutorials.py has a third variant that uses self.fail. A single helper in test_helpers.py would keep them in sync, following the pattern JsonSerializationSchemaTest.setUpClass already uses:

def require_couchdb() -> None:
    if not COUCHDB_OKAY:
        if REQUIRE_COUCHDB_INTEGRATION_TESTS:
            raise RuntimeError(COUCHDB_UNREACHABLE_MESSAGE)
        raise unittest.SkipTest(COUCHDB_UNREACHABLE_MESSAGE)

COUCHDB_OKAY = False
COUCHDB_ERROR = e

COUCHDB_UNREACHABLE_MESSAGE = "No CouchDB is reachable at {}/{}: {}".format(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: this message is always built, so when CouchDB is reachable it reads No CouchDB is reachable at ...: None. It could be built only in the failure path (e.g. inside the helper suggested in test_couchdb.py).

)


def env_flag(name: str) -> bool:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice helper. server/test/_helper/test_helpers.py still parses REQUIRE_SERVER_INTEGRATION_TESTS with the same inline expression, though. Should it use the same accepted values, so the two flags don't drift apart?


# Likewise, the CouchDB tests are skipped when no CouchDB is reachable. Set this environment variable to fail them
# instead.
REQUIRE_COUCHDB_INTEGRATION_TESTS = env_flag("REQUIRE_COUCHDB_INTEGRATION_TESTS")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: module-level constants could be annotated Final[bool] / Final[str] (this one and COUCHDB_UNREACHABLE_MESSAGE).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants