Repository navigation
sdk: Add REQUIRE_COUCHDB_INTEGRATION_TESTS to fail CouchDB tests - #653
paul-gerber-svg wants to merge 2 commits into
Conversation
zrgt
left a comment
There was a problem hiding this comment.
- Since #639 is merged, this now conflicts in
test/_helper/test_helpers.pyand in the unittest step ofpr.yml. Please rebase ontodevelopand keep both flags (REQUIRE_SCHEMA_TESTSandREQUIRE_COUCHDB_INTEGRATION_TESTS). Ideally both should share one small helper to parse the env var instead of duplicating.lower() in {"1", "true", "yes"}. test/examples/test_tutorials.pyalso usesskipUnless(COUCHDB_OKAY)for the CouchDB tutorial test, so it can still skip silently in CI. Could you cover it with the same flag?
|
ping @paul-gerber-svg |
ef11093 to
dfe1177
Compare
|
Rebased onto |
zrgt
left a comment
There was a problem hiding this comment.
Thanks, failing instead of silently skipping is a good change. A few points inline. The first four were already wrong before this PR, but they sit in code it touches, so this seems like a good moment to fix them.
| ), | ||
| ) | ||
| @unittest.skipUnless(COUCHDB_OKAY or REQUIRE_COUCHDB_INTEGRATION_TESTS, COUCHDB_UNREACHABLE_MESSAGE) | ||
| def test_tutorial_backend_couchdb(self): |
There was a problem hiding this comment.
test_tutorial_backend_couchdb still has only pass as its body. It never imports basyx.aas.examples.tutorial_backend_couchdb, so with REQUIRE_COUCHDB_INTEGRATION_TESTS=1 it passes even if the tutorial is broken. Requiring CouchDB here adds no coverage. The import was probably removed by ruff's unused-import fix. Could you import the tutorial inside the test body (with a # noqa: F401 if needed)? The other tutorial tests in this file are empty the same way.
| @@ -53,3 +62,7 @@ | |||
| except urllib.error.URLError as e: | |||
There was a problem hiding this comment.
Now that CI relies on this probe, it should be more robust:
urlopen(request)on L59 has notimeout. A CouchDB that accepts the connection but never answers stalls the job until the CI timeout.- Only
URLErroris caught.ConnectionResetError,http.client.RemoteDisconnectedandTimeoutErrorare notURLErrorsubclasses. They escape at import time and break every test module that importstest_helpers, including the schema tests, instead of producing the clear message.
Suggestion: urllib.request.urlopen(request, timeout=5) and except OSError as e: (URLError is an OSError).
|
|
||
| If no CouchDB server at the configured URL, the script will exit with exit code 1. To avoid the error exit code (for use | ||
| in CI), provide the ``--failsafe`` option. | ||
| If no CouchDB server is reachable at the configured URL, or the test database already exists, the script exits with |
There was a problem hiding this comment.
The docstring presents --failsafe as safe for idempotent re-runs, but it has two side effects that should be mentioned:
- It also exits 0 when the server is unreachable.
- If the database already exists, the script exits before it creates the test user and the
_securityentry. If an earlier run failed after creating the DB, re-running with--failsafeleaves the user missing, the probe gets a 401, and the tests skip or fail.
| COUCHDB_OKAY, | ||
| COUCHDB_UNREACHABLE_MESSAGE, | ||
| REQUIRE_COUCHDB_INTEGRATION_TESTS, | ||
| TEST_CONFIG, |
There was a problem hiding this comment.
.lstrip("http://") strips the character set {h, t, p, :, /}, not the prefix. With http://test-couchdb:5984 the result is couchdb://est-couchdb:5984/.... source_core is not used anywhere, so I'd delete it (or use removeprefix if it's needed).
| TEST_CONFIG["couchdb"]["url"], TEST_CONFIG["couchdb"]["database"], COUCHDB_ERROR | ||
| ), | ||
| ) | ||
| @unittest.skipUnless(COUCHDB_OKAY or REQUIRE_COUCHDB_INTEGRATION_TESTS, COUCHDB_UNREACHABLE_MESSAGE) |
There was a problem hiding this comment.
The same condition is checked twice here: once in skipUnless(COUCHDB_OKAY or REQUIRE_...) and again in setUpClass. test_tutorials.py has a third variant that uses self.fail. A single helper in test_helpers.py would keep them in sync, following the pattern JsonSerializationSchemaTest.setUpClass already uses:
def require_couchdb() -> None:
if not COUCHDB_OKAY:
if REQUIRE_COUCHDB_INTEGRATION_TESTS:
raise RuntimeError(COUCHDB_UNREACHABLE_MESSAGE)
raise unittest.SkipTest(COUCHDB_UNREACHABLE_MESSAGE)| COUCHDB_OKAY = False | ||
| COUCHDB_ERROR = e | ||
|
|
||
| COUCHDB_UNREACHABLE_MESSAGE = "No CouchDB is reachable at {}/{}: {}".format( |
There was a problem hiding this comment.
Minor: this message is always built, so when CouchDB is reachable it reads No CouchDB is reachable at ...: None. It could be built only in the failure path (e.g. inside the helper suggested in test_couchdb.py).
| ) | ||
|
|
||
|
|
||
| def env_flag(name: str) -> bool: |
There was a problem hiding this comment.
Nice helper. server/test/_helper/test_helpers.py still parses REQUIRE_SERVER_INTEGRATION_TESTS with the same inline expression, though. Should it use the same accepted values, so the two flags don't drift apart?
|
|
||
| # Likewise, the CouchDB tests are skipped when no CouchDB is reachable. Set this environment variable to fail them | ||
| # instead. | ||
| REQUIRE_COUCHDB_INTEGRATION_TESTS = env_flag("REQUIRE_COUCHDB_INTEGRATION_TESTS") |
There was a problem hiding this comment.
Nit: module-level constants could be annotated Final[bool] / Final[str] (this one and COUCHDB_UNREACHABLE_MESSAGE).
CouchDB tests in
test/backend/test_couchdb.pyare skipped wheneverno CouchDB is reachable. In CI, the
sdk-testjob always starts aCouchDB service container, so a broken setup there just skipped the
tests instead of failing the build.
This adds a
REQUIRE_COUCHDB_INTEGRATION_TESTSenvironment variable,same idea as
REQUIRE_SERVER_INTEGRATION_TESTSfor the server'sDocker integration tests: set it to fail instead of skip when no
CouchDB is reachable. CI now sets it for the
sdk-testjob.Also fixes the
setup_testdb.pydocstring, which said CI passes--failsafeto avoid failing on an unreachable CouchDB, which was false.