Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,9 @@ jobs:
env:
# Fail instead of silently skipping if the schema files above are missing (see test_helpers.py).
REQUIRE_SCHEMA_TESTS: "1"
# A CouchDB service container is always started for this job (see `services:` above), so a
# missing or misconfigured CouchDB should fail the build, not silently skip the CouchDB tests.
REQUIRE_COUCHDB_INTEGRATION_TESTS: "1"
run: |
python -m coverage run --source=basyx -m unittest
- name: Report test coverage
Expand Down
6 changes: 4 additions & 2 deletions sdk/test/_helper/setup_testdb.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,10 @@

setup_testdb.py -u admin -p admin_password

If no CouchDB server at the configured URL, the script will exit with exit code 1. To avoid the error exit code (for use
in CI), provide the ``--failsafe`` option.
If no CouchDB server is reachable at the configured URL, or the test database already exists, the script exits with

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docstring presents --failsafe as safe for idempotent re-runs, but it has two side effects that should be mentioned:

  • It also exits 0 when the server is unreachable.
  • If the database already exists, the script exits before it creates the test user and the _security entry. If an earlier run failed after creating the DB, re-running with --failsafe leaves the user missing, the probe gets a 401, and the tests skip or fail.

exit code 1 by default. CI relies on this to fail fast if the CouchDB service container isn't healthy, rather than
silently continuing into a test run that would just skip the CouchDB-backed tests. For idempotent/local re-runs
where an already-existing database is expected and fine, pass ``--failsafe`` to exit with code 0 instead.
"""

import argparse
Expand Down
17 changes: 15 additions & 2 deletions sdk/test/_helper/test_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,22 @@
)


def env_flag(name: str) -> bool:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice helper. server/test/_helper/test_helpers.py still parses REQUIRE_SERVER_INTEGRATION_TESTS with the same inline expression, though. Should it use the same accepted values, so the two flags don't drift apart?

"""Return whether the environment variable ``name`` is set to "1", "true" or "yes" (case-insensitive)."""
return os.environ.get(name, "false").lower() in {"1", "true", "yes"}


# By default, the schema validation tests are skipped when the AAS schema files are missing, so that a plain local
# `python -m unittest` run doesn't require downloading them first. Set this environment variable to "1"/"true"
# (e.g. in CI) to instead make those tests fail if the files are missing, so a broken download can't silently cause
# the tests to be skipped without anyone noticing.
REQUIRE_SCHEMAS = os.environ.get("REQUIRE_SCHEMA_TESTS", "false").lower() in {"1", "true", "yes"}
REQUIRE_SCHEMAS = env_flag("REQUIRE_SCHEMA_TESTS")

# Likewise, the CouchDB tests are skipped when no CouchDB is reachable. Set this environment variable to fail them
# instead.
REQUIRE_COUCHDB_INTEGRATION_TESTS = env_flag("REQUIRE_COUCHDB_INTEGRATION_TESTS")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: module-level constants could be annotated Final[bool] / Final[str] (this one and COUCHDB_UNREACHABLE_MESSAGE).


# Check if CouchDB database is available. Otherwise, skip tests.
# Check if CouchDB database is available. Otherwise, skip tests (unless REQUIRE_COUCHDB_INTEGRATION_TESTS is set).
try:
request = urllib.request.Request(
"{}/{}".format(
Expand All @@ -53,3 +62,7 @@
except urllib.error.URLError as e:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that CI relies on this probe, it should be more robust:

  • urlopen(request) on L59 has no timeout. A CouchDB that accepts the connection but never answers stalls the job until the CI timeout.
  • Only URLError is caught. ConnectionResetError, http.client.RemoteDisconnected and TimeoutError are not URLError subclasses. They escape at import time and break every test module that imports test_helpers, including the schema tests, instead of producing the clear message.

Suggestion: urllib.request.urlopen(request, timeout=5) and except OSError as e: (URLError is an OSError).

COUCHDB_OKAY = False
COUCHDB_ERROR = e

COUCHDB_UNREACHABLE_MESSAGE = "No CouchDB is reachable at {}/{}: {}".format(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: this message is always built, so when CouchDB is reachable it reads No CouchDB is reachable at ...: None. It could be built only in the failure path (e.g. inside the helper suggested in test_couchdb.py).

TEST_CONFIG["couchdb"]["url"], TEST_CONFIG["couchdb"]["database"], COUCHDB_ERROR
)
20 changes: 13 additions & 7 deletions sdk/test/backend/test_couchdb.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,12 @@
from basyx.aas.backend import couchdb
from basyx.aas.examples.data.example_aas import *

from test._helper.test_helpers import COUCHDB_ERROR, COUCHDB_OKAY, TEST_CONFIG
from test._helper.test_helpers import (
COUCHDB_OKAY,
COUCHDB_UNREACHABLE_MESSAGE,
REQUIRE_COUCHDB_INTEGRATION_TESTS,
TEST_CONFIG,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.lstrip("http://") strips the character set {h, t, p, :, /}, not the prefix. With http://test-couchdb:5984 the result is couchdb://est-couchdb:5984/.... source_core is not used anywhere, so I'd delete it (or use removeprefix if it's needed).

)

source_core: str = (
"couchdb://"
Expand All @@ -21,13 +26,14 @@
)


@unittest.skipUnless(
COUCHDB_OKAY,
"No CouchDB is reachable at {}/{}: {}".format(
TEST_CONFIG["couchdb"]["url"], TEST_CONFIG["couchdb"]["database"], COUCHDB_ERROR
),
)
@unittest.skipUnless(COUCHDB_OKAY or REQUIRE_COUCHDB_INTEGRATION_TESTS, COUCHDB_UNREACHABLE_MESSAGE)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same condition is checked twice here: once in skipUnless(COUCHDB_OKAY or REQUIRE_...) and again in setUpClass. test_tutorials.py has a third variant that uses self.fail. A single helper in test_helpers.py would keep them in sync, following the pattern JsonSerializationSchemaTest.setUpClass already uses:

def require_couchdb() -> None:
    if not COUCHDB_OKAY:
        if REQUIRE_COUCHDB_INTEGRATION_TESTS:
            raise RuntimeError(COUCHDB_UNREACHABLE_MESSAGE)
        raise unittest.SkipTest(COUCHDB_UNREACHABLE_MESSAGE)

class CouchDBBackendTest(unittest.TestCase):

@classmethod
def setUpClass(cls) -> None:
if not COUCHDB_OKAY:
raise RuntimeError(f"REQUIRE_COUCHDB_INTEGRATION_TESTS is set. {COUCHDB_UNREACHABLE_MESSAGE}")

def setUp(self) -> None:
self.couch_identifiable_store = couchdb.CouchDBIdentifiableStore(
TEST_CONFIG["couchdb"]["url"], TEST_CONFIG["couchdb"]["database"]
Expand Down
14 changes: 4 additions & 10 deletions sdk/test/examples/test_tutorials.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

from basyx.aas import model

from .._helper.test_helpers import COUCHDB_ERROR, COUCHDB_OKAY, TEST_CONFIG
from .._helper.test_helpers import COUCHDB_OKAY, COUCHDB_UNREACHABLE_MESSAGE, REQUIRE_COUCHDB_INTEGRATION_TESTS


class TutorialTest(unittest.TestCase):
Expand All @@ -35,16 +35,10 @@ def test_tutorial_storage(self):
pass
# The tutorial already includes assert statements for the relevant points. So no further checks are required.

@unittest.skipUnless(
COUCHDB_OKAY,
"No CouchDB is reachable at {}/{}: {}".format(
TEST_CONFIG["couchdb"]["url"],
TEST_CONFIG["couchdb"]["database"],
COUCHDB_ERROR,
),
)
@unittest.skipUnless(COUCHDB_OKAY or REQUIRE_COUCHDB_INTEGRATION_TESTS, COUCHDB_UNREACHABLE_MESSAGE)
def test_tutorial_backend_couchdb(self):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_tutorial_backend_couchdb still has only pass as its body. It never imports basyx.aas.examples.tutorial_backend_couchdb, so with REQUIRE_COUCHDB_INTEGRATION_TESTS=1 it passes even if the tutorial is broken. Requiring CouchDB here adds no coverage. The import was probably removed by ruff's unused-import fix. Could you import the tutorial inside the test body (with a # noqa: F401 if needed)? The other tutorial tests in this file are empty the same way.

pass
if not COUCHDB_OKAY:
self.fail(f"REQUIRE_COUCHDB_INTEGRATION_TESTS is set. {COUCHDB_UNREACHABLE_MESSAGE}")
# The tutorial already includes assert statements for the relevant points. So no further checks are required.

def test_tutorial_serialization_deserialization_json(self):
Expand Down