Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Inside a container the usual answers describe the machine, not the process. `psu

Reading the cgroup files directly is not enough either. The limit that applies is rarely written where the process is, and a cgroup that restricts nothing does not say so by leaving its limit file empty. A consumer that reads one file and takes it at face value believes it has 8 EB of memory.

This package handles both, and reports only what actually restricts the process. `None` means nothing visible to the process restricts it - the machine is then the honest answer, and `get_machine_cpu_count()` and `get_machine_memory_bytes()` below give it.
This package handles both, and reports only what actually restricts the process. `None` means nothing visible to the process restricts it, or a limit could not be read or used. `describe().notices` says which. The machine is then what is left to size from, and `get_machine_cpu_count()` and `get_machine_memory_bytes()` below give it.

Off Linux and Windows every limit reads as `None`. macOS still reports the memory and the CPU cores of the machine. On other platforms the machine facts can read as `None`. Some examples below pair this package with `psutil`, which it does not require - `psutil` answers what the machine is using, which is not a question about limits.

Expand Down Expand Up @@ -193,6 +193,8 @@ A limit of `None` with no notice about it means nothing went wrong: the mechanis
| `CPU_USAGE_SCOPE_MISMATCH` | the level the CPU limit applies to counts no CPU time, so no rate can be measured | cgroup v1 with `cpu` and `cpuacct` on separate mounts, where the level holding the quota has no counterpart under the accounting one |
| `MEMORY_METRICS_UNAVAILABLE` | no mechanism here is present to carry a memory limit | a host with no cgroup filesystem mounted, or a platform this package has no backend for |
| `CPU_METRICS_UNAVAILABLE` | no mechanism here is present to carry a CPU limit | the same |
| `MEMORY_METRICS_UNREADABLE` | where a memory limit would be read from cannot be located, because a `/proc` file cannot be read; the next reading tries again | a `/proc` file that could not be read while no discovery is kept: before the first one succeeded, or after `clear_cache()` or a fork |
| `CPU_METRICS_UNREADABLE` | the same for a CPU limit | the same |
| `MEMORY_ANCESTORS_HIDDEN` | only part of the memory hierarchy is visible, so a limit above it is enforced but unreadable | a container given a subtree as its whole `/sys/fs/cgroup` |
| `CPU_ANCESTORS_HIDDEN` | only part of a CPU hierarchy is visible | the same |
| `MEMORY_LIMIT_UNREADABLE` | a level holds a memory limit that says nothing usable, so what it enforces is unknown | a hand-built or mocked cgroup tree; no kernel writes such a value |
Expand Down
192 changes: 161 additions & 31 deletions src/proclimits/_cgroup.py

Large diffs are not rendered by default.

108 changes: 78 additions & 30 deletions src/proclimits/_sensor.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from dataclasses import dataclass

from . import _backend
from ._types import Notice, NoticeCode, RawMemory, Source
from ._types import Notice, NoticeCode, RawCpu, RawMemory, Source

_SHORTEST_WINDOW_SECONDS = 0.01
"""Below this a rate says more about the counter's own resolution than about the load."""
Expand Down Expand Up @@ -96,6 +96,9 @@ class Description:

A limit of `None` next to no notice about it means the mechanism was there and nothing limited this
process in a way that kills it. Only hard limits are read.

The sources are located after the readings, so beside a `*_METRICS_UNREADABLE` notice they can show what a
later lookup found.
"""

memory_budget: MemoryBudget | None
Expand All @@ -113,18 +116,21 @@ class Description:
"""

memory_source: Source | None
"""Where the memory limit and usage are read from. `None` where no mechanism is present."""
"""Where the memory limit and usage are read from. `None` where no mechanism is present, or where it
cannot be located right now."""

cpu_quota_source: Source | None
"""Where the CPU bandwidth quota is read from. `None` where no mechanism is present."""
"""Where the CPU bandwidth quota is read from. `None` where no mechanism is present, or where it cannot be
located right now."""

cpu_set_source: Source | None
"""Where the set of allowed cores is read from. `None` where no mechanism is present."""
"""Where the set of allowed cores is read from. `None` where no mechanism is present, or where it cannot
be located right now."""

cpu_usage_source: Source | None
"""Where the consumed CPU time is read from, as a mechanism and not as a level. Which level is counted is
under `cpu_usage`, and `cpu_rate_level` names where a rate is measured. `None` where no mechanism is
present."""
present, or where it cannot be located right now."""

raw_memory_limit: int | None
"""The tightest memory limit before filtering. Sentinels included."""
Expand Down Expand Up @@ -327,14 +333,13 @@ def describe() -> Description:
"""Explain how the sensor arrives at its readings.

Every reading is taken again, so a limit changed between two calls shows up here. Where the readings come
from was discovered once - `clear_cache()` is what forgets that.
from was discovered once - `clear_cache()` is what forgets that. A lookup that cannot read `/proc` is not
kept.

Returns:
The readings, the source of each metric, the raw values before filtering, the machine facts, and a
notice for whatever is worth knowing about them.
"""
sources = _backend.sources()

# One read of each filtering fact, here and in the report, so what decided a rejection is the number
# reported beside it.
machine_memory = get_machine_memory_bytes()
Expand All @@ -343,6 +348,9 @@ def describe() -> Description:
memory = _evaluate_memory(ceiling)
cpu = _evaluate_cpu_limit(machine_cpus)

# After the readings: a lookup that succeeded is kept, so a limit read above has its source here.
sources = _backend.sources()

return Description(
memory_budget=memory.effective,
cpu_limit=cpu.effective,
Expand Down Expand Up @@ -370,8 +378,9 @@ def describe() -> Description:
def clear_cache() -> None:
"""Forget the discovered metric sources.

Where a mechanism has anything to discover, it is discovered once and kept. Call this after the process
was moved into another group, and the next reading locates the sources again.
Where a mechanism has anything to discover, it is discovered once and kept. A lookup that cannot read
`/proc` is not kept. Call this after the process was moved into another group, and the next reading
locates the sources again.

A child of `fork` clears it on its own, so a pre-fork server whose supervisor puts each worker into a
group of its own needs no call here.
Expand Down Expand Up @@ -590,15 +599,7 @@ def _memory_rejection(raw: RawMemory, ceiling: int | None) -> Notice | None:
if raw.limit is None:
# No limit and no mechanism read the same way from the outside, and only one of them is a fact about
# this machine.
return (
Notice(
code=NoticeCode.MEMORY_METRICS_UNAVAILABLE,
message='No mechanism here carries a memory limit, so nothing was read. A Linux machine with '
'no cgroup filesystem mounted looks like this, and so does a platform with no backend here.',
)
if _backend.sources().memory is None
else None
)
return _missing_memory_mechanism(raw)

if ceiling is None:
return Notice(
Expand Down Expand Up @@ -691,15 +692,9 @@ def _evaluate_cpu_limit(machine_cpus: int | None) -> _CpuEvaluation:
notices = []
candidates = []

if not restrictions and _no_cpu_mechanism():
if not restrictions and (missing := _missing_cpu_mechanism(raw)) is not None:
# As for memory: "nothing limits the CPU here" and "nothing here can say" are different answers.
notices.append(
Notice(
code=NoticeCode.CPU_METRICS_UNAVAILABLE,
message='No mechanism here carries a CPU limit, so nothing was read. A Linux machine with no '
'cgroup filesystem mounted looks like this, and so does a platform with no backend here.',
)
)
notices.append(missing)

for restriction in restrictions:
# A reading in cores is trusted where the machine core count is unknown: cores have no sentinel, so
Expand Down Expand Up @@ -751,11 +746,64 @@ def _no_cpu_limit(notice: Notice) -> _CpuEvaluation:
)


def _no_cpu_mechanism() -> bool:
"""Whether no mechanism here could carry a CPU limit of either kind."""
def _missing_memory_mechanism(raw: RawMemory) -> Notice | None:
"""Say why no source of a memory limit is here: none is present, or none can be located right now.

`None` where one is.
"""
# A lookup that failed leaves no source either, and says nothing about this machine. The reading's own lookup
# decides first, so a lookup that succeeds after it cannot hide the failure.
if raw.unlocated_file is not None:
return _unlocated_notice(NoticeCode.MEMORY_METRICS_UNREADABLE, 'memory', raw.unlocated_file)

# The reading's lookup succeeded or found no `/proc`, and either answer is kept, so this one is usually the
# same. Another thread can clear the cache in between, and this lookup then looks again and can fail.
sources = _backend.sources()

if sources.unlocated_file is not None:
return _unlocated_notice(NoticeCode.MEMORY_METRICS_UNREADABLE, 'memory', sources.unlocated_file)

if sources.memory is not None:
return None

return Notice(
code=NoticeCode.MEMORY_METRICS_UNAVAILABLE,
message='No mechanism here carries a memory limit, so nothing was read. A Linux machine with '
'no cgroup filesystem mounted looks like this, and so does a platform with no backend here.',
)


def _missing_cpu_mechanism(raw: RawCpu) -> Notice | None:
"""Say why no source of a CPU limit of either kind is here: none is present, or none can be located right now.

`None` where one is.
"""
# As for memory: the reading's own lookup decides first, and this one can still fail after a cleared cache.
if raw.unlocated_file is not None:
return _unlocated_notice(NoticeCode.CPU_METRICS_UNREADABLE, 'CPU', raw.unlocated_file)

sources = _backend.sources()

return sources.cpu_quota is None and sources.cpu_set is None
if sources.unlocated_file is not None:
return _unlocated_notice(NoticeCode.CPU_METRICS_UNREADABLE, 'CPU', sources.unlocated_file)

if sources.cpu_quota is not None or sources.cpu_set is not None:
return None

return Notice(
code=NoticeCode.CPU_METRICS_UNAVAILABLE,
message='No mechanism here carries a CPU limit, so nothing was read. A Linux machine with no '
'cgroup filesystem mounted looks like this, and so does a platform with no backend here.',
)


def _unlocated_notice(code: NoticeCode, metric: str, unlocated_file: str) -> Notice:
"""Say that where a limit would be read from cannot be located, because a file cannot be read."""
return Notice(
code=code,
message=f'Where a {metric} limit would be read from cannot be located, because {unlocated_file} cannot '
'be read. Nothing is reported, and the next reading tries again.',
)


def get_machine_memory_bytes() -> int | None:
Expand Down
42 changes: 40 additions & 2 deletions src/proclimits/_types.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,20 @@ class NoticeCode(str, Enum):
CPU_METRICS_UNAVAILABLE = 'cpu-metrics-unavailable'
"""No mechanism here is present to carry a CPU limit: an unsupported platform, or none is mounted."""

MEMORY_METRICS_UNREADABLE = 'memory-metrics-unreadable'
"""Where a memory limit would be read from cannot be located, because a `/proc` file cannot be read.

Nothing is reported. A read that failed, so the next reading tries again. The message names the file and
the reason.
"""

CPU_METRICS_UNREADABLE = 'cpu-metrics-unreadable'
"""Where a CPU limit would be read from cannot be located, because a `/proc` file cannot be read.

Nothing is reported. A read that failed, so the next reading tries again. The message names the file and
the reason.
"""

MEMORY_LIMIT_UNREADABLE = 'memory-limit-unreadable'
"""A level holds a memory limit that says nothing usable, so what it enforces is unknown."""

Expand Down Expand Up @@ -145,10 +159,19 @@ class Sources:
cpu_usage: Source | None
"""Carries the consumed CPU time, which a mechanism may count somewhere other than where it limits."""

unlocated_file: str | None
"""The file that kept the sources from being located, and why, as `<path> (<reason>)`. `None` where none did.

Set, every source is `None`, and the next lookup tries again.
"""


@dataclass(frozen=True)
class RawMemory:
"""The memory limit and usage as the mechanism spells them."""
"""The memory limit and usage as the mechanism spells them.

Where `unlocated_file` is set, no level was read and every other field is `None`.
"""

limit: int | None
"""The tightest limit along the chain, in bytes. `None` when no level holds one.
Expand Down Expand Up @@ -200,6 +223,13 @@ class RawMemory:
no level to name.
"""

unlocated_file: str | None
"""The `/proc` file this reading could not read to locate its controller, and why, as `<path> (<reason>)`.

`None` where the lookup succeeded or found no `/proc`. A job object is not located through `/proc`, so it is
always `None` there.
"""


@dataclass(frozen=True)
class RawCpuQuota:
Expand Down Expand Up @@ -247,7 +277,8 @@ class RawCpuSet:
class RawCpu:
"""Everything the CPU controls say, read in one go.

A level that says nothing usable concerns both restrictions, so they are read together.
A level that says nothing usable concerns both restrictions, so they are read together. Where
`unlocated_file` is set, no level was read and every other field is `None`.
"""

quota: RawCpuQuota | None
Expand All @@ -270,3 +301,10 @@ class RawCpu:
mechanisms that express a limit that way ever set this - a cgroup states cores outright, so it is always
`None` there.
"""

unlocated_file: str | None
"""The `/proc` file this reading could not read to locate its controllers, and why, as `<path> (<reason>)`.

`None` where the lookup succeeded or found no `/proc`. A job object is not located through `/proc`, so it is
always `None` there.
"""
17 changes: 12 additions & 5 deletions src/proclimits/_windows.py
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,7 @@ class _SystemInfo(ctypes.Structure):
limit_level=None,
unreadable_level=None,
usage_unreadable_level=None,
unlocated_file=None,
)
"""What is reported where no job limits the memory."""

Expand All @@ -304,6 +305,7 @@ class _SystemInfo(ctypes.Structure):
limit_level=None,
unreadable_level=_LEVEL,
usage_unreadable_level=None,
unlocated_file=None,
)
"""What is reported where this process is in a job that will not say what it enforces."""

Expand Down Expand Up @@ -337,6 +339,7 @@ def read_memory() -> RawMemory:
limit_level=_LEVEL,
unreadable_level=None,
usage_unreadable_level=None,
unlocated_file=None,
)

if job_limit is None:
Expand All @@ -358,22 +361,23 @@ def read_cpu() -> RawCpu:
cpu_set=None,
unreadable_level=None if error == _ERROR_NO_JOB else _LEVEL,
unconvertible_level=None,
unlocated_file=None,
)

# One unreadable answer drops both restrictions, because the unread one may be the tighter. No machine
# separates the two: rate control is a job information class wherever a supported CPython runs, and both
# were measured to answer or fail together.
control, _ = _query(_JOB_CPU_RATE_CONTROL, _RateControl)
if control is None:
return RawCpu(quota=None, cpu_set=None, unreadable_level=_LEVEL, unconvertible_level=None)
return RawCpu(quota=None, cpu_set=None, unreadable_level=_LEVEL, unconvertible_level=None, unlocated_file=None)

rate = _rate_cap(control)
machine_cores = machine_cpu_count()

if rate is not None and machine_cores is None:
# A rate is a share of the machine, so without the size of the machine it says nothing about cores.
# The affinity mask goes with it: the share that cannot be sized may be the tighter of the two.
return RawCpu(quota=None, cpu_set=None, unreadable_level=None, unconvertible_level=_LEVEL)
return RawCpu(quota=None, cpu_set=None, unreadable_level=None, unconvertible_level=_LEVEL, unlocated_file=None)

quota = (
RawCpuQuota(cores=rate * machine_cores, limit_level=_LEVEL, usage_level=_LEVEL)
Expand All @@ -391,11 +395,13 @@ def read_cpu() -> RawCpu:
if allowed is None:
# The job restricts the cores and will not say which, so what it allows is unknown rather than
# unrestricted.
return RawCpu(quota=None, cpu_set=None, unreadable_level=_LEVEL, unconvertible_level=None)
return RawCpu(
quota=None, cpu_set=None, unreadable_level=_LEVEL, unconvertible_level=None, unlocated_file=None
)

cpu_set = RawCpuSet(cores=allowed, limit_level=_LEVEL, usage_level=_LEVEL)

return RawCpu(quota=quota, cpu_set=cpu_set, unreadable_level=None, unconvertible_level=None)
return RawCpu(quota=quota, cpu_set=cpu_set, unreadable_level=None, unconvertible_level=None, unlocated_file=None)


def read_cpu_usage(level: str | None = None) -> float | None:
Expand Down Expand Up @@ -430,7 +436,7 @@ def sources() -> Sources:
"""
source = Source(interface=Interface.WINDOWS_JOB_OBJECT, levels=(_LEVEL,) if _in_job() else ())

return Sources(memory=source, cpu_quota=source, cpu_set=source, cpu_usage=source)
return Sources(memory=source, cpu_quota=source, cpu_set=source, cpu_usage=source, unlocated_file=None)


def mechanism_notices() -> tuple[Notice, ...]:
Expand Down Expand Up @@ -574,6 +580,7 @@ def _job_memory(limit: int) -> RawMemory:
# Nothing charged takes two unrelated APIs refusing at once, and has never been seen on a machine.
# A job-wide limit does have a usage figure, so this is a guard rather than a path.
usage_unreadable_level=_LEVEL if charged is None else None,
unlocated_file=None,
)


Expand Down
Loading
Loading