Repository navigation
Improve handling of FASTAPI Asyncio in API #4924
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Chris Chapman (ChrisChapman-gh)
merged 31 commits into
microsoft:main
from
JC-wk:asyncio-prevent-garbage-collection
Oct 26, 2026
Merged
Changes from all commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
74c89c8
implement app-scoped lifecycle management to prevent silent garbage c…
7be55fe
changelog
4f4d616
Merge remote-tracking branch 'origin/main' into asyncio-prevent-garba…
eb3bf72
bump api
c2a4be7
implement copilot suggestions
a51b3eb
Merge branch 'main' into asyncio-prevent-garbage-collection
JC-wk 4cd1a7b
Merge branch 'main' into asyncio-prevent-garbage-collection
JC-wk f3f0b01
Merge branch 'main' into asyncio-prevent-garbage-collection
JC-wk 422d029
Merge branch 'main' into asyncio-prevent-garbage-collection
JC-wk 2d043f3
Bump version to 0.25.27 and implement BackgroundTaskManager for impro…
d819444
Merge branch 'main' into asyncio-prevent-garbage-collection
JC-wk 8a403f8
suppress bandit warning
95075a9
Merge branch 'main' into asyncio-prevent-garbage-collection
JC-wk 8a5e234
Merge branch 'main' into asyncio-prevent-garbage-collection
JC-wk 619344f
Merge branch 'main' into asyncio-prevent-garbage-collection
JC-wk 1c7e021
Refactor shutdown handling in lifespan to improve task exception logg…
baaf3b8
Bump version to 0.26.9
JC-wk 0e46e29
Merge branch 'main' into asyncio-prevent-garbage-collection
JC-wk 582bcb1
Prevent background tasks from being garbage collected during shutdown
2fca0e9
Merge branch 'main' into asyncio-prevent-garbage-collection
JC-wk 2e2d61f
Bump version to 0.26.10
JC-wk 37fefb5
Refactor shutdown task exception handling and update tests for logging
JC-wk f16f6e4
Merge branch 'main' into asyncio-prevent-garbage-collection
JC-wk efa00ce
Merge branch 'main' into asyncio-prevent-garbage-collection
JC-wk 72d62e5
Merge branch 'main' into asyncio-prevent-garbage-collection
ChrisChapman-gh ecfbd8e
Merge branch 'main' into asyncio-prevent-garbage-collection
JC-wk 9f11bd1
Apply Ruff formatting to API lifecycle code and tests
ca1283f
Merge branch 'main' into asyncio-prevent-garbage-collection
ChrisChapman-gh f02bc6f
bump version
0a8cdca
Merge branch 'main' into asyncio-prevent-garbage-collection
JC-wk 11485a7
Merge branch 'main' into asyncio-prevent-garbage-collection
ChrisChapman-gh File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| __version__ = "0.28.7" | ||
| __version__ = "0.28.8" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,267 @@ | ||
| import asyncio | ||
| from unittest.mock import AsyncMock, MagicMock, patch | ||
| import pytest | ||
| from fastapi import FastAPI | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_background_task_manager(): | ||
| from main import BackgroundTaskManager | ||
|
|
||
| manager = BackgroundTaskManager() | ||
| assert manager.is_shutting_down is False | ||
|
|
||
| async def dummy(): | ||
| pass | ||
|
|
||
| task = asyncio.create_task(dummy()) | ||
| manager.add(task) | ||
| assert task in manager.get_tasks() | ||
|
|
||
| manager.discard(task) | ||
| assert task not in manager.get_tasks() | ||
|
|
||
| await task | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| @patch("main.bootstrap_database", return_value=True) | ||
| @patch("main.DeploymentStatusUpdater") | ||
| @patch("main.AirlockStatusUpdater") | ||
| async def test_lifespan_lifecycle_normal(mock_airlock, mock_deployment, mock_bootstrap): | ||
| from main import BackgroundTaskManager, lifespan | ||
|
|
||
| mock_deployment_instance = MagicMock() | ||
| mock_deployment_instance.init_repos = AsyncMock() | ||
| mock_deployment_instance.receive_messages = AsyncMock() | ||
| mock_deployment.return_value = mock_deployment_instance | ||
|
|
||
| mock_airlock_instance = MagicMock() | ||
| mock_airlock_instance.init_repos = AsyncMock() | ||
| mock_airlock_instance.receive_messages = AsyncMock() | ||
| mock_airlock.return_value = mock_airlock_instance | ||
|
|
||
| app = FastAPI() | ||
|
|
||
| async with lifespan(app): | ||
| assert hasattr(app.state, "background_tasks") | ||
| assert isinstance(app.state.background_tasks, BackgroundTaskManager) | ||
| tasks = app.state.background_tasks.get_tasks() | ||
| assert len(tasks) == 2 | ||
| names = {t.get_name() for t in tasks} | ||
| assert "deployment-status-updater" in names | ||
| assert "airlock-status-updater" in names | ||
|
|
||
| assert app.state.background_tasks.is_shutting_down is True | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| @patch("main.bootstrap_database", return_value=True) | ||
| @patch("main.DeploymentStatusUpdater") | ||
| @patch("main.AirlockStatusUpdater") | ||
| @patch("main.logger") | ||
| async def test_lifespan_task_failure_during_runtime(mock_logger, mock_airlock, mock_deployment, mock_bootstrap): | ||
| from main import lifespan | ||
|
|
||
| mock_deployment_instance = MagicMock() | ||
| mock_deployment_instance.init_repos = AsyncMock() | ||
|
|
||
| async def fail_immediately(): | ||
| raise ValueError("Runtime failure") | ||
|
|
||
| mock_deployment_instance.receive_messages = fail_immediately | ||
| mock_deployment.return_value = mock_deployment_instance | ||
|
|
||
| mock_airlock_instance = MagicMock() | ||
| mock_airlock_instance.init_repos = AsyncMock() | ||
| mock_airlock_instance.receive_messages = AsyncMock() | ||
| mock_airlock.return_value = mock_airlock_instance | ||
|
|
||
| app = FastAPI() | ||
|
|
||
| async with lifespan(app): | ||
| await asyncio.sleep(0.1) | ||
|
|
||
| # Verify that the runtime failure was logged as an error | ||
| error_logs = [ | ||
| call | ||
| for call in mock_logger.error.call_args_list | ||
| if "Background task deployment-status-updater failed" in call[0][0] | ||
| ] | ||
| assert len(error_logs) == 1 | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| @patch("main.bootstrap_database", return_value=True) | ||
| @patch("main.DeploymentStatusUpdater") | ||
| @patch("main.AirlockStatusUpdater") | ||
| @patch("main.logger") | ||
| async def test_lifespan_task_failure_during_shutdown(mock_logger, mock_airlock, mock_deployment, mock_bootstrap): | ||
| from main import lifespan | ||
|
|
||
| mock_deployment_instance = MagicMock() | ||
| mock_deployment_instance.init_repos = AsyncMock() | ||
|
|
||
| async def raise_teardown_exception(): | ||
| try: | ||
| await asyncio.sleep(5) | ||
| except asyncio.CancelledError: | ||
| raise RuntimeError("Database connection closed abruptly") | ||
|
|
||
| mock_deployment_instance.receive_messages = raise_teardown_exception | ||
| mock_deployment.return_value = mock_deployment_instance | ||
|
|
||
| mock_airlock_instance = MagicMock() | ||
| mock_airlock_instance.init_repos = AsyncMock() | ||
| mock_airlock_instance.receive_messages = AsyncMock() | ||
| mock_airlock.return_value = mock_airlock_instance | ||
|
|
||
| app = FastAPI() | ||
|
|
||
| async with lifespan(app): | ||
| await asyncio.sleep(0.1) | ||
|
|
||
| # The error should NOT be logged as error in _done_callback because it was gated on is_shutting_down | ||
| for call in mock_logger.error.call_args_list: | ||
| assert "Background task deployment-status-updater failed" not in call[0][0] | ||
|
|
||
| # Instead, it should be caught during shutdown inspection and logged as warning | ||
| warning_logs = [ | ||
| call for call in mock_logger.warning.call_args_list if "raised exception during shutdown" in call[0][0] | ||
| ] | ||
| assert len(warning_logs) == 1 | ||
| assert "deployment-status-updater" in warning_logs[0][0][0] | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| @patch("main.bootstrap_database", return_value=True) | ||
| @patch("main.DeploymentStatusUpdater") | ||
| @patch("main.AirlockStatusUpdater") | ||
| @patch("main.logger") | ||
| @patch("main.SHUTDOWN_TIMEOUT", 0.05) | ||
| async def test_lifespan_timeout_preserves_completed_task_exceptions( | ||
| mock_logger, mock_airlock, mock_deployment, mock_bootstrap | ||
| ): | ||
| from main import lifespan | ||
|
|
||
| mock_deployment_instance = MagicMock() | ||
| mock_deployment_instance.init_repos = AsyncMock() | ||
|
|
||
| # Deployment updater completes quickly during cancellation by raising an exception | ||
| async def raise_teardown_exception(): | ||
| try: | ||
| await asyncio.sleep(5) | ||
| except asyncio.CancelledError: | ||
| raise RuntimeError("Deployment teardown failed") | ||
|
|
||
| mock_deployment_instance.receive_messages = raise_teardown_exception | ||
| mock_deployment.return_value = mock_deployment_instance | ||
|
|
||
| hang_event = asyncio.Event() | ||
|
|
||
| # Airlock updater suppresses cancellation and hangs past the timeout | ||
| async def hang_on_cancellation(): | ||
| try: | ||
| await asyncio.sleep(5) | ||
| except asyncio.CancelledError: | ||
| await hang_event.wait() | ||
| raise RuntimeError("Airlock teardown failed late") | ||
|
|
||
| mock_airlock_instance = MagicMock() | ||
| mock_airlock_instance.init_repos = AsyncMock() | ||
| mock_airlock_instance.receive_messages = hang_on_cancellation | ||
| mock_airlock.return_value = mock_airlock_instance | ||
|
|
||
| app = FastAPI() | ||
|
|
||
| try: | ||
| async with lifespan(app): | ||
| await asyncio.sleep(0.01) | ||
|
|
||
| # 1. Completed task exception is still inspected and logged as warning despite timeout | ||
| warning_logs = [ | ||
| call for call in mock_logger.warning.call_args_list if "raised exception during shutdown" in call[0][0] | ||
| ] | ||
| assert len(warning_logs) == 1 | ||
| assert "deployment-status-updater" in warning_logs[0][0][0] | ||
|
|
||
| # 2. Timeout error is logged | ||
| error_logs = [ | ||
| call | ||
| for call in mock_logger.error.call_args_list | ||
| if "Timeout waiting for background tasks to shutdown" in call[0][0] | ||
| ] | ||
| assert len(error_logs) == 1 | ||
|
|
||
| # 3. Pending task is logged with warning | ||
| pending_logs = [ | ||
| call | ||
| for call in mock_logger.warning.call_args_list | ||
| if "did not terminate in time during shutdown" in call[0][0] | ||
| ] | ||
| assert len(pending_logs) == 1 | ||
| assert "airlock-status-updater" in pending_logs[0][0][0] | ||
| finally: | ||
| hang_event.set() | ||
| await asyncio.sleep(0.01) | ||
| late_shutdown_logs = [ | ||
| call | ||
| for call in mock_logger.warning.call_args_list | ||
| if "Background task airlock-status-updater raised exception during shutdown" in call[0][0] | ||
| ] | ||
| assert len(late_shutdown_logs) == 1 | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| @patch("main.bootstrap_database", return_value=True) | ||
| @patch("main.DeploymentStatusUpdater") | ||
| @patch("main.AirlockStatusUpdater") | ||
| @patch("main.logger") | ||
| @patch("main.SHUTDOWN_TIMEOUT", 0.05) | ||
| async def test_lifespan_cancellation_suppression_strictly_bounded( | ||
| mock_logger, mock_airlock, mock_deployment, mock_bootstrap | ||
| ): | ||
| import time | ||
| from main import lifespan | ||
|
|
||
| mock_deployment_instance = MagicMock() | ||
| mock_deployment_instance.init_repos = AsyncMock() | ||
|
|
||
| hang_event = asyncio.Event() | ||
|
|
||
| # Worker suppresses cancellation and ignores cancel signal | ||
| async def suppress_cancellation(): | ||
| try: | ||
| await asyncio.sleep(5) | ||
| except asyncio.CancelledError: | ||
| await hang_event.wait() | ||
|
|
||
| mock_deployment_instance.receive_messages = suppress_cancellation | ||
| mock_deployment.return_value = mock_deployment_instance | ||
|
|
||
| mock_airlock_instance = MagicMock() | ||
| mock_airlock_instance.init_repos = AsyncMock() | ||
| mock_airlock_instance.receive_messages = suppress_cancellation | ||
| mock_airlock.return_value = mock_airlock_instance | ||
|
|
||
| app = FastAPI() | ||
|
|
||
| try: | ||
| start_time = time.monotonic() | ||
| async with lifespan(app): | ||
| await asyncio.sleep(0.01) | ||
| elapsed = time.monotonic() - start_time | ||
|
|
||
| # Teardown should return promptly near SHUTDOWN_TIMEOUT and NOT hang | ||
| assert elapsed < 0.5 | ||
|
|
||
| # Both tasks should be reported as pending | ||
| pending_logs = [ | ||
| call | ||
| for call in mock_logger.warning.call_args_list | ||
| if "did not terminate in time during shutdown" in call[0][0] | ||
| ] | ||
| assert len(pending_logs) == 2 | ||
| finally: | ||
| hang_event.set() | ||
| await asyncio.sleep(0.01) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.