Repository navigation
air list: cache terminal runs' MLflow links on the Jobs scan path - #7627
stevenchen-db wants to merge 2 commits into
Conversation
The Jobs scan (default list, --all-users, user= filters) builds its rows from runs/list on every call and re-resolves each row's MLflow link with a get-output and an MLflow request, about 40 requests for 20 rows. A terminal run's link does not change, so cache it, like the index path already does for its rows. The scan uses its own cache component holding only the MLflow fields, so its entries never stand in for the index path's full rows. An entry is used only while the run's task runs and attempts match the ones it was resolved from, so a run repaired after it was cached is refetched. Active runs are never cached. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
✅ Integration test reportCommit: 38e141a
Top 3 slowest tests (at least 2 minutes):
|
| // putMLflowLinks caches terminal entries that have an MLflow link. | ||
| func putMLflowLinks(ctx context.Context, c *cache.Cache, host string, entries []listedRun) { | ||
| for _, e := range entries { | ||
| if !e.terminal || e.row.MLflowURL == "" || e.row.MLflowURL == "-" { |
There was a problem hiding this comment.
nit: a failed MLflow name lookup gets cached here. fetchMLflowRunName (mlflow.go:146) returns "" on any error and mlflowRunLabel then falls back to the run ID, so the ID is cached as the label for up to 60 days. The index path already does the same (list_index.go:146), so this isn't new, but this PR extends it to the scan path. Could we skip the put when the name lookup failed (e.g. a nameKnown flag on listedRun)? A test where GetRun errors would pin it.
| @@ -0,0 +1 @@ | |||
| * `air list --all-users` and `air list --filter user=` cache finished runs' MLflow links, so repeat listings are about 3x faster. ([#7627](https://github.com/databricks/cli/pull/7627)) | |||
There was a problem hiding this comment.
nit: the default list is server-side ActiveOnly, so finished rows only appear with --all-status. Could the entry mention --all-status rather than --all-users / --filter user=?
| assert.Equal(t, []string{"70"}, outputRunIDs, "get-output must use the task run id") | ||
| } | ||
|
|
||
| func TestJobsScanCachesTerminalMLflowLinks(t *testing.T) { |
There was a problem hiding this comment.
nit: the repair and active-run cases are well covered. Two cases are missing: a terminal run whose first lookup has no link (-) shouldn't be cached and should resolve once the link appears, and a corrupted cache file should fall back to a refetch.
pardis-beikzadeh-db
left a comment
There was a problem hiding this comment.
LGTM. One non-blocking comment inline.
| // MLflow links appear only in the text table, so the per-run get-output | ||
| // lookups are skipped for JSON output (which omits the column anyway). | ||
| if f.fetchMLflow { | ||
| scan, isScan := f.strategy.(*jobsScanStrategy) |
There was a problem hiding this comment.
problem: The scan's caching is hard to find. Someone reading or changing jobsScanStrategy won't see that it caches, because the cache reads and writes happen here in the shared runFetcher.next, behind scan-only type checks. The index path keeps all of its caching inside its own strategy.
suggested fix: Move it into the strategy the way the index path does: apply cached links in jobsScanStrategy.next() where the row is built, and write them back from a (s *jobsScanStrategy) cacheMLflowLinks method called next to the index one.
Changes
air listmodes served by the Jobs scan (--all-users,--filter user=…, and the default active list) build their rows fromruns/liston every call and re-resolve each row's MLflow link with aget-outputand an MLflow request. A terminal run's link never changes, so it is now cached on disk, like the index path (--all-statusfor your own runs) already caches its rows.Why
On df1,
air list --all-users --all-statusandair list --all-status --filter user=<me>go from ~2.0s to ~0.6s once warm (20 terminal rows, ~40 requests skipped). The first call is unchanged.Tests
TestJobsScanCachesTerminalMLflowLinks: a terminal run is resolved once and then served from the cache; a repaired run (new attempt) is refetched; an active run is never cached; nothing is written to the index path's cache. Removing the cache hook or the task-run check each fails it.go test ./cmd/air(list tests also with-race),TestAccept/air, lint pass.--all-status, JSON, and the default list are identical to main.This pull request and its description were written by Isaac.