Repository navigation
[air] Package code_source without git on PATH - #7625
stevenchen-db wants to merge 3 commits into
Conversation
`air run` packaged every code_source with the git binary, so a host without git failed with a raw exec error. A directory outside any repository now packages without git: files come from libs/git's ignore rules, regular files and symlinks are listed (symlinks not followed), and a nested repository is one "dir/" entry. With git on PATH the git listing is unchanged. Known differences of the git-free listing, only used without git: - Rule precedence: the first matching rule from the root down wins, so a root "*.log" excludes nested/keep.log even with "!keep.log" in nested/.gitignore. - include_paths globs use path.Match, where "*" does not cross "/". A code_source inside a repository, or pinned with code_source.git, still needs git. Without it, `air run` fails before config validation and packaging with an actionable error. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
✅ Integration test reportCommit: e44a0bf
Top 11 slowest tests (at least 2 minutes):
|
git ls-files lists only regular files and symlinks, and packaging a FIFO would block. Also sharpen the fallback's doc comments. Co-authored-by: Isaac <no-reply@databricks.com>
| } | ||
|
|
||
| var names []string | ||
| err = filepath.WalkDir(repoPath, func(p string, d fs.DirEntry, err error) error { |
There was a problem hiding this comment.
If root_path is a symlink to a directory, WalkDir Lstats the root and never descends, so this returns no entries and the user sees the "directory is empty or all of its files are excluded by .gitignore" error. git -C follows the link, so the git path packages it fine. Could we filepath.EvalSymlinks(repoPath) before the walk? The emitted names are relative, so nothing else changes. A test with a symlinked root would pin it.
| return true | ||
| } | ||
| for p := strings.TrimSuffix(name, "/"); p != ""; { | ||
| if ok, _ := path.Match(spec, p); ok { |
There was a problem hiding this comment.
include_paths: ["src/*.py"] selects src/x/y.py under git, since git's pathspec * crosses /. Here path.Match's * doesn't cross / and the ancestor loop only tries prefixes, so the file is dropped without any message. The doc comment calls this a known difference, but on a packaging path a silent omission is worse than an error. Could we either match git's semantics, or fail in the fallback when include_paths contain glob metacharacters? Either way, a row in TestSnapshotFilesWithoutGit_MatchesGitListing would catch it.
| // | ||
| // Known differences from `git ls-files --others --exclude-standard`, which only matter | ||
| // on hosts without git (with git on PATH the git listing is used unchanged): | ||
| // - Rule precedence: libs/git returns the first matching rule from the root down, so |
There was a problem hiding this comment.
Root *.log plus nested !keep.log silently diverges: git includes nested/keep.log, while Repository.Ignore returns on the first positive match from the root down. The fixture in writeFallbackFixture has no conflicting root rule, so this is untested. Could we evaluate the deepest .gitignore first, or at least log a one-line warning when the fallback runs in a tree with nested .gitignore files, so users know the snapshot may differ from what git would select?
| return false | ||
| } | ||
|
|
||
| // emptySnapshotError explains why no file was selected for the snapshot. An empty |
There was a problem hiding this comment.
nit: in the non-repo branch below, the hint suggests git check-ignore -v, which the git-free path can't run, and it's shown whether or not git is installed. Could that branch give a git-agnostic hint, such as which .gitignore file to check?
| @@ -0,0 +1,23 @@ | |||
| //go:build unix | |||
There was a problem hiding this comment.
nit: Windows CI skips this whole file, so the fallback's separator handling (ToSlash on rel names, path.Match on include paths) has no Windows coverage. Would a case in TestSnapshotFilesWithoutGit_MatchesGitListing that runs on every platform be feasible?
|
|
||
| // checkCodeSourceGit fails before the config is validated or anything is packaged when | ||
| // git is missing and the code_source cannot be packaged without it. Runs for dry runs too. | ||
| func checkCodeSourceGit(ctx context.Context, cfg *runConfig, configPath string) error { |
There was a problem hiding this comment.
nit: this re-runs resolveRootPath and gitRequiredError, which resolveSnapshotPlan repeats right after. The early check is justified by running before validation, so a short comment saying so would keep the duplication from looking accidental.
pardis-beikzadeh-db
left a comment
There was a problem hiding this comment.
The listing used when git isn't installed can be simplified a lot. Please fold these in when you rebase on main.
|
|
||
| // matchesIncludePath reports whether name, or a directory containing it, matches one of | ||
| // the include_paths. | ||
| func matchesIncludePath(name string, includePaths []string) bool { |
There was a problem hiding this comment.
Building on Maggie's glob thread on line 320: if the fallback rejects glob include_paths (isPathspecPattern on main already detects them), it can walk each include_paths entry directly, the way libs/fileset.FileSet does. Then this function and the *.py known-difference test can go.
| if ignored { | ||
| return fs.SkipDir | ||
| } | ||
| if hasGitMetadata(p) { |
There was a problem hiding this comment.
On main, snapshotFiles drops directory entries (#7643), so after the rebase this dir/ entry never reaches the archive. Returning fs.SkipDir here is enough.
|
|
||
| // insideGitRepository is the git-free stand-in for isRepository, used only when no git | ||
| // binary is available: repository metadata in dir or one of its ancestors marks a work tree. | ||
| func insideGitRepository(dir string) bool { |
There was a problem hiding this comment.
folders.FindDirWithLeaf(dir, git.GitDirectoryName) already does this upward walk; libs/git/info.go uses it the same way. FindDirWithLeaf treats an empty .git directory as a repository, which is fine here, so hasGitMetadata and its test can go.
Changes
databricks air runno longer needsgiton PATH to package acode_sourcedirectory that isn't in a git repository.libs/git's ignore rules (root and nested.gitignore,info/exclude,core.excludesFile). Symlinks are listed, not followed; a nested repository is one entry, as with git.code_source.gitset, no git: fails before validation, packaging, or any API call, with an actionable error.Known differences of the git-free listing (only used without git; pinned by tests): a root rule wins over a nested negation (
*.logat the root excludesnested/keep.logeven with!keep.loginnested/.gitignore), andinclude_pathsglobs usepath.Match, where*doesn't cross/.Why
Hosts without git (containers, CI images, agent sandboxes) couldn't use
code_sourceat all, though git is only needed for its ignore rules outside a repository. The quickstart lists git as a requirement; a docs follow-up can narrow that to code inside a git repository.No Jira ticket.
Tests
git ls-filesfor whole trees, directory and fileinclude_paths, nested.gitignore, negation, ignored dirs, dotfiles and symlinks; nested repo as one entry; the two known differences; the early error for in-repo and pinned code (with an injected git lookup, also asserting no lookup withoutcode_source).code_sourcerun succeeded and printed its output (run 892749685192587); the in-repo case failed with 0 API calls; a workload withoutcode_sourceis unchanged. With git on PATH, the dry-run file count and request list match main.TestAccept/air/20/20, lint clean,./task checkspasses.TestLogsCommandRepeatsRunLinkAfterStreamingfails the same way on main and is unrelated.This pull request and its description were written by Isaac.