Skip to content

loader: implement package maps#62239

Open
arcanis wants to merge 4 commits intonodejs:mainfrom
arcanis:mael/package-maps
Open

loader: implement package maps#62239
arcanis wants to merge 4 commits intonodejs:mainfrom
arcanis:mael/package-maps

Conversation

@arcanis
Copy link
Contributor

@arcanis arcanis commented Mar 13, 2026

This PR adds a new --experimental-package-map=<path> flag letting Node.js resolve packages using a static JSON file instead of walking node_modules directories.

node --experimental-package-map=./package-map.json app.js

Why?

The node_modules resolution algorithm predates npm and its clear definition of the concept of packages. It works well enough and is widely supported, but has known issues:

  • Phantom dependencies - packages can accidentally import things they don't declare, because hoisting makes transitive dependencies visible

  • Peer dependency resolution is broken in monorepos - if website-v1 uses react@18 and website-v2 uses react@19, and both use a shared component-lib with React as a peer dep, there's no node_modules layout that resolves correctly. The shared lib always gets whichever React was hoisted.

  • Hoisting is lossy - runtimes can't tell if an import is legitimate or accidental

  • Resolution requires I/O - you have to hit the filesystem to resolve packages

Package managers have tried workarounds (pnpm symlinks, Yarn PnP), but are either limited by what the filesystem itself can offer (like symlinks) or by their complexity and lack of standardization (like Yarn PnP). This PR offers a mechanism for such tools to solve the problems listed above in tandem with Node.js.

How it works

A package-map.json declares packages, their locations (relative to the package map), and what each can import:

{
  "packages": {
    "my-app": {
      "name": "my-app",
      "path": "./src",
      "dependencies": ["lodash", "react"]
    },
    "lodash": {
      "name": "lodash",
      "path": "./node_modules/lodash"
    },
    "react": {
      "name": "react",
      "path": "./node_modules/react"
    }
  }
}

When resolving a bare specifier:

  1. Find which package contains the importing file (by checking paths)
  2. Look up the specifier in that package's dependencies
  3. If found, resolve to the target's path
  4. If not found but exists elsewhere in the map → ERR_PACKAGE_MAP_ACCESS_DENIED
  5. If not in the map at all → MODULE_NOT_FOUND

Compatibility

An important aspect of the package maps feature that separates it from competing options like Yarn PnP is its builtin compatibility with node_modules installs. Package managers can generate both node_modules folders AND package-map.json files, with the later referencing paths from the former.

Tools that know how to leverage package-map.json can then use this pattern for both static package resolution and strict dependency checks (with optional fallbacks to hoisting if they just wish to use the package map information to emit warnings rather than strict errors), whereas tools that don't will fallback to the classical node_modules resolution.

Differences with import maps

Issue #49443 requested to implement import maps. In practice these aren't a good fit for runtimes like Node.js for reasons described here and which can be summarized as: import maps take full ownership of the resolution pipeline by spec, thus preventing implementing additional runtime-specific behaviours such as exports or imports fields.

This PR comes as close from implementing import maps as possible but with a very light difference in design making it possible to stay compatible with other Node.js resolution features.

Why not a loader?

The ecosystem now has to deal with a variety of third-party resolvers, most of them not implementing the loader API for many different reasons: too complex, turing-complete, or dependent on a JS runtime.

After I've been following this path for more than six years I can confidently say that loaders would work for Node.js itself but wouldn't be standard enough to be included in at least some of those popular third-party tools.

Questions

  • The current implementation makes package maps strict: if they find an issue, they throw and refuse the resolution. Should we instead delegate to the default resolution unless an additional --experimental-strict-package-maps is set? Or via a strict field in package-map.json.

@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/config
  • @nodejs/loaders

@nodejs-github-bot nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Mar 13, 2026
@zkochan
Copy link

zkochan commented Mar 13, 2026

I like the idea, it would greatly reduce the amount of filesystem operations that pnpm has to do in order to create an isolated node_modules layout using symlinks.

I also suggested arcanis to possibly go one layer deeper and allow to map the individual files of packages. This would allow to map node_modules directly from a content-addressable store (that consists of package files). Of course, that would increase the size of the file several times but it would also make installation even faster.

@codecov
Copy link

codecov bot commented Mar 13, 2026

Codecov Report

❌ Patch coverage is 99.28058% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.69%. Comparing base (66a687f) to head (74bac97).
⚠️ Report is 10 commits behind head on main.

Files with missing lines Patch % Lines
lib/internal/modules/cjs/loader.js 96.00% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #62239      +/-   ##
==========================================
+ Coverage   89.66%   89.69%   +0.02%     
==========================================
  Files         676      677       +1     
  Lines      206462   206954     +492     
  Branches    39533    39621      +88     
==========================================
+ Hits       185128   185627     +499     
+ Misses      13461    13452       -9     
- Partials     7873     7875       +2     
Files with missing lines Coverage Δ
lib/internal/errors.js 97.63% <100.00%> (+0.01%) ⬆️
lib/internal/modules/esm/resolve.js 98.96% <100.00%> (+0.01%) ⬆️
lib/internal/modules/package_map.js 100.00% <100.00%> (ø)
src/node_options.cc 76.37% <100.00%> (-0.08%) ⬇️
src/node_options.h 97.93% <ø> (ø)
lib/internal/modules/cjs/loader.js 98.24% <96.00%> (+0.09%) ⬆️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@aduh95
Copy link
Contributor

aduh95 commented Mar 14, 2026

This seems quite close to the importmap HTML feature, but using a different syntax. Have you considered reusing the same syntax, or at least a compatible JSON structure?

@arcanis
Copy link
Contributor Author

arcanis commented Mar 14, 2026

I did, but felt that the semantics were too different; import maps have two fields:

  • the imports field is a global resolution map, keyed by bare identifiers. It wouldn't work for packages as that field is a flat map of all packages in the project, and thus must be keyed by arbitrary package IDs to allow for multiple packages sharing the same name (ie multiple versions of a same package in the same dependency tree).

  • the scopes field is keyed by filesystem path. This is a problem because it precludes a same folder from having multiple package IDs each with their own dependency set, necessary to represent peer dependencies with workspaces.

Neither of those match the semantics we need, and reusing them just for their name but with different semantics would have been imo misleading for third-party resolver implementors.

@jasnell
Copy link
Member

jasnell commented Mar 14, 2026

Love it. I'll try to give a detailed review on the flight home today if the in flight wifi treats me kindly.

Comment on lines +1021 to +1024
1. Node.js determines which package contains the importing file by checking
if the file path is within any package's `path`.
2. If the importing file is not within any mapped package, standard
`node_modules` resolution is used.
Copy link
Contributor

@bakkot bakkot Mar 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is surprising to me. Is there a way to make the project's root directory be "within a mapped package" without also including its node_modules? Or alternatively, can there be a way to list the dependencies of the project itself, which would apply to any files which are not within a mapped package?

That is, I would like to be able to have a project with JS in its root directory with that JS being governed by the package map (because I would like all bare specifier resolution in that project to be governed by the package map). You could do "root": { path: "." }, but then every file would be within root's path, including files in node_modules.

Possibly the check to see if a file is "within a package's path" should not includes files within a node_modules subdirectory of that path?

Copy link
Contributor Author

@arcanis arcanis Mar 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way it works is that the longest package path wins. So if you have a package mapped on . (which will usually be the case) you will also map its nested packages:

{
  "packages": {
    "root": {
      "path": "."
    },
    "react": {
      "path": "./node_modules/react"
    }
}

Paths in ./node_modules/react will be recognized as belonging to the React package, not the root.


// Walk up to find containing package
let checkPath = filePath;
while (isPathContainedIn(checkPath, this.#basePath)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I'm reading this right, it looks like this will not allow you to have packages which live outside of the directory where the package-map.json file lives. That's an unfortunate limitation: it would be nice if this let you have a cache of packages shared across projects, where the resolution would still be governed by each individual project's package-map.json.

That is, imagine I have a project foo which depends on direct@1.0.0 and indirect@1.0.0, and a project bar which depends on direct@1.0.0 and indirect@1.1.0 (where direct has a listed dependency on indirect@^1.0.0).

It would be nice if I could have /code/foo/package-map.json with

{ "packages": {
  "app": {
    "path": "./src",
    "dependencies": ["direct"]
  },
  "direct": {
    "name": "direct",
    "path": "/shared/direct@1.0.0",
    "dependencies": ["indirect"]
  },
  "indirect": {
    "name": "indirect",
    "path": "/shared/indirect@1.0.0"
  }
} }

and /code/bar/package-map.json with

{ "packages": {
  "app": {
    "path": "./src",
    "dependencies": ["direct"]
  },
  "direct": {
    "name": "direct",
    "path": "/shared/direct@1.0.0",
    "dependencies": ["indirect"]
  },
  "indirect": {
    "name": "indirect",
    "path": "/shared/indirect@1.1.0"
  }
} }

and have that just work. But it looks like the code will never find a package outside the directory where the package-map.json lives.

Copy link
Contributor Author

@arcanis arcanis Mar 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll double-check and add tests (edit: you were right, the isPathContainedIn check was extraneous), but you're meant to have two options:

  • you can have a relative path that point outside the project (../../../.cache/react@18)

  • or you can have an absolute path

The first option is a little more awkward but works better if the package-map.json is shared between systems (you only need to make sure the relative path between project and cache is kept the same between map generation and runtime).

Comment on lines +987 to +1005
{
"packages": {
"app": {
"name": "my-app",
"path": "./packages/app",
"dependencies": ["utils", "ui-lib"]
},
"utils": {
"name": "@myorg/utils",
"path": "./packages/utils",
"dependencies": []
},
"ui-lib": {
"name": "@myorg/ui-lib",
"path": "./packages/ui-lib",
"dependencies": ["utils"]
}
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you support multiple version resolutions of the same package name here? I'd have expected something like "dependencies" to map from a package name to a path to achieve that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants