Skip to content

Support directory references in <filepath@store> #1410

@dimitri-yatsenko

Description

@dimitri-yatsenko

Feature Proposal

Extend <filepath@store> to reference existing directories (folders with files), not just individual files.

Motivation

<filepath@store> currently only supports individual files. Users who need to reference existing directory structures in a store (e.g., multi-file recordings, dataset folders managed by acquisition software) must use <object@store>, which copies/uploads content — the opposite of filepath's "reference only" philosophy.

Directory support would be consistent with filepath's design: no copying, no lifecycle management, just a path reference.

Current Behavior

  • FilepathCodec.encode() hardcodes is_dir: False
  • StorageBackend.exists() uses Path.is_file() for local stores, rejecting directories
  • Inserting a directory path raises FileNotFoundError

Proposed Changes

1. storage.py — Add isdir(), widen exists()

  • Add isdir(remote_path) method: Path.is_dir() for local, fs.isdir() for cloud
  • Change exists() from Path.is_file() to Path.exists() for local protocol (safe — the only other caller, hash_registry, operates under _hash/ paths which filepath rejects as reserved)

2. filepath.py — Detect directories in encode()

  • After confirming path exists, call backend.isdir(path) to detect type
  • For directories: compute size as sum of contained files, track item_count
  • Set is_dir dynamically instead of hardcoded False

3. objectref.py — Graceful _verify_folder() without manifest

_verify_folder() currently requires a .manifest.json (created by object@). Filepath directories won't have one. Degrade gracefully: if no manifest exists, just verify the directory itself exists.

4. What does NOT change

  • decode() — already returns ObjectRef.from_json() which reads is_dir from metadata
  • ObjectRef directory methods (listdir(), walk(), download()) — already implemented and gated on is_dir
  • validate() — already accepts str/Path
  • Reserved section and filepath_prefix validation — works the same for directories

Files to Modify

  1. src/datajoint/storage.py
  2. src/datajoint/builtin_codecs/filepath.py
  3. src/datajoint/objectref.py
  4. tests/unit/test_codecs.py

Metadata

Metadata

Labels

enhancementIndicates new improvementsfeatureIndicates new features

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions