Skip to content

Update production_file.md: update RUCIO commands wrt deprecation and dataset removal #67

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
veprbl wants to merge 1 commit into
base: master
Choose a base branch
from
+30 −30
Open

Update production_file.md: update RUCIO commands wrt deprecation and dataset removal #67

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 30 additions & 30 deletions _resources/production_file.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ layout: default

# Production Files Access

Starting from production campaign 25.01.1 onwards. Now files are registred to a catalog in Rucio (Data Mangement Software).
Starting from production campaign 25.10.0 onwards. Now files are registred to a catalog in Rucio (Data Mangement Software).
All old files will be registred to rucio, notification to follow

To see what files are available and how to access it use Rucio.
Expand All @@ -34,49 +34,49 @@ To see what files are available and how to access it use Rucio.
- **List datasets for a campaign** :
command: `rucio list-dids --short <dids>`
```shell
$ rucio list-dids --short epic:/RECO/25.01.1/\*

epic:/RECO/25.01.1/epic_craterlake/DIS/NC/18x275/minQ2=100
epic:/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=1
epic:/RECO/25.01.1/epic_craterlake/DIS/NC/18x275/minQ2=1000
epic:/RECO/25.01.1/epic_craterlake/DIS/NC/18x275/minQ2=1
epic:/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10
epic:/RECO/25.01.1/epic_craterlake/BACKGROUNDS/SYNRAD/dataprod_rel_1.0.0/18x275
epic:/RECO/25.01.1/epic_craterlake/DIS/NC/18x275/minQ2=10
....
$ rucio did list --short epic:/RECO/25.10.0/\*

epic:/RECO/25.10.0/epic_craterlake/DIS/NC/18x275/minQ2=1000
epic:/RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=1
epic:/RECO/25.10.0/epic_craterlake/SIDIS/pythia6-eic/1.0.0/18x275/q2_0to1
epic:/RECO/25.10.0/epic_craterlake/DIS/NC/5x41/minQ2=100
epic:/RECO/25.10.0/epic_craterlake/SIDIS/pythia6-eic/1.0.0/10x100/q2_0to1
epic:/RECO/25.10.0/epic_craterlake/DIS/CC/18x275/minQ2=1000
epic:/RECO/25.10.0/epic_craterlake/DIS/CC/5x41/minQ2=100
epic:/RECO/25.10.0/epic_craterlake/DIS/NC/18x275/minQ2=1
...
```
Note: The list shown above is truncated for preview.
Replace `25.01.1` with any other campaign.
Replace `25.10.0` with any other campaign.

- **List files within a dataset**:
command: `rucio list-content --short <did>`
```shell
$ rucio list-content --short epic:/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10

epic:/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.1105.eicrecon.tree.edm4eic.root
epic:/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.1106.eicrecon.tree.edm4eic.root
epic:/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.1107.eicrecon.tree.edm4eic.root
epic:/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.1108.eicrecon.tree.edm4eic.root
epic:/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.1109.eicrecon.tree.edm4eic.root
epic:/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.1110.eicrecon.tree.edm4eic.root
epic:/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.1111.eicrecon.tree.edm4eic.root
$ rucio did content list --short epic:/RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=10

epic:/RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_1.0000.eicrecon.edm4eic.root
epic:/RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_1.0001.eicrecon.edm4eic.root
epic:/RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_1.0002.eicrecon.edm4eic.root
epic:/RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_1.0003.eicrecon.edm4eic.root
epic:/RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_1.0004.eicrecon.edm4eic.root
...
```
Note: The list shown above is truncated for preview.

- **Find location of files**:
command: `rucio list-file-replicas --protocol root --pfns <did>`
```shell
$ rucio list-file-replicas --protocol root --pfns --rses isopenaccess epic:/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.0255.eicrecon.tree.edm4eic.root
$ rucio replica list file --protocols root --pfns --rses isopenaccess epic:/RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_1.0000.eicrecon.edm4eic.root

root://dtn-rucio.jlab.org:1094//volatile/eic/EPIC//RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.0255.eicrecon.tree.edm4eic.root
root://dtn-rucio.jlab.org:1094//volatile/eic/EPIC//RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_1.0000.eicrecon.edm4eic.root
```
If you see multiple replicas you can use any one of those. You can check using `rucio list-rses --rses isopenaccess` for a full list of RSE's.

#### Access methods for production files include:

- **Download all files in a dataset at once with `rucio download <dataset>`**
```shell
rucio download epic:/RECO/25.01.1/epic_craterlake/DIS/NC/18x275/minQ2=100
rucio download epic:/RECO/25.10.0/epic_craterlake/DIS/NC/18x275/minQ2=100
```

or multiple datasets at once
Expand All @@ -86,33 +86,33 @@ To see what files are available and how to access it use Rucio.

- **Download single file with `rucio download <file_did>`**
```shell
rucio download epic:/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.1105.eicrecon.tree.edm4eic.root
rucio download epic:/RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_1.0000.eicrecon.edm4eic.root
```

- **Copying with `xrdcp`:**
```shell
xrdcp root://dtn-rucio.jlab.org:1094//volatile/eic/EPIC/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.0255.eicrecon.tree.edm4eic.root <local destination>
xrdcp root://dtn-rucio.jlab.org:1094//volatile/eic/EPIC/RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_1.0000.eicrecon.edm4eic.root <local destination>
```

- **Copying with `gfal-copy`:**
```shell
gfal-copy root://dtn-rucio.jlab.org:1094//volatile/eic/EPIC/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.0255.eicrecon.tree.edm4eic.root <local destination>
gfal-copy root://dtn-rucio.jlab.org:1094//volatile/eic/EPIC/RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_1.0000.eicrecon.edm4eic.root <local destination>
```

- **Opening directly with ROOT:**
```c++
auto f = TFile::Open("root://dtn-rucio.jlab.org:1094//volatile/eic/EPIC/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.0255.eicrecon.tree.edm4eic.root")
auto f = TFile::Open("root://dtn-rucio.jlab.org:1094//volatile/eic/EPIC/RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_1.0000.eicrecon.edm4eic.root")
```
or using Python
```python
import uproot
file_path = "root://dtn-rucio.jlab.org:1094//volatile/eic/EPIC/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.0255.eicrecon.tree.edm4eic.root"
file_path = "root://dtn-rucio.jlab.org:1094//volatile/eic/EPIC/RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_1.0000.eicrecon.edm4eic.root"
root_file = uproot.open(file_path)
```

```python
import ROOT
file_path = "root://dtn-rucio.jlab.org:1094//volatile/eic/EPIC/RECO/25.01.1/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_5.0255.eicrecon.tree.edm4eic.root"
file_path = "root://dtn-rucio.jlab.org:1094//volatile/eic/EPIC/RECO/25.10.0/epic_craterlake/DIS/NC/10x100/minQ2=10/pythia8NCDIS_10x100_minQ2=10_beamEffects_xAngle=-0.025_hiDiv_1.0000.eicrecon.edm4eic.root"
file = ROOT.TFile.Open(file_path, "READ")
```