The `fdp-*` tool suite

Thirteen command-line tools share one library, one configuration and one set of options. Every tool has a man page (man fdp-get) and --help.

Tool	Purpose
`fdp-cd`, `fdp-pwd`	remote working directory for relative paths
`fdp-ls`	list directories (`-l`, `-R`, `-H`, `-d`, `--json`)
`fdp-stat`	size, time, checksums and URLs of a file
`fdp-get`	download files or directories, adler32-verified
`fdp-put`	upload files or directories, checksum sent with the upload
`fdp-mkdir`, `fdp-rm`	create and remove directories and files
`fdp-tpc`	WebDAV third-party copy between another dCache and FDP
`fdp-meta`	MetaCat: query, generate, validate, declare, update
`fdp-globus`	Globus transfers through the AmSC data movement API
`fdp-token`	obtain, inspect, renew and remove bearer tokens
`fdp-diag`	endpoints, clock, htgettoken, token coverage, configuration

Paths and the working directory

A remote path is an absolute namespace path (/amsc/dune/scratch/x), a path relative to the remote working directory (x, ../y), or a URL (https://, davs://, root://, roots://). The working directory is /amsc until changed:

fdp-cd /amsc/dune/scratch/$USER    # checked on the server: must be a directory
fdp-pwd                            # /amsc/dune/scratch/anorman
fdp-ls -l                          # lists it
fdp-put results.h5                 # -> /amsc/dune/scratch/anorman/results.h5
fdp-cd run42 && fdp-get f.h5        # -> ./f.h5
fdp-cd -                           # previous directory;  fdp-cd  alone: back to /amsc
export FDP_SESSION=$$              # one working directory per shell instead of one per user

Precedence: --cwd DIR > $FDP_STORAGE_CWD > the fdp-cd state file (~/.config/fdp/cwd) > storage.cwd in the configuration > /amsc.

Listing and inspecting

fdp-ls                                         # the working directory
fdp-ls -lH /amsc/public/treasure/aoj/data      # long format, human sizes
fdp-ls -R --json /amsc/dune/scratch/$USER      # recursive, JSON
fdp-stat /amsc/public/treasure/aoj/data/RunG_batch0.h5

path:     /amsc/public/treasure/aoj/data/RunG_batch0.h5
type:     file
size:     2379411451 (2.2G)
modified: 2025-11-06T21:41:14Z
adler32:  be111976
url:      https://amsc.fnal.gov:2880/amsc/public/treasure/aoj/data/RunG_batch0.h5
xrootd:   root://amsc.fnal.gov//amsc/public/treasure/aoj/data/RunG_batch0.h5

Downloading

fdp-get /amsc/public/treasure/aoj/data/RunG_batch0.h5        # into the current directory
fdp-get -R -j 8 /amsc/public/treasure/aoj/data ./aoj-data    # recursive, 8 files in parallel
fdp-get -Rn /amsc/public/treasure/aoj ./aoj                  # dry run: the plan only
fdp-get --skip-existing -R /amsc/dune/scratch/run42 ./       # only missing files
fdp-get --resume /amsc/dune/big/file.root .                  # continue a partial download

Each file is written to DEST.fdp-part; its adler32 is compared with dCache's, and only a verified file is renamed into place. A failed flush to disk or a size mismatch fails the file and leaves an existing destination untouched. Existing local files are kept unless -f. A missing source fails on its own; the others still transfer (exit 12).

Uploading

fdp-put results.h5 /amsc/dune/scratch/$USER/                 # dunewrite
fdp-put -R -j 8 ./run42 /amsc/dune/scratch/$USER/            # tree -> .../run42
fdp-put -R --skip-existing ./run42 /amsc/dune/scratch/$USER/
fdp-put --replace results.h5 /amsc/dune/scratch/$USER/results.h5

dCache never overwrites a file: an existing destination is an error (exit 6) unless --replace (delete, then upload) or --skip-existing. The local adler32 travels with the upload so dCache rejects a corrupted transfer, and size and checksum are compared again afterwards. In recursive uploads a symbolic link to a file is uploaded under the link's name; links to directories and dangling links are skipped and reported; nothing is ever written outside the destination.

Directories and removal

fdp-mkdir -p /amsc/dune/scratch/$USER/run43/raw
fdp-rm /amsc/dune/scratch/$USER/results.h5      # asks for confirmation
fdp-rm -Rn /amsc/dune/scratch/$USER/run42       # what would go
fdp-rm -Rf /amsc/dune/scratch/$USER/run42       # no prompt

Without -f the answer is read from stdin; no answer at all (a script, </dev/null) removes nothing and exits 10.

Third-party copy

fdp-tpc https://fndcadoor.fnal.gov:2880/pnfs/fnal.gov/usr/dune/x.root /amsc/dune/scratch/$USER/x.root
fdp-tpc -m push SOURCE DEST          # the source server sends instead of FDP pulling

The data goes directly between the servers. The FDP side uses the roles map; the other side's token comes from --other-issuer / --other-role or --other-token-file.

MetaCat and AmSC metadata

fdp-meta login                                   # session from your amsc token
fdp-meta find /amsc/public/treasure/aoj/data/RunG_batch0.h5
fdp-meta query "files from treasure:collide-1m limit 3"
fdp-meta generate --data-directory /amsc/dune/scratch/$USER/run42 \
    --namespace dune_fdp --dataset run42 --extension .h5 -o run42.json
fdp-meta validate --namespace dune_fdp --metadatafile run42.json
fdp-meta declare  --namespace dune_fdp --dataset run42 --metadatafile run42.json

generate reads a local directory (size, sha256, adler32) or lists a remote one (adler32 from dCache, sha256 by streaming unless --no-sha256), skipping hidden files and metadata/ directories. Names must be unique in a namespace (--nsubdirs N adds directory levels) and use only a-z A-Z 0-9 - . _ /; --ignore-bad-files or --fix-bad-files decide what happens to others. declare creates the namespace and dataset when missing and declares in chunks of 500.

Globus

fdp-globus login -C fdp_public -C nersc --domain nersc.gov
fdp-globus submit fdp_public:/amsc/public/treasure/aoj nersc:/path/on/nersc/aoj   # NAME:/path, UUID:/path or globus:// URL
fdp-globus status TASK_ID
fdp-globus cancel TASK_ID

Diagnostics and configuration

fdp-diag                          # endpoints, clock skew, htgettoken, cached tokens
fdp-diag /amsc/dune/scratch       # + which token would read and write there, and whether it covers the path
fdp-diag --show-config            # the effective configuration as YAML

Configuration is YAML: --config FILE, else $FDP_CONFIG, ./config/fdp.yaml, ~/.config/fdp/fdp.yaml. Precedence: command line (--set key=value and dedicated options) > FDP_* environment (transfer.parallel → FDP_TRANSFER_PARALLEL) > .env > YAML > built-in defaults.

endpoints: {webdav: https://amsc.fnal.gov:2880, xrootd: root://amsc.fnal.gov}
storage:   {base_path: /amsc, public_path: /amsc/public, default_protocol: webdav}
token:     {vault: htvaultprod.fnal.gov, issuer: amsc, htgettoken: htgettoken, min_lifetime: 300}
transfer:  {parallel: 4, retries: 3, timeout: 600, stall_timeout: 120, verify_checksum: true}

Options every tool accepts

Option	Meaning
`--config FILE`, `--env-file FILE`	configuration and `.env` file
`--set KEY=VALUE`	override one configuration key (repeatable)
`--role ROLE`, `-i NAME`, `--token-file FILE`	force a vault role, the issuer, or one token file
`--protocol webdav\|xrootd`, `--server URL`	protocol for bare paths; WebDAV door
`--cwd DIR`	remote working directory for this command
`--no-browser`	never start a browser login
`--json`, `-v`, `-q`	machine-readable output, verbose, quiet
`-j N`, `--no-verify`	(transfers) parallel files; skip checksum verification
`-R`	recursive (`fdp-ls`, `-get`, `-put`, `-rm`); `-r` is rejected everywhere

Exit status

Code	Meaning	Code	Meaning
0	success	7	network or transport failure
1	generic error	8	unexpected server response
2	usage error	9	checksum mismatch
3	configuration error	10	cancelled
4	authentication (no token, 401/403)	11	unsupported (e.g. XRootD not built)
5	not found	12	partial: some items failed
6	destination exists

TAB completion

zsh and bash complete options, subcommands, local files and remote FDP paths (listed from the server, never logging in, cached for 30 s):

fdp-cd /amsc/public/tr<TAB>      # -> /amsc/public/treasure/
fdp-get ao<TAB>                  # aoj/  aoj_tokenized_Feb2026/
fdp-token get --role dune<TAB>   # duneread  dunewrite

Setup per shell and install location is on the install page.

The fdp-* tool suite