bwdatools

Some tools to simplify file syncronization between LSDF@KIT and the bwDataArchive.

Getting started

Prerequisites

The tool lftp is used to access bwDataArchive and needs to be available in your $PATH. The tool is installed on os-login.lsdf.kit.edu or can be obtained from here.

In order to minimize password inputs, the bwda sftp server can be configured in ~/.netrc in the following way.

~/.netrc
---
machine archive-sftp.lsdf.kit.edu
login <xyz>
password <password>

The tools included in this package require further machine-specific configurations to determine the path on bwda automatically. This is done using the file ~/.bwda. Here is an example configuration for the machine os-login.lsdf.kit.edu.

~/.bwda
---
bwda_acc="xyz"
bwda_url="archive-sftp.lsdf.kit.edu"
ldir_base="/lsdf02/lsdf/kit/ifh/projects/<your-name>" # no trailing slash!
rdir_base="private"                                   # no trailing slash!

By default, it is assumed that the directory tree (starting from ldir_base on LSDF and rdir_base on bwDataArchive) is the same.

Installing

Clone this repository using git clone https://git.mkray.de/mwtkrayer/bwdatools.git and add the directory to your $PATH. Do not forget to setup $HOME/.bwda.

Usage

Interactive session

Usage: bwda [-h] [path]
Opens an interactive lftp session on bwDataArchive.
 path                     directory on LSDF, session will cd to the corresponding bwda directory
 -h | --help              display this message

List directory content on bwda

Usage: bwdals [opt]
List the contents of the current directory on bwda.
Options are directly passed to lftp's ls command with the following exceptions:
 -h | --help              display this message

Transfer files from bwda to LSDF (in parallel)

Usage: bwdaget files
 files                    files to be downloaded
 -c | --channel N         use N channels per file
 -n | --dry-run           print commands to stdout instead of executing them
 -P | --parallel N        transfer N files in parallel
 -h | --help              display this message

Syncronize files from LSDF to bwda

Usage: bwdasync [opt] [dir]
 dir                      path to local directory on LSDF
 --include-size-mismatch  include files which mismatch in size to sync list
 -n | --dry-run           print commands to stdout instead of executing them
 -e | --extension ext     only sync files with specified extension
 -P | --parallel N        transfer N files in parallel
 -h | --help              display this message