bwdatools

Some tools to simplify file syncronization between LSDF@KIT and the bwDataArchive.

Getting started

Prerequisites

The tool lftp is used to access bwDataArchive and needs to be available in your $PATH. The tool is installed on os-login.lsdf.kit.edu or can be obtained from here.

In order to minimize password inputs, the bwda sftp server can be configured in ~/.netrc in the following way. Do not forget to restrict file permissions to this file (chmod 600 ~/.netrc) as it contains your password in clear text.

~/.netrc
---
machine archive-sftp.lsdf.kit.edu
login <xyz>
password <password>

If you feel uncomfortable doing this, an alternative is to ask the staff at bwDataArchive to install your ssh key, as we do not have the permissions to do so ourselves.

The tools included in this package require further machine-specific configurations to determine the path on bwda automatically. This is done using the file ~/.bwda. Here is an example configuration for the machine os-login.lsdf.kit.edu.

~/.bwda
---
bwda_acc="xyz"
bwda_url="archive-sftp.lsdf.kit.edu"
ldir_base="/lsdf02/lsdf/kit/ifh/projects/<your-name>" # no trailing slash!
rdir_base="private"                                   # no trailing slash!

By default, it is assumed that the directory tree (starting from ldir_base on LSDF and rdir_base on bwDataArchive) is the same.

Installing

Clone this repository using git clone https://git.mkray.de/mwtkrayer/bwdatools.git and add the directory to your $PATH. Do not forget to setup $HOME/.bwda.

Usage

Interactive session

Usage: bwda [-h] [path]
Opens an interactive lftp session on bwDataArchive.
 path                     directory on LSDF, session will cd to the corresponding bwda directory
 -h | --help              display this message

List directory content on bwda

Usage: bwdals [opt]
List the contents of the current directory on bwda.
Options are directly passed to lftp's ls command with the following exceptions:
 -h | --help              display this message

Transfer files from bwda to LSDF (in parallel)

Usage: bwdaget files
 files                    files to be downloaded
 -c | --channel N         use N channels per file
 -n | --dry-run           print commands to stdout instead of executing them
 -P | --parallel N        transfer N files in parallel
 -h | --help              display this message

Syncronize files from LSDF to bwda

Usage: bwdasync [opt] [dir]
 dir                      path to local directory on LSDF
 --include-size-mismatch  include files which mismatch in size to sync list
 -n | --dry-run           print commands to stdout instead of executing them
 -e | --extension ext     only sync files with specified extension
 -P | --parallel N        transfer N files in parallel
 -h | --help              display this message