# bwdatools
Some tools to simplify file syncronization between LSDF@KIT 
and the bwDataArchive.

## Getting started
### Prerequisites
The tool `lftp` is used to access bwDataArchive and needs to be available
in your `$PATH`. The tool is installed on `os-login.lsdf.kit.edu` or can be 
obtained from [here](https://lftp.yar.ru/).

In order to minimize password inputs, the bwda sftp server can be
configured in `~/.netrc` in the following way. Do not forget to restrict file 
permissions to this file (```chmod 600 ~/.netrc```) as it contains your password 
in clear text.
```text
~/.netrc
---
machine archive-sftp.lsdf.kit.edu
login <xyz>
password <password>
```
If you feel uncomfortable doing this, an alternative is to ask the 
staff at bwDataArchive to install your ssh key, as we do not have the 
permissions to do so ourselves.

The tools included in this package require further machine-specific 
configurations to determine the path on bwda automatically. This is 
done using the file `~/.bwda`. Here is an example configuration for 
the machine `os-login.lsdf.kit.edu`.
```text
~/.bwda
---
bwda_acc="xyz"
bwda_url="archive-sftp.lsdf.kit.edu"
ldir_base="/lsdf02/lsdf/kit/ifh/projects/<your-name>" # no trailing slash!
rdir_base="private"                                   # no trailing slash!
```
By default, it is assumed that the directory tree (starting from `ldir_base` on LSDF
and `rdir_base` on bwDataArchive) is the same.

### Installing
Clone this repository using `git clone https://git.mkray.de/mwtkrayer/bwdatools.git` 
and add the directory to your `$PATH`. Do not forget to setup `$HOME/.bwda`.

## Usage
### Interactive session
```text
Usage: bwda [-h] [path]
Opens an interactive lftp session on bwDataArchive.
 path                     directory on LSDF, session will cd to the corresponding bwda directory
 -h | --help              display this message
```

### List directory content on bwda
```text
Usage: bwdals [opt]
List the contents of the current directory on bwda.
Options are directly passed to lftp's ls command with the following exceptions:
 -h | --help              display this message
```

### Transfer files from bwda to LSDF (in parallel)
```text
Usage: bwdaget files
 files                    files to be downloaded
 -c | --channel N         use N channels per file
 -n | --dry-run           print commands to stdout instead of executing them
 -P | --parallel N        transfer N files in parallel
 -h | --help              display this message
```

### Syncronize files from LSDF to bwda 
```text
Usage: bwdasync [opt] [dir]
 dir                      path to local directory on LSDF
 --include-size-mismatch  include files which mismatch in size to sync list
 -n | --dry-run           print commands to stdout instead of executing them
 -e | --extension ext     only sync files with specified extension
 -P | --parallel N        transfer N files in parallel
 -h | --help              display this message
```