# bwdatools
Some tools to simplify file syncronization between LSDF@KIT 
and the bwDataArchive.

## Getting started
### Prerequisites
The tool `lftp` is used to access bwDataArchive and needs to be available
in your `$PATH`. The tool is installed on `os-login.lsdf.kit.edu` or can be 
obtained from [here](https://lftp.yar.ru/).

In order to minimize password inputs, the bwda sftp server can be
configured in `~/.netrc` in the following way.
```
~/.netrc
---
machine archive-sftp.lsdf.kit.edu
login <xyz>
password <password>
```

The tools included in this package require further machine-specific 
configurations to determine the path on bwda automatically. This is 
done using the file `~/.bwda`. Here is an example configuration for 
the machine `os-login.lsdf.kit.edu`.
```
~/.bwda
---
bwda_acc="xyz"
bwda_url="archive-sftp.lsdf.kit.edu"
ldir_base="/lsdf02/lsdf/kit/ifh/projects/<your-name>" # no trailing slash!
rdir_base="private"                                   # no trailing slash!
```
By default, it is assumed that the directory tree (starting from `ldir_base` on LSDF
and `rdir_base` on bwDataArchive) is the same.

### Installing
Clone this repository using `git clone https://git.mkray.de/mwtkrayer/bwdatools.git` 
and add the directory to your `$PATH`. Do not forget to setup `$HOME/.bwda`.

## Usage
### Interactive session
```
Usage: bwda [-h] [path]
Opens an interactive lftp session on bwDataArchive.
 path                     directory on LSDF, session will cd to the corresponding bwda directory
 -h | --help              display this message
```

### List directory content on bwda
```
Usage: bwdals [opt]
List the contents of the current directory on bwda.
Options are directly passed to lftp's ls command with the following exceptions:
 -h | --help              display this message
```

### Transfer files from bwda to LSDF (in parallel)
```
Usage: bwdaget files
 files                    files to be downloaded
 -c | --channel N         use N channels per file
 -n | --dry-run           print commands to stdout instead of executing them
 -P | --parallel N        transfer N files in parallel
 -h | --help              display this message
```

### Syncronize files from LSDF to bwda 
```
Usage: bwdasync [opt] [dir]
 dir                      path to local directory on LSDF
 --include-size-mismatch  include files which mismatch in size to sync list
 -n | --dry-run           print commands to stdout instead of executing them
 -e | --extension ext     only sync files with specified extension
 -P | --parallel N        transfer N files in parallel
 -h | --help              display this message
```