82 lines
2.9 KiB
Markdown
82 lines
2.9 KiB
Markdown
# bwdatools
|
|
Some tools to simplify file syncronization between LSDF@KIT
|
|
and the bwDataArchive.
|
|
|
|
## Getting started
|
|
### Prerequisites
|
|
The tool `lftp` is used to access bwDataArchive and needs to be available
|
|
in your `$PATH`. The tool is installed on `os-login.lsdf.kit.edu` or can be
|
|
obtained from [here](https://lftp.yar.ru/).
|
|
|
|
In order to minimize password inputs, the bwda sftp server can be
|
|
configured in `~/.netrc` in the following way. Do not forget to restrict file
|
|
permissions to this file (```chmod 600 ~/.netrc```) as it contains your password
|
|
in clear text.
|
|
```text
|
|
~/.netrc
|
|
---
|
|
machine archive-sftp.lsdf.kit.edu
|
|
login <xyz>
|
|
password <password>
|
|
```
|
|
If you feel uncomfortable doing this, an alternative is to ask the
|
|
staff at bwDataArchive to install your ssh key, as we do not have the
|
|
permissions to do so ourselves.
|
|
|
|
The tools included in this package require further machine-specific
|
|
configurations to determine the path on bwda automatically. This is
|
|
done using the file `~/.bwda`. Here is an example configuration for
|
|
the machine `os-login.lsdf.kit.edu`.
|
|
```text
|
|
~/.bwda
|
|
---
|
|
bwda_acc="xyz"
|
|
bwda_url="archive-sftp.lsdf.kit.edu"
|
|
ldir_base="/lsdf02/lsdf/kit/ifh/projects/<your-name>" # no trailing slash!
|
|
rdir_base="private" # no trailing slash!
|
|
```
|
|
By default, it is assumed that the directory tree (starting from `ldir_base` on LSDF
|
|
and `rdir_base` on bwDataArchive) is the same.
|
|
|
|
### Installing
|
|
Clone this repository using `git clone https://git.mkray.de/mwtkrayer/bwdatools.git`
|
|
and add the directory to your `$PATH`. Do not forget to setup `$HOME/.bwda`.
|
|
|
|
## Usage
|
|
### Interactive session
|
|
```text
|
|
Usage: bwda [-h] [path]
|
|
Opens an interactive lftp session on bwDataArchive.
|
|
path directory on LSDF, session will cd to the corresponding bwda directory
|
|
-h | --help display this message
|
|
```
|
|
|
|
### List directory content on bwda
|
|
```text
|
|
Usage: bwdals [opt]
|
|
List the contents of the current directory on bwda.
|
|
Options are directly passed to lftp's ls command with the following exceptions:
|
|
-h | --help display this message
|
|
```
|
|
|
|
### Transfer files from bwda to LSDF (in parallel)
|
|
```text
|
|
Usage: bwdaget files
|
|
files files to be downloaded
|
|
-c | --channel N use N channels per file
|
|
-n | --dry-run print commands to stdout instead of executing them
|
|
-P | --parallel N transfer N files in parallel
|
|
-h | --help display this message
|
|
```
|
|
|
|
### Syncronize files from LSDF to bwda
|
|
```text
|
|
Usage: bwdasync [opt] [dir]
|
|
dir path to local directory on LSDF
|
|
--include-size-mismatch include files which mismatch in size to sync list
|
|
-n | --dry-run print commands to stdout instead of executing them
|
|
-e | --extension ext only sync files with specified extension
|
|
-P | --parallel N transfer N files in parallel
|
|
-h | --help display this message
|
|
```
|