معرفی شرکت ها
bq-tools-dirupload-0.8.9.1
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشترتوضیحات
Upload/Download files to bisque/ViQi in parallel
| ویژگی | مقدار |
|---|---|
| سیستم عامل | - |
| نام فایل | bq-tools-dirupload-0.8.9.1 |
| نام | bq-tools-dirupload |
| نسخه کتابخانه | 0.8.9.1 |
| نگهدارنده | [] |
| ایمیل نگهدارنده | [] |
| نویسنده | ViQi Inc |
| ایمیل نویسنده | info@viqi.org |
| آدرس صفحه اصلی | - |
| آدرس اینترنتی | https://pypi.org/project/bq-tools-dirupload/ |
| مجوز | - |
# Tools
bq-dirupload : upload local files to remote system
bq-dirfetch : download remote files to local system
## Configuration
Create a file config file containing you login info at `~/.bisque/config`
```
[default]
host=https://data.viqi.org/
user=demo
password=<demopass>
```
# bq-dirupload
1. Upload directories and files to a ViQi/BisQue server
```
usage: Upload files to bisque [-h] [-c CONFIG] [--profile PROFILE] [-n] [-d [DEBUG]] [--debug-file DEBUG_FILE] [-q] [-a CREDENTIALS] [--host HOST] [--user USER] [--password PASSWORD] [--alias ALIAS] [--tag TAG]
[--path-tags PATH_TAGS] [--re-tags RE_TAGS] [--re-only] [--mustmap] [--include INCLUDE] [--exclude EXCLUDE] [--dataset DATASET] [--threads THREADS] [--empty] [-s] [-r]
[--tagmap TAGMAP] [--fixedtags FIXEDTAGS] [--json-args JSON_ARGS] [--protocol {binary,fsxlustre,multipart}] [--destination-dir DESTINATION_DIR] [--transfer-only]
[--register-only REGISTER_ONLY] [--archive-type {zip-bisque,zip-multi-file,zip-time-series,zip-z-stack,zip-dicom}] [--watch]
[directories ...]
positional arguments:
directories director(ies) to upload
optional arguments:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
bisque config
--profile PROFILE Profile to use in bisque config
-n, --dry-run report actions w/o changes
-d [DEBUG], --debug [DEBUG]
set debug level: debug,info,warn,error
--debug-file DEBUG_FILE
output filename for debug messages
-q, --quiet print actions
-a CREDENTIALS, --credentials CREDENTIALS
A bisque login.. admin
--host HOST Default bisque server to connect to
--user USER User to connect with
--password PASSWORD passwor to connect with
--alias ALIAS Use admin credentials to login as alias
--tag TAG Add name:value pair, can be templated with other values mycode:$site$season
--path-tags PATH_TAGS
tag names for a parsable path i.e. /root/project/date//subject/ or \root\project\data\subject
--re-tags RE_TAGS re expressions for tags i.e. (?P<location>\w+)--(?P<date>[\d-]+)
--re-only Accept files only if match re-tags
--mustmap Contextual tag must have a value in a tagmap
--include INCLUDE shell expression for files to include. Can be repeated
--exclude EXCLUDE shell expression for files to exclude. Can be repeated
--dataset DATASET create dataset and add files to it
--threads THREADS set number of uploader threads
--empty Allow empty files to be uploaded
-s, --skip-loaded Skip upload if there is file with the same name already present on the server
-r, --replace-uploaded
Force upload of metadata even if file exists on server
--tagmap TAGMAP Supply a map tag/value -> tag/value found in tag path and re decoder. [context_key:]carp=carpenteria or [context_key:]@tagmap.csv
--fixedtags FIXEDTAGS
key:tag=value or key:@fileoftags fixed tags to add to resource: First column is key: including filename or image_path
--json-args JSON_ARGS
Load default arguments from json file i.e { "--path-tags" : "project/date" }
--protocol {binary,fsxlustre,multipart}
Set the upload protocol or let tyhe system choose
--destination-dir DESTINATION_DIR
Upload to destination-dir on server .. must be valid mount i.e /home/my-uploads
--transfer-only Upload files only without registration
--register-only REGISTER_ONLY
register files without actually uploading them use argument as prefix path for
--archive-type {zip-bisque,zip-multi-file,zip-time-series,zip-z-stack,zip-dicom}
zip archive will be given a type: bisque, z-stack, t-stack
--watch Watch directories listed for new files and transfer
bq-dirupload -n --threads 1 --re-tags "(?P<photo_site_code>\w+)_(?P<target_assemblage>\D+)(?P<plot>\d+)_(?P<season>\D+)(?P<year>\d+).+\.JPG" --dataset upload --tagmap target_assemblage:@speciesmap.csv --tagmap photo_site_code:@locationmap.csv --tagmap season:fa=fall --tagmap year:15=2015 --fixedtags photo_site_code:@photo_code_reference_2019_0912.csv TopLevelDir
Magic decoder ring:
-n : dry run
--threads 1: one thread for debugging
--retags : use filename to create tags: photo_site_code, target_assemblage, season and year.
--dataset : create a dataset "upload"
--tagmap target_assemblage:@speciesmap.csv: use value ins speciesmap.csv to rename tag/values for target_assemblage
--tagmap photo_site_code:@locationmap: Use location map to rename tag/value from photo_site_code
--tagmap season:fa=fall : rename season 'fa' to 'fall'
--tagmap year:15=2015 : remame year from '15' to 2015
--fixedtags photo_site_code:@photo_code_reference_2019_0912.csv : use photo_site_code to read a set of fixed tags to be applied to the resource
A map is consists of [context_tag:]oldval=newval or [context_tag:]@map.csv where csv is a two column table of old value, new value
Other interesting Arguments
--debug-file somefile : write actions to somefile
--path-tags map components of the file path to metadata tags i.e. on the path ghostcrabs/manua/winter/somefile.jpg
--path-tags=project/site/season -> { project:ghostcrabs, site:manua, season:winter} as tags on somefile.jpg
--path-tags=/site// -> {site:manua } skipping root and season elements
```
## Examples
1. Simple decoding of path/filename into metadata tags
```bq-dirupload --skip --include *.JPG --tag activity:bear --re-tags "(?P<location>\w+)_(?P<camera>\w+)__(?P<date>[\d-]+)__(?P<time>[\d-]+).*" ~/Desktop/pics/bears/```
2. Complex mapping of path/filename metadata into tag
`bq-dirupload -n --threads 1 --re-tags "(?P<photo_site_code>\w+)_(?P<target_assemblage>\D+)(?P<plot>\d+)_(?P<season>\D+)(?P<year>\d+).+\.JPG" --dataset upload --tagmap target_assemblage:@speciesmap.csv --tagmap photo_site_code:@locationmap.csv --tagmap season:fa=fall --tagmap year:15=2015 --fixedtags photo_site_code:@photo_code_reference_2019_0912.csv`
Magic decoder ring:
```
-n : dry run to see action without doing them
--threads 1: one thread for debugging
--retags : use filename to create tags: photo_site_code, target_assemblage, season and year.
--dataset : create a dataset "upload"
--tagmap target_assemblage:@speciesmap.csv: use value ins speciesmap.csv to rename tag/values for target_assemblage
--tagmap photo_site_code:@locationmap: Use location map to rename tag/value from photo_site_code
--tagmap season:fa=fall : rename season 'fa' to 'fall'
--tagmap year:15=2015 : remame year from '15' to 2015
--fixedtags photo_site_code:@photo_code_reference_2019_0912.csv : use photo_site_code to read a set of fixed tags to be applied to the resource
A map is consists of [context_tag:]oldval=newval or [context_tag:]@map.csv where csv is a two column table of old value, new value
Other interesting Arguments
--debug-file somefile : write actions to somefile
--path-tags map components of the file path to metadata tags i.e. project/site/season would map ghostcrabs/manua/winter/somefile.jpg
would get { project:ghostcrabs, site:manua, season:winter} as tags on somefile.jpg
```
# bq-dirfetch
Downoad files to the local fileystsem
```
usage: Fetch files from bisque [-h] [-c CONFIG] [--profile PROFILE] [-n]
[-d [DEBUG]] [--debug-file DEBUG_FILE] [-q]
[-a CREDENTIALS] [--bisque-host BISQUE_HOST]
[--alias ALIAS] [--threads THREADS]
[--meta {false,xml,tablecsv,json}]
[--view VIEW]
[datasets [datasets ...]]
positional arguments:
datasets datasets to download
optional arguments:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
bisque config
--profile PROFILE Profile to use in bisque config
-n, --dry-run report actions w/o changes
-d [DEBUG], --debug [DEBUG]
set debug level: debug,info,warn,error
--debug-file DEBUG_FILE
output filename for debug messages
-q, --quiet print actions
-a CREDENTIALS, --credentials CREDENTIALS
A bisque login.. admin
--bisque-host BISQUE_HOST
Default bisque server to connect to
--alias ALIAS Use admin credentials to login as alias
--threads THREADS set number of uploader threads
--meta {false,xml,tablecsv,json}
fetch metadata of resources: xml, tablecsv or false
--view VIEW Metadata view i.e. deep,clean or view=TAG1,TAG2
bq-dirfetch -n --threads 1 dataset_uri
```
## Examples
```
$ bq-dirfetch https://data.viqi.org/data_service/00-aGLFAaVudK2SesxZbFgtSA
$ ls My\ data
30PKCVampNFp1Exte_DSP.tif 30PKCVampNFp6Exte_DSP.tif 30PKCVampNFz5Live5-21-2004_2-21-06_PM.tif 'My Data.tablecsv'
30PKCVampNFp3Exte_DSP.tif 30PKCVampNFz1Live5-21-2004_1-36-13_PM.tif D4P1.TIF
$ cat My\ data/My\ Data.tablecsv
name,filename,tag1,tag1.subtag1,upload_datetime
D4P1.TIF,D4P1.TIF,,,2021-04-06 11:45:25.939292
30PKCVampNFp1Exte_DSP.tif,30PKCVampNFp1Exte_DSP.tif,,,2021-04-06 11:47:12.354264
30PKCVampNFp3Exte_DSP.tif,30PKCVampNFp3Exte_DSP.tif,,,2021-04-06 11:53:07.626649
30PKCVampNFp6Exte_DSP.tif,30PKCVampNFp6Exte_DSP.tif,val1,subval1,2021-04-06 11:54:38.046633
30PKCVampNFz1Live5-21-2004_1-36-13_PM.tif,30PKCVampNFz1Live5-21-2004_1-36-13_PM.tif,,,2021-04-06 12:07:28.216727
30PKCVampNFz5Live5-21-2004_2-21-06_PM.tif,30PKCVampNFz5Live5-21-2004_2-21-06_PM.tif,,,2021-04-06 12:12:12.523292
```
## Advanced Configuration and Arguments
Pass a file of arguments with --json-args=path-to-file.json
where path-to-file.json has:
```
{
"directories" : [ "tmp", "tmp1" ],
"--path-tags" : "top/d1/d2",
"--profile" : "hq"
}
```
Furthermore you can use an environment variable to pass this information
```
export BQ_OEM_TRANSFER=path-to-file.json
```
نیازمندی
| مقدار | نام |
|---|---|
| <0.6,>=0.5.10.14 | bisque-api |
| >=0.6.4.8 | viqi-api |
| - | bisque-metadoc |
| - | configparser |
| - | lxml |
| >=2020.11.13 | regex |
| - | watchdog |
| - | exif |
زبان مورد نیاز
| مقدار | نام |
|---|---|
| >=3.8 | Python |
نحوه نصب
نصب پکیج whl bq-tools-dirupload-0.8.9.1:
pip install bq-tools-dirupload-0.8.9.1.whl
نصب پکیج tar.gz bq-tools-dirupload-0.8.9.1:
pip install bq-tools-dirupload-0.8.9.1.tar.gz