# Amazon EFS (amazon-efs)
Amazon EFS (amazon-efs) allows programmatically manipulate EFS data (create, read, delete, list files) from any machine.
# Prerequisites
* python
* pip
* boto3
* AWS Account
* AWS Credentials
# Install
pip install amazon-efs
# Warning
EFS should have at least one mount target in a Private subnet
# Limits
## Lambda compute env
*list_files*, *upload*, *download*, *delete* actions are limited by 15 minutes execution time (AWS Lambda works under the hood)
## Batch compute env
*list_files*, *upload*, *download* actions are not implemented yet
# Basics
Supported compute environments:
* Lambda (Default)
* Batch
Lambda compute environment (by default):
```
efs = Efs('<file_system_id>')
```
Lambda compute environment:
```
efs = Efs('<file_system_id>', compute_env_name='lambda')
```
Batch compute environment:
| The "batch_queue" option is required |
|------------------------------------|
```
efs = Efs('<file_system_id>', {
'batch_queue': '<batch_queue>',
}, compute_env_name='batch')
```
# Lambda compute environment
This computing environment is used for lightweight operations (lasting no more than 15 minutes).
```
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
# Deploying required underlying resources
efs.init()
# Actions (e.g. list_files, upload, download, delete)
files_list = efs.list_files()
# Don't forget to destroy underlying resources at the end of the session
efs.destroy()
```
## Actions
### List files
```
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
efs.init()
files_list = efs.list_files()
print(files_list)
files_list = efs.list_files('dir1')
print(files_list)
files_list = efs.list_files('dir1/dir2')
print(files_list)
efs.destroy()
```
### Upload
```
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
efs.init()
efs.upload('file.txt')
efs.upload('file.txt', 'dir1/new_file.txt')
efs.upload('file.txt', 'dir1/dir2/new_file.txt')
efs.upload('file.txt', 'dir1/dir3/new_file.txt')
efs.upload('file.txt', 'dir2/dir3/new_file.txt')
efs.upload('file.txt', 'dir2/dir4/new_file.txt')
efs.destroy()
```
### Download
```
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
efs.init()
efs.download('dir1/dir3/new_file.txt', 'file1.txt')
efs.destroy()
```
### Delete
#### Delete file
```
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
efs.init()
efs.delete('dir2/dir3/new_file.txt')
efs.destroy()
```
#### Delete folder
```
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
efs.init()
efs.delete('dir1/dir2/*')
efs.delete('dir1/*')
efs.destroy()
```
### Async delete
Use it if you want to schedule deletion and monitor progress yourself.
```
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
state = efs.init()
http_response_status_code = efs.delete('dir2/dir3/new_file.txt')
```
Then, after the job is completed, destroy the infrastructure.
```
efs = Efs(efs_id, { 'state': state })
efs.destroy()
```
# Batch compute environment
This computing environment is used for heavy operations (lasting more than 15 minutes).
## Actions
### Delete
| The "batch_queue" option is required |
|--------------------------------------|
#### Delete file
```
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
batch_queue = '<batch_queue>'
efs = Efs(efs_id, {
'batch_queue': batch_queue,
}, compute_env_name='batch')
efs.init()
efs.delete('dir2/dir3/new_file.txt')
efs.destroy()
```
#### Delete folder
```
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
batch_queue = '<batch_queue>'
efs = Efs(efs_id, {
'batch_queue': batch_queue,
}, compute_env_name='batch')
efs.init()
efs.delete('dir1/dir2/*')
efs.delete('dir1/*')
efs.destroy()
```
### Async delete
Use it if you want to schedule deletion and monitor progress yourself.
```
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
batch_queue = '<batch_queue>'
efs = Efs(efs_id, {
'batch_queue': batch_queue,
}, compute_env_name='batch')
state = efs.init()
batch_job_arn = efs.delete('dir2/dir3/new_file.txt')
```
Then, after the job is completed, destroy the infrastructure.
```
efs = Efs(efs_id, { 'state': state }, compute_env_name='batch')
efs.destroy()
```
# State
You can destroy underlying infrastructure even after destroying EFS object from RAM if you saved the **state**
```
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id)
state = efs.init()
# Destroy object
del efs
efs = Efs(efs_id, { 'state': state })
files_list = efs.list_files()
print(files_list)
efs.destroy()
```
# Tags
You can add custom tags to underlying resources
```
from amazon_efs import Efs
efs_id = 'fs-0d74736bfc*******'
efs = Efs(efs_id, {
'tags': {
'k1': 'v1',
'k2': 'v2'
}
})
efs.init()
files_list = efs.list_files()
print(files_list)
efs.destroy()
```
# Logging
```
from amazon_efs import Efs
import logging
fs_id = 'fs-0d74736bfc*******'
logger = logging.getLogger()
logging.basicConfig(level=logging.ERROR, format='%(asctime)s: %(levelname)s: %(message)s')
efs = Efs(fs_id, logger=logger)
```