s3file.py allows you to manipulate buckets and objects on S3 storages. It is written in pure Python 3 and needs no additional modules.
From its built-in help (s3file.py -h):
s3file.py -- manipulate buckets and objects on S3 storages
Usage:
s3file.py [<Generic Parameters> ...]
<GET|HEAD|PUT|DELETE> <host> /bucket/path/to/file [<local file>]
s3file.py [<Generic Parameters> ...]
-b <GET|HEAD|PUT|DELETE> <bucket.host> /path/to/file [<local file>]
Convenience shortcuts:
s3file.py [<Generic Parameters> ...] LS <host> </|/bucket|/bucket/pre/fix>
s3file.py [<Generic Parameters> ...] -b LS <bucket.host> </|/pre/fix>
s3file.py [<Generic Parameters> ...] <COPY|MOVE> <host>
</srcbucket/srcpath/srcfile> </dstbucket/dstpath/dstfile>
s3file.py [<Generic Parameters> ...] -b <COPY|MOVE> <bucket.host>
</srcpath/srcfile> </dstpath/dstfile>
s3file.py [<Generic Parameters> ...] <shortcut> <host> /bucket
s3file.py [<Generic Parameters> ...] -b <shortcut> <bucket.host> /
s3file.py --version
s3file.py --examples
Parameters:
-b Hostname is prefixed by the name of the bucket
--version Display version number
--examples Show a tutorial
Generic parameters:
-e Print the HTTP ETag header returned by the server
-s Use HTTP instead of HTTPS
-p<port> Connect to <port> instead of 443 or 80
-h<header:value> Add an additional HTTP header to the request; maybe given
multiple times
-t<content_type> Set the content type for a PUT request
-t Show the content type returned by GET or HEAD
-i<inifile> Read accesskey and secretkey from the [s3file] section of
the given ini file
-v Show all HTTP headers returned by the request
Shortcuts:
<shortcut> One of:
CREATE-BUCKET
CREATE-BUCKET-LOCKING
GET-LOCKING
COMPLIANCE-YEARS-<years>
COMPLIANCE-DAYS-<days>
GOVERNANCE-YEARS-<years>
GOVERNANCE-DAYS-<days>
ENABLE-VERSIONING
GET-VERSIONING
DISABLE-VERSIONING
GET-LOGGING
LS
LS-<lsflag>
<lsflag> Any combination of:
L long output
A show object versions
H human readable filesize
N numeric user id
Examples (see s3file.py --examples):
Examples:
1. Create a bucket:
$ export S3ACCESSKEY="someuser"
$ export S3SECRETKEY="somekey"
$ echo -n "" | s3file.py PUT s3.somehost.invalid /new-bucket
Or:
$ s3file.py CREATE-BUCKET s3.somehost.invalid /new-bucket
2. Create a bucket, but read credentials from an ini file:
$ echo '[s3file]
accesskey = someuser
secretkey = somekey' > s3file.ini
$ echo -n "" | s3file.py -is3file.ini PUT s3.somehost.invalid /new-bucket
Or:
$ s3file.py -is3file.ini CREATE-BUCKET s3.somehost.invalid /new-bucket
3. Delete a bucket:
$ s3file.py -is3file.ini DELETE s3.somehost.invalid /my-bucket
4. List all buckets:
$ s3file.py -is3file.ini GET s3.somehost.invalid /
Or:
$ s3file.py -is3file.ini LS s3.somehost.invalid /
Or:
$ s3file.py -is3file.ini LS-L s3.somehost.invalid /
Or:
$ s3file.py -is3file.ini LS-LN s3.somehost.invalid /
5. Upload a local file named bar.txt and save it as foo.txt:
$ s3file.py -is3file.ini PUT s3.somehost.invalid /my-bucket/foo.txt bar.txt
6. Delete a file:
$ s3file.py -is3file.ini DELETE s3.somehost.invalid /my-bucket/foo.txt
7. Write the content of a remote file to stdout:
$ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/readme.txt
8. Read from stdin and save it as remote file:
$ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/readme.txt
...some content...
<Ctrl+D>
9. Copy a file, possibly from one bucket to another:
$ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/readme.txt | \
s3file.py -iconfig2.ini PUT s3.otherhost.invalid /bucket2/cool.txt
Or:
$ s3file.py -is3file.ini COPY s3.somehost.invalid /my-bucket/readme.txt \
/bucket2/cool.txt
10. Move a file, possibly from one bucket to another:
$ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/src.txt | \
s3file.py -iconfig2.ini PUT s3.otherhost.invalid /bucket2/dst.txt
$ s3file.py -is3file.ini DELETE s3.somehost.invalid /my-bucket/src.txt
Or:
$ s3file.py -is3file.ini MOVE s3.somehost.invalid /my-bucket/src.txt \
/bucket2/dst.txt
11. Download a remote file and save it locally, but assume the first part of
the hostname is the name of the bucket:
$ s3file.py -is3file.ini GET my-bucket.s3.somehost.invalid /read.me \
ohey.txt
12. Create a bucket with object locking:
$ echo -n "" | s3file.py -is3file.ini \
-hx-amz-bucket-object-lock-enabled:True PUT s3.somehost.invalid \
/newbucket
Or:
$ s3file.py -is3file.ini CREATE-LOCKED-BUCKET s3.somehost.invalid \
/newbucket
13. Show whether a bucket has object locking enabled (HTTP error 404 means
object locking is disabled):
$ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/?object-lock
Or:
$ s3file.py -is3file.ini GET-LOCKED-BUCKET s3.somehost.invalid /my-bucket
14. Enable object locking in compliance mode for a bucket and set the default
retention to 100 days (this requires a bucket with object locking enabled):
$ echo '<?xml version="1.0""?>
<ObjectLockConfiguration>
<ObjectLockEnabled>Enabled</ObjectLockEnabled>
<Rule>
<DefaultRetention>
<Mode>COMPLIANCE</Mode>
<Days>100</Days>
</DefaultRetention>
</Rule>
</ObjectLockConfiguration>' | \
s3file.py -is3file.ini PUT s3.somehost.invalid /my-bucket/?object-lock
Or:
$ s3file.py -is3file.ini COMPLIANCE-DAYS-100 s3.somehost.invalid /my-bucket
15. Enable object locking in governance mode for a bucket and set the default
retention to 2 years (this requires a bucket with object locking enabled):
$ echo '<?xml version="1.0"?>
<ObjectLockConfiguration>
<ObjectLockEnabled>Enabled</ObjectLockEnabled>
<Rule>
<DefaultRetention>
<Mode>GOVERNANCE</Mode>
<Years>2</Years>
</DefaultRetention>
</Rule>
</ObjectLockConfiguration>' | \
s3file.py -is3file.ini PUT s3.somehost.invalid /my-bucket/?object-lock
Or:
$ s3file.py -is3file.ini GOVERNANCE-YEARS-2 s3.somehost.invalid /my-bucket
16. Enable object versioning for a bucket:
$ echo '<?xml version="1.0"?>
<VersioningConfiguration>
<Status>Enabled</Status>
<MfaDelete>Disabled</MfaDelete>
</VersioningConfiguration>' | \
s3file.py -is3file.ini PUT s3.somehost.invalid /my-bucket/?versioning
Or:
$ s3file.py -is3file.ini ENABLE-VERSIONING s3.somehost.invalid /my-bucket
17. Show whether a bucket has object versioning enabled:
$ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/?versioning
Or:
$ s3file.py -is3file.ini GET-VERSIONING s3.somehost.invalid /my-bucket
18. List all versioned objects:
$ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/?versions
Or:
$ s3file.py -is3file.ini LS-A s3.somehost.invalid /my-bucket
19. Disable object versioning for a bucket:
$ echo '<?xml version="1.0"?>
<VersioningConfiguration>
<Status>Suspended</Status>
<MfaDelete>Disabled</MfaDelete>
</VersioningConfiguration>' | \
s3file.py -is3file.ini PUT s3.somehost.invalid /my-bucket/?versioning
Or:
$ s3file.py -is3file.ini DISABLE-VERSIONING s3.somehost.invalid /my-bucket
20. Show whether a bucket has access logging enabled:
$ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/?logging
Or:
$ s3file.py -is3file.ini GET-LOGGING s3.somehost.invalid /my-bucket
21. List all objects:
$ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket
Or:
$ s3file.py -is3file.ini LS s3.somehost.invalid /my-bucket
Or:
$ s3file.py -is3file.ini LS-L s3.somehost.invalid /my-bucket
Or:
$ s3file.py -is3file.ini LS-LH s3.somehost.invalid /my-bucket
Or:
$ s3file.py -is3file.ini LS-LHN s3.somehost.invalid /my-bucket
Or:
$ s3file.py -is3file.ini LS-LN s3.somehost.invalid /my-bucket
The flags shown leftmost in the long format output (LS-L and so on) are:
b: Object is a bucket
d: Object is deleted
l: This is the latest version of the object
22. List all objects of a certain prefix (think of a folder, but in fact the
slash is part of the filename):
$ s3file.py -is3file.ini GET s3.somehost.invalid \
/my-bucket?prefix=sub/folder/
Or:
$ s3file.py -is3file.ini LS s3.somehost.invalid /my-bucket/sub/folder/
Please note that without the trailing slash files in "parent folder" (sub)
are listed if their names start with "folder":
$ s3file.py -is3file.ini LS s3.somehost.invalid /my-bucket/sub/folder
23. Upload a local file and assign some metadata:
$ s3file.py -is3file.ini -hx-amz-meta-key1:"value 1" \
-hx-amz-meta-key-2:value2 PUT s3.somehost.invalid \
/my-bucket/foo.txt bar.txt
24. Show an object's metadata (any header starting with "x-amz-meta-"):
$ s3file.py -is3file.ini -v HEAD s3.somehost.invalid /my-bucket/foo.txt
25. Set new metadata for an object:
$ s3file.py -is3file.ini -hx-amz-metadata-directive:replace \
-hx-amz-meta-key3:value3 \
-hx-amz-meta-key-2:new_value2 COPY s3.somehost.invalid \
/my-bucket/foo.txt /my-bucket/foo.txt
26. Remove any metadata from an object:
$ s3file.py -is3file.ini -hx-amz-metadata-directive:replace \
COPY s3.somehost.invalid /my-bucket/foo.txt /my-bucket/foo.txt