s3file.py

s3file.py allows you to manipulate buckets and objects on S3 storages. It is written in pure Python 3 and needs no additional modules.

Features

Installation

  1. Put s3file.py to e.g. your /usr/local/bin/ directory
  2. Make it executable: chmod 0755 /usr/local/bin/s3file.py
  3. Optionally, edit s3file.py and adjust the very first line to point to your Python executable.

Usage

From its built-in help (s3file.py -h):

s3file.py -- manipulate buckets and objects on S3 storages

Usage:
  s3file.py [<Generic Parameters> ...]
            <GET|HEAD|PUT|DELETE> <host> /bucket/path/to/file [<local file>]
  s3file.py [<Generic Parameters> ...]
            -b <GET|HEAD|PUT|DELETE> <bucket.host> /path/to/file [<local file>]

Convenience shortcuts:
  s3file.py [<Generic Parameters> ...] LS <host> </|/bucket|/bucket/pre/fix>
  s3file.py [<Generic Parameters> ...] -b LS <bucket.host> </|/pre/fix>

  s3file.py [<Generic Parameters> ...] <COPY|MOVE> <host>
            </srcbucket/srcpath/srcfile> </dstbucket/dstpath/dstfile>
  s3file.py [<Generic Parameters> ...] -b <COPY|MOVE> <bucket.host>
            </srcpath/srcfile> </dstpath/dstfile>

  s3file.py [<Generic Parameters> ...] <shortcut> <host> /bucket
  s3file.py [<Generic Parameters> ...] -b <shortcut> <bucket.host> /

  s3file.py --version
  s3file.py --examples

Parameters:
  -b                Hostname is prefixed by the name of the bucket
  --version         Display version number
  --examples        Show a tutorial

Generic parameters:
  -e                Print the HTTP ETag header returned by the server
  -s                Use HTTP instead of HTTPS
  -p<port>          Connect to <port> instead of 443 or 80
  -h<header:value>  Add an additional HTTP header to the request; maybe given
                    multiple times
  -t<content_type>  Set the content type for a PUT request
  -t                Show the content type returned by GET or HEAD
  -i<inifile>       Read accesskey and secretkey from the [s3file] section of
                    the given ini file
  -v                Show all HTTP headers returned by the request

Shortcuts:
  <shortcut>        One of:
                    CREATE-BUCKET
                    CREATE-BUCKET-LOCKING
                    GET-LOCKING
                    COMPLIANCE-YEARS-<years>
                    COMPLIANCE-DAYS-<days>
                    GOVERNANCE-YEARS-<years>
                    GOVERNANCE-DAYS-<days>
                    ENABLE-VERSIONING
                    GET-VERSIONING
                    DISABLE-VERSIONING
                    GET-LOGGING
                    LS
                    LS-<lsflag>
  <lsflag>          Any combination of:
                    L  long output
                    A  show object versions
                    H  human readable filesize
                    N  numeric user id

Examples (see s3file.py --examples):

Examples:

1.  Create a bucket:
    $ export S3ACCESSKEY="someuser"
    $ export S3SECRETKEY="somekey"
    $ echo -n "" | s3file.py PUT s3.somehost.invalid /new-bucket
  Or:
    $ s3file.py CREATE-BUCKET s3.somehost.invalid /new-bucket

2.  Create a bucket, but read credentials from an ini file:
    $ echo '[s3file]
accesskey = someuser
secretkey = somekey' > s3file.ini
    $ echo -n "" | s3file.py -is3file.ini PUT s3.somehost.invalid /new-bucket
  Or:
    $ s3file.py -is3file.ini CREATE-BUCKET s3.somehost.invalid /new-bucket

3.  Delete a bucket:
    $ s3file.py -is3file.ini DELETE s3.somehost.invalid /my-bucket

4.  List all buckets:
    $ s3file.py -is3file.ini GET s3.somehost.invalid /
  Or:
    $ s3file.py -is3file.ini LS s3.somehost.invalid /
  Or:
    $ s3file.py -is3file.ini LS-L s3.somehost.invalid /
  Or:
    $ s3file.py -is3file.ini LS-LN s3.somehost.invalid /

5.  Upload a local file named bar.txt and save it as foo.txt:
    $ s3file.py -is3file.ini PUT s3.somehost.invalid /my-bucket/foo.txt bar.txt

6.  Delete a file:
    $ s3file.py -is3file.ini DELETE s3.somehost.invalid /my-bucket/foo.txt

7.  Write the content of a remote file to stdout:
    $ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/readme.txt

8.  Read from stdin and save it as remote file:
    $ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/readme.txt
...some content...
<Ctrl+D>

9.  Copy a file, possibly from one bucket to another:
    $ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/readme.txt | \
      s3file.py -iconfig2.ini PUT s3.otherhost.invalid /bucket2/cool.txt
  Or:
    $ s3file.py -is3file.ini COPY s3.somehost.invalid /my-bucket/readme.txt \
      /bucket2/cool.txt

10. Move a file, possibly from one bucket to another:
    $ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/src.txt | \
      s3file.py -iconfig2.ini PUT s3.otherhost.invalid /bucket2/dst.txt
    $ s3file.py -is3file.ini DELETE s3.somehost.invalid /my-bucket/src.txt
  Or:
    $ s3file.py -is3file.ini MOVE s3.somehost.invalid /my-bucket/src.txt \
      /bucket2/dst.txt

11. Download a remote file and save it locally, but assume the first part of
    the hostname is the name of the bucket: 
    $ s3file.py -is3file.ini GET my-bucket.s3.somehost.invalid /read.me \
      ohey.txt

12. Create a bucket with object locking:
    $ echo -n "" | s3file.py -is3file.ini \
      -hx-amz-bucket-object-lock-enabled:True PUT s3.somehost.invalid \
      /newbucket
  Or:
    $ s3file.py -is3file.ini CREATE-LOCKED-BUCKET s3.somehost.invalid \
      /newbucket

13. Show whether a bucket has object locking enabled (HTTP error 404 means
    object locking is disabled):
    $ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/?object-lock
  Or:
    $ s3file.py -is3file.ini GET-LOCKED-BUCKET s3.somehost.invalid /my-bucket

14. Enable object locking in compliance mode for a bucket and set the default
    retention to 100 days (this requires a bucket with object locking enabled):
    $ echo '<?xml version="1.0""?>
<ObjectLockConfiguration>
<ObjectLockEnabled>Enabled</ObjectLockEnabled>
<Rule>
<DefaultRetention>
<Mode>COMPLIANCE</Mode>
<Days>100</Days>
</DefaultRetention>
</Rule>
</ObjectLockConfiguration>' | \
      s3file.py -is3file.ini PUT s3.somehost.invalid /my-bucket/?object-lock
  Or:
    $ s3file.py -is3file.ini COMPLIANCE-DAYS-100 s3.somehost.invalid /my-bucket

15. Enable object locking in governance mode for a bucket and set the default
    retention to 2 years (this requires a bucket with object locking enabled):
    $ echo '<?xml version="1.0"?>
<ObjectLockConfiguration>
<ObjectLockEnabled>Enabled</ObjectLockEnabled>
<Rule>
<DefaultRetention>
<Mode>GOVERNANCE</Mode>
<Years>2</Years>
</DefaultRetention>
</Rule>
</ObjectLockConfiguration>' | \
      s3file.py -is3file.ini PUT s3.somehost.invalid /my-bucket/?object-lock
  Or:
    $ s3file.py -is3file.ini GOVERNANCE-YEARS-2 s3.somehost.invalid /my-bucket

16. Enable object versioning for a bucket:
    $ echo '<?xml version="1.0"?>
<VersioningConfiguration>
<Status>Enabled</Status>
<MfaDelete>Disabled</MfaDelete>
</VersioningConfiguration>' | \
      s3file.py -is3file.ini PUT s3.somehost.invalid /my-bucket/?versioning
  Or:
    $ s3file.py -is3file.ini ENABLE-VERSIONING s3.somehost.invalid /my-bucket

17. Show whether a bucket has object versioning enabled:
    $ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/?versioning
  Or:
    $ s3file.py -is3file.ini GET-VERSIONING s3.somehost.invalid /my-bucket

18. List all versioned objects:
    $ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/?versions
  Or:
    $ s3file.py -is3file.ini LS-A s3.somehost.invalid /my-bucket

19. Disable object versioning for a bucket:
    $ echo '<?xml version="1.0"?>
<VersioningConfiguration>
<Status>Suspended</Status>
<MfaDelete>Disabled</MfaDelete>
</VersioningConfiguration>' | \
      s3file.py -is3file.ini PUT s3.somehost.invalid /my-bucket/?versioning
  Or:
    $ s3file.py -is3file.ini DISABLE-VERSIONING s3.somehost.invalid /my-bucket

20. Show whether a bucket has access logging enabled:
    $ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket/?logging
  Or:
    $ s3file.py -is3file.ini GET-LOGGING s3.somehost.invalid /my-bucket

21. List all objects:
    $ s3file.py -is3file.ini GET s3.somehost.invalid /my-bucket
  Or:
    $ s3file.py -is3file.ini LS s3.somehost.invalid /my-bucket
  Or:
    $ s3file.py -is3file.ini LS-L s3.somehost.invalid /my-bucket
  Or:
    $ s3file.py -is3file.ini LS-LH s3.somehost.invalid /my-bucket
  Or:
    $ s3file.py -is3file.ini LS-LHN s3.somehost.invalid /my-bucket
  Or:
    $ s3file.py -is3file.ini LS-LN s3.somehost.invalid /my-bucket

  The flags shown leftmost in the long format output (LS-L and so on) are:
    b: Object is a bucket
    d: Object is deleted
    l: This is the latest version of the object

22. List all objects of a certain prefix (think of a folder, but in fact the
    slash is part of the filename):
    $ s3file.py -is3file.ini GET s3.somehost.invalid \
      /my-bucket?prefix=sub/folder/
  Or:
    $ s3file.py -is3file.ini LS s3.somehost.invalid /my-bucket/sub/folder/
  Please note that without the trailing slash files in "parent folder" (sub)
  are listed if their names start with "folder":
    $ s3file.py -is3file.ini LS s3.somehost.invalid /my-bucket/sub/folder

23. Upload a local file and assign some metadata:
    $ s3file.py -is3file.ini -hx-amz-meta-key1:"value 1" \
      -hx-amz-meta-key-2:value2 PUT s3.somehost.invalid \
      /my-bucket/foo.txt bar.txt

24. Show an object's metadata (any header starting with "x-amz-meta-"):
    $ s3file.py -is3file.ini -v HEAD s3.somehost.invalid /my-bucket/foo.txt

25. Set new metadata for an object:
    $ s3file.py -is3file.ini -hx-amz-metadata-directive:replace \
      -hx-amz-meta-key3:value3 \
      -hx-amz-meta-key-2:new_value2 COPY s3.somehost.invalid \
      /my-bucket/foo.txt /my-bucket/foo.txt

26. Remove any metadata from an object:
    $ s3file.py -is3file.ini -hx-amz-metadata-directive:replace \
      COPY s3.somehost.invalid /my-bucket/foo.txt /my-bucket/foo.txt

Releases