使用 boto python 递归地从 s3 下载文件。

Question

我在 s3 中有一个桶，它有很深的目录结构。我希望我可以一次下载它们。我的文件如下所示：

 foo/bar/1. .
foo/bar/100 . .

有什么方法可以使用 python 中的 boto lib 从 s3 存储桶递归下载这些文件？

提前致谢。

原文由 sriram 发布，翻译遵循 CC BY-SA 4.0 许可协议

Answer 1

您可以像这样下载存储桶中的所有文件（未经测试）：

 from boto.s3.connection import S3Connection

conn = S3Connection('your-access-key','your-secret-key')
bucket = conn.get_bucket('bucket')
for key in bucket.list():
    try:
        res = key.get_contents_to_filename(key.name)
    except:
        logging.info(key.name+":"+"FAILED")

请记住，S3 中的文件夹只是另一种写入密钥名称的方式，只有客户端会将其显示为文件夹。

原文由 j0nes 发布，翻译遵循 CC BY-SA 3.0 许可协议

Answer 2

#!/usr/bin/env python

import boto
import sys, os
from boto.s3.key import Key
from boto.exception import S3ResponseError

DOWNLOAD_LOCATION_PATH = os.path.expanduser("~") + "/s3-backup/"
if not os.path.exists(DOWNLOAD_LOCATION_PATH):
    print ("Making download directory")
    os.mkdir(DOWNLOAD_LOCATION_PATH)

def backup_s3_folder():
    BUCKET_NAME = "your-bucket-name"
    AWS_ACCESS_KEY_ID= os.getenv("AWS_KEY_ID") # set your AWS_KEY_ID  on your environment path
    AWS_ACCESS_SECRET_KEY = os.getenv("AWS_ACCESS_KEY") # set your AWS_ACCESS_KEY  on your environment path
    conn  = boto.connect_s3(AWS_ACCESS_KEY_ID, AWS_ACCESS_SECRET_KEY)
    bucket = conn.get_bucket(BUCKET_NAME)

    #goto through the list of files
    bucket_list = bucket.list()

    for l in bucket_list:
        key_string = str(l.key)
        s3_path = DOWNLOAD_LOCATION_PATH + key_string
        try:
            print ("Current File is ", s3_path)
            l.get_contents_to_filename(s3_path)
        except (OSError,S3ResponseError) as e:
            pass
            # check if the file has been downloaded locally
            if not os.path.exists(s3_path):
                try:
                    os.makedirs(s3_path)
                except OSError as exc:
                    # let guard againts race conditions
                    import errno
                    if exc.errno != errno.EEXIST:
                        raise

if __name__ == '__main__':
    backup_s3_folder()

原文由 pitaside 发布，翻译遵循 CC BY-SA 3.0 许可协议