Python – 使用请求直接下载文件到内存

目标是从互联网上下载文件,并从中创建一个文件对象,或者像对象这样的文件,而不必触摸硬盘.这只是为了我的知识,想知道它是否可行或实用,特别是因为我想看看我是否可以避免编写文件删除行.这就是我通常从网上下载内容...

目标是从互联网上下载文件,并从中创建一个文件对象,或者像对象这样的文件,而不必触摸硬盘.这只是为了我的知识,想知道它是否可行或实用,特别是因为我想看看我是否可以避免编写文件删除行.

这就是我通常从网上下载内容并将其映射到内存的方式：

import requests
import mmap

u = requests.get("http://www.pythonchallenge.com/pc/def/channel.zip")

with open("channel.zip", "wb") as f: # I want to eliminate this, as this writes to disk
    f.write(u.content)

with open("channel.zip", "r+b") as f: # and his as well, because it reads from disk
    mm = mmap.mmap(f.fileno(), 0)
    mm.seek(0)
    print mm.readline()
    mm.close() # question: if I do not include this, does this become a memory leak?

解决方法:

r.raw(HTTPResponse)已经是一个类似文件的对象(只是传递stream = True)：

#!/usr/bin/env python
import sys
import requests # $pip install requests
from PIL import Image # $pip install pillow

url = sys.argv[1]
r = requests.get(url, stream=True)
r.raw.decode_content = True # Content-Encoding
im = Image.open(r.raw) #NOTE: it requires pillow 2.8+
print(im.format, im.mode, im.size)

通常,如果你有一个bytestring;您可以将其包装为f = io.BytesIO(r.content),以获取类似文件的对象而不触及磁盘：

#!/usr/bin/env python
import io
import zipfile
from contextlib import closing
import requests # $pip install requests

r = requests.get("http://www.pythonchallenge.com/pc/def/channel.zip")
with closing(r), zipfile.ZipFile(io.BytesIO(r.content)) as archive:
    print({member.filename: archive.read(member) for member in archive.infolist()})

您无法直接将r.raw传递给ZipFile(),因为前者是不可搜索的文件.

I would like to see if I can circumvent having to code a file deletion line

tempfile可以自动删除文件f = tempfile.SpooledTemporaryFile(); f.write(u.content).直到调用.fileno()方法(如果某些api需要真实文件)或达到maxsize;数据保存在内存中.即使数据写在磁盘上;文件一关闭就会被删除.