在Python中的大文件中拆分行的内存问题

I'm trying to read from disk a huge file (~2GB) and split each line into multiple strings:

我试图从磁盘读取一个巨大的文件(~2GB)并将每行分成多个字符串:

def get_split_lines(file_path):
    with open(file_path, 'r') as f:
        split_lines = [line.rstrip().split() for line in f]
    return split_lines

Problem is, it tries to allocate tens and tens of GB in memory. I found out that it doesn't happen if I change my code in the following way:

问题是,它试图在内存中分配数十和数十GB。我发现如果我按以下方式更改代码不会发生:

def get_split_lines(file_path):
    with open(file_path, 'r') as f:
        split_lines = [line.rstrip() for line in f]    # no splitting
    return split_lines

I.e., if I do not split the lines, memory usage drastically goes down. Is there any way to handle this problem, maybe some smart way to store split lines without filling up the main memory?

即,如果我不拆分线路,内存使用量会大幅下降。有没有办法解决这个问题,也许有一些聪明的方法来存储分割线而不填满主存储器?

Thank you for your time.

感谢您的时间。

2 个解决方案

#1

After the split, you have multiple objects: a tuple plus some number of string objects. Each object has its own overhead in addition to the actual set of characters that make up the original string.

拆分后,您有多个对象:元组加上一些字符串对象。除了构成原始字符串的实际字符集外,每个对象都有自己的开销。

Rather than reading the entire file into memory, use a generator.

不是将整个文件读入内存,而是使用生成器。

def get_split_lines(file_path):
    with open(file_path, 'r') as f:
        for line in f:
            yield line.rstrip.split()

for t in get_split_lines(file_path):
    # Do something with the tuple t

This does not preclude you from writing something like

这并不妨碍你写一些类似的东西

lines = list(get_split_lines(file_path))

if you really need to read the entire file into memory.

如果你真的需要将整个文件读入内存。

2 个解决方案

#1

更多相关文章

随机推荐