I see files being opened, but not closed. Could that be the leak?
I'm not convinced. There's only one open() (line 10) and that's part of a context manager (with open([...]) as f: so open files should be automatically closed when that block completes. IIRC even if an exception occurs.
Same applies to line 53 (with ZipFile([...]).
I do suspect a memory leak though.
Having the files being processed on a NAS may add additional complications too and those may vary depending on the protocol in use (SMB, NFS, FTP, SFTP, etc). Though that may depend if the script is running directly on the NAS or on a client of the NAS.*
This needs more debugging. I'd be trying the following:
- Regularly writing python free memory/heap, system free memory, and system swap usage to a log file (and stdout)
- Testing with both the files and script on the same machine. If it doesn't happen it's highly likely to be network related.
- Have the script flush python's I/O buffers after zipf.write([...]
- Have the script flush the system's I/O buffers after zipf.write([...]
I'd also consider adding more swap space but that may only delay the problem not fix it.
*: if the latter it's going to perform poorly anyway. There's going to be a lot of data going over your network in both directions.
Statistics: Posted by thagrol — Fri Jan 17, 2025 11:58 pm
