Photos Asset.get_image_data: Is there a memory leak?

[billbris]

I have written a script that will move my photo assets from one, more, or all albums to an FTP server. Basically, it is functional. When I select an album (standard iOS albums), the files get moved to the FTP server in tact. The problem I am running into is when I select all the albums, read: a lot of assets (images and video) the process fails on asset.get_image_data.

The error is:
<class 'SystemError'>-<built-in method get_image_data of _photos2.Asset object at 0x112539fc0> returned a result with an error set. When I try to get any additional information from the io.BytesIO class, I get the same error as above (error set).

Once this happens, the script breaks down and all kinds of other errors occur. After a number of errors occur Pythonista breaks and I am returned to the iOS home.

As for quantity, the scripts tend to break down after 1400-ish files have moved successfully. It doesn't seem as if the actual image or video is corrupted as the same file copies without a problem during a smaller transfer.

I have uploaded no code as it crosses a few files. It is actually very simple.

1. Get an album or list of albums
2. Create an FTP connection
3. For each album step through each "asset" in the album.
4. For each asset create a filename to be used on the FTP server
5. Get the asset.get_image_data buffer and use ftplib storbinary to send the asset to the FTP server.

Like I said, this works for something under 1400 images/videos.

Anyone else have similar issues and might be able to point me in a direction to solve the problem?

Thanks in advance.
Bill

[JonB]

You might try
https://forum.omz-software.com/topic/3146/share-code-get-available-memory/5
to trend memory and o see if there is a leak.

You might try a loop that just creates the asset, without otherwise using it, to see if the problem is with your code or the method. For instance if you are opening but not closing files, or are creating reference cycles. You might also try calling gc.collect inside your loop-- I have found that certain ObjCInstances create reference cycles to their instance methods, which cause issues when doing video processing, but that in some cases calling collect manually helps.

[ccc]

I have uploaded no code as it crosses a few files.

A GitHub repo deals well with a few files. This kind of workflow would benefit from asyncio like https://pypi.python.org/pypi/aioftp delivers. I am not sure if it will work in Pythonista but if so, it could really speed up the transfer of 1400 images / movies.

[jumpbeen]

I have the same problem.

import photos
from get_available_memory import get_free_mem
import gc
from PIL import Image
photo_count = 500 
photo_index = 0
all_assets = photos.get_assets()
print('Start', get_free_mem()) #Start 814.9 MB
while photo_index < photo_count:
	ass = all_assets[photo_index]
	img_data = ass.get_image_data()
	img_data.__del__()
	del img_data
	del ass
	gc.collect()
	photo_index += 1
del all_assets
gc.collect()
print('Done', get_free_mem()) #Done 165.4 MB

[ccc]

This post is deleted!

[JonB]

Yes, this does seem to leak, although sometimes the free memory can be misleading ( memory allocated might not be truly "freed").

I have an Objc version of this, which leaks somewhat more slowly, and it seems to give back much of the leak once the script ends.

https://gist.github.com/c26545118d79b505ebf6425bedebad71

I suspect maybe there is a way to force an autorelease pool drain, i am a little fuzzy on where than happens (it seems to not happen while a script is running?)

[JonB]

Ok, a workaround for the original code:

First, make a copy of objc_util.py, and place in site packages, for instance as objc_util2

At around line 532, change the if ptr: to

		if ptr and not b'NSAutoreleasePool' in class_getName(object_getClass(ptr)):
			# Retain the ObjC object, so it doesn't get freed while a pointer to it exists:
			objc_instance.retain(restype=c_void_p, argtypes=[])
			objc_instance._cached_methods = {}
		return objc_instance

Now, in the photos code, you can wrap your loop in an autorelease pool to make sure objc memory is getting drained.

while [...]:
    pool=ObjCClass('NSAutoreleasePool').new()
    # get image data, etc
    [...]
    pool.drain()

This seems to have squashed the leak, as the memory usage never grows.

[billbris]

JonB:
First of all, excellent work! Thank you for your time and effort.

I added the above fix in the manner you prescribed. It worked in both the pared down test (step through all photos/videos/etc, calling get_image_data for each asset) as well as the full on script where the problem initially occurred.

The script used to die at asset #1440-ish each and every time (on my iPad). It now runs through a normal completion copying 1628 files across FTP (using local wifi connection to my home FTP server).

Well done!
Bill