How does the iOS backup work?
In Pythonista, I want to maintain a local .db file of 2 GB.
My iOS backup is on iCloud and the file is already saved in a previous backup.
The "next backup size" (settings/iCloud/iCloud storage/manage storage) is some MB, this is normal.
I only add one little record in the dB file and this "next backup size" just goes to 2 GB.
I hoped that only some bytes need to be backuped, thus I don't understand how this iOS backup works...It seems that if an app file is modified, the entire file would be in the next backup. Thus, I should abandon this way 😢
Looks like it is file based, and and doesn't do file diff's, at least not for binary files.
I don't do much with sqlite, but is there a way to have it limit the size of any one file, thus splitting a DB over multiple files?
ccc last edited by
Would it be possible to read and write to a sqlite db that is store on iCloud, Box, Dropbox, GitHub? Or do you need that data to be local?
I think most of those methods require transferring the whole file (I don't think you can get a file handle to a Dropbox file for example).
You could do something like Google cloud datastore, which is probably free if you are not writing to it too often. Then keep track of DB operations, and play those back to the cloud version at backup time.
@JonB and @ccc Sorry to answer so lately (3:00 night for me) and thanks to try to help me.
I also think about iCloud, in Pythonista iCloud drive, but I think it will be the same.
If the file is modified, even for one byte, the entire file would be uploaded.
That does not solve my problem.
Actually, I have 30.000 files in 10.000 folders, and my Pythonista becomes very slow.
For example, if I want to create a new file, I have to wait 10" before I get the dialog to select the directory where to save my file. I hoped to save my files (books covers as png) in a dB, to have one file instead of 30.000. But this backup problem will force me to search for another solution.
I post this but I'll not be present/able to follow now the discussion. Thanks for all.
Re slowness, you might be able to place those many files in folders prefixed by period, in which case I think pythonista doesn't display them (not sure if it still has to crawl them though). You can still get to them programatically, etc.
If you don't want to back up those files, but do want to backup everything else... I wonder if you can stick files in the non-appgroup folder. I don't know whether you can set iCloud to backup one but not the other.
If you don't want to backup certain files, it appears you can call an obi function to exclude a file from backup. I haven't tried it.
@JonB Good to know but my problem is that I want the file be part of the backup but I would have imagined that the backup did work by modified blocks only. I already have written such code 40 years ago on mainframe to only save on disk modified parts of big files.
Perhaps, I could
- backup the entire file
- then set it as 'not to be in the backup'
- at each modification, save this in a journal file, which will be in the backup
- when this journal file becomes too big, save the entire file in the backup
- delete the journal file
... and so on
Édit: or perhaps:
set the file permanently 'not to be in the backup'
copy the file in iCloud Drive/Pythonista3
at each modification, save this in a journal file, which will be in the backup
when this journal file becomes too big, copy the entire file in iCloud
delete the journal file
Perhaps easier way to check date/time/size of the copied version
mikael last edited by mikael
@cvp, one more option could be to use Python standard dbm instead of sql. They can be set to automatically split the storage into several files.
@mikael Thanks for the info, but anyway, if I split, for instance, 2GB in 100 files of 20MB, each insert of update of a record could oblige the backup of a lot of bytes instead of a little bit. I'll check that.
@mikael I don't know dbm but I've read " A dbm database can only store strings, both as keys and values".
If it is still true, my keys are string but my values are images...
Could you keep files on disk, and just store file name? That would keep your DB to small text, and images themselves probably change rarely
@JonB yes but in this cases, the number of files will stay 30.000, isn't it?
Folders names are authors names
Files names are books names
Files are covers png
@JonB you are right, the images never change. The only modification in the dB or files tree is to add an author/book/cover. I'm not sure that this info could help to find the right way but so you have all infos
mikael last edited by
@cvp, yeah. I guess you could encode/decode the images, but would have to experiment what kind of control you have on the number of files.
If you place the 10000 folders underneath a folder called .DB or something, I wonder if that still causes slowness. I'm thinking I have git repos with hundreds of commits resulting in thousands of files stored in . git, and don't necessarily have massive slowdown in creating or moving files, but I'm not sure.
Could maybe have heirarchys to keep single folders small -- for instance git deals with tons of commits/blobs/trees, each which is its own folder based on the hash. But top level folders are first 2 letters of the hash, thus making it fast to lookup a particular hash -- top folder has < 256 subfolders, etc.
@JonB You are right (as usual 🙄), all my 10.000 folders are under a folder that I prefixed with a dot, and creating or moving a file has retrieved its normal speed.
Thus, thanks a lot. I'll keep my 30.000 files without a dB...
I'll have to modify some scripts to access these sub-folders or files because the upper folder is not visible.
Although, I still think that iOS backup should not be so designed, by file...But, ok, it is not vital.
A last time (for today), thanks for your time.
Unforeseen consequence: uploads via SFTP to my backup server ADrive.com does not work anymore from sub_folder of hidden folder 😢 (error 2 "no such file").
Exactly same process to my iMac (where standard SFTP server runs) works successfully.
Edit: ADrive server seems to have (temporary, I hope) problems. The same error occurs actually even for an upload from a visible folder.
Just when I test my modification...😢and😅
ccc last edited by
Folder rename (drop/add the leading dot) before and after sftp operations might be a workaround.