About using CFFI in Pythonista (and my CFFI-based libraries)

[dgelessus]

This is in response to the questions/replies from @cvp and @JonB in https://forum.omz-software.com/topic/5660/install-pymunk-on-pythonista-get-cffi-error about how to use CFFI in Pythonista. I'm putting it in a separate thread since it's not really relevant to the original question (getting a specific library working in Pythonista).

For context: CFFI is a library for calling C code from Python. It has a similar purpose as ctypes, with one big difference being that CFFI lets you write type and function declarations using standard C syntax, without having to "translate" them to another syntax/API first as with ctypes.

By default, CFFI uses its own custom C extension to call C code, and also has various features that make use of a proper C preprocessor/compiler. Of course this doesn't work by default on Pythonista, since there is no C toolchain and you can't compile or load custom extension modules. However, in addition to the default C extension backend, CFFI also includes a ctypes-based pure-Python backend (cffi.backend_ctypes.CTypesBackend), which works in Pythonista, with no modifications needed to CFFI itself.

The installation process is very straightforward: you download the source and copy the cffi package into site-packages, as with any other module. We can safely ignore the C parts, since we don't use them. (You can probably also install CFFI via Stash, but I haven't tried it.) Once it's installed, you can create an FFI object with the ctypes backend like this:

import cffi.backend_ctypes

ffi = cffi.FFI(backend=cffi.backend_ctypes.CTypesBackend())

After that, you can use the ffi object as you normally would - see the CFFI docs. Note that the docs sometimes refer to different modes of operation for CFFI (ABI/API and in-line/out-of-line). Only in-line ABI mode can be used with the ctypes backend in Pythonista, all other modes require a working C compiler.

Unfortunately, this often doesn't help with getting CFFI-based libraries to work on Pythonista - in many cases these libraries rely on third-party or custom C code that isn't available in Pythonista. CFFI is simply the interface between Python and C, so even if you have CFFI working it won't be of any use without the C libraries that you want to call.

That's not to say that CFFI is of no use in Pythonista. Almost everything that you can do with ctypes can also be done with CFFI, and usually with a much nicer and more convenient API, which makes it very useful for calling C APIs (Unix/iOS and CPython).

[dgelessus]

There are also a couple of CFFI-based libraries that I wrote, which have been linked in the other post. It seems that I never made a proper post on the forums about them, so I'll do that here:

The first one is cffipp (found in my pythonista-c-utils repo), which adds a proper C preprocessor to CFFI. By default CFFI's C parsing understands almost no C preprocessing features (only simple constant #defines are supported). This library adds support for all important C preprocessor directives and features, which makes it possible to load standard iOS headers into CFFI (as long as they are pure C, such as Core Foundation). The repo includes patched versions of a few headers, to make them compatible with CFFI and improve parsing performance.

Unfortunately, cffipp is now quite outdated - I wrote it three years ago based on CFFI 1.6 and the headers for iOS 9. The CFFI and header patches include a lot of copied code, so updating it to newer CFFI and iOS SDK versions would take a bit of work. If anyone is interested in using this, let me know, I might be able to get it updated.

The second library is objc-cffi, which is basically a CFFI-based version of objc_util with a few extra convenience features. This library is fairly incomplete - although it has a few features that objc_util doesn't (introspection, better automatic type detection for methods) it's missing some very important features for practical use (class definitions, block support, collection syntax).

I also found out quite soon that CFFI is not well suited as the base for this library. To interact with the Objective-C runtime I need to create a lot of types dynamically, which is very slow with CFFI, because every type needs to be constructed as C source code and then parsed by CFFI. I also ran into name collision issues, because every FFI object has a single namespace and does not allow multiple definitions (the Objective-C runtime sometimes returns different type information for the same type, depending on context).

Around this time I also started contributing to rubicon-objc, which has also been mentioned a few times on this forum - it's basically objc_util, but not Pythonista-specific (also works on Mac and other platforms with Objective-C). I've ported most of objc-cffi's convenience features over to rubicon-objc now, and I'm still working on improving rubicon-objc, so I would definitely recommend that over objc-cffi.

[mikael]

@dgelessus, big thank you for this write-up. Good to understand the concepts even though I have limited need for C interaction at the moment.