3D rotations in a UImage

madivad

G'day guys, it's been a while since I've played with Pythonista and I was stoked today when I opened up the forums and found a beta out there. That's great!

I have some UI ImageViews that I would like to transform the images in, about one of the 3D axis. The image has the silhouette of a phone in it and I want it to be representative of the orientation the phone is being held.

We have access to CGAffineTransform the transforms of which cover rotation, translation, skew and scale, but I can't get it to rotate about the axis I'd like because they're 2D related.

I have started looking at how to write the wrapper function and would seriously like some help to do it :)

This example code:

CATransform3D transform = CATransform3DIdentity;
transform.m34 = 1.0 / -500;
transform = CATransform3DRotate(transform, 45.0f * M_PI / 180.0f, 0, 1, 0.0f); 
_bgView.layer.transform = transform;

I believe that gives the rotation I am after (at this point I'm only doing a single axis, not the whole device, however I may do that later).

I've opened up https://omz-software.com/pythonista/docs/ios/objc_util.html
and now I'm here.

If someone can point me into the right direction for researching the transform I actually need, I'd appreciate that. I've had my head stuck in CGAffineTransform for the last few days and I've skewed my own searching results and I just don't have the right terminology under my belt.

This page has a lot of information that is WAY beyond my knowledge, mathematically, practically and programmatically. I realise I have NO idea which one or where to start. But if the above function is a good place to start and I need any others later, I can always extend the code and include more of the other functions myself (hopefully).

https://developer.apple.com/documentation/quartzcore/transforms

Any help appreciated.

I have read that this is easy enough to achieve in scenekit(?) but I'm designing a GUI style app. It's a page of data with some icons, one of which I want to display an orientation. I don't think that scenekit wouldn't fit my needs.

edit: I had to remove all of the links except the one I really wanted as this question got flagged as spam. I have done the research, I'm just confused about what it is I actually need.

Thanks.

cvp

@madivad said

I have read that this is easy enough to achieve in scenekit(?) but I'm designing a GUI style app. It's a page of data with some icons, one of which I want to display an orientation. I don't think that scenekit wouldn't fit my needs.

I think that SceneKit is the best way to do what you want and you can always add a scene as a subview of an ui.View

You would find a lot of samples of SceneKit in the forum

JonB

@madivad This should be doable.

Something like this though I haven't tried yet

from objc_util.import *
from ctypes import *
import math

'''we need to wrap the CATransform structure '''
def CATransform (ctypes.Structure):
    _fields_ = 
        [{'m11', c_float},{'m12', c_float},{'m13', c_float},{'m14', c_float},
        {'m21', c_float},{'m22', c_float},{'m23', c_float},{'m24', c_float},
        {'m31', c_float},{'m32', c_float},{'m33', c_float},{'m34', c_float}]


CATransform3DMakeTranslation = c.CATransform3DMakeTranslation 
CATransform3DMakeTranslation.argtypes = [c_float, c_float, c_float]
CATransform3DMakeTranslation.restype = CATransform

CATransform3DRotate = c.CATransform3DRotate
CATransform3DRotate.restype = CATransform3D
CATransform3DRotate.argtypes = [CATransform c_float, c_float, c_float, c_float]

identity= CATransform3DMakeTranslation(0,0,0)
identity.m34 = -1/500
rot = CATransform3DRotate(identity,  math.radians(45), 0,1,0)

v=ui.View(frame=(0,0,800,800))
sv=ui.View(frame = (50,5,300,300))
sv.bgcolor = 'blue'
v.add_subview(sv)
v.present()

SV = ObjCInstance(sv)
layer = SV.layer()
layer.transform = rot

Pythonista interface with ctypes structures returned from ObjC methods can be kind of screwed up, so we might have to modify the argtypes of the setLayer_ method manually to get things to work in the last statement.
I can give it a try tonight

madivad

@cvp said in 3D rotations in a UImage:

you can always add a scene as a subview of an ui.View

I'm sorry, but this is exactly the sort of "stupid" (from me) that drives me nuts. I did not think of that, I made an assumption that it was ui or scenekit, I didn't realise you could mix and match like that. That's awesome and I'll definitely play with that idea tonight.

@JonB excellent, that a great place for me to start. I'll definitely give that a go too, hopefully report back tonight.

Thanks guys, cheers.

cvp

@madivad In @JonB code, some little typing errors:

not objc_util.import (no ".")
class CATransform instead of def CATransform
not [{'m11', c_float} but [('m11', c_float)
CATransform3DRotate.argtypes = [CATransform, c_float, c_float, c_float, c_float]
CATransform structure goes up to m44

madivad

@JonB said in 3D rotations in a UImage:

CATransform3DMakeTranslation = c.CATransform3DMakeTranslation
CATransform3DMakeTranslation.argtypes = [c_float, c_float, c_float]
CATransform3DMakeTranslation.restype = CATransform

CATransform3DRotate = c.CATransform3DRotate
CATransform3DRotate.restype = CATransform3D
CATransform3DRotate.argtypes = [CATransform c_float, c_float, c_float, c_float]

identity= CATransform3DMakeTranslation(0,0,0)
identity.m34 = -1/500
rot = CATransform3DRotate(identity, math.radians(45), 0,1,0)

I'm looking through the above code and I want to understand it. This is what was going wrong when I looked at the "wrapper how to" page, it makes little sense to me.

For starters:

the 1st and 4th lines above:

CATransform3DMakeTranslation = c.CATransform3DMakeTranslation
CATransform3DRotate = c.CATransform3DRotate
where does the "c." come from?

How are the inputs of CATransform3DRotate and CATransform3DMakeTranslation defined? Are those functions a part of the struct? I was thinking I'd need wrappers for each and every function but I'm starting to see (but not yet understand) some of it.
I am getting an error at ctypes.Structure with the carrot pointed to the dot in between. invalid syntax. weird, probably a conflict with other code in the module? import wrong? I'll post full code tonight.
this line:

CATransform3DRotate.argtypes = [CATransform c_float, c_float, c_float, c_float]
should there by a comma between the first arguments, or is the first argument not required?
I am getting an error about argtypes item 1 not having a from type.
Looking through what you've done there and cursory look at the docs, I've become confused re CATransform and CATransform3D and the relationship between them all. I'll focus on it tonight. Thanks for your inputs gents. cheers.

cvp

@madivad said

I am getting an error at ctypes.Structure with the carrot pointed to the dot in between. invalid syntax. weird, probably a conflict with other code in the module? import wrong? I'll post full code tonight.

Try

class CATransform (Structure):
    _fields_ = [('m11', c_float),('m12', c_float),('m13', c_float),('m14', c_float),('m21', c_float),('m22', c_float),('m23', c_float),('m24', c_float), ('m31', c_float),('m32', c_float),('m33', c_float),('m34', c_float), ('m41', c_float),('m42', c_float),('m43', c_float),('m44', c_float)]

cvp

@madivad said

How are the inputs of CATransform3DRotate and CATransform3DMakeTranslation defined? Are those functions a part of the struct? I was thinking I'd need wrappers for each and every function but I'm starting to see (but not yet understand) some of it.

See Apple doc, like https://developer.apple.com/documentation/quartzcore/1436524-catransform3drotate?language=objc

cvp

@madivad I run this code without syntax error but it crashes

from objc_util import *
from ctypes import *
import math

'''we need to wrap the CATransform structure '''
class CATransform (Structure):
    _fields_ = [('m11', c_float),('m12', c_float),('m13', c_float),('m14', c_float),('m21', c_float),('m22', c_float),('m23', c_float),('m24', c_float), ('m31', c_float),('m32', c_float),('m33', c_float),('m34', c_float), ('m41', c_float),('m42', c_float),('m43', c_float),('m44', c_float)]


CATransform3DMakeTranslation = c.CATransform3DMakeTranslation 
CATransform3DMakeTranslation.argtypes = [c_float, c_float, c_float]
CATransform3DMakeTranslation.restype = CATransform

CATransform3DRotate = c.CATransform3DRotate
CATransform3DRotate.restype = CATransform
CATransform3DRotate.argtypes = [CATransform, c_float, c_float, c_float, c_float]


identity= CATransform3DMakeTranslation(0,0,0)
identity.m34 = -1/500
rot = CATransform3DRotate(identity,  math.radians(45), 0,1,0)

v=ui.View(frame=(0,0,800,800))
sv=ui.View(frame = (50,5,300,300))
sv.bgcolor = 'blue'
v.add_subview(sv)
v.present()

SV = ObjCInstance(sv)
layer = SV.layer()
layer.transform = rot

Error

Fatal Python error: Segmentation fault

Current thread 0x000000016be37000 (most recent call first):
  File "/private/var/mobile/Containers/Shared/AppGroup/1B829014-77B3-4446-9B65-034BDDC46F49/Pythonista3/Documents/a8.py", line 20 in <module>

Extension modules: pykit_io, _ui, _appex, _frameworksimporter, console, _debugger_ui (total: 6)

madivad

TLDR; I have been researching this for hours and in checking out the image.transform options I can see that I think the transforms I need are already within pythonista, but I had chosen the wrong one initially and had never thought to look at the others. I feel quite stupid about now and am going back to the drawing board. I'm going to leave this post here because I've thought about and learned a lot and this has kind of formed my notes and I want to take the time to thank @JonB and @cvp for their time. I might revisit this code as an ObjC wrapper if I'm wrong, because I feel like I'm close to nailing it, but there is no NEED to reinvent the wheel.

I have to revisit the sample code I pasted at the origin because that was just plucked from SE and now I want to work out how I want to apply it. What I pasted isn't actually the code, I'm working on that part now. This is more about the learning for me at the moment and I am trying to get my head around the terminology for the transforms and matrixes. The maths is really beyond me and I'm starting to confuse a lot of these terms.

Looking at what @jonb has put together is based on some mistakes I think I have misstated.

This is where I'm thinking atm:

The code I pasted in the OT was from a SE site and not exactly right for my application. In learning the terms (mostly tonight) I need to MAKE the translation. At least, that's what I think I want.

I want the 3D equiv to CGAffineTransform, which doesn't take in the original instance (hence why I think I need the create/make version).

CATransform3DMakeRotation vs
CATransform3DRotate vs
CGAffineTransform <-- the 2D version
I think I need to use first one over the second.

I'm still confused by the c.CATrans.... where does the c come from in JonB's code?

Swapping out ctypes.Structure for just Structure worked

CATransform3DMakeTranslation declaration:

CATransform3D CATransform3DMakeTranslation(CGFloat tx, CGFloat ty, CGFloat tz);

CATranform3DMakeRotation declaration

   CATransform3D CATransform3DMakeRotation(CGFloat angle, CGFloat x, CGFloat y, CGFloat z);

They are both of type: CATransform3D, declared here:

typedef struct CATransform3D {
    ...
} CATransform3D;

Based on the code you guys have posted, I believe this is going to translate to:

class CATransform3D (Structure): ### NOTE 3D
    _fields_ = [('m11', c_float),('m12', c_float),('m13', c_float),('m14', c_float),('m21', c_float),('m22', c_float),('m23', c_float),('m24', c_float), ('m31', c_float),('m32', c_float),('m33', c_float),('m34', c_float), ('m41', c_float),('m42', c_float),('m43', c_float),('m44', c_float)]

Now I have got it crashing too, and I think it's to do with the returned 3DTransform, I need to convert it back to Affine... which means I think I would need to take the original image and trying to 3DRotate it. It's the image from a ui.ImageView which is..... it was at this point he knew he'd stuffed up

I'll post more when I recover.

I'll be back. :)

cvp

@madivad said

where does the c come from in JonB's code?

Part of Pythonista...(try "print(c) in a script which does not import anything)

ccc

Now add from objc_util.import * and try print(c) again.

This is exactly why PEP8 advises against wildcard imports.

cvp

@madivad Could you try this script https://github.com/cvpe/Pythonista-scripts/blob/master/Gists/a4.py.

It will ask you first to select a photo from your camera roll, for example an iPhone face, but any photo would be ok.
Then, the script displays an empty full screen ui.View with a SceneKit subview.
This SceneKit shows a box representing an iPhone with your selected photo as a face.
You can use two fingers to rotate or scale the box.
And if you move or rotate your device, you will see the box doing the same.
There is still a limitation (which I can't actually remove), the box comes back at its initial orientation.
But anyway, you can see what Pythonista offers to you.

Feedback would be appreciated

JonB

@cvp
Looks like @cvp came to the rescue with my untested code. But here is the original, marked up with the intent. You can do google search for iOS headers CATransform3D, and get the c header files describing the various structure and functions, then just have to manually translate. The way CFUNCTION wrappers work, the doll contains the symbol, but you have to manually set return type (restype) and argument types argtypes.

The low level stuff is slightly trickier to work it’s compared to actual ObjC. When converting objc code, you usually don’t need to mess with any of that, because the method encodings define everything, and are built into the runtime. But when passing structures instead of objects, you often have to override restypes. Rubicon handles this cleaner than the current objc_util…


from objc_util import *
from ctypes import *
import ui
import math, copy

'''we need to wrap the CATransform structure
 struct CATransform3D
{
  CGFloat m11, m12, m13, m14;
  CGFloat m21, m22, m23, m24;
  CGFloat m31, m32, m33, m34;
  CGFloat m41, m42, m43, m44;
};
this could also have been done with a pack
 '''
 
class CATransform3D(Structure):
    _fields_ =   [('m11', CGFloat),('m12', CGFloat),('m13', CGFloat),('m14', CGFloat),
        ('m21', CGFloat),('m22', CGFloat),('m23', CGFloat),('m24', CGFloat),
        ('m31', CGFloat),('m32', CGFloat),('m33', CGFloat),('m34', CGFloat),
        ('m41', CGFloat),('m42', CGFloat),('m43', CGFloat),('m44', CGFloat),
        ]

''' 
note, c is a variable imported in objc_util, which is the cDLL for the static library.
'''

'''
/* Returns a transform that translates by '(tx, ty, tz)':
 * t' =  [1 0 0 0; 0 1 0 0; 0 0 1 0; tx ty tz 1]. */

CA_EXTERN CATransform3D CATransform3DMakeTranslation (CGFloat tx,
    CGFloat ty, CGFloat tz)
    '''
CATransform3DMakeTranslation = c.CATransform3DMakeTranslation 
CATransform3DMakeTranslation.argtypes = [CGFloat, CGFloat, CGFloat]
CATransform3DMakeTranslation.restype = CATransform3D

'''
/* Rotate 't' by 'angle' radians about the vector '(x, y, z)' and return
 * the result. If the vector has zero length the behavior is undefined:
 * t' = rotation(angle, x, y, z) * t. */

CA_EXTERN CATransform3D CATransform3DRotate (CATransform3D t, CGFloat angle,
    CGFloat x, CGFloat y, CGFloat z)
    '''
CATransform3DRotate = c.CATransform3DRotate
CATransform3DRotate.restype = CATransform3D
CATransform3DRotate.argtypes = [CATransform3D, CGFloat, CGFloat, CGFloat, CGFloat]

''' named const can often be retrieved using in_dll.  however, we need a mutable copy, so we dont wat to use this as identity'''
CATransform3DIdentity=CATransform3D.in_dll(c,'CATransform3DIdentity')
identity=copy.deepcopy(CATransform3DIdentity)


'''other ways to get identity:

identity = CATransform3D()
identity.m11=1
identity.m22=1
identity.m33=1
identity.m44=1

or

identity=CATransform3DMakeTranslation(0,0,0)

note, we cant use CATransformMake because that is a c macro, not an exported function
'''



v=ui.View(frame=(0,0,500,500))
v.bg_color='white'
sv=ui.ImageView()

sv.image=ui.Image.named('card:Clubs3')
sv.size_to_fit()
v.add_subview(sv)
v.present('sheet')

layer = sv.objc_instance.layer() 

#another way to get current transrorm
#identity=layer.transform(argtypes=[],restype=CATransform3D)

identity.m34 = -1/500 #sets perspective
#rotate 
rot = CATransform3DRotate(identity,  math.radians(55), 0,1,0)
layer.setTransform_(rot,argtypes=[CATransform3D],restype=None)

cvp

@JonB Marvelous, as usual... Sometimes, I ask me why I try to do something here.
Thanks for him and for me for your clear explanations.

JonB

@cvp by the way, the crash was because I was using c_float instead of CGFloat. objc_util defines CGFloat, along with some other useful iOS c types. CGFloat is double on 64 bit (all modern version of iOS) and float on 32 bit machines (ipad 3 or maybe 4 and earlier). Also, as you guys found, I mistakenly forgot the last row of the transform.

There are ways to animate the rotations.
But I suspect SceneKit is ultimately easier to work with.

cvp

@JonB Thanks. Hoping that @madivad will find an answer to his request.

pulbrich

Have a look at this wrapper for sceneKit. There are plenty of examples, too. Though I made no regression tests lately it should still work, or at least help with understanding the bridging.

link to the GitHub repo