FBXLoader not working with many skeleton animations (e.g., from Mixamo.com)

[radman-x]

Description of the problem

The FBXLoader does not work with many skeleton animations. e.g., from Mixamo.com. One example from Mixamo that fails is the following:

Character: WHITECLOWN N HALLIN
Animation: SAMBA DANCING

You can get this model by downloading it from Mixamo directly but I have also attached it to this issue.
WhiteClownSambaDancing.zip

Here is the displayed result using the webgl_loader_fbx.html sample in THREE v96 modified to load the WhiteClownSamaDancing.fbx:

Screen shot using webgl_loader_fbx.html

Here is the same FBX file displayed in AutoDesk FBX Review:

Screen shot using AutoDesk FBX Review

There are many other Mixamo character/animations pairings that work fine with the FBXLoader but many that do not. It is possible (though not confirmed) that the ones that fail were created with Maya (this has been stated as a possible problem in some other issues).

Also, the WhiteClownSambaDancing.fbx file loads correctly in many other software including the Mixamo site itself and http://www.open3mod.com/. The later is fully open source so you can see the exact code they use to perform the node/bone transforms and animations. They actually use AssImp for their conversion and you can see the exact code there. In particular, see the following for their FBX import 3D transform handling:

https://github.com/assimp/assimp/blob/master/code/FBXConverter.cpp#L644

After digging into the FBXLoader code a bit, it seems there may be several areas where the issue could lie:

• There does not seem to be any code to honor the FBX inherit type on the nodes.
• There does not seem to be any code to implement rotate/scale pivot points on the nodes (NOT the geometry transforms which are implemented).

The following code from the AutoDesk FBX SDK may be of some help for implementing both of the above, esp. the code and comments in CalculateGlobalTransform():

AutoDesk FBX source code Transformations/main.cxx

There are some other THREE.js issues which have not been fully addressed regarding incorrect FBX animations loaded via the FBXLoader (#11895, #13466, #13821). This issue is about improving the FBXLoader, not a particular file or asset pipeline. Please do not suggest use of FBX2GLTF (which just bakes the animations) or other converters.

Also, we are willing to provide some help with design and coding, if need be, but doing a full solution with a PR is beyond our bandwidth at this time (ping @looeee @Kyle-Larson @takahirox ?).

Three.js version

• Dev
• r96
• ...

Browser

• All of them
• Chrome
• Firefox
• Internet Explorer

OS

• All of them
• Windows
• macOS
• Linux
• Android
• iOS

Hardware Requirements (graphics card, VR Device, ...)

[looeee]

Thanks for the detailed bug report. This seems to be the same issue as #14864.

There does not seem to be any code to honor the FBX inherit type on the nodes.

Can you explain what you mean by the above line?

There does not seem to be any code to implement rotate/scale pivot points on the nodes (NOT the geometry transforms which are implemented).

Three.js doesn't support transformed pivot points, so there is no way to do this directly. The only solution that I am aware of would be to create a parent Group for each model and apply the pivot transforms using that. However this is obviously not desirable, since it involves creating extra nodes in the scene graph, and would also probably add a lot of complexity to the animation section of the loader. Ideally we'll be able to solve this issue without resorting to this approach.

The data on FBX transforms are taken from this page. It's not completely clear from that, but I've found that FBX files can contain either the FBX and Maya transform type detailed there or the 3ds Max type.

It seems likely that this issue stems from incomplete / incorrect application of the FBX and Maya transform type.

[radman-rt]

Thanx for the fast reply. The "inherit type" issue I discussed is a property on the FBX node -- see:

http://docs.autodesk.com/FBX/2014/ENU/FBX-SDK-Documentation/cpp_ref/class_fbx_transform.html#a0affdd70d8df512d82fdb6a30112bf0c

(The above link is to the 2014 FBX spec but I think it's the same on the 2017). This property, essentially, determines the equation to use for a node's complete, global 3D transformation. There are 3 values for this:

• eInheritRrSs (0)
• eInheritRSrs (1)
• eInheritRrs (2)

I think if the letter is cap, then that transform is global (e.g., "R" => global rotation) or if it's small, then that is local (e.g., "r" => local rotation). And that gives the order of the matrix multiplications. In any event, if you search through the CalculateGlobalTransform() method in this page (same "main.cxx" link as in my first comment above), you will find the final matrix equation is determined by this property (search for the "lInheritType" variable). I believe these are the correct equations and code we need to implement. Both the failing Mixamo asset file and some succeeding asset files have these properties on their nodes (we found this by dumping some data in FBXLoader) so this may not be problem but it is something to consider.

Also the code in the AssImp FBXConverter::GenerateTransformationNodeChain method here does something similar though they do not seem to use the InheritType property. It seems they are just doing tests for non-zero values of some properties (pivot vectors, etc.). One of the more interesting aspects of the AssImp code id that they have the same problem as we do with THREE -- they must insert extra nodes in the hierarchy to accomplish the full 3D transform. Recall that the AssImp code is used Open3Mod and that correctly loads the offending Mixamo file.

I am not sure we can implement this in the current state of THREE without inserting additional nodes at the points where a pivot exists (though we might be able to). Of course, if THREE added pivot points (rotation and scale) to their 3D transform matrix calculations, this would be a lot easier.

Your hunch on Maya vs. 3DS Max may in fact be right. Maybe the Mixamo characters created in 3DS Max work in FBXLoader and those created in Maya do not. If I get some time, I can dump some more data in FBXLoader to determine the authoring application of the bad vs. good Mixamo FBX files.

[looeee]

Thanks for the info. I've looked over the links you provided, especially AutoDesk FBX source code Transformations/main.cxx, which I hadn't seen before.

I think I was incorrect about the meaning of pivot in the context of FBX transforms. It seems like pivot actually means "global transform matrix" here - so it's possible to apply all the pivot transforms via matrix transforms.

The exception is the 3DS Max style Geometric transform data, referred to as the 3DS Max object-offset concept in the FBX docs. I had been implementing this by transforming the geometry of each mesh, thinking it was a hack to get around the lack of pivot points, but actually it seems like that may be the correct interpretation after all.

I'll have a go at implementing the transform code from that link. It will be sometime next week before I have time to do it, however.

[radman-rt]

OK, that sounds great.

BTW, one thing I found handy is the AutoDesk FBX Converter -- 2013 is the latest version I found here. It is very nice since you can convert to ASCII and view the data much easier. I will continue some further research to find the differences between the working and failing FBX files.

Let me know if I can provide any further info or assistance.

[looeee]

After #14933, model (Group, Mesh, etc ) transforms are applied correctly according to the FBX spec.
However, it seems like there are still issues with how transforms are applied to bones and / or animation nodes. I'm going to link some reference info here that may help me or someone else figure out what the problem is:

Bones (cluster nodes in FBX) have three additional matrices and a "mode" associated with them:

• transform - the initial global transform of the mesh that the bone is controlling. These should be safe to ignore, apparently they are largely legacy and will be the same for every bone in the case of a single skinned mesh.
• transformLink - the bone's global transform at the moment of binding.
• TransformAssociateModel - this also appears to be ignored in FBX SDK example code, and is only needed dependant on the value of ELinkMode according to docs.
• ELinkMode - this specifies whether vertex weights need to be normalized, and is currently ignored.

There is some information in the following links regarding how to apply transformations to bone nodes:

• Useful things you might want to know about FBXSDK
• Calculating bind pose matrix and animation bone transforms
• Getting the local transformation matrix for the vertices of a cluster?

It's mentioned in a couple of places that the correct matrix for the bone at the time of binding is

lClusterRelativeInitPosition = lClusterGlobalInitPosition.Inverse() * lReferenceGlobalInitPosition;

Which I am interpreting to mean:

bone.matrix = transformLink.inverse() * model.matrixWorld

However, so far, attempting to do this in the loader has not given good results.

[looeee]

Apologies @mrdoob, I was probably unclear in the last comment - this model still has issues, probably related to bone transforms.

[mrdoob]

It closed automatically 🤔

[looeee]

Oh, apologies to the AI system that closed it then 😆
I'll be more careful when using the word 'fixed' and referencing an issue in the same sentence in future.

[yazancash]

i have the same problem. for 3 days im trying to find where is the problem.

i believe the problem is in array
Transform: *16 {

not
TransformLink: *16 {

FBXLoader sounds reading Transform{ } incorrectly.
while TransformLink{ } is not important. like we can ignore it.

im glad to donate for developers who solve this. i really need this.

thanks for your time!

*Edit: here is an example copy of my FBX file: http://rigmodels.com/del/test.zip
you can see Animation works on all programs, except ThreeJS.