Skeleton Retarget for ReadyPlayerMe avatars picks incorrect bones

[Malcolmnixon]

Tested versions

4.3-dev5

System information

Windows 11, Vulkan Mobile, NVidia RTX 3070 TI

Issue description

ReadyPlayerMe produce customized avatars for game players. Importing these avatars using the Skeleton Retarget system results in the wrong finger bones being selected and renamed. The user can correct this by manually picking every bone in the left and right hands; however due to the popularity of ReadyPlayerMe avatars it would be preferrable if these could be identified and handled correctly with minimal effort.

Steps to reproduce

This Zip file 65fa409029044c117cbd3e3c.zip contains a ReadyPlayerMe avatar.

Importing through the standard Skeleton Retarget method results in the following finger-bone issues:

Actual Bone Type	ReadyPlayerMe Name	Name after Skeleton Retarget
Left Thumb Metacarpal	LeftHandThumb1	LeftHandThumb1
Left Thumb Proximal	LeftHandThumb2	LeftThumbMetacarpal
Left Thumb Distal	LeftHandThumb3	LeftThumbProximal
Left Thumb Tip	LeftHandThumb4	LeftThumbDistal
Left Index Proximal	LeftHandIndex1	LeftHandIndex1
Left Index Intermediate	LeftHandIndex2	LeftIndexProximal
Left Index Distal	LeftHandIndex3	LeftIndexIntermediate
Left Index Tip	LeftHandIndex4	LeftIndexDistal
Left Middle Proximal	LeftHandMiddle1	LeftHandMiddle1
Left Middle Intermediate	LeftHandMiddle2	LeftMiddleProximal
Left Middle Distal	LeftHandMiddle3	LeftMiddleIntermediate
Left Middle Tip	LeftHandMiddle4	LeftMiddleDistal
Left Ring Proximal	LeftHandRing1	LeftHandRing1
Left Ring Intermediate	LeftHandRing2	LeftRingProximal
Left Ring Distal	LeftHandRing3	LeftRingIntermediate
Left Ring Tip	LeftHandRing4	LeftRingDistal
Left Pinky Proximal	LeftHandPinky1	LeftHandPinky1
Left Pinky Intermediate	LeftHandPinky2	LeftLittleProximal
Left Pinky Distal	LeftHandPinky3	LeftLittleIntermidate
Left Pinky Tip	LeftHandPinky4	LeftLittleDistal

Minimal reproduction project (MRP)

EDIT(lyuma): Added some more test files which do not contain any mesh. The mesh vertex-based approach is not sufficient to solve these files:
simple_vrm_rig.glb.gz - This one is in T-pose
simple_vrm_rig_anim4.blend
simple_vrm_rig_nlaonly2.fbx - this is animated and not in t-pose

[BastiaanOlij]

cc @fire @lyuma

[Malcolmnixon]

I also confirmed this happens with Mixamo avatars.
And with the AlicialSolid VRM character included in the Godot VRM Plugin.

[TokageItLab]

I remember we make it to handle the case of a model with a Palm, where the fingers could exist too close to the wrist, so that the tips are mapped in priority.

It might be handled to some extent by making sure that auto-mapping excludes bones named "Tip" and "Leaf". However, if the bone names are sequentially numbered as Finger 1, 2, 3, 4, I don't think there is anything that can be done.

This is because the human hand has 4 finger bones from the wrist, but some 3D models ignore the actual first bone , and only a few of them add TipBone that don't exist in real, so it cause an offset.

Model type	Finger bones
In Real or Realistic 3D model	Palm, Metacarpal, Proximal, Distal
General 3D model	Metacarpal, Proximal, Distal
3D model specialized for FingerIK or some proprietary software	Metacarpal, Proximal, Distal, Tips

IMO, most models created by designers do not have a TipBone unless they specialize in IK. We should avoid making special accommodations for some proprietary software, so the only thing we can discuss is to consider ways to simplify manual fixes.

[Malcolmnixon]

IMO, most models created by designers do not have a TipBone unless they specialize in IK. We should avoid making special accommodations for some proprietary software, so the only thing we can discuss is to consider ways to simplify manual fixes.

I reviewed the following avatars, and at least form this limited set we need manual correction 75% of the time:

AVATAR	BONES	MAPPING RESULTS
VRM AliciaSolid	4-bones (named)	Bone mapping does not use the friendly names, and gets the fingers wrong (off by 1)
Avaturn.me	3-bones	Mapped correctly
Mixamo	4-bones (1..4)	Bone mapping wrong (off by 1)
ReadyPlayerMe	4-bones (1..4)	Bone mapping wrong (off by 1)

In this group every model with 4-bone fingers is incorrect. The possibilities for four-bone fingers seems to be either:

1. Metacarpal, Proximal, Intermediate, Distal
2. Proximal, Intermediate, Distal, Tip

The bone mapper can't differentiate these - even if the bones have friendly names (metacarpal, proximal, distal, etc.)

Consider the following from Mixamo:

It might be possible to differentiate If we calculate the ratio of the finger length (hand <--> bone4) to the position of the first bone (hand <--> bone1) then we can work out if bone 1 is very close to the hand (Metacarpal) or roughly half way along (Proximal).

So we could use position-metrics, or failing that inspect the bone names and see if they contain metacarpal, proximal, intermediate, distal, or tip "hints".

[TokageItLab]

The methods using position and length may work to some extent, but I assume it can't be guaranteed to be accurate for thumbs.

I just came up with the idea that the most accurate way to identify a Tips bone is to see if it is bound to Skin or not. It would work well to add this to the factor for mapping. Although it may not work for animations that do not have Skin, that would be rare case.