Currently I'm trying to implement GPU skinning in my project.

So far I have achieved single joint translation and rotation, and multi-jointed translation. The problem arises when I try to rotate a multi-jointed skeleton.

The image above shows the current progress.

The left image shows how the model should deform. The middle image shows how it deforms in my project. The right shows a better deform (still not right) inverting a certain value, which I will explain below.

The way I get my animation data is by exporting it to the MD5 format (MD5mesh for mesh data and MD5anim for animation data).

When I come to parse the animation data, for each frame, I check if the bone has a parent, if not, the data is passed in as is from the MD5anim file.

If it does have a parent, I transform the bones position by the parents orientation, and the add this with the parents translation. Then the parent and child orientations get concatenated. This is covered at this website.

if (Parent < 0){
    ...
    // Save this data without editing it
} else {
    Math3::vec3 rpos;
    Math3::quat pq = Parent.Quaternion;

    Math3::quat pqi(pq);
    pqi.InvertUnitQuat();
    pqi.Normalise();

    Math3::quat::RotateVector3(rpos, pq, jv);

    Math3::vec3 npos(rpos + Parent.Pos);
    this->Translation = npos;
    Math3::quat nq = pq * jq;
    nq.Normalise();

    this->Quaternion = nq;
}

And to achieve the image to the right, all I need to do is to change Math3::quat::RotateVector3(rpos, pq, jv); to
Math3::quat::RotateVector3(rpos, pqi, jv);, why is that?

And this is my skinning shader.

SkinningShader.vert

#version 330 core

smooth out vec2 vVaryingTexCoords;
smooth out vec3 vVaryingNormals;
smooth out vec4 vWeightColor;

uniform mat4 MV;
uniform mat4 MVP;
uniform mat4 Pallete[55];
uniform mat4 invBindPose[55];

layout(location = 0) in vec3 vPos;
layout(location = 1) in vec2 vTexCoords;
layout(location = 2) in vec3 vNormals;
layout(location = 3) in int vSkeleton[4];
layout(location = 4) in vec3 vWeight;

void main()
{
    vec4 wpos = vec4(vPos, 1.0);
    vec4 norm = vec4(vNormals, 0.0);
    vec4 weight = vec4(vWeight, (1.0f-(vWeight[0] + vWeight[1] + vWeight[2])));
    normalize(weight);

    mat4 BoneTransform;

    for(int i = 0; i < 4; i++) {
        if(vSkeleton[i] != -1) {
            if(i == 0) {
                // These are interchangable for some reason
                // BoneTransform = ((invBindPose[vSkeleton[i]] * Pallete[vSkeleton[i]]) * weight[i]);
                BoneTransform = ((Pallete[vSkeleton[i]] * invBindPose[vSkeleton[i]]) * weight[i]);
            } else {
                // These are interchangable for some reason
                // BoneTransform += ((invBindPose[vSkeleton[i]] * Pallete[vSkeleton[i]]) * weight[i]);
                BoneTransform += ((Pallete[vSkeleton[i]] * invBindPose[vSkeleton[i]]) * weight[i]);
            }
        }
    }

    wpos = BoneTransform * wpos;

    vWeightColor = weight;
    vVaryingTexCoords = vTexCoords;
    vVaryingNormals = normalize(vec3(vec4(vNormals, 0.0) * MV));
    gl_Position = wpos * MVP;
}

The Pallete matrices are the matrices calculated using the above code (a rotation and translation matrix get created from the translation and quaternion).

The invBindPose matrices are simply the inverted matrices created from the joints in the MD5mesh file.

Update 1

I looked at GLM to compare the values I get with my own implementation. They turn out to be exactly the same. So now i'm checking if there's a problem with matrix creation...

Update 2

Looked at GLM again to compare matrix creation using quaternions. Turns out that's not the problem either.