Цей сайт найкраще переглядати в сучасному браузері з включеним JavaScript.

Using spine-player (spine-ts) with custom fragment shader

MattSalsa

Howdy! In our game engine we use a shader to let our players change the color of their character. We also have a web based (spine-ts/spine-player) preview of the character and would like to reuse this shader there.

Naively, I tried hooking spine-player's success callback and calling player.context.gl.attachShader. The shader compiles and these calls don't throw, but there is no change in what's rendered.

After looking at the source, I'm assuming this is because the gl->canvas pipeline is inititalized and managed outside of any of the lifecycle hooks that are exposed in spine-player.

Is there a way to do this without "diving deeper" into spine-webgl, if not, is there an example showing usage of a custom frag shader with spine-webgl?

Thanks so much for taking a look!

Here is where I left off:

playerRef.current = new SpinePlayer("container", {
        jsonUrl: "https://thumbnails.r2.allout.game/playercharacter-8.json",
        atlasUrl: "https://thumbnails.r2.allout.game/playercharacter-8.atlas.txt",
        animations: ["Player_Shop/Idle"],
        showControls: false,
        backgroundColor: "#00000000",
        alpha: true,
        preserveDrawingBuffer: true,
        showLoading: false,
        defaultMix: 0,
        viewport: {
          x: -350,
          y: -120,
          width: 600,
          height: 700,
          padLeft: "0%",
          padRight: "15%",
          padTop: "10%",
          padBottom: "0%",
          debugRender: false,
        },
        success: (player: SpinePlayer) => {
          const gl = player.context?.gl as WebGLRenderingContext;

          const vertexShaderSource = `
              attribute vec4 a_position;
              void main() {
                  gl_Position = a_position;
              }
          `;
          
          const fragmentShaderSource = `
              void main() {
                  gl_FragColor = vec4(0.0, 0.0, 0.0, 1.0); // all black for testing
              }
          `;
          
          const compileShader = (gl: WebGLRenderingContext, source: string, type: number) => {
              const shader = gl.createShader(type)!;
              gl.shaderSource(shader, source);
              gl.compileShader(shader);
              if (!gl.getShaderParameter(shader, gl.COMPILE_STATUS)) {
                  console.error(gl.getShaderInfoLog(shader));
                  gl.deleteShader(shader);
                  return null;
              }
              return shader;
          };
          
          const vertexShader = compileShader(gl, vertexShaderSource, gl.VERTEX_SHADER);
          const fragmentShader = compileShader(gl, fragmentShaderSource, gl.FRAGMENT_SHADER);
          
          if (!vertexShader || !fragmentShader) {
              console.error("Error compiling shaders");
              return;
          }
          
          const shaderProgram = gl.createProgram()!;
          gl.attachShader(shaderProgram, vertexShader);
          gl.attachShader(shaderProgram, fragmentShader);
          gl.linkProgram(shaderProgram);
          
          if (!gl.getProgramParameter(shaderProgram, gl.LINK_STATUS)) {
              console.error("Error linking shader program", gl.getProgramInfoLog(shaderProgram));
              return;
          }
          
          gl.useProgram(shaderProgram);
          
          setSkins(skins);
          setReady(true);
        },
      });

Davide

MattSalsa

If you only need to change the color, isn't it sufficient to use skeleton.color like this:

success: (player) => {
    player.skeleton.color.set(1, 0, 0, 1); // tint the skeleton red
}

Alternatively, you could try to monkey patch the Shader.newTwoColoredTextured method, as the player uses that one.
If you need to do more than that, it would be better if you showed us your fragment shader.

Spinebot

To use a custom fragment shader with spine-player (spine-ts), you will indeed need to dive a bit deeper into the spine-webgl rendering pipeline. The spine-player does not expose direct hooks for modifying the WebGL context in the way you are attempting. However, you can achieve this by using the spine-webgl runtime directly, which provides more control over the rendering process.

Here is an example of how you can integrate a custom shader with spine-webgl:

Set up the basic rendering pipeline using spine-webgl:

import { SpineCanvas, AssetManager, SkeletonRenderer, PolygonBatcher, Shader } from '@esotericsoftware/spine-webgl';

// Create the canvas and WebGL context
const canvas = document.getElementById('canvas');
const gl = canvas.getContext('webgl');

// Load the skeleton and atlas
const assetManager = new AssetManager(gl);
assetManager.loadText('path/to/your/skeleton.json');
assetManager.loadTextureAtlas('path/to/your/atlas.atlas');

assetManager.loadAll().then(() => {
    const atlas = assetManager.get('path/to/your/atlas.atlas');
    const skeletonJson = assetManager.get('path/to/your/skeleton.json');
    const skeletonData = new SkeletonJson(new AtlasAttachmentLoader(atlas)).readSkeletonData(skeletonJson);

    const skeleton = new Skeleton(skeletonData);
    const animationState = new AnimationState(new AnimationStateData(skeletonData));

    // Set up the custom shader
    const vertexShaderSource = `
        attribute vec4 a_position;
        void main() {
            gl_Position = a_position;
        }
    `;
    const fragmentShaderSource = `
        void main() {
            gl_FragColor = vec4(0.0, 0.0, 0.0, 1.0); // all black for testing
        }
    `;
    const shader = new Shader(gl, vertexShaderSource, fragmentShaderSource);

    // Set up the renderer
    const batcher = new PolygonBatcher(gl);
    const skeletonRenderer = new SkeletonRenderer(gl);
    skeletonRenderer.premultipliedAlpha = true;

    // Render loop
    function render() {
        gl.clear(gl.COLOR_BUFFER_BIT);
        shader.bind();
        batcher.begin(shader);
        skeletonRenderer.draw(batcher, skeleton);
        batcher.end();
        requestAnimationFrame(render);
    }

    render();
});

Integrate this with your existing setup:

You can replace the SpinePlayer initialization with the above code, ensuring you load the same skeleton and atlas files. This approach gives you full control over the WebGL context and allows you to use custom shaders as needed.

Handling multiple skeletons:

If you need to render multiple skeletons, you can reuse the same WebGL context, shaders, and batcher. Just ensure you update the skeleton and animation state for each render call.

By using spine-webgl directly, you can achieve the desired customization with your shaders while maintaining the flexibility to manage the rendering pipeline as needed.

MattSalsa

Davide Thanks for looking! We do a bit more, the tint approach isn't quite there.

This is roughly what our shader does:

!!ps
#define _ForegroundCutoff (1.0f)
#define _ShadowCutoff (0.7f)
#define _OutlineCutoff (0.21f)
#define _HueThreshold (0.03f)
#define _SatThreshold (0.03f)
#define _RigCount (1)

float3 rgb2hsv(float3 c) {
    float4 K = float4(0.0, -1.0 / 3.0, 2.0 / 3.0, -1.0);
    float4 p = c.g < c.b ? float4(c.bg, K.wz) : float4(c.gb, K.xy);
    float4 q = c.r < p.x ? float4(p.xyw, c.r) : float4(c.r, p.yzx);

    float d = q.x - min(q.w, q.y);
    float e = 1.0e-10;
    return float3(abs(q.z + (q.w - q.y) / (6.0 * d + e)), d / (q.x + e), q.x);
}

// We _only_ replace the designated fuchsia color, regardless of Brightness value
bool is_fuchsia(float3 c1, float3 c2) {
    if (abs(c1.x - c2.x) > _HueThreshold) return false;
    if (abs(c1.y - c2.y) > _SatThreshold) return false;
    return true;
}

// -> We have cutoff values for FG/Shadow/Outline, and the is just used to normalize between those values.
float normalize_value(float value, float max, float min) {
    return (value - min) / (max - min);
}

float4 color_replace(float4 color) {
    float4 result = color;

    // Convert the input color to hsv so we can do comparisons on the hue value.
    float3 hsvColor = rgb2hsv(color.rgb);

    int width;
    int height;
    color_replace_texture.GetDimensions(width, height);

    // We -1 the height here because the below code uses it to index into the texture
    // Otherwise each call to height in the loop below would need an additional -1
    height -= 1;

    // It would be nice to do a general optimization pass of this shader. I'm sure there is a lot of room for improvement
    for (int i = 0; i < width && i < _RigCount; i++) {
        float3 refHsvColor = rgb2hsv(color_replace_texture.Load(int3(i, height, 0)).rgb);
        if (is_fuchsia(hsvColor, refHsvColor)) {
            if (hsvColor.z >= _ForegroundCutoff) {
                result.rgb = color_replace_texture.Load(int3(i, height - 1, 0)).rgb;
            }
            else if (hsvColor.z >= _ShadowCutoff) {
                // Color foreground/shadow blend
                float normalizedBlendAmount = normalize_value(hsvColor.z, _ForegroundCutoff, _ShadowCutoff);
                result.rgb = lerp(color_replace_texture.Load(int3(i, height - 2, 0)), color_replace_texture.Load(int3(i, height - 1, 0)), normalizedBlendAmount).rgb;
            }
            else if (hsvColor.z >= _OutlineCutoff) {
                // Color is shadow/outline blend
                float normalizedBlendAmount = normalize_value(hsvColor.z, _ShadowCutoff, _OutlineCutoff);
                result.rgb = lerp(color_replace_texture.Load(int3(i, height - 3, 0)), color_replace_texture.Load(int3(i, height - 2, 0)), normalizedBlendAmount).rgb;
            }
            else {
                // Color is pure outline
                result.rgb = color_replace_texture.Load(int3(i, 0, 0)).rgb;
            }
        }
    }

    return result;
}

float4 ps_main(float4 color: COLOR0, float2 _uv: UV, float2 shadow_uv: TEXCOORD0): SV_Target0 {
    float4 result = albedo.Sample(albedo_smp, _uv);
    result = color_replace(result) * color;

    if (mask_in_shadow != 0.0f) {
        float2 suv = float2(shadow_uv.x, 1 - shadow_uv.y);
        result.a *= shadow.Sample(shadow_smp, suv).a;
    }
    result.rgb *= result.a;

    return result;
}

Davide

MattSalsa

I expected the color change to be easier. That is way more complicated than I expected.
Have you tried monkey patching the shader? Doing that you can modify the shader using the already available attributes/uniforms.

Anyway, I expect the easier way to test it is by cloning the repo and directly modifying the newTwoColoredTextured shader.

Once you are satisfied, you can do a custom build or use the monkey patching approach mentioned above.

MattSalsa

Davide Thanks! Adding our shader there worked like a charm! Of course, it would be nice to not maintain a fork, so native support for customer shaders down the line would be amazing, but either way I appreciate the support.

Davide

MattSalsa

I've quickly looked into it, and allowing a custom shader should only require a couple of lines.

Before thinking to implement that, I need to know if you've added any custom attributes or uniforms to your shader, or if your modifications are based on the current attributes and uniforms.

The easiest thing would probably be for you to paste your custom shader here so that I can directly try what I have in mind. 🙂