Overlapping Alphas in Unity 3D

In my last blog post, I unveiled the pheromone system and it’s accompanying overlay as a core game mechanic. The overlay was basic at best, and had one really irritating flaw – where two node’s pheromone indicators overlapped, the transparency is doubled. At best, this is pretty ugly, and at worst, it completely obscures the terrain behind it and ruins the data the overlay is trying to show altogether.

Note the hideous overlaps…

I searched high and low over the internet looking for an answer, but couldn’t find any decent solution to the problem. That is, until I posted on reddit asking for help, and a genius who goes by the name ‘Wompipomp’ came to my rescue. With his help, I came to a solution I’m pretty happy with.

Introducing the Stencil Buffer

The solution was to create a custom shader based on the standard Unity sprite shader which makes use of the stencil buffer to conditionally render pixels depending on whether anything has already been drawn there.

The stencil buffer is an additional data buffer available to shaders during the rendering process, consisting of one 8-bit integer per pixel. By adding data to this as we render each circle and then using that data when rendering others, we can ensure that we draw to each pixel at most once.

Let’s look at it in some more detail.

The original problem

We have two overlapping circles. We want to render both of them, but without the overlap in the middle, and we still want to see the terrain behind it.

Rendering the first circle

When we render the first circle, the stencil buffer is filled with zero values. For each pixel, we test that a reference value of 1 is greater than the value in the buffer (still 0). If this test passes, then we increment the value in the buffer to mark that we are going to draw to this pixel, and we do so. The ShaderLab code for this stencil buffer behaviour is as follows:

Stencil {
    Ref 1 // The reference value to test
    Comp Greater // Is the Ref greater than the value in the buffer for this pixel?
    Pass IncrSat // If so, then increment the value in the buffer

Rendering the second circle

Now we come to render the second circle, and the stencil buffer contains the data we added when we rendered the first. We do exactly the same test (it’s the same shader after all) – is the reference value of 1 greater than the value in the buffer? Well, for most of the circle, that test still passes, so we increment the value in the buffer and draw a pixel. However, for areas that are occupied by the other circle, the test fails. When we fail the stencil test, we skip rendering that pixel altogether, resulting in the shape we see above.

The result

As we can see, the result will be that every pixel will be drawn to at most once. We can see the background, both circles are the correct colour and alpha level, but there’s no nasty overlap. In game it looks pretty good. We get a nice gradient across all the pheromone circles as the values rise and fall.


The full shader code to achieve this effect is shown below.

Shader "Custom/Sprites/OverlayStencil"
        [PerRendererData] _MainTex("Sprite Texture", 2D) = "white" {}
        _Color("Tint", Color) = (1, 1, 1, 1)
        [MaterialToggle] PixelSnap("Pixel snap", Float) = 0
            "Queue" = "Transparent"
            "IgnoreProjector" = "True"
            "RenderType" = "Transparent"
            "PreviewType" = "Plane"
            "CanUseSpriteAtlas" = "True"
        Cull Off
        Lighting Off
        ZWrite Off
            Mode Off
        Blend One OneMinusSrcAlpha
                Ref 1
                Comp Greater
                Pass IncrSat
#pragma vertex vert
#pragma fragment frag
#pragma multi_compile DUMMY PIXELSNAP_ON
#include "UnityCG.cginc"

            struct appdata_t
                float4 vertex : POSITION;
                float4 color : COLOR;
                float2 texcoord : TEXCOORD0;
            struct v2f
                float4 vertex : SV_POSITION;
                fixed4 color : COLOR;
                half2 texcoord : TEXCOORD0;
            fixed4 _Color;
            v2f vert(appdata_t IN)
                v2f OUT;
                OUT.vertex = mul(UNITY_MATRIX_MVP, IN.vertex);
                OUT.texcoord = IN.texcoord;
                OUT.color = IN.color * _Color;
                OUT.vertex = UnityPixelSnap(OUT.vertex);
                return OUT;
            sampler2D _MainTex;
            fixed4 frag(v2f IN) : SV_Target
                fixed4 c = tex2D(_MainTex, IN.texcoord) * IN.color;
                c.rgb *= c.a;
                return c;