Unity Shader Gotcha #8 - Custom Deferred Lighting

Background
This is not well-known, but Unity provides a way for you to specify a custom deferred lighting shader. All you have to do is download Unity's internal shaders collection, grab "Internal-PrePassLighting.shader", import it into your Unity project, then close and restart the editor. After this, Unity will now be using the version referenced in your project.

From here, you can modify the lighting equation inside the CalculateLight() function to change how it calculates the lighting contributions.

Problems
You are still limited to having the same input and output limitations as Unity's built-in deferred lighting shader. So that means you only have (normal XYZ, specular) in the range [0,1] as inputs, and (diffuse RGB, specular) as outputs. More importantly, this shader's output is shared by all material shaders. So changing it will make all of their outputs wrong, forcing you to have to write a complete set of replacement shaders that can work with your new custom deferred lighting results.

So this is not something for the faint of heart.

Benefits
Obviously the biggest benefit is being able to use a custom lighting function for deferred lighting. You can also change how the specular power input is interpreted, possibly packing more data into it or simply changing the mapping. Finally, you can change how the specular lighting is calculated so you can potentially use a better specular light color approximation in the material shaders. At the very least, you will probably want to fix the inadequacies of Unity's default lighting equation.

So if you are in any way serious about getting good lighting and materials out of Unity, this is the only way to go.

Unity Shader Gotcha #7 - World Space & Detail Textures

Sometimes you want to use world-space vectors for things like normal mapped light probe sampling and cube map reflections. The problem is they eat up a lot of channels for passing data:

struct Input {
    float2 uv_MainTex;
    float3 viewDir;
    float3 worldRefl;
    float3 worldNormal;
    INTERNAL_DATA
};

As a result, it is now impossible on pre-SM4.0 targets to add any more channels if you want to do anything that involves multiple texture coordinates or custom vertex data (ex. detail textures, terrain). You will get an error if you try to do something like:

struct Input {
    float2 uv_MainTex;
    float2 uv2_Detail;
    float3 viewDir;
    float3 worldRefl;
    float3 worldNormal;
    INTERNAL_DATA
};

The closest thing you can do is add an extra uniform parameter to the shader to do texture transformations, and that only really helps with detail mapping. So while this feature is great for making cool per-pixel lighting effects, it comes with a steep cost in terms of flexibility. :(

Unity Shader Gotcha #6 - Lighting 2.0

For whatever reason, Unity requires that we apply a scale of 2 to lighting in some places and not others. It is implicit in the _PrePass and _DirLightmap function inputs. However, in the forward custom lighting function and whenever you use ShadeSH9() you must explicitly multiply them by 2.

This is especially important for when you handle ShadeSH9() yourself, since this is not well-explained in the documentation. Your dynamic objects would end up looking much darker than your lightmapped objects with no clear explanation as to why.

Unity Shader Gotchas #5 - DirLightMap & Defered

Background
For those who don't know, when you make a custom lighting function for your shaders you can specify how it handles Unity's directional lightmaps. This is mostly to allow you to pick how it generates specular lighting using said lightmaps.

The Problem
Unity decided to handle output of this information in a way that is awkward and causes the results to differ between forward and deferred mode. In forward mode, it gets diffuse lighting by the return value, and the specular lighting by an out parameter.

Example:
float4 LightingCustom_DirLightmap (SurfaceOutput s, float4 color, float4 scale, float3 viewDir, bool surfFuncWritesNormal, out float3 specColor) {
  // Lighting code

    specColor = specularLighting;
    return float4(diffuseLighting, 0);
}

In deferred mode, it is handled by passing the specular lighting as a single component value in the return value's alpha channel.

Example:
float4 LightingCustom_DirLightmap (SurfaceOutput s, float4 color, float4 scale, float3 viewDir, bool surfFuncWritesNormal, out float3 specColor) {
  // Lighting code

    return float4(diffuseLighting, luminance(specularLighting));
}

Worse yet, in deferred mode the results of the _DirLightmap function get fed into the _PrePass function with the accumulated deferred lighting. So you are restricted in what you can do in the _PrePass function in terms of custom lighting. Even more than that, it will cause all your dynamic light specular highlights to be heavily shifted toward the dominant lightmap color as it will pollute the dynamic light specular color approximation.

The Solution
This is a hack I discovered while inspecting the results of the shader after applying the "#pragma debug" directive. Essentially, you can put an "inout" qualifier on the "SurfaceOutput s" parameter, allowing you to modify its contents. Then you just apply all your lighting output by adding it to the Emission field of "s". Finally, you can safely zero out all the other outputs so they don't apply the lighting twice.

Example:
float4 LightingCustom_DirLightmap (inout SurfaceOutput s, float4 color, float4 scale, float3 viewDir, bool surfFuncWritesNormal, out float3 specColor) {
  // Lighting code

  s.Emission += s.Albedo * diffuseLighting + s.Gloss * specularLighting.
  specColor = 0;
    return 0;
}

This allows you to have consistent results between forward and deferred mode, improves the quality of the specular light approximation in deferred mode, and allows you more flexibility in your _PrePass function.

Unity Shader Gotchas #4 - Custom Alpha Blending

Be careful when using both custom alpha blending (eg. premultiplied alpha) and a custom lighting function. You need to make sure to return the alpha in the custom lighting function or it will not be applied. Basically all that means is the following:

half4 LightingCustom (SurfaceOutput s, half3 lightDir, half atten) {
    half4 c;
    
    // Lighting Code
    
    // Don't forget this part!
    c.a = s.Alpha;
    return c;
}