CI/CD for Game Development

At present, the field of Game CI/CD is in its nascent stages, and there are numerous issues that need resolution. Our development process is complex, necessitating a robust procedure and deployment structure. This approach aims to resolve several challenges:

Developing collaboratively as a team.
Automating server builds and deployments.
Managing multiple environments such as development, staging, and production.

During my search for game CI solutions, I discovered Game CI, an excellent resource that typically works out of the box without customization. However, it did not fully meet our needs. This blog post outlines how I addressed the challenges encountered during our setup.

We maintain three branches on GitHub for our game: dev, staging, and production. The goal is to automate deployments to our servers upon any commits to these branches. To this end, I crafted a GitHub action in .github/workflows/main.yml, inspired by the GameCI configuration.

Initially, the action failed, providing no useful error messages—a common issue with Unity. To circumvent this, I implemented a custom Builder to output errors effectively.

///////////////////////////////////////////////////////////////////////////
/// Build menu.
///
/// Creates a "Wodo" menu in Unity with options to build the game,
/// switch build targets without building, and automatically manage the
/// build number according to SemVer standards (see <https://semver.org/>).
///
///////////////////////////////////////////////////////////////////////////
#if UNITY_EDITOR
using System;
using UnityEditor;
using UnityEditor.Build.Reporting;
using UnityEngine;
using System.Linq;

namespace UnityBuilderAction{
    public class BuildMenu
    {
        private static readonly string s_Eol = Environment.NewLine;
        /// <summary>
        /// Scenes to include in the build.
        /// </summary>
        public static string[] scenes = {
            "Assets/WodoMiniFPS/Scenes/DashboardWork/Dashboard.unity",
            "Assets/WodoMiniFPS/Scenes/Intro.unity",
            "Assets/WodoMiniFPS/Scenes/MainMenu.unity",
            "Assets/WodoMiniFPS/Scenes/World1.unity",
            "Assets/WodoMiniFPS/Scenes/NewMap.unity",
        };
    
        public static string ProductName = PlayerSettings.productName;
    
        ///////////////////////////////////////////////////////////////////////////
        /// Debug build menu handlers.
        ///////////////////////////////////////////////////////////////////////////
    
        /// <summary>
        /// Build all targets.
        /// </summary>
        [MenuItem("Wodo/Build Dev Package")]
        public static void BuildAll_DevPackage()
        {
            BuildLinuxDedicatedServer(true);
            BuildMacOSXClient(true);
            BuildWebGLClient(true);
            BuildWindowsClient(true);
            // BuildWindowsDedicatedServer(true);
            //BuildAndroidClient(true);
    
            //SetWindowsTarget();
    
            DoIncreaseBuild();
        }
    
        /// <summary>
        /// Build all targets.
        /// </summary>
        [MenuItem("Wodo/Build/All Targets")]
        public static void BuildAll_All()
        {
            Debug.Log("Bulding all Development targets");
            BuildAll(true);
    
            Debug.Log("Bulding all Release targets");
            BuildAll(false);
    
            DoIncreaseBuild();
        }
    
        /// <summary>
        /// Build all targets.
        /// </summary>
        [MenuItem("Wodo/Build/Debug/All Targets")]
        public static void BuildAll_Debug()
        {
            Debug.Log("Bulding all Development targets");
            BuildAll(true);
            DoIncreaseBuild();
        }
    
        /// <summary>
        /// Android Client target.
        /// </summary>
        [MenuItem("Wodo/Build/Debug/Android Client")]
        public static void BuildAndroid_Debug()
        {
            BuildAndroidClient(true);
            DoIncreaseBuild();
        }
    
        /// <summary>
        /// Windows Client target.
        /// </summary>
        [MenuItem("Wodo/Build/Debug/Windows Client")]
        public static void BuildWindows_Debug()
        {
            BuildWindowsClient(true);
            DoIncreaseBuild();
        }
    
        /// <summary>
        /// Linux Dedicated Server target.
        /// </summary>
        [MenuItem("Wodo/Build/Debug/Linux Dedicated Server")]
        public static void BuildLinuxDedicatedServer_Debug()
        {
            BuildLinuxDedicatedServer(true);
            DoIncreaseBuild();
        }
    
        /// <summary>
        /// Windows Dedicated Server target.
        /// </summary>
        [MenuItem("Wodo/Build/Debug/Windows Dedicated Server")]
        public static void BuildWindowsDedicatedServer_Debug()
        {
            BuildWindowsDedicatedServer(true);
            DoIncreaseBuild();
        }
    
        ///////////////////////////////////////////////////////////////////////////
        /// Release build menu handlers.
        ///////////////////////////////////////////////////////////////////////////
    
        /// <summary>
        /// Build all targets.
        /// </summary>
        [MenuItem("Wodo/Build/Release/All Targets")]
        public static void BuildAll_Release()
        {
            Debug.Log("Bulding all targets");
            BuildAll(false);
            DoIncreaseBuild();
        }
    
        /// <summary>
        /// Android Client target.
        /// </summary>
        [MenuItem("Wodo/Build/Release/Android Client")]
        public static void BuildAndroid_Release()
        {
            BuildAndroidClient(false);
            DoIncreaseBuild();
        }
    
        /// <summary>
        /// Windows Client target.
        /// </summary>
        [MenuItem("Wodo/Build/Release/Windows Client")]
        public static void BuildWindows_Release()
        {
            BuildWindowsClient(false);
            DoIncreaseBuild();
        }
    
        /// <summary>
        /// MacOSX Client target.
        /// </summary>
        [MenuItem("Wodo/Build/Release/MacOSX Client")]
        public static void BuildMacOSX_Release()
        {
            BuildMacOSXClient(false);
            DoIncreaseBuild();
        }
    
        /// <summary>
        /// WebGL Client target.
        /// </summary>
        [MenuItem("Wodo/Build/Release/WebGL Client")]
        public static void BuildWebGL_Release()
        {
            BuildWebGLClient(false);
            DoIncreaseBuild();
        }
    
        /// <summary>
        /// Linux Dedicated Server target.
        /// </summary>
        [MenuItem("Wodo/Build/Release/Linux Dedicated Server")]
        public static void BuildLinuxDedicatedServer_Release()
        {
            Console.WriteLine("Wodo build started here -----------");
            BuildLinuxDedicatedServer(false);
            DoIncreaseBuild();
        }
    
        /// <summary>
        /// Windows Dedicated Server target.
        /// </summary>
        [MenuItem("Wodo/Build/Release/Windows Dedicated Server")]
        public static void BuildWindowsDedicatedServer_Release()
        {
            BuildWindowsDedicatedServer(false);
            DoIncreaseBuild();
        }
    
        ///////////////////////////////////////////////////////////////////////////
        /// Build scripts.
        ///////////////////////////////////////////////////////////////////////////
    
        /// <summary>
        /// Build all targets.
        /// </summary>
        public static void BuildAll(bool inDebug)
        {
            Debug.Log("Bulding Android client");
            BuildAndroidClient(inDebug);
    
            Debug.Log("Bulding Windows client");
            BuildWindowsClient(inDebug);
    
            Debug.Log("Bulding Linux dedicated server");
            BuildLinuxDedicatedServer(inDebug);
    
            Debug.Log("Bulding Windows dedicated server");
            BuildWindowsDedicatedServer(inDebug);
        }
    
        /// <summary>
        /// Build the Android client.
        /// </summary>
        /// <param name="inDebug">True to build a Development (Debug) build.</param>
        public static void BuildAndroidClient(bool inDebug)
        {
            BuildOptions opts = BuildOptions.None;
            string loc;
    
            SetAndroidTarget();
    
            if (inDebug)
            {
                Debug.Log($"Building Android client, Debug");
                opts = BuildOptions.Development;
                loc = $"Build/autobuild/Debug/Android/{ProductName}.aab";
            }
            else
            {
                Debug.Log($"Building Android client, Release");
                loc = $"Build/autobuild/Release/Android/{ProductName}.aab";
            }
    
            /* We absolutely do not want to ever store secrets in code (or even add them to
            version control), so instead we'll fetch them from the system environment.
            Don't forget to set these environment variables before invoking the build script! */
            Debug.Log("Reading environment KEY_STORE_PASS");
            PlayerSettings.Android.keystorePass = Environment.GetEnvironmentVariable("KEY_STORE_PASS");
            Debug.Log("Reading environment KEY_ALIAS_PASS");
            PlayerSettings.Android.keyaliasPass = Environment.GetEnvironmentVariable("KEY_ALIAS_PASS");
    
            var buildPlayerOptions = new BuildPlayerOptions()
            {
                scenes = scenes,
                locationPathName = loc,
                target = BuildTarget.Android,
                subtarget = (int)StandaloneBuildSubtarget.Player,
                options = opts
            };
    
            BuildReport report = BuildPipeline.BuildPlayer(buildPlayerOptions);
            PostBuild(report);
        }
    
        /// <summary>
        /// Build the Windows client.
        /// </summary>
        /// <param name="inDebug">True to build a Development (Debug) build.</param>
        public static void BuildWindowsClient(bool inDebug)
        {
            BuildOptions opts = BuildOptions.None;
            string loc;
    
            SetWindowsTarget();
    
            if (inDebug)
            {
                Debug.Log($"Building Windows Client, Debug");
                opts = BuildOptions.Development;
                loc = $"Build/autobuild/Debug/Windows/{ProductName}.exe";
            }
            else
            {
                Debug.Log($"Building Windows Client, Release");
                loc = $"Build/autobuild/Release/Windows/{ProductName}.exe";
            }
    
            var buildPlayerOptions = new BuildPlayerOptions()
            {
                scenes = scenes,
                locationPathName = loc,
                target = BuildTarget.StandaloneWindows64,
                subtarget = (int)StandaloneBuildSubtarget.Player,
                options = opts
            };
    
            BuildReport report = BuildPipeline.BuildPlayer(buildPlayerOptions);
            PostBuild(report);
        }

        /// <summary>
        /// Build the MacOSX client.
        /// </summary>
        /// <param name="inDebug">True to build a Development (Debug) build.</param>
        public static void BuildMacOSXClient(bool inDebug)
        {
            BuildOptions opts = BuildOptions.None;
            string loc;
    
            SetMacOSXTarget();
    
            if (inDebug)
            {
                Debug.Log($"Building MacOSX Client, Debug");
                opts = BuildOptions.Development;
                loc = $"Build/autobuild/Debug/MacOSX/{ProductName}";
            }
            else
            {
                Debug.Log($"Building MacOSX Client, Release");
                loc = $"Build/autobuild/Release/MacOSX/{ProductName}";
            }
    
            var buildPlayerOptions = new BuildPlayerOptions()
            {
                scenes = scenes,
                locationPathName = loc,
                target = BuildTarget.StandaloneOSX,
                subtarget = (int)StandaloneBuildSubtarget.Player,
                options = opts
            };
    
            BuildReport report = BuildPipeline.BuildPlayer(buildPlayerOptions);
            PostBuild(report);
        }

        /// <summary>
        /// Build the WebGL client.
        /// </summary>
        /// <param name="inDebug">True to build a Development (Debug) build.</param>
        public static void BuildWebGLClient(bool inDebug)
        {
            BuildOptions opts = BuildOptions.None;
            string loc;
    
            SetWebGLTarget();
    
            if (inDebug)
            {
                Debug.Log($"Building WebGL Client, Debug");
                opts = BuildOptions.Development;
                loc = $"Build/autobuild/Debug/WebGL/{ProductName}";
            }
            else
            {
                Debug.Log($"Building WebGL Client, Release");
                loc = $"Build/autobuild/Release/WebGL/{ProductName}";
            }
    
            var buildPlayerOptions = new BuildPlayerOptions()
            {
                scenes = scenes,
                locationPathName = loc,
                target = BuildTarget.WebGL,
                subtarget = (int)StandaloneBuildSubtarget.Player,
                options = opts
            };
    
            BuildReport report = BuildPipeline.BuildPlayer(buildPlayerOptions);
            PostBuild(report);
        }
    
        /// <summary>
        /// Build Linux dedicated server.
        /// </summary>
        /// <param name="inDebug">True to build a Development (Debug) build.</param>
        public static void BuildLinuxDedicatedServer(bool inDebug)
        {
            BuildOptions opts = BuildOptions.None;
            string loc;
    
            // SetLinuxDSTarget();
    
            if (inDebug)
            {
                // Debug.Log($"Building Linux dedicated server, Debug");
                opts = BuildOptions.Development;
                loc = $"Build/autobuild/Debug/linuxds/BUILD_NAME.x86_64";
            }
            else
            {
                // Debug.Log($"Building Linux dedicated server, Release");
                Console.Write($"Building Linux dedicated server, Release");
                loc = $"Build/autobuild/Release/linuxds/BUILD_NAME.x86_64";
            }
    
            var buildPlayerOptions = new BuildPlayerOptions()
            {
                scenes = scenes,
                locationPathName = loc,
                target = BuildTarget.StandaloneLinux64,
                subtarget = (int)StandaloneBuildSubtarget.Server,
                options = opts
            };
            Console.WriteLine(Environment.NewLine);
            Console.Write("Build report start");
            BuildReport report = BuildPipeline.BuildPlayer(buildPlayerOptions);
            Console.WriteLine(Environment.NewLine);
            Console.Write("Build report end");
            Console.WriteLine(Environment.NewLine);
            PostBuild(report);
        }
    
        /// <summary>
        /// Build Windows dedicated server.
        /// </summary>
        /// <param name="inDebug">True to build a Development (Debug) build.</param>
        public static void BuildWindowsDedicatedServer(bool inDebug)
        {
            BuildOptions opts = BuildOptions.None;
            string loc;
    
            SetLinuxDSTarget();
    
            if (inDebug)
            {
                Console.Write($"Building Windows dedicated server, Debug");
                opts = BuildOptions.Development;
                loc = $"Build/autobuild/Debug/WindowsDS/{ProductName}.exe";
            }
            else
            {
                Console.Write($"Building Windows dedicated server, Release");
                loc = $"Build/autobuild/Release/WindowsDS/{ProductName}.exe";
            }
    
            var buildPlayerOptions = new BuildPlayerOptions()
            {
                scenes = scenes,
                locationPathName = loc,
                target = BuildTarget.StandaloneWindows,
                subtarget = (int)StandaloneBuildSubtarget.Server,
                options = opts
            };
    
            BuildReport report = BuildPipeline.BuildPlayer(buildPlayerOptions);
            PostBuild(report);
        }
    
        ///////////////////////////////////////////////////////////////////////////
        /// HELPERS
        ///////////////////////////////////////////////////////////////////////////
    
        /// <summary>
        /// Perform post-build steps and print build results.
        /// </summary>
        private static void PostBuild(BuildReport report)
        {
            BuildSummary summary = report.summary;
    
            if (summary.result == BuildResult.Succeeded)
            {
                Console.WriteLine(Environment.NewLine);
                Console.WriteLine("Build succeeded --------------------------------");
            }
    
            if (summary.result == BuildResult.Failed)
            {
                Console.WriteLine(Environment.NewLine);
                Console.WriteLine("Build failed -----------------------------------");
            }

            // This format is required by the game-ci build action
            Console.WriteLine(
                $"{Environment.NewLine}" +
                $"###########################{Environment.NewLine}" +
                $"#      Build results      #{Environment.NewLine}" +
                $"###########################{Environment.NewLine}" +
                $"{Environment.NewLine}" +
                $"Duration: {summary.totalTime.ToString()}{Environment.NewLine}" +
                $"Warnings: {summary.totalWarnings.ToString()}{Environment.NewLine}" +
                $"Errors: {summary.totalErrors.ToString()}{Environment.NewLine}" +
                $"Size: {summary.totalSize.ToString()} bytes{Environment.NewLine}");

            // Ignore steps that have no errors.
            foreach (var step in report.steps.Where(r => r.messages.Any(m => m.type is LogType.Error or LogType.Exception)))
            {
                Console.WriteLine(
                    $"{s_Eol}" +
                    $"#####################################{s_Eol}" +
                    $"#      Step {step.name} Results{s_Eol}" +
                    $"#{s_Eol}" +
                    $"# Duration: {step.duration.ToString()}{s_Eol}" +
                    $"# Errors or Exceptions:{s_Eol}" +
                    "#"
                );

                foreach (var msg in step.messages.Where(msg => msg.type is LogType.Error or LogType.Exception))
                {
                    Console.WriteLine($"#   [{msg.type.ToString().ToUpper()}] {msg.content}");
                }

                Console.WriteLine($"#{s_Eol}#####################################{s_Eol}");
            }
            
        }
    
    }
}
#endif

Within main.yml, I modified the buildMethod to reference our new custom Builder:

buildMethod: UnityBuilderAction.BuildMenu.BuildLinuxDedicatedServer_Release

Fortuitously, this change began to yield clear error messages, such as:

[ERROR] Failed to find a suitable OpenCL device for the GPU Lightmapper. Please change your lightmapper or install the latest graphics driver.
[ERROR] Falling back to CPU lightmapper.

The issue stems from Unity’s inability to detect an appropriate GPU within a Docker container. I attempted to utilize the NVIDIA runtime as detailed here, to no avail. Ultimately, I resorted to forking the Unity Builder and creating a custom Docker image—a solution I had hoped to avoid.

The fork can be viewed here: wodo-platform/[email protected]

And the Docker image here: ghcr.io/bekirdag/unity-editor-custom:v3.0.1

The Builder Changes

I modified index.js to include a method called getLinuxCommand, responsible for running the Docker command with additional variables:

--gpus all \
-v /tmp/.X11-unix:/tmp/.X11-unix \
-e DISPLAY=$DISPLAY \

A new Docker image was created, based on `unityci/editor``:

# Use the existing Docker image as the base
FROM unityci/editor:ubuntu-2022.3.8f1-linux-il2cpp-3.0.0

RUN apt-get update && apt-get -y upgrade \
    && apt-get install -y \
    apt-utils \
    unzip \
    tar \
    curl \
    xz-utils \
    ocl-icd-libopencl1 \
    opencl-headers \
    clinfo \
    nvidia-opencl-dev \
    ;

RUN mkdir -p /etc/OpenCL/vendors && \
    echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd

ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility

This custom image is then referenced in the .yml file:

customImage: 'ghcr.io/bekirdag/unity-editor-custom:v3.0.1'

Finally, the configuration for the headless server build is as follows:

# Build
- name: Build headless server
  uses: wodo-platform/[email protected]
  env:
    UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
    UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
    UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}
  with:
    customImage: 'ghcr.io/bekirdag/unity-editor-custom:v3.0.1'
    targetPlatform: StandaloneLinux64 # Build a macOS standalone (Intel 64-bit).
    customParameters: -standaloneBuildSubtarget Server
    buildMethod: UnityBuilderAction.BuildMenu.BuildLinuxDedicatedServer_Release

Optionally, upload your build artifact to your server:

# Upload unix dev server
- name: change directory to build
  run: |
    cd /full_path/_work/metal-impact/metal-impact/Build/autobuild/Release/
    rm -rf linuxds.tar
    tar -zcvf linuxds.tar linuxds
    ssh user@IP_ADDRESS 'rm -rf /FULL_PATH/linuxds.tar && rm -rf /FULL_PATH/linuxds'
    scp /FULL_PATH/metal-impact/metal-impact/Build/autobuild/Release/linuxds.tar USER@IP_ADDRESS:/full_path/linuxds.tar
    ssh user@IP_ADDRESS 'cd /FULL_PATH && tar -xvf linuxds.tar && cd linuxds && chmod 755 build_name.x86_64'

You may also want to configure your .yml to save builds for Mac and Windows as artifacts, allowing your testers to access the development builds without needing to request them from developers.

Bekir Dağ

CI CD for Game Development

CI/CD for Game Development

The Builder Changes