Loop 2: Build Failure Investigation in Minutes, Not Hours

This is Part 4 of the “Agentic Game Dev” series. Part 1: The Manifesto → Part 2: The Framework → Part 3: Asset Validation → Part 4: Build Failure Investigation → Part 5: Pipeline Insights → Part 6: Dogfooding Our Perforce Agent → Part 7: Butter Up! Devlog #1 → Part 8: Butter Up! Devlog #2.

A studio I worked with had what they called “the build guy.” His name doesn’t matter. His role was that he was the only person who understood the Jenkins configuration. When a build failed, he debugged it. When a Jenkins agent went down, he fixed it. When someone needed to add a new platform target, he did it.

He went on paternity leave. The studio couldn’t ship a build for three weeks.

Three weeks. Not because the code was broken. Because nobody else could read the Jenkins logs fast enough to know what to do.

I’ve told this story before. It comes up in every conversation about CI/CD with a small studio, and it always lands the same way. Nervous laughter, then someone in the room gets quiet and admits they have their own build guy. Sometimes their own build guy is them.

The build failure investigation loop is the one that gives that knowledge back to the team. Not by replacing the build guy. By making the first hour of debugging legible to everyone else.

What “Investigation” Actually Means

When a Jenkins build fails, an experienced engineer does roughly the following:

1. Reads the last few hundred lines of the log to find the actual error.
2. Categorizes the error type: compile, linker, asset cook, shader, test, dependency, infrastructure.
3. Looks at recent changelists to identify likely suspects.
4. Cross-references with Jira to see what the changelists were trying to do.
5. Checks if this same failure has happened before.
6. Forms a hypothesis.
7. Either fixes it or reaches out to whoever owns the affected code.

Steps 1 through 5 are mechanical. They take an experienced engineer fifteen to thirty minutes. They take a less-experienced engineer an hour or more. Step 6 is judgment. Step 7 is action.

The loop’s job is steps 1 through 5. Not 6, not 7.

What the Agent Reads

The minimum viable inputs:

• The build log. Not all of it. Agents have context windows, and Unreal cook logs can run to gigabytes. The relevant slice: the last 500 to 2000 lines, plus any matched error patterns earlier in the log.
• The triggering changelist. Which CL kicked the build off, the file list, and the description.
• Recent build history. Did this same build configuration succeed yesterday? Has this error type appeared before in this project? Is there a known fix?
• The Jenkins job configuration. What platform, what build type, what stage failed.

If the agent can read across the seams from Part 2 (Perforce, Jenkins, Jira), it can also pull:

• The Jira ticket the CL references. What was this work trying to do? What’s the acceptance criteria?
• Related CLs. What else has touched the affected modules in the last week?
• The author’s recent activity. Is this their first commit to this module, or do they own it?

That last bullet is sensitive. The agent isn’t grading the engineer. It’s just answering the question an experienced lead would answer in their head: who do I ask about this?

What the Agent Outputs

The shape of the output matters more than the contents.

A good failure summary looks something like this:

Build #4831 failed at the Cook stage on PS5 Shipping.

Likely category: asset cook failure, missing texture reference.

Probable triggering CL: 47892 by alex@studio (12 minutes before failure). Modified Content/UI/Inventory/T_Slot_Common.png and 3 related assets.

Evidence:

• Cook log: LogTextureCompressor: Error: Missing source for /Game/UI/Inventory/T_Slot_Common
• CL 47892 deleted T_Slot_Common.png (renamed to T_Slot_Default.png)
• 14 levels reference T_Slot_Common

Same failure history: No prior occurrence in this project.

Linked task: PROJ-892, “Rename inventory slot textures for consistency”

The engineer reads that and starts at the answer. They confirm the diagnosis, fix the references, and move on. Forty minutes saved. Maybe more.

What’s not in there is also important. The agent didn’t say “fix the references.” It didn’t open a PR with a proposed fix. It didn’t auto-revert the CL. It read, classified, correlated, and surfaced. The engineer still owns the work.

The Categorization Taxonomy

We documented categories in Part 4 of the pipeline series:

• compile_error: code compilation failures
• linker_error: missing symbols, library issues
• asset_cook_failure: asset cooking or processing failures
• shader_error: shader compilation failures
• test_failure: automated test failures
• timeout: build exceeded time limits
• oom: out of memory during build
• dependency_issue: missing or incompatible dependencies
• config_error: configuration or setup issues

The taxonomy is the agent’s vocabulary. Everything it surfaces gets one of these labels. Categorization isn’t sophisticated AI. It’s pattern matching against known error signatures, with an LLM filling the gaps where signatures don’t catch the full picture.

The value of categorization isn’t the label itself. It’s what the label enables: routing, history, and learning.

• Routing. A shader_error goes to the rendering team’s Slack channel. A dependency_issue goes to the build engineer. A test_failure goes to whoever owns the test.
• History. “We’ve seen 14 asset_cook_failure events in the last 30 days, mostly on PS5, mostly tied to texture renames” is a sentence your TD wants in their weekly review.
• Learning. Once the agent has seen a pattern enough times (“shader errors in this project are usually caused by missing permutation defines”), it can add that to the next surfacing as a probable cause.

Why Humans Are Bad at This and Agents Aren’t

Reading build logs is a skill. It atrophies when you don’t use it. New engineers have it least. The build guy has it most. Distributing this skill is the loop’s whole point.

There are a few reasons humans struggle here that agents don’t.

Volume. A full Unreal cook log can be larger than the codebase. No human reads all of it. Experienced engineers know which sections to skim and which to focus on. Less-experienced engineers don’t, and they either miss the error or grep for the wrong terms.

Cross-referencing. The error is in the build log. The CL is in Perforce. The ticket is in Jira. The recent author activity is in your task system. A human assembles this manually. Six tabs, ten minutes. The agent reads from one query.

Pattern memory. “This is the same failure as last Tuesday” is a sentence the engineer who saw last Tuesday’s failure can say. Anyone else can’t. The agent’s pattern memory is project-wide, not engineer-specific.

Time pressure. The engineer hit by a build failure is often mid-task on something else. Context switching costs them ten minutes before they even start reading. The agent has no other tasks.

The agent’s job is the assembly. The engineer’s job is the diagnosis.

Where the Agent Will Be Wrong

Be honest about this. The agent will be wrong.

Wrong attribution. CL 47892 modified files that look related to the failure, but the actual cause is CL 47885 from two hours earlier. An obscure dependency change that took two builds to manifest. The agent’s confidence on attribution should be visible. “Most likely CL” is honest. “CL X caused this” is not.

Plausible-but-wrong category. A test failure caused by a memory leak might get categorized as test_failure when the underlying issue is oom. The agent should show evidence for the category, and the engineer should correct the agent when it’s wrong. Over time, the corrections feed back into pattern improvement.

Missed context. The agent doesn’t know that yesterday’s submit was a temporary checkpoint with the real fix coming in a follow-up CL. Or that the build server’s disk is at 99% and that’s the actual cause of the cook failure. Human context wins. The agent surfaces what it can read.

The way to stay safe is the same as Part 3: surface evidence, not verdicts. A summary that says “the cook log contains this error, CL X modified files matching the pattern, prior failures of this type were caused by Y” is robust to being wrong because the engineer can see the reasoning. A summary that says “CL X caused this, fix the references” is fragile and will mislead.

What This Loop Could Enable

The basic loop saves an hour per build failure. Multiply by the number of failures per week. That’s the floor.

The aspirational version is more interesting:

• Distributed expertise. Every engineer reads agent summaries the same way the build guy reads Jenkins logs. The studio’s worst-case becomes “the build guy is on vacation,” not “the studio can’t ship for three weeks.”
• Pattern libraries that compound. After a year of use, the agent has seen most of the failure modes specific to your project: your engine version, your build farm, your dependency tree. New engineers benefit from accumulated pattern memory immediately, not after they’ve debugged each failure type once.
• Cross-project learning. Engineers who move between projects don’t have to relearn each project’s quirks. The agent’s pattern library does the carrying.
• Failure trend reporting. “We had 23 dependency_issue failures this sprint, 18 of them caused by the same upstream library version” is a planning input. Without categorization, this report is “the build broke a lot.”

None of these require sophisticated AI. They require reading across the seams (the same Part 2 idea) and consistent categorization. The agent is the smallest piece. The connective tissue is what makes it work.

What the Agent Should Never Do

Never auto-merge a fix. Even if the fix looks obvious. Even if the test suite passes. The Auto-merge level of the trust spectrum is not for game runtime code.

Never rerun a Shipping build without approval. Reruns cost compute. They also create artifact ambiguity. Which build is the “real” one? Reruns should be a human decision.

Never edit Jenkins configuration. The build guy’s configuration is sacred until it’s reviewed and committed by a human. Drift in CI configuration is one of the most painful debugging scenarios.

Never close a Jira ticket. “The agent thinks this is fixed” isn’t the same as “this is fixed.” Tickets stay open until a human verifies.

Never delete or modify build artifacts. Cleanup is a separate, scheduled job. Conflating cleanup with investigation is how you lose the evidence that would have helped you debug.

Where ButterStack Fits

ButterStack ships both halves of this loop. The fast single-shot analysis categorizes failures by the taxonomy above and attributes them to the likely triggering changelist. The agentic Build Investigator does the rest of what this post describes: it reads the changelists since the last green build, pulls diffs and file contents, matches the error signature against your account’s prior failures and their recorded fixes, and files a finding with cited evidence. It shipped at launch. The connective tissue across Perforce, Jenkins, and Jira is what makes it possible.

If your studio has a build guy (or if your studio is a build guy), the loop in this post is a sketch of what it would look like to share that knowledge across your team. I’d genuinely like to hear how you handle build failures today, and where the next hour you’d like back is hiding.

Get started with ButterStack

Where to Start

You don’t need an agent to start. The taxonomy alone is useful. Pick a label set (the nine categories from this post are a fine starting place) and start labeling failures by hand. Track which categories happen most. After a month, you’ll know which patterns are worth automating first.

When you do build the loop, start at Draft level. The agent posts summaries to a Slack channel. Engineers read them. Engineers correct them when they’re wrong. Don’t auto-route to individuals until the agent’s accuracy is high enough that you trust it not to wake up the wrong person at 2 AM.

Part 5 covers the insights loop. The publisher report, the ownership map, the trend question. Same pattern as this one: the agent does the assembly. The human reads the answer.

Thanks!
Ryan L’Italien
Founder and CEO of ButterStack

Want to see what pipeline observability looks like? Try ButterStack free and connect your first integration in minutes.

Or just email me at: ryan@butterstack.com.