The reason why I went with an FSM for my game, the problems that came with it, and how I managed to resolve them without switching approaches altogether.

Context

There are many ways to approach enemy AI in video games. Early video games could get away with simple if/else or switch statements to determine what the enemy / npc should be doing at any given moment, but as the demand increased for smarter, more challenging enemies, managing their state became increasingly more difficult.

For many years, the default choice to address this was Finite State Machines (FSMs). This allowed behavior to be represented as a series of discrete states where each state could have one or more transitions to other states. However, as the demand for smart agents continued to increase, the number of states and transitions between them exploded. Hierarchical Finite State Machines resolved this quite well for a while, and in fact, have been used in more recent games including DOOM 2016 and DOOM Eternal, but I think it’s safe to say this wasn’t your standard HFSM setup.

By far the industry standard for many years now has been Behavior Trees. While there are plenty of other solutions, such as Goal-Oriented Action Planning, Utility AI, etc. behavior trees are well established and allow game designers to build complex behavior through visual interfaces. The thing is though, for many games, especially indie titles, these systems are akin to hanging a picture using a sledgehammer.

Why FSMs May Be The Right Choice For Indie Games

Death’s Door is a great game, but you’re never going to see an NPC decide it needs to stock up on ammo, navigate to the nearest ammo supply shop, bring the ammo back home, decide it’s now hungry, go and hunt rabbits... you get my point.

Enemies in many games don’t need to be hyper intelligent to make a fun experience. In fact, many enemies only really need:

1. Idle / patrol -> chase.
2. Chase -> attack.
3. React to interrupts such as hits / stuns.
4. Search if they lose sight of the target.
5. Die.

Game developers leverage a large bag of tricks to make enemies appear smarter than they actually are. Even simply increasing their HP can make players perceiving an enemy to be smarter. Animations also play a huge role in selling this illusion, and in fact, many enemies may actually use the exact same underlying logic, but appear to behave differently because of their unique animations.

A small indie game could easily get away with extremely simple enemy behavior, especially if combat isn’t a major part of the game. The issue with a basic FSM however becomes apparent the moment you find yourself creating a new enemy type that has different behavior. The problem is due to the fact that state transitions are tightly coupled with the state machine. What works for a simple melee enemy, may not work for a ranged enemy, and will almost certainly not work for a boss.

Now, games are frequently known for not exactly being examples of high quality codebases, and nowadays with AI, you can generate multiple enemy types in a few minutes. However, regardless of whether you’re hand-crafting every line of code yourself or prompting an LLM to generate it, traditional state machines still break down the moment you try to share logic across different enemy types. The problem here is that now you have a dozen different places to make changes or multiple paths that may all share the same bugs. Reusability is essential to actually finishing a project.

So how could we resolve these issues?

Thin States

First of all, make your states as thin as possible. A state should execute some action, but not know how or why. The more assumptions your states make, the less reusable they are. Instead, logic should be split out into controllers.

Imagine you have a MeleeAttackState. It’s deciding which attack to use, determining when the animation finishes, enabling and disabling hit boxes, triggering vfx and sound fx.. Even if the available attacks are data-driven, you’re still going to end up creating a RangedAttackState and then a ChargeAttackState, a DiveBombAttackState, etc. Instead, what if all you needed was a single AttackState for all enemies?

This can be done by creating an AttackController. You can make this an abstract base class or create an IAttackController interface. Your attack state then simply has to relay events such as:

• StateEntered
• StateExited
• AttackCanceled
• AttackPhaseChanged (animation events).

Your attack controller can then listen for these events and perform whatever logic it needs without the attack state ever knowing the implementation details.

Zero Transitions

A typical FSM setup may look something like this:

fsm.AddState(Idle);
fsm.AddState(Chase);
fsm.AddState(Attack);
...

fsm.AddTransition(Idle, Chase, () => HasTarget);
fsm.AddTransition(Idle, Death, () => stats.Health <= 0);
fsm.AddTransition(Idle, Attack, () => HasTarget && InAttackRange);

// repeat for every state

This is exactly where the coupling is created. Each state has to know about each other state through explicit transitions. However, most FSMs have a convenient method available to you:

fsm.ChangeState(Death, true) // true to force immediate change

By removing the transitions and directly deciding when to change states, you remove all decision making from the state machine, making the state machine a convenient way to organize your code.

The Brain

If state machines don’t own the decision making through transitions, then something else needs to make those decisions instead. This is where the “brain” comes in.

First off, the brain can be created as a completely separate component external to your enemy class. In fact, by doing so, your enemy class can be reused across every enemy without ever needing to create a sub-class of it. As mentioned earlier, by separating logic from your states into controllers, you can swap out which controllers are used for which enemies. Your enemy class then doesn’t care if you select a MeleeAttackController or a RangedAttackController.

The brain becomes your central hub for decision making. You can create an abstract EnemyBrain if you want (which is often more convenient in Unity over interfaces) that can contain the state machine instance and even add the core states such as Idle, Chase, Death, Stunned, etc. (basically anything that will always be used by all enemies). However, the base brain’s job is to take care of the standard initialization, but makes very few to zero decisions itself.

Event-Driven Brain

In a typical FSM, each transition for the current state is constantly checked each frame to see if the conditions are met. However, since we’ve removed all of these transitions in order to make our code reusable, something has to drive state changes. That’s where events shine here.

Each brain, whether an archetype brain or a specific boss-brain, acts as the decision maker and orchestrator using events as decision making points. For example, our health system says “the enemy just died”, the brain can decide “Ok, let’s stop what we’re doing and immediately change to the death state”. Since we’ve already split our logic up into multiple controllers, each controller can emit events, then the brain can decide if it cares about them right now, or at all.

Example:

private void OnAttackStateEntered() {
    attackController.Execute();

    // The attack controller could even directly subscribe itself, especially if we have a single
    // attack state.
}

private void OnAttackPhaseChanged(AttackPhase phase) {
    if (phase == AttackPhase.Finished) {
        attackController.Finish();

        // Decide which state to change to..
    }
}

private void OnTargetAcquired(Target target) {
   fsm.ChangeState(AlertedState);
   // fsm.ChangeState(ChaseState);
}

private void OnLostSight() {
    fsm.ChangeState(SearchState);
}

private void OnTargetHealthLow() {
    // I'm a grunt, I don't care.
}

Smarter Enemies

I created a custom utility AI system a while back. It’s been collecting dust for a while now. What I realized though about this architecture is that, if the brain gets to make the decisions, I could easily create a boss-brain that uses the utility AI system to make those decisions. Not much else has to change. We can still use states for execution, but the utility AI decides which state we should even be in.

Combining this with a custom attack controller, the utility system can even decide which attack makes the most sense right now. Perhaps a ground slam, or perhaps an energy beam attack depending on range. The attack controller then executes exactly the same way it did for the dumb grunt enemy.

Conclusion

Event-driven architectures are by no means a novel topic, nor is having a “brain” control the behavior of an enemy. In fact, I don’t think any individual part of this architecture is novel in any way on its own, but I arrived at this architecture because of my constraints:

1. I wanted reusability.
2. I didn’t want to deal with transition hell again.
3. I didn’t want half of the enemies using an FSM while others used a behavior tree.
4. I wanted to keep things as simple as possible while still being able to have a large variety of enemies.

So far, I’ve been extremely happy with this approach. I’m no longer pulling my hair out debugging a strange edge case only to find that it’s because of a transition priority issue after 3 hours of debugging. I have a very clear separation of concerns across the entire enemy codebase, and each brain is on average about 80 lines of event handling boilerplate which makes it extremely easy to instruct an LLM how to create a new variation.