Allocations on the Wire: Building a Low-Allocation MQTT Broker with Trie Routing and .NET 10

From bytes to behaviors: the protocol challenge

In the previous post, we laid down the foundations of Beskar.Networking an ultra-fast transport layer utilizing System.IO.Pipelines and PinnedBlockMemoryPool to move raw bytes over sockets with zero GC pressure.

But raw transport is only half the battle. If you want to build a fully fledged server framework, you need to handle application-layer protocols. For this project, that protocol is MQTT (supporting both MQTT v3.1.1 and MQTT v5.0 specifications).

Moving from raw streams to a stateful message broker introduces a mountain of high-allocation operations:

1. MQTT packet parsing & encoding: Serializing headers, user properties, payloads, and variable integers.
2. Subscription matching (Trie): Figuring out which client sessions match a published topic filter (like sensors/+/temperature or factory/#) under high concurrency.
3. Session state tracking: Managing Quality of Service (QoS) flow state, unacknowledged publishes, offline message queues, and session takeover rules.
4. Keep-alive monitoring: Tracking heartbeats for thousands of connected clients.

If you resolve these with naive heap-allocated lists, string splits, and timers, your garbage collector will spend more time cleanup cycles than the network card does shifting packets.

Let's dive into how we tamed the heap to build a zero-allocation MQTT broker.

Topic routing without string allocations

The core job of an MQTT broker is routing messages. When a message is published to factory/line1/sensor3/temperature, the broker must locate all matching subscriber filters (e.g. factory/+/+/temperature or factory/#).

A naive approach splits the topic string by /, walks the parts, and matches them using regular expressions. Under a load of 100,000 messages per second, this creates millions of short-lived string and string[] allocations.

To eliminate this, Beskar.Networking implements a highly optimized, concurrent, UTF-8 byte-based Trie router: the MqttTrieSubscriptionRouter.

1. Stack-only topic level enumeration

Instead of splitting strings, we walk the raw UTF-8 bytes of the topic. We use a custom TopicLevelEnumerator (a stack-only ref struct enumerator) to slice the bytes at each / separator without extracting substrings onto the heap.

2. Alternate Lookups for zero-allocation dictionary access

Inside each node of our Trie (MqttTrieNode), we store children in a dictionary:

internal sealed class MqttTrieNode(byte[]? level)
{
   public byte[]? Level { get; } = level;

   public Dictionary<byte[], MqttTrieNode> Children =>
      field ??= new Dictionary<byte[], MqttTrieNode>(ByteArrayEqualityComparer.Instance);
   
   // ...
}

Normally, looking up a node in Dictionary<byte[], MqttTrieNode> with a slice of the incoming topic (represented as a ReadOnlySpan<byte>) would require converting the span to a byte[] array, allocating memory.

We bypass this entirely by using Alternate Lookups a powerful optimization feature in modern .NET. This allows us to query our byte[] dictionary using a ReadOnlySpan<byte> directly:

var children = node.Children;
var lookup = children.GetAlternateLookup<ReadOnlySpan<byte>>();

if (!lookup.TryGetValue(level, out var child))
{
   // Only allocate byte[] if we must add a new node
   var levelBytes = level.ToArray();
   child = new MqttTrieNode(levelBytes);
   children.Add(levelBytes, child);
}

3. The Visitor Pattern for matching

When a message is published, we traverse the Trie to collect matches. If we had to allocate a new List<MqttSubscription> for every routing check, it would flood the GC.

Instead, we use a visitor pattern (ISubscriptionVisitor). Matching traverses the Trie and invokes the visitor's Visit() method inline:

public interface ISubscriptionVisitor
{
   void Visit(in MqttSubscription subscription);
}

By making the visitor a stack-allocated struct, the matching process invokes zero allocations and executes inside a thread-safe read lock:

public void Route<TVisitor>(ReadOnlySpan<byte> topic, ref TVisitor visitor) 
   where TVisitor : struct, ISubscriptionVisitor
{
   using var disposer = _lock.EnterReadLock();

   var enumerator = new TopicLevelEnumerator(topic);
   MatchRecursive(_rootNode, ref enumerator, ref visitor);
}

private static void MatchRecursive<TVisitor>(
   MqttTrieNode node,
   ref TopicLevelEnumerator levels,
   ref TVisitor visitor)
   where TVisitor : struct, ISubscriptionVisitor
{
   // Check multi-level wildcard (#)
   if (node.MultiLevelWildcardChild is { Subscriptions: { } hashSubs })
   {
      for (var i = 0; i < hashSubs.Count; i++)
      {
         visitor.Visit(hashSubs[i]);
      }
   }

   // Exact match or single-level (+) wildcard traversal...
   if (!levels.MoveNext())
   {
      if (node.Subscriptions is not { } exactSubs) return;
      for (var i = 0; i < exactSubs.Count; i++)
      {
         visitor.Visit(exactSubs[i]);
      }
      return;
   }

   var currentLevel = levels.Current;

   // Query child node with zero allocations using alternate lookup
   var alternateLookup = node.Children.GetAlternateLookup<ReadOnlySpan<byte>>();
   if (alternateLookup.TryGetValue(currentLevel, out var exactChild))
   {
      var nextLevels = levels;
      MatchRecursive(exactChild, ref nextLevels, ref visitor);
   }

   if (node.SingleLevelWildcardChild is not null)
   {
      var nextLevels = levels;
      MatchRecursive(node.SingleLevelWildcardChild, ref nextLevels, ref visitor);
   }
}

Session lifecycle, QoS, and offline queuing

MQTT demands strict state handling for QoS 1 and 2 messages. If a client connects with a persistent session (CleanSession = false), the broker must:

• Deduplicate incoming packets (QoS 2).
• Queue outgoing publishes if the client is currently offline.
• Deliver those queued messages once the client reconnects.
• Support Session Takeover safely disconnecting an old, stale connection when a new client connects with the same Client ID.

1. Packet deduplication and tracking

To track unacknowledged publishes, each session maintains a registry of active packet IDs. If a packet is sent, we store its identifier and track its status (Published, Acknowledged, Received, Released). These lookups are mapped to pooled state objects, avoiding new object allocations during standard confirmation flows.

2. Session Takeover and event pipelines

When a client reconnects, the broker must coordinate session takeover:

1. Locate the existing active session.
2. Signal the old transport connection to gracefully close (awaiting the DISCONNECT packet or forcefully severing the socket).
3. Transfer the queued offline messages to the new session.
4. Clean up resources associated with the old connection without leaking rented buffers.

Here is the start of the cleanup routine hooked into our server lifecycle events:

private void HandleSessionTakeover(MqttSession oldSession, MqttSession newSession)
{
   TraceLogger.LogNeutralInfo("Session takeover initiated for Client ID: {0}", oldSession.ClientId);
   
   // Await current pending publish queues and copy them
   newSession.TransferOfflineQueueFrom(oldSession);
   
   // Trigger disconnect of the old session
   oldSession.DisconnectGracefully();
}

Optimizing Keep-Alives without task bloat

Each MQTT client negotiates a keep-alive timeout interval (e.g., 60 seconds). If the broker doesn't receive a packet within that window, it must disconnect the client.

Creating a dedicated System.Threading.Timer or spawning a long-running Task.Delay loop for every connected client would consume huge amounts of memory and CPU cycles when scaling to tens of thousands of active connections.

In Beskar.Networking, we manage heartbeats with a centralized, single-loop KeepAliveService.

• Each connection session records a LastPacketReceivedTime timestamp using a fast system clock.
• The centralized service runs on a single background timer, checking active sessions in batches.
• We minimize heap allocation inside this check loop by utilizing pre-allocated or rented arrays to track stale connection IDs that need to be dropped.

Combined with our use of ValueTask for async execution paths, this allows the broker to keep track of connection heartbeats with virtually zero CPU overhead.

Conclusion: clean, allocation-aware protocols

Protocols like MQTT are historically demanding to implement efficiently because they are stateful, packet-driven, and highly dynamic.

By layering MqttTrieSubscriptionRouter on top of our pipeline-driven transport, we proved that you can write a complex server with clean, interface-driven abstractions without compromising on speed or memory usage. Our topic routing matches wildcards entirely on UTF-8 bytes, lookups avoid string conversion, and session objects are recycled through structured lifecycle pipelines.

With the core MQTT protocol server and client now fully operational, we have a foundation capable of handling extreme connection rates under tight memory constraints.

Keep your loops tight, your allocations pooled, and your networks fast!

• GitHub: github.com/MarvinDrude/Beskar.Networking
• GitHub: github.com/MarvinDrude/Beskar.Memory