Resiliency in the Chaos: Source-Generated Framing and Auto-Reconnecting Client-Server Pipelines

The quest for bulletproof pipelines

In our first post, we built an ultra-fast TCP socket transport utilizing System.IO.Pipelines and PinnedBlockMemoryPool to move raw bytes with zero GC pressure. In the second post, we layered an MQTT broker on top of it, optimizing topic routing via a UTF-8 Trie and Alternate Lookups.

But when you write networking code that leaves localhost and runs across real, unstable networks, your biggest enemy isn't performance.

It's chaos.

Clients drop offline. Sockets time out silently. Wi-Fi connections disconnect and reconnect. OS level buffers fill up. If your client and server cannot negotiate handshakes, detect silent disconnects, recover state, and auto-reconnect without leaking memory or spinning threads, your low-allocation design is worthless.

To tackle this, I designed Beskar.Networking.Resilient, a highly resilient, transport-agnostic client-server framework. Along the way, I encountered two significant challenges:

1. Packet Framing Boilerplate: Writing code to serialize, deserialize, and length-prefix packet structures by hand is error-prone and tedious.
2. State Coordination: Synchronizing handshakes, keep-alives, and reconnection loops across asynchronous tasks without deadlocking.

Here is the story of how we used Roslyn Source Generators to eliminate packet framing code entirely, built a robust async state machine, and verified the whole system under extreme simulated conditions using a custom-built Chaos Simulator.

Roslyn Source Generators: eradicating framing boilerplate

Every protocol needs a framing format to distinguish where one packet ends and another begins. A typical frame contains magic bytes, version info, flags, a length prefix, and a payload:

+-------------------+---------+-------------+-----------+----------------+----------------+
| Magic (2B: 0xBE5C)| Ver (1B)| Type ID (2B)| Flags (2B)| PayloadLen (vB)| Payload (data) |
+-------------------+---------+-------------+-----------+----------------+----------------+

Writing the parsing code (TryRead) and serializing code (TryWrite) for this frame by hand usually involves a wall of bit-shifting, big-endian conversions, and validation logic.

To solve this once and for all, I built a Roslyn Source Generator (FramingProtocolGenerator) that implements the serialization contracts at compile time.

1. Declarative packet definition

Now, defining a high-performance network packet is as simple as declaring a partial struct and decorating it with custom attributes:

using System.Buffers;
using Beskar.Memory.Flags;
using Beskar.Networking.Protocol.Attributes;

namespace Beskar.Networking.Protocol.Frames;

[GenerateFramingProtocol]
public partial struct BeskarPacket
{
   [MagicBytes(0xBE, 0x5C, Order = 0)]
   public partial bool HasValidMagicBytes { get; }

   [VersionField(Order = 1)]
   public byte Version { get; set; }

   [ProtocolField(Order = 2)]
   public BeskarPacketType PacketType { get; set; }

   [FlagsField(Order = 3)]
   public PackedBools16 Flags { get; set; }

   [VarNumberField(Order = 4)]
   public int PayloadLength { get; set; }

   [ByteSequenceField(nameof(PayloadLength), safeCopyData: false, Order = 5)]
   public ReadOnlySequence<byte> Payload { get; set; }
}

The source generator intercepts this definition, parses the attributes, and emits the partial implementation of the struct, forcing it to implement IFramingProtocol<BeskarPacket>.

2. High-performance code emission

Let's look at a slice of the code emitted by our generator (FramingProtocolGenerator.Rendering.cs). It writes highly optimized, allocation-free methods for writing and reading from buffers:

• Fast bitwise flag casting: Casting helper structs like PackedBools16 directly into big-endian integers using Unsafe.As.
• Zero-copy sequence reading: Slicing the incoming ReadOnlySequence<byte> directly without heap-allocating byte arrays.

Here is the logic our generator uses to render the TryRead method:

// Emitted code generated by FramingProtocolGenerator.Rendering.cs
public static bool TryRead(ref SequenceReader<byte> reader, out BeskarPacket result)
{
   result = default;
   var pkt = new BeskarPacket();

   // 1. Read & Verify Magic Bytes
   if (!reader.TryRead(out byte b0) || b0 != 0xBE) return false;
   if (!reader.TryRead(out byte b1) || b1 != 0x5C) return false;

   // 2. Read Version
   if (!reader.TryRead(out byte ver)) return false;
   pkt.Version = ver;

   // 3. Read Packet Type (2-Bytes Big Endian)
   if (!reader.TryReadBigEndian(out short pTypeRaw)) return false;
   pkt.PacketType = (BeskarPacketType)(ushort)pTypeRaw;

   // 4. Read Flags with zero-allocation Unsafe casting
   if (!reader.TryReadBigEndian(out short flags16Raw_Flags)) return false;
   ushort flagsU16_Flags = (ushort)flags16Raw_Flags;
   pkt.Flags = System.Runtime.CompilerServices.Unsafe.As<ushort, PackedBools16>(ref flagsU16_Flags);

   // 5. Read Variable-Length Integer (VarNumber)
   if (!VarNumber.TryRead(ref reader, out int varVal_PayloadLength)) return false;
   pkt.PayloadLength = varVal_PayloadLength;

   // 6. Slice Payload Sequence directly (Zero-allocation)
   int seqLen = pkt.PayloadLength;
   if (reader.UnreadSequence.Length < seqLen) return false;
   var seq = reader.UnreadSequence.Slice(0, seqLen);
   reader.Advance(seqLen);
   pkt.Payload = seq;

   result = pkt;
   return true;
}

By generating this boilerplate at compile time, we completely eliminate manual serialization errors and guarantee that frame parsing runs at native speed.

The Resilient Client-Server Choreography

Writing a client that can auto-reconnect or a server that can handle disconnects gracefully is deceptively difficult because they are multi-threaded state machines.

               +------------------+
               |   Disconnected   |<-----------------------+
               +------------------+                        |
                 |              ^                          | Connection Limit
                 | ConnectAsync | Disconnected / Timeout   | / Handshake Denied
                 v              |                          |
               +------------------+                        |
               |    Connecting    |------------------------+
               +------------------+
                 |
                 | Handshake Ok
                 v
               +------------------+
+------------->|    Connected     |
|              +------------------+
|                |
|                | Heartbeat Timeout / Conn Lost
|                v
|              +------------------+
|              |   Reconnecting   |
|              +------------------+
|                |
+----------------+ Reconnect Ok

1. The Client Reconnection Loop

Inside ResilientClient.cs, when a socket exception occurs, we trigger the auto-reconnect loop. We must ensure that only one reconnection task runs at a time, and that we yield execution paths cleanly during backoffs:

private Task TriggerAutoReconnectAsync(Exception? cause)
{
   if (State is ResilientClientState.Disconnecting or ResilientClientState.Disconnected)
      return Task.CompletedTask;

   lock (_reconnectLock)
   {
      if (_isReconnecting)
         return _reconnectTask ?? Task.CompletedTask;

      _isReconnecting = true;
      _reconnectTask = Task.Run(async () =>
      {
         try
         {
            TraceLogger.LogClientWarning("ResilientClient: Connection lost. Auto-reconnect triggered...");
            State = ResilientClientState.Reconnecting;
            await DisconnectInternalAsync(null, raiseDisconnectedEvent: false);

            var newReconnectCts = new CancellationTokenSource();
            var oldReconnectCts = Interlocked.Exchange(ref _reconnectCts, newReconnectCts);
            oldReconnectCts?.Dispose();
            var masterCt = newReconnectCts.Token;

            var attempt = 0;
            var maxRetries = Options.Reconnecting.MaxRetries;

            while (!masterCt.IsCancellationRequested && State is ResilientClientState.Reconnecting)
            {
               attempt++;
               if (maxRetries > 0 && attempt > maxRetries) break;

               var retryInterval = Options.Reconnecting.RetryInterval;
               
               // Delay retry block using linked cancellation tokens
               try
               {
                  await Task.Delay(retryInterval, masterCt);
               }
               catch (OperationCanceledException) { break; }

               if (State is not ResilientClientState.Reconnecting) break;

               // Attempt connection & handshake
               var result = await ConnectInternalAsync(_remoteEndPoint, masterCt, isReconnect: true);
               if (!result.Failed)
               {
                  TraceLogger.LogClientInfo("ResilientClient: Reconnect attempt #{0} succeeded!", attempt);
                  return; // Reconnection successful
               }
            }

            // Exceeded retry limits: return to completely disconnected
            State = ResilientClientState.Disconnected;
            await DisconnectInternalAsync(null, raiseDisconnectedEvent: true);
         }
         finally
         {
            lock (_reconnectLock)
            {
               _isReconnecting = false;
               _reconnectTask = null;
            }
         }
      });

      return _reconnectTask;
   }
}

2. The Server Handshake Buffer

One classic race condition in socket servers is the Handshake Race.

When a client establishes a connection, it immediately starts sending application data. However, the server might still be executing asynchronous authentication checks or initializing the session. If the server reads the application data before finishing the handshake, the packet parser might crash or reject the frames because the client is not yet flagged as Connected.

To solve this, ResilientServer.cs buffers any incoming application message frames in a fast Channel until the handshake is complete. If the handshake succeeds, we drain this channel sequentially to the application event handlers:

// Inside ResilientServer.cs's read loop
if (frameKind is ResilientFrameKind.Message)
{
   if (client.IsHandshakeCompleted)
   {
      // Handshake is done: process the message frame immediately
      if (!client.DrainingTask.IsCompleted) await client.DrainingTask;
      await Events.FrameReceived.ExecuteAsync(eventContext, ct);
   }
   else
   {
      // Handshake in progress: buffer the message to prevent packet loss
      if (!client.PreHandshakeFrameChannel.Writer.TryWrite((frame, streamContext.Stream)))
      {
         // Fallback if the channel is full
         if (client.IsHandshakeCompleted)
         {
            await Events.FrameReceived.ExecuteAsync(eventContext, ct);
         }
      }
   }
}

Welcome to the Arena: The Chaos Simulator

To verify this complex dancing of state machines under heavy pressure, standard unit tests aren't enough. We needed to simulate real network conditions.

We wrote the Chaos Simulator (Experiments/Beskar.Resilient.ChaosSimulator). It binds to TCP, WebSocket, and QUIC listener addresses simultaneously, spawns hundreds of concurrent clients, and starts throwing random hurdles at them:

1. Client Personalities

To thoroughly test our state machines, I created various client roles that behave differently:

1. Sender: Sends high-frequency message packets to flood the server.
2. Echoer: Sends messages and awaits responses, testing bidirectional streams.
3. KeepAliveOnly: Connects and stays idle, forcing the keep-alive tickers to track it.
4. Flaky: Randomly sleeps, throws exceptions, or drops connections to force the server's cleanups.
5. SlowReceiver: Consumes packets at a snail's pace, forcing TCP window sizing and pipeline backing.
6. ChannelCongestor: Spawns multiple concurrent streams over QUIC to overload session-level multiplexing.

2. Random Severing & Crash Simulation

Every client connection randomly decides whether to disconnect gracefully (sending a DisconnectPacketPayload with reason codes) or abruptly (crashing, killing the socket, and abandoning rented buffers).

Here is the dashboard printed during a typical run:

======================================================================
               BESKAR RESILIENT CHAOS SIMULATOR RUNNING
======================================================================
[12:45:15] [SERVER] [TCP  ] [CONNECT   ] Client 'chaos-se' connected successfully.
[12:45:16] [CLIENT] [WS   ] [DISCONN_G ] Client 'chaos-flaky-12' disconnecting gracefully...
[12:45:18] [SERVER] [QUIC ] [DISCONNECT] Client 'chaos-ch' disconnected abruptly!
[12:45:20] [CLIENT] [TCP  ] [CONN_OK   ] Client 'chaos-echoer-45' connected successfully.

----------------------------------------------------------------------
                           DASHBOARD STATS
----------------------------------------------------------------------
Active Connections:
  TCP:   42  |  WebSockets: 38  |  QUIC:  20  (Total: 100)

Server Stats:
  Total Connections Accepted: 1,489
  Graceful Disconnects:       1,123
  Abrupt Disconnects:         266
  Total Messages Received:     845,920

Client Stats:
  Connect Attempts:           1,602
  Connect Successes:          1,489
  Connect Failures:           113
  Messages Transmitted:       845,920
  Pings Sent:                 8,349
======================================================================

Through hundreds of thousands of simulated connections, we proved that:

• The client state machine successfully re-negotiated handshakes and resumed message pipelines.
• The server correctly cleared connections, handled session takeovers (disconnecting old connections when duplicate Client IDs connected), and reclaimed rented buffers without leaking.
• Zero tasks were left dangling in background loops, keeping CPU usage flat when connections dropped.

Conclusion: bulletproof protocol building

Building resilience into C# networking layer is a journey of managing state concurrency.

By combining the Roslyn Source Generator (FramingProtocolGenerator) to compile p acket parsing logic directly into CPU instructions, and coordinating the client-server connection lifecycles through buffered channels and sequential event pipelines, we created a networking framework that handles network failures smoothly.

Testing our state machine against the Chaos Simulator was key: it highlighted edge cases like handshake race conditions and keep-alive thread exhaustion before they could impact production code.

Keep your pipelines resilient, your allocations low, and let the generator do the heavy lifting!

• GitHub: github.com/MarvinDrude/Beskar.Networking
• GitHub: github.com/MarvinDrude/Beskar.Memory