Client receiving lingering packets from previous server

Hello!

I am the creator of Nexus, which is a server-side world clustering/sectoring torch plugin that many big servers are currently using. This allows all kinds of sandbox-related data to sync live between servers, and allow players to seamlessly switch to those other servers.

The original bug I have been trying to fix the past few months is actually a double requestrespawn event after you switch to another server. This results in the client getting either an empty respawn screen or the famous spectator bug at 0,0,0. In logs, it would appear like shown below:

2021-11-15 09:20:44.993 - Thread:   1 ->  Updating continues.
2021-11-15 09:20:45.072 - Thread:   1 ->  GC Memory: 1900.852 / 1900.852 MB
2021-11-15 09:20:45.530 - Thread:   1 ->  RequestRespawn
2021-11-15 09:20:45.606 - Thread:   1 ->  RequestRespawn
2021-11-15 09:20:45.614 - Thread:   1 ->  Error: Missing session or local player in SetIdentityBlockTypesBuilt
2021-11-15 09:20:45.614 - Thread:   1 ->  Error: Missing session or local player in SetIdentityBlockTypesBuilt

Hooking up dnspy to clients and testing this issue with breakpoints to view the call stack yielded an event that the server was telling the client that they were dead before the normal requestrespawn event occurred. This led me to server-side debugging that to see if it was a random plugin causing this issue... After doing the same call-stack tracing on the server, it wasn't sending anything to the client on join that would tell the client it was dead even after stripping all plugins from torch.

It seemed like this client was getting a requestrespawn event from thin air. But code-wise, this is impossible, right? I finally gave in and traced the packet that this event was coming from and the endpoint it was coming from. Turns out this packet wasn't even being sent from the server they were joining, but actually from the previous server they left.

Now this made perfect sense because, on player disconnect, that server broadcasts to all clients that the player is dead. Especially if that player's character is auto removed or cleaned up. After hooking into sent packets on the server-side, there seems to be a few packets actually being sent to client endpoints even after they have left the game. See below for my test pic:

Now, what looks to be happening is that packets still in the queue are not checking if their recipient is still online, then pushes those packets to the steam networking layer. What steam does with those I have no clue and there must be some kind of client lingering state that tries to send those once the client is back available.

On the client-side, there is no check to make sure that packets received are actually from the same server they are currently on. So once it starts processing messages, the previous server is free to send the last bits of packets that were left from the queue. This also relates to the block limit errors and crashes in the error above. Because after patching into the sendpacket, and checking to make sure that the packets being sent are actually to clients online, both issues seem to be fixed and the client no longer gets stray packets on join.

Now, what I think should be done is a client-side check to make sure received packets are matched to the server they are on, or maybe adjust the connection/linger state of the steam networking layer?

Not sure about the best way to go about it but I think this is definitely a big issue as if you repeatedly un-log and relog quickly to another server, you can get an error like this. It was only noticeable more in Nexus because of the amount of switching servers that happens.

(I should also note that torch or nexus, the only plugin running during these tests, does not interfere with anything of client disconnects/reconnects/packet sending) The nature that the client is even getting these lingering packets should confirm that it has not been tampered with and is a true bug in the game networking layer.

If you require any more information or logs, or how I fixed it serverside, please let me know!