Hotfix 195.024 Breaks Dedicated Servers

Alex L shared this bug 30 days ago
Solved

WHAT IS HAPPENING

The Space Engineers dedicated server is reporting 300% CPU utilization and running at a reduced simulation speed (As low as 0.5). We run the server 24 hours a day and this did not start happening until immediately after updating to 195.024. Players are commenting that the server is running slowly then catching up, making it difficult to play.


WHAT SHOULD HAPPEN


Server CPU load should correlate with the actual CPU load of the machine. Simulation speed should be a constant 1.0 as it always was before the latest update.

HOW TO REPRODUCE

Start a dedicated server and let at least one player join. Connect to the remote client and monitor the simulation speed.

Comments (22)

photo
2

I just tested on another machine and the server simulation speed stayed at 1.0 for about an hour then dropped and never recovered. So it seems that this bug develops over time after updating to the recent hotfix, as opposed to happening right away.

photo
2

Quick update. It seems like this had something to do with floating objects. We've had a lot of game breaking bugs with the recent updates. Each time I will slowly remove things from the server until we see a change in behavior. This time I removed all asteroids, planets, grids, mods, etc and no difference. Then, when I removed the floating objects the server became more stable.

You can see from the attached screenshot that the server is closer to 1.0 sim speed now, but it's really unstable. Prior to this most recent update it seldomly went below 1.0 and was usually a straight line along 1.0 sim speed.

Something has definitely changed, but until this gets addressed by Keen, you can save your server by not allowing floating objects (by doing frequent cleanups).

Files: 01.png
photo
3

Can confirm the same thing happening on our dedicated servers. Unfortunately, in our instance cleaning floating objects didn't do much. Our servers that were 0.9 - 1.0 under load before the update are now 0.5 - 0.6 even with a low amount of players. One of our servers with less than 15 grids and 6 players was even experiencing severe sim loss.

photo
3

We have the same issue but floating objects have nothing to do with it. We also had a server, no planets, 15 grids, 5 players, and it was at 0.5 sim. Prior to the hotfix that same server, same players, same grids was at 1.0 for weeks now.

photo
2

Same issue on our server, cleaning floating objects didn't help. 12 players with about 25 grids, .9-1.0 sim for weeks, after the update - same grids and players and sim is fluctuating as well as server CPU load. Jumping up to almost 300% at times for no apparent reason.

photo
1

Hello, Engineers!


Thank you for contacting us. Could you please provide us a save of your server where you are experiencing the issue? So far I haven't been able to reproduce this issue.

Thank you!


Kind Regards

Keen Software House: QA Department

photo
3

https://drive.google.com/file/d/1oPU35s1TZ36dvRSF3KMU00ADbbRLMkFI/view?usp=sharing


This world has issues as soon as 6-8 people join, without adding any additional grids to the configuration. It is a world with mods, not vanilla.

photo
2

Did a dot trace profile on one of our servers with issues:

https://i.imgur.com/XmSMi5i.png

Digging down into the busiest CLR Worker after that :

https://i.imgur.com/M2uyhZW.png

photo
2

i have emailed support@support.keenswh.com with the archive password

photo
1

Thank you!

photo
photo
1

@keen, here you go.

photo
1

Hmm, it's not allowing me to upload. It says there's a 9mb limit, which is not much. How would you like me to share?

photo
1

Hello, Alex!


If you are having difficulty attaching files you can optionally use Google Drive. When sharing a google drive link please make sure it is set to be downloadable by anyone with the link.


Thank you.

photo
photo
4

"vanilla survival - the expanse" server is reporting very bad sim speed. Limiting hinges seems to help us stay above .6

but unless the server is nearly empty, we dont see better than .6-.8


Before the update we were locked at 1.0 even with 30 players online. Its a unbelievably noticable differance. does keen not Q/A multiplayer? it is not good to neglect your fanbase who play on dedicated servers.

photo
3

The Expanse is Vanilla with some plugins; we reboot the server and delete asteroids & reset planet voxels every 2h 15m


One more detail is weather was disabled and active weather entries were edited out of the game files due to the initial lighting bug issue.


There are about 500 grids on the server. Performance has been noticeably worse since the "weather" patch and it seems to have scaled up with player activity. Previous to this the server ran for 2 months with only reasonable sim speed drops (like when a fight was going on, or someone had crashed into terrain)

photo
2

Please add your world save to this issue so that keen can investigate the issue - you might be the only vanilla server reporting it and they need to see it on an un-modded world they say ala https://support.keenswh.com/spaceengineers/pc/topic/hotfix-195-024-breaks-dedicated-servers#comment-37348

photo
2

So we are Vanilla, but we run using a 3rd party dedicated server software with plugins (which I'm not going to name, but I'm sure Keen developers are aware of)


If Keen Developers want our world file we will provide directly to them. Please send request to gertrone@gmail.com or give me an email address where I can send the link.


Thanks and good luck fixing this issue; We have a lot of DLC buying players on our server and want to support the game but it's hard when our stability gets disrupted.

photo
photo
3

Between both my servers, only my squeakly clean creative testing server is showing normal simspeed and CPU load after the update. My previously stable survival server is now bouncing around at roughly 0.6 Simspeed.

photo
2

For clarification, this is with no players online on either, but with considerable voxel deformation and grids. Approx 400,000 PCU on the survival server.

photo
2

I would recommend you attach a copy of the world save as they requested for reproduction, might help us get this issue fixed easier.

https://support.keenswh.com/spaceengineers/pc/topic/hotfix-195-024-breaks-dedicated-servers#comment-37244

photo
2

Survival Save

Just to add, not a single CPU thread is even being fully utilized on the server.

photo
photo
2

Hello,


SkunkWorks is experiencing the same massive sim speed. It started recently and it is random. We used to run a fairly large map (1300 Grids, max players 30) at a .75-8 sim. Now we are lucky to get a .5 with 10 players on. It seems to be related to the server thread CPU load. The load used to settle around 75%-115% now if fluctuates between 150% - 300%. We run an i9-9900K on a dedicated box and we only started having issues after the last few hotfixes.


WORLD FILE

https://drive.google.com/file/d/1PLbACC1LofynG2iXxaegxRjvkRzl02gz/view?usp=sharing

photo
2

Hello, Engineers!


Thank you for your information and saves. We will investigate it.


Kind Regards

Keen Software House: QA Department

photo
2

Thank you :)

photo
photo
1

Hello, Engineers!


Can you please confirm that this issue also occurs on vanilla servers? Can you please provide us with vanilla server save?


Thank you for your help.


Kind Regards

Keen Software House: QA Department

photo
2

Ms. Červená,


I'm not sure what you mean by this. Every major dedicated server (and all of the ones listed in this comment thread) is modded or has plug-ins in some way. Are you asking us to setup a pure vanilla server and recreate the issue?

photo
1

Hello, Darien!


Yes, unless the issue can be reproduced on a clean, un-modded server, it is usually caused by a certain mod or by combination of mods which we have no control over. Can you please confirm this issue also occurs without mods?


Thank you.


Kind Regards

Keen Software House: QA Department

photo
2

We will boot up a clean server to test with in the morning. (SkunkWorks). A lot of use use the same stuff but the Expanse is the closest to vanilla that I know of.

photo
1

Thank you Asy93!

photo
4

Just to add my two cents on this Kateřina, while I appreciate Keen are entirely focussed on Vanilla for bugfixing, the fact remains:

Prior to this latest hotfix we had nowhere near as many issues with sim speed as we all are now.

Post this hotfix - which incorporates some significant modifications with regards to CPU functionality, AMD processors in particular - we are all seeing major drops in sim speed.

Part of me wonders if the fix for AMD processor based lockups and crashes involved some sort of thread throttling which has had a knock on effect on Intel based processors - all 4 of ours are Intel i9-9900k.

photo
2

@Keen, your own hosted servers are having the issue. I logged into several of them last night and 0.7 was the highest sim speed.

I also tried logging into about a dozen other player run servers and only two had a 1.0 sim speed. All had mods, so this has nothing to do with modded or unmodded. This is an issue with the latest update.

photo
3

Hello,


We ran a vanilla server on our box. We had about 500 grids placed. 8 players, the sim was .35-.45 server thread CPU load was 250%-500% @Keen


https://drive.google.com/file/d/1Uc4VTUNfU5mvJTlUJXBOgTkHcRAgOwgP/view?usp=sharing

photo
2

You should provide that save.

photo
2

@Kateřina, it looks like this is still marked as "Needs More Information." There are a handful of saves that have been provided for you to test with. I also mentioned in a separate comment that this bug is occurring on Keen's dedicated servers. Is there anything else you need from us?

photo
1

Hello everyone!


Thank you for your vanilla save, Asy93, I tried it also on v194 and the sim speed was similar (0.4 – 0.5).

We would appreciate more vanilla saves of servers to investigate this issue. All other saves attached here contain mods.


Thank you for your cooperation.


Kind Regards

Keen Software House: QA Department

photo
2

Ms. Kateřina Červená,


It should be noted that the same issues are occurring on the Official SE servers as well.

photo
2

Dudes please check the vanilla servers its happening everywhere

photo
photo
9

Keen,


We know you don't "officially" provide support for modded servers, we've all heard it before, but please for the sake of the community, go and conduct a half decent investigation of what the hotfix broke rather than falling back and hiding behind the same throw away line like it's a shield.


It has been mentioned multiple times on this thread that your official servers appear to be experiencing the same issue. Might I suggest you start there rather than asking for more vanilla saves because we all know the community is held together by mods, so that ask is a stretch.


Please don't be that team that is too proud to admit fault, you're better than that, if the hotfix has reported issues, conduct a proper investigation and roll it back if need be. Dedicated servers are the lifeblood of this game, so please assist us by identifying and resolving this issue quickly.


Thank you.

photo
5

Kateřina,


We, as a community, appreciate all of the hard work and dedication that Keen has tirelessly implemented in this game. It is truly a labor of love. That being said, as a server admin, I find myself at an impasse. Our server is losing player base to constant drops in sim speed, we are trying every fix we can to bandaid the situation but it is persistent. These people love your game, they have spent hundreds of hours messing about on the server. When you guys push a major update or paid DLC, we sit together and watch the live streams and eagerly buy the content to build more.


Surely there can be a way the community and the development team can communicate about this? Why not gather the server owners/admins together in a call to discuss these issues? We can all test together in real-time or show you what exactly is happening on our servers. Why not log onto the official keen servers or pull reports from them to test the validity of our claims? Its been over a week, and while I understand that you are only a QA rep, there needs to be something that can be done here.

photo
2

I've noticed a drop in singleplayer too.

Weird thing is that, as soon as I start all is fine, after a minute or so it starts dropping sitting at around 0.5 sim speed.

photo
2

Hello Keen,

Attached is my world file. I will email support with the archive password. World is vanilla with many many grids.


https://drive.google.com/file/d/1e4OCTiSeypqXoA-GWevRozaoOVcYky-0/view?usp=sharing

Thank you for the support.

photo
1

Hello!


Thank you for your save, could you please send me the archive password to email katerina.cervena@keenswh.com?

Thank you!


Kind Regards

Keen Software House: QA Department

photo
photo
13

I don't have a vanilla save to provide, but I wanted to re-iterate that while this report may not have the highest number of votes, that is simply a reflection of the fact that single-player players are not aware of it. Please, take this bug report seriously.


Our player numbers have tanked in the past week directly as a result of this issue. As soon as we have more than about 12 players on server the sim drop issues make the server borderline unplayable where it would usually be sitting at 1.0 until we hit the 25-30 player mark (with the exception of combat or ships crashing into voxels).


Of all the bugs reported in these forums, this is the biggest and most pressing issue outside of our control that our server has faced in the past 6-12 months. Like many others, we spend an enormous amount of time trying to optimise our server to run smoothly and seeing these kind of issues still marked as More Information Needed with no one actively communicating about the issue being worked on honestly make server admins question why they even bother pouring hundreds of hours into developing and maintaining their servers.

photo
4

Well said Rob11 - and I'd like to echo that comment 100%.

photo
4

Hello, everyone!


Thank you for your saves and all information, we are investigating this issue. We would still welcome any other vanilla saves. Thank you for your patience.


Kind Regards

Keen Software House: QA Department

photo
2

Katerina, sources are saying that Keen have identified and fixed this bug in 196, can you confirm?

photo
1

Hello, everyone!

This issue should be fixed in version 196.011. Can you please confirm?


Thank you.

Keen Software House: QA Department

photo
1

So far so good on 196.011, will comment tomorrow once it's been running longer.

photo
1

No issues as best I can see post fix

photo
1

The performance gains from this patch have been significant; The only sim speed drops we've seen was during a fight involving about about 5 or 6 large ships and our players are quite happy.


We didn't even require a wipe so that was nice.


If you need an updated save for testing at any time please let either myself or Garret Sidzaka know.


Thanks!

photo
1

Not seeing this issue anymore with v196

photo
1

Hello everyone!


Thank you for your information.


Kind Regards

Keen Software House: QA Department

photo