server SIM speed across all servers is bad, please help

Boromir ●

What we noticed was, after I picked up my bag after dieing, the server sim speed would begin dropping. The local sim speed would drop too. When I say drop, here's what I mean: every 2 seconds it would spin down to like 0.3 for a second and then spin back up to 1.0. The local sim speed would do this too. Sometimes the server sim speed would be at a steady 1.0 but the local would definitely do that. At least until I died. If I died then everything would be fine until I picked up my bag again. Both players would experience this at the same time: the effects of me having my bag.

Here's our mods: PVE, and the Vegetables mod. Neither of us are carrying any vegetable items from that mod.

It seems like whatever debugging tool was added last week and then hotfixed to fix the performance, it wasn't completely fixed. It was better before last Tuesday's patch.

Reply URL

1

Boromir ●

Not to disregard @ibisgrunk's concern, but mine turned out to be a hosting center problem. I did a lot of testing this weekend with much older backups and the problem appeared there as well.

Thank you!

URL

1

shad ●

we cannot melee combat each other on MP because of the lag

I cannot confirm this. The first 1-2 weeks after 0.6.4 launched, we were fighting a lot on the server and had tournaments on an already old world with 25+ people online simultaneously. Server and local simspeed were absolutely stable at 1.0 and combat worked smoothly.

The only thing I noticed that defnitely is an issue, is your latency. Everything over a 100ms ping gives you a huge disatvantage, obviously. This, of course, does not take doing snapshots and connecting players into account, which definitely is an annoying issue, especially on very populated servers.

i also see less than 1% of all activity in ME as player melee PvP

Obviously, since ME is still a building and engineering focused game after all, and the combat overhaul was not trying to lose this focus, as the developers also cleary stated in the update stream and the recent Dev-Diaries. I consider the new combat as a "cherry on top" kind of feature, which is great if it comes to PvP or organised tournaments etc.

the game, ME, is unplayable. every time the server saves and a player connects or disconnects... and that's just what we know causes it...

I think this is completely exaggerated. Yes, it is annoying, but far from being unplayable. Furthermore, disconnecting doesn't do anything, only connecting players cause the simulation to come to a more or less long halt, depending on the world size.

What I personally want to see resolved is the constantly recurring Havok freezes, which indeed make the game unplayable after a certain amount of time, and which force servers to wipe sooner or later, but rather sooner, depending on the player activity. Within one year, we had to wipe the server 10 times now, which makes consistent worlds impossible.

we cannot melee combat each other on MP because of the lag

I cannot confirm this. The first 1-2 weeks after 0.6.4 launched, we were fighting a lot on the server and had tournaments on an already old world with 25+ people online simultaneously. Server and local simspeed were absolutely stable at 1.0 and combat worked smoothly.

The only thing I noticed that defnitely is an issue, is your latency. Everything over a 100ms ping gives you a huge disatvantage, obviously. This, of course, does not take doing snapshots and connecting players into account, which definitely is an annoying issue, especially on very populated servers.

i also see less than 1% of all activity in ME as player melee PvP

Obviously, since ME is still a building and engineering focused game after all, and the combat overhaul was not trying to lose this focus, as the developers also cleary stated in the update stream and the recent Dev-Diaries. I consider the new combat as a "cherry on top" kind of feature, which is great if it comes to PvP or organised tournaments etc.

the game, ME, is unplayable. every time the server saves and a player connects or disconnects... and that's just what we know causes it...

I think this is completely exaggerated. Yes, it is annoying, but far from being unplayable. Furthermore, disconnecting doesn't do anything, only connecting players cause the simulation to come to a more or less long halt, depending on the world size.

What I personally want to see resolved is the constantly recurring Havok freezes, which indeed make the game unplayable after a certain amount of time, and which force servers to wipe sooner or later, but rather sooner, depending on the player activity. Within one year, we had to wipe the server 10 times now, which makes consistent worlds impossible.

Reply URL

1

ibisgrunk ●

while i consider your criticism of every paragraph interesting, in essence telling me i am wrong and the experiences i had are false, you mention Havok freezing without any detail, what on Clang's green earth is Havok freezing? ME is not in this list, https://en.wikipedia.org/wiki/List_of_games_using_Havok.

Shad you don't need to argue my comments like this imho. Different players have different experiences, it doesn't help to toss in saying according to your experiences my experiences cant have happened or my conclusion is wrong. This thread isnt meant to be combative... but it is meant to make someone at Keen angry enough to get fired up and feel the pain we go through... :)

Shad i deleted some chippy comments i had because i don't want Support forum to be opinion - i would take that to the regular Forum, but can you post the Server Specs for the server you had no issues on - this information is very very important because i think a Server Admin thread is needed now. too many folks starting ME servers without a bevvy of useful info. we had bad performance issues with our hosting company, have experience with servers, and would think a lot of new players don't need to be turned off to ME just because a server is weak.

What if its just the RAM size and all these complaints are because folks are running ME on bad servers? Let's start an info pool and put more data and less opinion into it. Maybe VM machines hate ME?

I had a wonderful chat with peaceman and his PlanetsNoLag which has high ping but no lag, i play it from USA and i believe he hosts it in Belgium. Great playing experience... better than hosted low cache VM servers miles closer... we found the CPU type makes or breaks the server SIM speed. also the closer folks build to each other the harder the SIM speed gets... Conco running Valanor has brilliant similar observations, same with Methyma and Androox... these guys are pros, lets trace their experiences for common use!

Honestly, and really honestly, if Keen just spent $500 a month on a rackspace official hosted server i think they wouldn't be putting out such a toxic playing experience. Keep one EU and one NAm, sorry ANZAC, official server so we know any issues are not hardware, or load balancing, or something tech. If Keen wants to put in a T1 to my house i would manage it for free!

Keen what say ye, stand by your game and host official Servers? not doing such is killing your revenue stream imo...

while i consider your criticism of every paragraph interesting, in essence telling me i am wrong and the experiences i had are false, you mention Havok freezing without any detail, what on Clang's green earth is Havok freezing? ME is not in this list, https://en.wikipedia.org/wiki/List_of_games_using_Havok.

Shad you don't need to argue my comments like this imho. Different players have different experiences, it doesn't help to toss in saying according to your experiences my experiences cant have happened or my conclusion is wrong. This thread isnt meant to be combative... but it is meant to make someone at Keen angry enough to get fired up and feel the pain we go through... :)

Shad i deleted some chippy comments i had because i don't want Support forum to be opinion - i would take that to the regular Forum, but can you post the Server Specs for the server you had no issues on - this information is very very important because i think a Server Admin thread is needed now. too many folks starting ME servers without a bevvy of useful info. we had bad performance issues with our hosting company, have experience with servers, and would think a lot of new players don't need to be turned off to ME just because a server is weak.

What if its just the RAM size and all these complaints are because folks are running ME on bad servers? Let's start an info pool and put more data and less opinion into it. Maybe VM machines hate ME?

I had a wonderful chat with peaceman and his PlanetsNoLag which has high ping but no lag, i play it from USA and i believe he hosts it in Belgium. Great playing experience... better than hosted low cache VM servers miles closer... we found the CPU type makes or breaks the server SIM speed. also the closer folks build to each other the harder the SIM speed gets... Conco running Valanor has brilliant similar observations, same with Methyma and Androox... these guys are pros, lets trace their experiences for common use!

Honestly, and really honestly, if Keen just spent $500 a month on a rackspace official hosted server i think they wouldn't be putting out such a toxic playing experience. Keep one EU and one NAm, sorry ANZAC, official server so we know any issues are not hardware, or load balancing, or something tech. If Keen wants to put in a T1 to my house i would manage it for free!

Keen what say ye, stand by your game and host official Servers? not doing such is killing your revenue stream imo...

URL

1

shad ●

I did not say you were wrong in any way, but rather shared my experience on this topic and tried to relativize your statement, that this is happening "across all servers" and that "we" cannot melee combat in MP. Actually, it is wrong to a certain degree, since I, and probably others as well, didn't experience this the same way you did.

What I am trying to say is - and that's the funny part about this, since you just said it so yourself - you shouldn't generalize "your" opinion/experience like as you were the spokesman of the community like you did in the OP and to avoid meaningless discussions like this. I felt like you included me into "your" opinion and I wanted to make clear, that this is not my point of view.

I am sorry if I have offended you, however, this is now the place to give feedback and discuss about things, and that's exactly what did imho. I get your point that you want to draw attention to server and MP issues, and believe me, I would love that too, but not with generalized statements, that drags other people into your argumentation.

Back to topic:

ME.eu and SE.de servers run on the following, dedicated remote host machine. It's located in Düsseldorf, Germany Iirc.

CPU: Intel i7-6700k @4.0 GHz
RAM: 32 GB DDR4, @2133 MHz
2 x 250 GB SSD in RAID 1
1000 MBit Up- & Downstream
OS: Windows Server 2012 R2

32GB RAM is fine for two large and active worlds and can even run a third, temporary server if needed. To run more than two instances, I'd recommend >32GB of system memory. Though, 16GB should be enough to run the OS and a single ME instance over a longer period of time, as of now.

The CPU clock is key for both games, ME and SE, since they only have one main thread and therefore heavily rely on single threaded performance. A low clocked server CPU, like a Xion, but with double the threads wouldn't help you anything with performance. The main thread workload for the ME server usually sits around 50-75% with up to 20 players, and up to 100% with up to 30 players -that's where simspeed starts to drop (not meaningful, maybe down to 0.8, but nevertheless noticable).

About the Havok freezes: ME does use Havok for its physics, and according to the devs, that is what basically stops working when we talk about the "simspeed bug".

Very active, and populated servers are suffering, either modded or vanilla ones, and it comes down to the amount or type of grids in the world or a combination of both. During the year, a certain pattern came to shine through: when the SANDBOX_0_0_0_.sbs file, which basically contains all the grid information, reaches a threshold of 180-200 MB, that "simspeed bug" starts to occur, at first after every 4-5 hours uptime, but rapidly starts to come back more frequently, as said file continues to grow.

The threshold and the triggering of the bug itself also depends on the amount of players actively playing and to a certain degree to the hardware specs of the server: we once had a very reliable repro to trigger the freeze on the remote hosted server, but failed to trigger it on a different, more powerful machine. Though, this is surely just a matter of additional time if this bug would occur on the second machine.

There is currently no solution to this problem yet, but grid purges are helping to get rid of this bug temporarily. I do regular, automated cleanups of all unclaimed grids in the world, which buys some additional time until the game becomes unplayable again. And by unplayable I mean the time the server needs to restart vs the actual playing time until it freezes again.

I have no experience with those rented game server VMs, but having Keen to rent their own, "official" servers, wouldn't solve that particular issue either. In the end, you'd have to wipe your world after x weeks again and this is what they'd have to work on in the very near future - make server worlds last longer than that short period of time.

I did not say you were wrong in any way, but rather shared my experience on this topic and tried to relativize your statement, that this is happening "across all servers" and that "we" cannot melee combat in MP. Actually, it is wrong to a certain degree, since I, and probably others as well, didn't experience this the same way you did.

What I am trying to say is - and that's the funny part about this, since you just said it so yourself - you shouldn't generalize "your" opinion/experience like as you were the spokesman of the community like you did in the OP and to avoid meaningless discussions like this. I felt like you included me into "your" opinion and I wanted to make clear, that this is not my point of view.

I am sorry if I have offended you, however, this is now the place to give feedback and discuss about things, and that's exactly what did imho. I get your point that you want to draw attention to server and MP issues, and believe me, I would love that too, but not with generalized statements, that drags other people into your argumentation.

Back to topic:

ME.eu and SE.de servers run on the following, dedicated remote host machine. It's located in Düsseldorf, Germany Iirc.

CPU: Intel i7-6700k @4.0 GHz
RAM: 32 GB DDR4, @2133 MHz
2 x 250 GB SSD in RAID 1
1000 MBit Up- & Downstream
OS: Windows Server 2012 R2

32GB RAM is fine for two large and active worlds and can even run a third, temporary server if needed. To run more than two instances, I'd recommend >32GB of system memory. Though, 16GB should be enough to run the OS and a single ME instance over a longer period of time, as of now.

The CPU clock is key for both games, ME and SE, since they only have one main thread and therefore heavily rely on single threaded performance. A low clocked server CPU, like a Xion, but with double the threads wouldn't help you anything with performance. The main thread workload for the ME server usually sits around 50-75% with up to 20 players, and up to 100% with up to 30 players -that's where simspeed starts to drop (not meaningful, maybe down to 0.8, but nevertheless noticable).

About the Havok freezes: ME does use Havok for its physics, and according to the devs, that is what basically stops working when we talk about the "simspeed bug".

Very active, and populated servers are suffering, either modded or vanilla ones, and it comes down to the amount or type of grids in the world or a combination of both. During the year, a certain pattern came to shine through: when the SANDBOX_0_0_0_.sbs file, which basically contains all the grid information, reaches a threshold of 180-200 MB, that "simspeed bug" starts to occur, at first after every 4-5 hours uptime, but rapidly starts to come back more frequently, as said file continues to grow.

The threshold and the triggering of the bug itself also depends on the amount of players actively playing and to a certain degree to the hardware specs of the server: we once had a very reliable repro to trigger the freeze on the remote hosted server, but failed to trigger it on a different, more powerful machine. Though, this is surely just a matter of additional time if this bug would occur on the second machine.

There is currently no solution to this problem yet, but grid purges are helping to get rid of this bug temporarily. I do regular, automated cleanups of all unclaimed grids in the world, which buys some additional time until the game becomes unplayable again. And by unplayable I mean the time the server needs to restart vs the actual playing time until it freezes again.

I have no experience with those rented game server VMs, but having Keen to rent their own, "official" servers, wouldn't solve that particular issue either. In the end, you'd have to wipe your world after x weeks again and this is what they'd have to work on in the very near future - make server worlds last longer than that short period of time.

URL

1

ibisgrunk ●

shad you sir are all right in my book, take back any tone that gave off a different impression and i certainly appreciate this as i want to link to this in discord sessions being run by ME server admins that want help... i dont want to whine, really want to give a path to help, and often my post tone comes off too snarky or immature. Thank you for your reply, it is very kind and informative.

URL

1

ibisgrunk ●

i also think, and it pains me to say this because of my love of the workshop, that certain mods simply combine to be toxic? For example Barbarians active on a Water mod server is much jumpier than one we played without, wondering about those connections as well. So we have technical set up things like RAM size but then we have game nuances like dont run X mod with Y mod and Z mod also running... and finally general experiences playing and running ME.

Hosted services we have done three of. Standard VM management cloud which means, and this is my IT background, that the VM machines are constantly being adjusted based on concurrent users which means a server with one guy playing from a hosting server has so little traffic they will throttle down its resources automatically, that dynamic management i believe can hurt ME server performance, because once users start joining the server causing more system needs then the VM adjusts with more resources and that can affect performance we found depending on how powerful the host is. Hosting companies like the workshop i am very grateful for but a lot have the Xeon weak CPU, but infinite virtual RAM - and imho SSD or not shouldn't affect SIM speed. There is a vast difference in hosting providers beyond GUI... i have found when you engage directly with the hosting company and push your desire to maintain a peak ready server you get more performance, for example we got the host to put our ME on its own silo without being tossed into a generic server pool for VMs like we were... i should document that and put it into wiki, things you can ask you MSP to provide for your server...

are you gesp_shad from the forum because you have brilliant posts there too, I want to link https://forum.keenswh.com/threads/ds-recurring-simspeed-freezes-after-three-hours.7396465/#post-1287073987 into some single resource available for server admins of ME, a compilation of information, a wiki.

Summary: Need a compilation wiki for server admins to come to in times of poor server performance, it could be something so simple and from that resource all players including Keen benefit?

Server hardware specs incl. VM/Hosting and very granular and as geeky as possible (like how much cache in the CPU is not enough?)
Mod conflicts and suspicions
What i call empirical observations such as too many folks building close to each other affects SIM speed and other experiences that help SIM speed like dont build a catchblock on a deer's arse...
Good habits for any ME server admin such as running around deleting damaged timber pieces...

i also think, and it pains me to say this because of my love of the workshop, that certain mods simply combine to be toxic? For example Barbarians active on a Water mod server is much jumpier than one we played without, wondering about those connections as well. So we have technical set up things like RAM size but then we have game nuances like dont run X mod with Y mod and Z mod also running... and finally general experiences playing and running ME.

Hosted services we have done three of. Standard VM management cloud which means, and this is my IT background, that the VM machines are constantly being adjusted based on concurrent users which means a server with one guy playing from a hosting server has so little traffic they will throttle down its resources automatically, that dynamic management i believe can hurt ME server performance, because once users start joining the server causing more system needs then the VM adjusts with more resources and that can affect performance we found depending on how powerful the host is. Hosting companies like the workshop i am very grateful for but a lot have the Xeon weak CPU, but infinite virtual RAM - and imho SSD or not shouldn't affect SIM speed. There is a vast difference in hosting providers beyond GUI... i have found when you engage directly with the hosting company and push your desire to maintain a peak ready server you get more performance, for example we got the host to put our ME on its own silo without being tossed into a generic server pool for VMs like we were... i should document that and put it into wiki, things you can ask you MSP to provide for your server...

are you gesp_shad from the forum because you have brilliant posts there too, I want to link https://forum.keenswh.com/threads/ds-recurring-simspeed-freezes-after-three-hours.7396465/#post-1287073987 into some single resource available for server admins of ME, a compilation of information, a wiki.

Summary: Need a compilation wiki for server admins to come to in times of poor server performance, it could be something so simple and from that resource all players including Keen benefit?

Server hardware specs incl. VM/Hosting and very granular and as geeky as possible (like how much cache in the CPU is not enough?)
Mod conflicts and suspicions
What i call empirical observations such as too many folks building close to each other affects SIM speed and other experiences that help SIM speed like dont build a catchblock on a deer's arse...
Good habits for any ME server admin such as running around deleting damaged timber pieces...

URL

1

ibisgrunk ●

this thread needs to be moved to DEDICATED SERVERS imho, and i can put more information from the other forum into it, doesnt need to be here if the moderator can help...

URL

1

ibisgrunk ●

well i have hit that wall as a user. mods kill the game, game kills mods and around and around i have gone... tired of it all as i lost another 100 hours on a server that is fading into death. tough life playing ME mp online?

problem is we can't find a server to run ME. been that way for two years or more? we got one in N America in Valanor and that's about it. some individual home ones, or like i did a couple of months of hosted games before fading off... Valanor is the only persistent server running these many years, tough to find a home!

here is what always happens when you run a server: mod updates competing with steam stuff mashed up with good old code development for an early access and all you can do is restart the server. i cant stand running an ME server because its like herding cats. three hotfixes in one day, no thanks... or steam re-ID'd all the mods and we have to relist them? been there, done that. or a mod is broken and conflicting but removing it means players lose xyz...

the unintended consequence of early access builder game is imho -- and perhaps reflected in the dismal online player counts -- that since ME is a builder game and since there are too many moving parts from mods and code its just not fun to play at this time. no catchblocks in motion work, which means all carts and siege machinery will fail over time. that means a base fun factor to the game is not usable. the larger and more complicated the buildings, like large villages, on a server actually drive that SIM speed lower with the more nearby users and that too deters the mp online folks who want to build near each other... it just all adds up into an unpleasant experience playing ME online in mp imho... but since its early access it only means the problem is for the user. its on me! no problem with modder or coder, its just a bitter sauce to eat. too early.

so i deleted ME and look forward to returning to game in a couple of more years when it finishes. i am guessing 2021?

the reason i tell all my friends not to buy ME is that it will just frustrate you in the hope for what it could be, then over time that will grind more painfully as it has for me, and as the game progresses through EA just skip the pain and wait for it to finish.

i am taking my own advice and wish Keen g*dspeed on moving ME into a finished stage!

reading this goodbye i completely understand now why so many old timers i first met pre version 0.4 went through this phase. and often why in these forums i read of folks coming back to check on it. now that is my turn, be back later! i will be that guy in later 2019 popping in saying something and you're like 'how did that guy have 2k hours playing and hasnt played in a year?'?