Server runs at 100% CPU, even with 1 empty area

Neverwinter Nights 1 Enhanced Edition
1

All of my experience stems from the 1.69 days where I could hyper optimize my server down to single percentages with tools like NWNX and resman, it appears those days are gone in EE?

Just simply running a server, whether NWNX or not, runs the core to 100% as soon as a player joins. I’ve tested this on 3 different machines, one being a windows 7, 3.4Ghz (original server I ran 1.69 for years on with 0 issues), an Ubuntu cloud server with 2.60Ghz server core, and a Windows Server R12 Virtual machine with 3.7Ghz. Each machine reacts the same, 100% cpu usage.

I’ve even gone as far as to test a 30KB module with only one 8x8 area, completely empty - same result.

Is this the new normal for neverwinter servers?

1 Like
2

There is two ways this can happen:

  1. There is a setting in the ini called “Clients Needed To Disable CPU Sleep”. If this is set to 0 (or less?) your server will be running at 100%. If you set it to 1, that means that if there is no one online, it will use little CPU, but as soon as 1 person logs on, it’ll go to 100%. Basically, NWN has “perf mode” and “power efficient mode”. If you have less players online than that setting, it’s in efficient mode, but as soon as you reach it, it’ll switch over to full power. (default setting is 1)
  2. Your stdin is constantly sending data. This can happen depending on how you start it (whether you pipe something to it, etc). Try running with -quiet flag.
3

Thanks for the heads up, turned out it was the CPU sleep option. I find it kind of… excessive that it would go to such lengths. Watching an ancient game eat up 4.7Ghz is pretty hillarious, especially when you compare it to modern game engines which are doing crazy things in comparison, and use less resources.

Seems like a nice option for unnecessarily pissing off your server host lol

4

We were thinking of adding a “target performance level” kind of setting to nwnx so that it sleeps as much as it can while still reaching that target. But there wasn’t really much demand, as most PWs are happy to run at 100%, and those that only have a few players online can just use this setting and the performance is still acceptable.

5

weird, my PW is running at 30% cpu…

6

Is it 30% of a single core, or 30% of all of it? Windows task manager bundles all cores together, so if you have 4 cores, and one is pegged to the max, it’ll show 25%. On linux, it’s common to say 100% in that case, and 400% if all cores are maxed out.

7

I have 2 cores, windows task manager doesn’t allow to show cpu usage per core though at least I haven’t found it anywhere so don’t know

8

I use Sysinternal’s process explorer. It’s freeware, and enables you to do some extended viewing.

image
image1920×1080 175 KB

9

What would be considered enough players that you would actually want to cap out your core? I was always under the impression that once your CPU hit 100%, you’d start getting some funky things happening in game. What issues were present before where increasing the tick rate by 10x was the solution?

Thankfully with it off I’m back to normal levels, which strangely seems to depend on the environment. On my 12 year old 3.4Ghz server, my module runs at 0-1%, but on my rented Vultr cloud server it runs at ~15% base load. Can’t really understand it considering the Vultr server is not only more powerful, but is running Linux which “should” be more optimized for this.

10

There’s no target in terms of CPU utilization, just server ‘frames’ per second. Most PWs run at 100% all the time, which is “gimme as much as possible”. Ideal target fps is around 100 (which is what you’ll get with an empty module and ~1% cpu utilization, as each frame is followed by a 10ms sleep). Below 50 and it becomes noticeable, below 30 and it becomes problematic.

Unfortunately, there’s currently no way to set a target fps and have the sleep times adapt to it. This is something that can be added to nwnx (request at github.com/nwnxee/unified/issues), or if you don’t want to use nwnx you can request this from BD, but they’re pretty busy so don’t hold your breath there.

11

How about letting it run at full throttle while capping the CPU utilization for the process?

#!/bin/sh
NUM_CORES=1
CPU_LIMIT=10
cpulimit -c "${NUM_CORES}" -l "${CPU_LIMIT}" -p "$(pgrep nwserver)" -v

This utility is quite crude with its SIGSTOP/SIGCONTs, but gets the job done and there’s no need to kill the target to change the limit. Of course, one would need to keep tweaking the value (10% is an example) to get a smooth operation.

1 Like
12

So if most people are used to their servers running at 100% all the time, how would someone performance optimize, and how would you even know when you’re ‘breaking the bank’ CPU wise? You could use nwnx profiling, but even if you shave ms off your worst offenders it won’t result in CPU% reductions. It’s hard to know where you’re really at or how much headroom you have, when you’re running at max out the gate.

Are CPU related issues even a realistic worry when it comes to this game, whether running at 100% or not? I’ve run many other game servers, and prioritizing CPU efficiency has always been of top importance, so I find all of this kind of odd.

13

From what I tried, I started hosting on lower end hardware (Intel Celeron N3050, 1,6 GHz duo core) which had CPU at 80%-100%, with many players in same area and many monsters CPU jumped to 100% and it caused huge lags on whole server

Switched to Core i3-8109U 3GHz duo core and now im running at 25-30 and no more problems unless there more are players downloading from sync (but that is another issue…).

I am hosting on windows, it is possible that things are different on linux…

14

On 1.69 I managed to get about 5% CPU usage from 10 players having massive fights with AoE spells, summons etc. It took a long time to get that level of optimization, and was reliant on plugins like nwnx_dynres + nwnx_areas, and hacky things like CutsceneGhostEffect on enemies to minimize pathfinding calcs - but sadly most of this isn’t available in EE.

15

(Answering in order)

@NWShacker:

How about letting it run at full throttle while capping the CPU utilization for the process?

That’s brilliant! Thanks!

@EzRemake:

So if most people are used to their servers running at 100% all the time, how would someone performance optimize, and how would you even know when you’re ‘breaking the bank’ CPU wise? You could use nwnx profiling, but even if you shave ms off your worst offenders it won’t result in CPU% reductions. It’s hard to know where you’re really at or how much headroom you have, when you’re running at max out the gate.

Are CPU related issues even a realistic worry when it comes to this game, whether running at 100% or not? I’ve run many other game servers, and prioritizing CPU efficiency has always been of top importance, so I find all of this kind of odd.

You need to think of it in terms similar to clientside. When you play an action game, your CPU is likely always at 100%, regardless of the CPU/GPU. What changes with stronger hardware is how many FPS you get, which controls how fast the game reacts to your inputs. Same logic applies to the server - you’re always at 100%, but the actual performance is measured in FPS (‘ticks per second’ in server parlance), and the more of those you get the better your performance.

If you shave some ms somewhere, it’ll be redistributed across everything else, and your ticks-per-second will go up.

@Shadooow:

Switched to Core i3-8109U 3GHz duo core and now im running at 25-30

i3-8109U is an 2core/4thread CPU, which means that if one thread is maxed out it will report as 25%. nwserver is entirely dominated by one thread, but it actually has other threads too that do stuff like the GUI updates or receiving network packages from the players, etc. I’d guess the extra 5% is from those.

@EzRemake

On 1.69 I managed to get about 5% CPU usage from 10 players having massive fights with AoE spells, summons etc.

Remember that “Clients Required To Disable CPU Sleep” setting? You can think of 1.69 as having that set to infinite. So, 1.69 would always sleep for 10ms after every tick. Which means the highest possible ticks-per-second in 1.69 was 100 (ideal), and any activity from your server would lower it. It also means that even when your server is struggling with only 10 ticks per second, it will still report only 90% CPU utilization (because it sleeps 10x10ms).
So, some servers used NWNX to disable that sleep in order to keep the CPU at 100% to get better performance.
If you want 1.69 behavior, just set the ini value higher than your max players.

It took a long time to get that level of optimization, and was reliant on plugins like nwnx_dynres + nwnx_areas, and hacky things like CutsceneGhostEffect on enemies to minimize pathfinding calcs - but sadly most of this isn’t available in EE.

CutsceneGhostEffect is a cool trick you can still do. However, unloading empty areas does not help nearly as much as one would expect. Objects (creatures, placeables, etc) in empty areas don’t compete for the same time-resources as those within populated areas. Every tick, 98% of the time is dedicated to things in areas that have players, and 2% to those without players (simplification, but close enough). If you remove empty areas, those 2% will still be spent on other stuff, and not on things with players.

EDIT: To elaborate on this last point, the empty areas themselves come from the 98%, but anything within the area comes from the 2%. So if you have an intensive HB script on the area, it’s worth checking if the area is empty and aborting early if so.

Hope this helps…

2 Likes
16

Thanks for taking the time to explain sherincall, really appreciate it! When you refer to an objects time resources, are you referring to their AI level in the engine?

So are you saying that (if nwnx_dynres + nwnx_areas still worked for EE), even removing empty areas from memory entirely has almost a negligible effect on performance? That’s pretty wild. I remember reaching ~300 areas in my old PW, and base CPU usage was about 30% even with 0 players, until I ripped everything out and instanced only what was necessary.

My new module uses 32x32 maps exclusively, but there’s only going to be 100 of them max. So far difference in load between 7 areas, and 15 is 15-30% base load. You’re saying this is nothing I should be worried about? (not using heartbeats, all creatures and containers destroyed when area empty)

EDIT: I’m coming to find that the nwserver process seems to be “stimulated” by other loads on the same core. If I run the nwserver by itself it will run at low (0%) CPU usage, but as soon as I run my Arma 3 server on the same machine nwserver immediately goes up to 40%. Very strange! Cloud servers may not be a good choice for this after all.

17

I’ve been doing even more testing and it honestly just has me more confused than ever.

Module Size: 7Mb
Area Count: 15 32x32 Maps
Creatures by default: 0
Place-ables per map: Average 50-60, 95% of which are static

Runs at 50% CPU on Linux server, 0-1% on Windows server. Both with the same settings, and the CPU sleep option set to 99. If I use the CPU sleep option at 1, I literally start getting rubber banding in game on the Linux server.

The only way for me to get the % down on the Linux server is to remove all placeables, which suddenly gets me down to 6% CPU. The Windows server manages 0-1% even with all the placeables. What’s going on here?

All of this for NWNX almost seems unreasonable.

18

That does sound really wrong. I can’t reproduce it.

Is it always exactly 50% on linux, or does it vary (say 40-60)? Do your placeables have any scripts attached to them?
Can you post the exact command line (without passwords) you’re using to run linux server? There is something weird going on in your setup (or less likely module), and I’ve no idea where to look.

As for the other things:

When you refer to an objects time resources, are you referring to their AI level in the engine?

Yes, AI levels. Each level gets a slice of a tick (highest gets 60%, next highest gets 60% of the remaining, etc) to process all objects at that level. The server keeps processing objects of a level in order until it runs out of time. Next tick, it continues where it left off, then wraps around.
This means that VERY_LOW priority objects (this includes all placeables and items, including static, as well as creatures in areas without players) will be processed last for whatever remaining time is. They will practically never all be processed in a single tick, so having more just means longer time to get to a particular one - this means that HB scripts for those things can fire more rarely than 6s - but it doesn’t affect how often higher priority things get checked.

15-30% base load. You’re saying this is nothing I should be worried about?

You can be worried about CPU load if:

  • You’re paying for the electricity/cycles on the cloud provider
  • You own the CPU and don’t want the wear and tear on it
  • The CPU is in your house and you don’t want the noise/heat
  • You are using the CPU for other things and this is making it harder to do that
  • You want to minimize global warming

But 100% CPU load does not mean you are having performance issues. Pretty much every PW out there is running at 100% all the time.

1 Like
19

#!/bin/sh
umask 002
export LD_PRELOAD=./NWNX_Core.so
./nwserver-linux
-module “Main”
-quiet
“$@”

Is what I’m using the run the server, pretty sure everything with stdin should be OK considering the quiet flag. Backslashes won’t show on the bottom lines.

I did a couple more tests which still have me pretty confused, but a little more hopeful. All I did was clone my 15 maps until I had 100 of them (target), and turns out the CPU will still end up running 35-55% average. Im almost starting to wonder if the Linux governor is running ondemand and I just can’t change it, and it’s the frequency scaling that’s throwing me through a loop.

Strangely though, a giant 120MB module like Nordock with 650 areas runs at 1-2% on the Linux server, so it doesn’t appear to be related to module size. Then again, Nordock uses very small maps and no placeables so who knows.