New Concept: Proximity Spawning (to reduce bandwidth usage)

[trevius]

While talking over performance issues with one of my guides, I came up with an idea that might be an awesome new feature to make a huge impact on the emulator. I think this feature could potentially reduce server load and bandwidth usage by a very noticeable amount.

I think this may also help reduce zone crashes or player LDs from raids in highly populated zones. I have tested and found that a zone with a large amount of spawns will almost always have lag if a raid of 3 groups or more is in it and fighting, but if they fought the same encounter in a zone with only a couple of NPCs there is no lag at all.

The idea is to have a way for spawns to only spawn if a player is within a set proximity to them. So, if a player is in 1 corner of the zone and no other players are in the zone, only the mobs in that corner of the zone would be spawned, which should considerably lower the traffic being sent to the client and possibly server load to manage pathing, scripts, etc on the other NPCs that aren't spawned. As players move throughout the zone, mobs set to use this system will spawn around them as they get within a set range.

The Details so far:

1. To make this work, we could use something similar to aggro radius or proximity, where the spawn point is waiting for a player to enter it's radius before actually spawning if doesn't have time left on the spawn timer. I think that the code for aggro radius might be a good place to start for getting this setup. The server will look at the spawn locations and set a radius to spawn for NPCs that are set to use this feature.

2. I am thinking that either the spawn_conditions table or maybe even just the npc_types table could be used to simply define 2 things to set this up. The first setting would be either 0 or 1 for if you want that particular NPC to use this system or if you want it to spawn normally. This way, you can have named mobs that are always up and just set trash spawns to use this system to reduce the bandwidth load. Any special NPCs with scripts or special pathing can all leave this feature disabled, to make this setup as customizable as anyone wants. The second setting will just be the radius range for the NPC to spawn in. If you are spawning small indoor zone like an LDoN dungeon, you could probably set all trash mobs to spawn within a 200 radius and no one would ever even be able to tell that this system was different than how it worked before (accept MQ users who would be like "WTF" lol). Or, if you were doing a large outdoor zone, you might set the range as far as 600 or so. I don't think mobs are drawn much further than that anyway. Even with a large range like that, a huge zone could still be over 50% empty most of the time, which should make a huge impact on bandwidth.

Post here if you have any concerns, suggestions or thoughts on this system. I think it should be doable and might make a huge difference on servers with limited bandwidth!

I am going to see if I can start figuring out where to even begin on coding this, but if any of the coders want to help out, you are more than welcome! If this ever gets finished, I would be extremely excited to try it out and see what if any impact it has on bandwidth and server load.

[ChaosSlayer]

I am gettying a feeling that in order to detect prox to spawn something you will need first to spawn something to detect proximity =)

Of course just a "watcher" trigger prabobbly will use less CPu/ram than actual mob, but then you running in number of issues to set a whole new way to respawn mobs, encounters, named etc who should be otherwise "UP" and perhaps even trackable (may end up been more complicated that you allready covered)

Another issue- a player running at high end SoW (60%+) speed will be covering a lot of ground VERY fast - this may result in mob spawnign right on top of him as he appraoches since delay in spawn is unavoidable, as well requring you to despawn as player moves away. Now if you got 5 players runing at sow speed in near prox after each other, the whole system turns into spawn fest and i would say MASSIVE server side CPU work as server will be requred to chain spawn/despawn/spawn/etc mobs as group of players runs by.

Not cirticizing idea itself - I just belive its a too complex approach to improve server performance

[trevius]

Well, consider that every NPC already has a radius check when they check for people entering aggro range. So, this system would essentially only be extending that radius, which means it probably wouldn't cause more server load in that aspect.

Then, consider that NPCs that might have running scripts, pathing, spell casting, and other checks running continuously would now only be running them when players are within range.

Also, I can repop an entire zone at once with only a blip of a performance hit. If only 2 or 3 spawns are popping per second while players are running around, I don't think that would cause any noticeable impact on performance. Even if they were moving fast enough to spawn 20 at once, it would only happen while they are running around, not while they are fighting, unless maybe if they are kiting. And, raid fights are my main concern for reducing lag. Raids and AE groups are the only real things that I ever see cause performance hits on my server.

And for players moving at really high speeds, normally that would be in open zones outdoors. In that case, you could either leave this system off for all NPCs in the zone, or set them to have a really large radius to ensure that mobs are popping on top of people. I don't think anyone moves faster than 600 feet per second without warping hehe.

Another bonus to this feature is that admins could setup zones that MQ maps are useless for. Players wouldn't be able to see which spawns are up or down on the map unless they actually entered the nearby area. I imagine that could be useful in many situations for those anti-MQ admins

The nice thing about this idea is that it would be a fully optional setting. You could leave important NPCs like Named and other scripted NPCs up all of the time by setting them to not use the system. This system is mainly to get rid of unneeded trash spawns.

I do think that the coding for this might be pretty easy to do. I am going to look into the aggroradius code and see if there is a way to possibly combine that with the spawn2 table to make the server watch for players to enter a set radius around each spawn point.

Yeah, not having an NPC there to base the radius check is probably my main concern, but I still think it should be possible. And the potential for very noticeable performance increases should be worth at least looking into this further.

[trevius]

After thinking about it a little more, there would probably have to be a few extra things done to make sure this system was pretty flawless. Most importanly is to make sure that if the NPC is following a running player that it won't poof once the player gets too far away from it's spawn point. So, if IsEngaged, then it negates it from despawning.

Also, we might want to consider adding extra code to make sure that people can't exploit the system by pulling a single mob that is just barely within range to spawn so they can avoid pulling the other spawns that would normally be near it. But, most likely this wouldn't even be a factor if we set the radius range to something like 300+, since I don't think anything can pull from that far anyway.

Here is some unmodified code from the source I found quickly to maybe start looking at to base this code on:

zone/aggro.cpp

//look around a client for things which might aggro the client.
void EntityList::CheckClientAggro(Client *around) {
	_ZP(EntityList_CheckClientAggro);

	LinkedListIterator<Mob*> iterator(mob_list);
	for(iterator.Reset(); iterator.MoreElements(); iterator.Advance()) {
		_ZP(EntityList_CheckClientAggro_Loop);
		Mob* mob = iterator.GetData();
		if(mob->IsClient())	//also ensures that mob != around
			continue;
		
		if(mob->CheckWillAggro(around)) {
			mob->AddToHateList(around);
		}
	}
}


	LinkedListIterator<Mob*> iterator(mob_list);
	for(iterator.Reset(); iterator.MoreElements(); iterator.Advance()) {
		Mob* mob = iterator.GetData();
		if(mob->IsClient())	//also ensures that mob != around
			continue;
		
		if(mob->DistNoRoot(*from_who) > d2)
			continue;
		
		if(engaged) {
			int32 amm = from_who->GetHateAmount(mob);
			if(amm == 0) {
				towho->Message(0, "... %s is not on my hate list.", mob->GetName());
			} else {
				towho->Message(0, "... %s is on my hate list with value %lu", mob->GetName(), amm);
			}
		} else if(!check_npcs && mob->IsNPC()) {
				towho->Message(0, "... %s is an NPC and my npc_aggro is disabled.", mob->GetName());
		} else {
			from_who->DescribeAggro(towho, mob, verbose);
		}
	}
}

Also, another idea that would help bandwidth and could replace this idea would be to just make it so that only mobs withing X radius of a player will send their current position information to them regularly. The entire zone would still be spawned exactly like it is now on the emu, but the client would only get updates from nearby mobs instead of every mob in the zone. It wouldn't really effect server load, but it should definitely help bandwidth and may be a better solution than the one I am suggesting in this thread. But I don't know how possible this second idea would be.

I am trying to figure out why WoW Emulator servers can handle 2000 players with only 1 or 2MBs of upload bandwidth (at least that is what I have heard). If it is true, then there definitely has to be something we can cut down by alot to help reduce bandwidth. And, I am almost positive that WoW doesn't send all spawn information to all players, I think they do a radius type thing like I am suggesting.

[trevius]

Still looking into this, it seems like this would maybe not be too hard to implement into the spawn_conditions code. Instead of just checking the time of day, it could have a separate option to just check player range from the spawn point before spawning and then depop if they get out of range of the NPC once it has spawned if they don't have aggro on it already.

I am trying to think of this on a per mob basis to make the concept easier to grasp which should hopefully make coding easier. Basically, all we need is a way for the server to spawn an NPC when a player gets near a spawngroup location without using any kind of placeholder to spawn it. It would have to check if an NPC is already spawned at that location before it tries to spawn another. Then, there needs to be a similar setup to depop the NPC when they get out of range if the NPC isn't engaged/aggroed.

I am sure this could be done with quests fairly easily, but it would be much easier to do it for entire zones at a time by setting a simple setting in the database. I think we could even potentially just have the check run as a client location check vs. the spawn group location and radius settings. Then, every spawn location in the zone wouldn't be looking for players to enter their radius, the check would only be done from around the client if they got within the radius which might reduce server load as well.

it looks like the spawn_conditions are checked before anything else when deciding to spawn an NPC, so that would be a good thing to reference when setting up this system.

Doing the coding work for this is probably a bit above my skill level, but I will keep looking into possibilities of how to write code for it.

[trevius]

Another thing I thought of is that the spawn group IDs, the spawn locations (X, Y, Z), and the spawn_radius setting will probably all need to be loaded into memory when the zone boots up. This way, the system can work without having to poll the database constantly for the spawn locations when it is checking the radius.

[ChaosSlayer]

I am trying to figure out why WoW Emulator servers can handle 2000 players with only 1 or 2MBs of upload bandwidth (at least that is what I have heard). If it is true, then there definitely has to be something we can cut down by alot to help reduce bandwidth. And, I am almost positive that WoW doesn't send all spawn information to all players, I think they do a radius type thing like I am suggesting.

its a good notice. A friend of mine plays wow his loading time is INSTANT from desktop into the game.
Thsi may have to do soemthign with wow inherited seamless zoning structure - the game is inheretly designed to be zoneless (with few exeptions) and this means any given player is only been sent information of some small surrounding area. However this does not mean that mobs are not up.

Now even with all those "wow" features (wow world is much smaller than eq, but on other hand a player is only present in a single zone at a time) the diffirence in numbers of players supported is massive - there got to do be something we missing in how to handle server performance

[KLS]

Personally I really hate this idea, not because i'm against the idea behind it but because I think it's the wrong approach. We really need a more involved spawn conditionals system so people can setup their events in an intelligent manner.

Perhaps a system where npcs can spawn and despawn based on certain conditions set by the quest system. Ex: In this case your boss is engaged and you have it set boss_one_engaged flag for the zone and all npcs that aren't essential despawn. Or perhaps you have a linear zone and only once a boss is killed the trash and next boss will spawn. etc etc.

[KLS]

Oh yeah and how wow works is just more efficient, it's also facilitated by the client.

Every continent is separated into zones which are separated into smaller areas which are all separated into smaller map nodes. Npcs in a node are only active if a player is nearby. At least this is how the emu basically works; not how the official servers work.

Perhaps we could have npcs freeze and not do ai if they're a certain distance from player characters. Might cause some problems for existing events.

[ChaosSlayer]

Originally Posted by KLS

Personally I really hate this idea, not because i'm against the idea behind it but because I think it's the wrong approach. We really need a more involved spawn conditionals system so people can setup their events in an intelligent manner.

Perhaps a system where npcs can spawn and despawn based on certain conditions set by the quest system. Ex: In this case your boss is engaged and you have it set boss_one_engaged flag for the zone and all npcs that aren't essential despawn. Or perhaps you have a linear zone and only once a boss is killed the trash and next boss will spawn. etc etc.

well this may work for sort of ldon/raid/encounter instance, but then you don't realy need anything special - when boss is engage you simply send a signal to depop entire zone exept the econuter


the real issue comes to handlign somethign as large as West Karana or Dread Lands which supose to have crap load of trash mobs running around

[KLS]

The more I think about it the more I think having npcs freeze if there's no pc nearby would help a lot with the situation and have less of a downside than popping and repopping based on proximity.

[trevius]

We have already been testing a quest system to depopall() on all trash mobs when a boss mob or named is being killed. It seems to completely stop all lag as compared to previously people would go LD and lag extremely bad in some cases.

But, I really don't like the quest solution, because then anyone else in the same zone will then see an empty zone, which not only looks bad, but also lets them roam freely without clearing anything and basically negates all of the work to create the trash mobs in the first place.

Any special settings like the radius_spawn idea or freezing them to stop all AI and pathing would have to be specified by a setting in the NPC_types table so it could be done on a per mob basis. Then you can decide which ones you want to be affected by any of these ideas, and which ones to remain normal all of the time. So, bosses, scripted NPCs and maybe some roamers might want to have this setting turned off, but other than that, almost all trash mobs aren't always needed.

By turning off the AI or freezing the mobs like KLS mentioned, that might help the server performance, but I honestly don't see any issues at all lag-wise on my local lan, ever. I only ever hear about the lag, so it is pretty obviously a bandwidth issue. I have 1MB upload bandwidth from my ISP, which is fairly decent as far as home connections go. If 1MB up can have lag during a raid of about 3 groups with maybe 70 people logged into the server, then I think there could be some definite improvements to improve bandwidth utilization.

Unfortunately, I don't know of any way to not send regular updates of the current location of every spawn in the zone every second to every character in the zone. So, the only solution I can think of is to not have trash mobs spawn unless a player is within range. The only other option would be if someone could figure out a way for the server to only send location updates for NPCs within a radius of the client.

[AndMetal]

I wonder if we might be able to look at this a different way, especially since this seems to be more a bandwidth issue than a processing issue (which could also explain the issues with Warrior Rampage in AoE situations).

While roaming around in a zone, what kind of information is transmitted to the client on a regular basis? Spawn information, etc? Also, are there any kind of limits? So, is everything being sent to everyone, or are there proximity limits on these? If there aren't proximity limits, I think that's where we need to start.

Anyways, just a thought...

[KLS]

Spawn info is sent on login and new spawn. Or if a spawn has to change shape for some reason, typically walking around you wont see many spawn infos. Movement updates are the big thing... which if it's a mob far away they only send movement updates once every 60 seconds, close npcs ie ones you can see send movement updates every few seconds.

[trevius]

Maybe if there was a way to adjust how far away the regular movement updates get sent. I use MQ which helps a ton with developing content and standing in the main part of Dreadspire, I can see probably 50 mobs in the entire main section all moving very accurately and seemingly sending multiple updates per second. It seems to be somewhere around 800 distance away. I can only ever see a max of 400 distance in the entire zone since it is all indoors. So, if I could somehow set that range on a per zone basis, that might help a bit. Also, if it was possible, I wouldn't mind the option to completely disable the zonewide 1 minute updates on a per zone basis. I don't see why my client even needs to be aware of mobs outside of my visual range other than for MQ purposes, which really has no part in actual game play.

I know that the rule for it could be adjusted, but for some zones, the location information might be more important than others. And I know for a fact that those updates are pretty heavy on the server or bandwidth. At one point, I tried setting those zonewide updates in the rules to be every 1 second, because I didn't know what it was doing exactly. With it set that way, my server was so insanely laggy that it was unplayable after a few people logged on. Even 1 time per minute, if you multiply that times 60 clients on the entire server, you are averaging a zone wide update to 1 client per second. Double the players to 120, which is about the max my server can handle and it goes up to an average of 2 zonewide updates per second.

Then, once you start factoring in combat spam that has to go out to each client, a big raid can cause a significant hit to bandwidth and performance. If all of those players were all fighting in separate areas, they would only be receiving their own combat spam, but all in a tight area, everyone gets everyone's combat spam, so it multiplies what needs to be sent by how many characters are there.

I think the AE issue with rampage is more of a server resource issue than it is on bandwidth, since AndMetal brought it up. The main reason to think this is because it is one of the few things that can an will crash a zone if you have too many mobs in the rampage pull. I don't think bandwidth can cause a zone crash, just cause lag and LDs. Only high running processes or bugs can cause crashes, or at least that is my guess. I think a rampage pull of 30+ mobs is just too much combat spam all hitting at the exact same time for the server to handle at once.

I am sure there are some solutions to help this stuff considerably. Even a 20% decrease in bandwidth could make a considerable difference on home run servers that don't have practically unlimited upload speeds. Right now, I think players seem to average about 5k per player on normal use. Though, that probably spikes up a lot during large raids.