Camelot Unchained - 500 Player Battles - Possible?

MightyUnclean

I'm pretty tech-ignorant, so I'm looking for some expert opinions.  The Kickstarter for Camelot Unchained states that their goal is to "Maintain an absolute minimum of 30 FPS in battles of up to 500 people."  That's why they're building a custom engine:  no one else, apparently, has built an available engine that can do this.  My question is, is this even feasible with modern technology?  For a small team working with a relatively small budget?  Have others, especially AAA developers, tried and failed to achieve such a lofty goal?  I'm wondering if problems achieving this base programming goal are part of the reason that the game is still in a pre-alpha state after four years.  Again, I'm very ignorant of how this all works, so I'm looking for the opinions of programmers or network pros that know a lot about this.

waynejr2

Well, I am going to wait and see how it turns out.   A the proof is in the pudding kind of thing.

Gdemami

If course it is possible.

The question is, whether it is worthy endeavor to pursue.

cameltosis

MightyUnclean said:

I'm pretty tech-ignorant, so I'm looking for some expert opinions.  The Kickstarter for Camelot Unchained states that their goal is to "Maintain an absolute minimum of 30 FPS in battles of up to 500 people."  That's why they're building a custom engine:  no one else, apparently, has built an available engine that can do this.  My question is, is this even feasible with modern technology?  For a small team working with a relatively small budget?  Have others, especially AAA developers, tried and failed to achieve such a lofty goal?  I'm wondering if problems achieving this base programming goal are part of the reason that the game is still in a pre-alpha state after four years.  Again, I'm very ignorant of how this all works, so I'm looking for the opinions of programmers or network pros that know a lot about this.

I work in IT - I was a programmer and have worked on things as small as apps and websites and as large as military software in aircraft. I currently manage a team of developers working on web technologies. 

I have also worked in the games industry, albeit only for 6 months in QA, so I have a small amount of insight into games development and used to chat to our dev team quite a bit. 

That said, I'm in no way an expert in game engines - I did build a physics engine for my university dissertation, but it sucked!


From my understanding of MMO games engines, there are 3 main areas of difficulty when scaling up the large amounts of players. 

1) Data Transfer

Every time I perform an action in game, my PC has to transmit my actions to the server, the server has to acknowledge receipt, and then the server has to send my action plus any further outcomes to everyone who needs to know. In a 500 player battle, my 1 action has to be transmitted to 499 other players. This needs to happen quickly, otherwise the game feels unresponsive. 

This is probably the easiest hurdle to overcome. The amount of data being transferred is actually small, it is just very frequent. You can optimise the data to keep it small (you're only transferring text messages essentially), you can probably compress it too and I imagine for security purposes, it is probably encrypted. Internet speeds are fast enough that bandwidth is unlikely to be an issue. 

So, on data transfer, yes, 500+ is easily doable. 


2) Data Processing

Lets say I use an AoE skill. Something has to process what my base damage is (so, need to process stats, skill modifiers etc), then calculate who is in range of me, then calculate the damage I do to those people. The more people involved in a battle, the more potential number crunching that needs to happen.

The primary challenge here is where does the processing take place and keeping it synchronised. If all the processing gets done server side, then it means you have greater control over the synchronisation (you can ensure all processing happens within the same GCD) and thus everyone gets the same results at the same time, but this does require some pretty beefy servers. 

The alternative is to move some of that processing to the client. For example, when using my AoE skill, my client could process the base damage and then transmit that to the server, which would then determine who it hits. The server could then transfer the hits to my enemies clients, which could then calculate the actual damage caused. 

This spreads the processing load out, however, it makes synchronising everything harder. If the processing happens on the server, then all calculations are handled at the point of arrival on the server, but by distributing the calculations, I get the first one on my machine, a second one on the server, a third on my enemies machine, plus the wait time for data to transfer from my pc -> server -> enemy pc -> back to server -> back to my pc. 

That said, this is all achievable. 

Computers are just big logic machines and combat mechanics are all logic based. A standard processor can process millions of calculations a second, so if you've properly coded your engine and have a suitable network of servers, there should be nothing standing in your way. It is still a challenge - optimisation of code is always a challenge - but I see nothing that would stand in their way. 


3) Graphics

This, in my opinion, is the toughest nut to crack. When we experience lag in MMOs, it is hardly ever the server or network speeds that cause it, it is almost always graphics lag. 

As I understand, when rendering a frame your computer has to first build a complete 3D representation of everything on your screen. This means calculating the relative positions of everything nearby by loading in the models, calculating positions and then sorting out things like overlaps / intersections etc. It then has to load in the textures for each face that is visible. Then, it will decide what you can actually see, then render that image. Then it applies post-processing effects like AA etc. 

With small scale games, the developer remains in almost complete control of what needs to be rendered. The devs know all of the environment and can make accurate guessing as to the number of people and gear you will need to render at any given moment. This makes optimising easier. It is also the reason why so many games are getting smaller in terms of level design etc - by keeping it small, they can increase the graphics quality as it means processing similar amounts of data. 

With an MMO, the developer is no longer in control. They can still control the environment, but the number of players is variable. It is far easier for your graphics card to process and render 1 person than it is 500+. This sort of graphics processing is also restricted to taking place on your PC - a server can't really help you out.

Now, there are various techniques that can be used to improve graphics performance. Sharing textures is one way - you don't need so much RAM to store textures if textures are shared, or using algorithms to process models, rather than loading a model itself (e.g. using tessellation). The easiest way is to simply reduce the polycount so there is less to process. 


Admittedly, my knowledge of graphics is pretty old, so everything I've just written may be wrong or simply out of date. But, I do believe that graphics is going to be the hardest thing to maintain when it comes to large scale battles. It is not a problem that CSE can solve simply by throwing hardware at the issue, because the hardware is our own PC hardware and thus out of their control. 

CSE also have an added complication with their CUBE system. This is going to allow us to design and build our own structures in the real world, so not only will CSE not control the number of players, they also won't control all of the environment. 


I still remain confident that they will achieve this. Working in their favour is the relative simplicity of their environments and lack of NPCs. With trees, forests, hills, streams etc, they can super-optimise the way those types of environments are processed, freeing up more time for processing players and player structures. 

I would definitely be curious to hear from CSE as to how their engine will handle the graphics of large scale fights. 

UOlover

It will be more than 500

collekt

MightyUnclean said:

 I'm wondering if problems achieving this base programming goal are part of the reason that the game is still in a pre-alpha state after four years.  Again, I'm very ignorant of how this all works, so I'm looking for the opinions of programmers or network pros that know a lot about this.

Everything is being designed from the ground up to support these large battles, and aside from developing their own engine it doesn't seem like this is the cause for delays. The main things we have heard in relation to delays are things like not being happy with the ability system, so they had to scrap it and rebuild it from scratch, etc.

Also, not sure if you're aware but it's not uncommon for MMOs to be in development for 5+ years. It just feels like a long time to you because companies don't normally announce their game on day one, before development has begun at all. They normally announce a few years after they start the development, so it feels like a shorter time before they are in beta/release.

meddyck

Possible? It already does that!

Yes making their own engine is the big reason aside from difficulty hiring programmers until they opened the Seattle office for how long development has taken.

Redoing the ability system took like 6 months. That's a big part of why Beta 1 still hasn't started rather than having started late last year, but it doesn't explain the previous 2 1/2 years.

Gavyne

Realistically, no they won't be able to achieve 30fps in a 500 players battle.  There are much larger budget games, for an example Guild Wars 2, or something more modern like BDO, they all have a threshold where the game starts to lag something fierce.

For GW2 in 3-way large scale WvW combat, things start to become a slideshow when you have more than 150-200 players in the same area.  But FPS isn't your only problem when you go large scale, it's the response time.  Calculations get choked up and you end up having server-side lag that delays your casts/melee combat.  So you would cast a standard spell, you press it, and it doesn't fire off for at least 10 seconds.  This server-side cast delay is worse than FPS issue.

Problem with CU is their ambition to create chained & reactionary combat.  If you have been following CU, you know what they're trying to do with their spells casting.  The first time I heard it, all I could think of is the bottleneck in computation during large scale combat.  There will most definitely be server-side lag if they want to achieve spell on spell combat.  Like they want you to be able to counter a spell that's traveling in the air, and make something else happen.   All that creates an issue with server-side computation because the server will have to track every lil movement of the spells.

So while I'm a founder for CU and have been supporting them, I don't believe they'll be able to achieve 500 players at 30 fps.  And like I said FPS isn't the only thing to worry about when you have that many players fighting, action delays become a huge problem, players warping becomes a problem.  You see this issue in GW2 and BDO today when you cross a certain threshold of number of players.

I understand why they want to set the ceiling high, like MJ said before if they weren't ambitious then there would be no reason for them to do it.  But I fear sometimes these smaller studios with small funding get too ambitious, it ends up hurting the game development as a result.

cameltosis

DMKano said:

cameltosis said:

1) Data Transfer

Every time I perform an action in game, my PC has to transmit my actions to the server, the server has to acknowledge receipt, and then the server has to send my action plus any further outcomes to everyone who needs to know. In a 500 player battle, my 1 action has to be transmitted to 499 other players. This needs to happen quickly, otherwise the game feels unresponsive. 

This is probably the easiest hurdle to overcome. The amount of data being transferred is actually small, it is just very frequent. You can optimise the data to keep it small (you're only transferring text messages essentially), you can probably compress it too and I imagine for security purposes, it is probably encrypted. Internet speeds are fast enough that bandwidth is unlikely to be an issue. 

So, on data transfer, yes, 500+ is easily doable. 

This is all pretty spot on - but I do want to point out one different thing with voxel based games - and that is voxel data stream overhead.

So 1 client sends actions to server - server sends back that client's action to the other 499 clients (in a 500 player battle) - yep minimal traffic.

Now add in destructible/buildable world geometry in Voxel MMOs - suddenly you start to choke at around 50 players due to huge voxel overhead.

CU uses voxels - to what extent I don't know - maybe they can pull it off

Yup, it is certainly going to be a challenge and I don't know enough about engines to be able to predict well enough. But, this is why they've created their own engine, so that it can handle large battles. 

My suspicion is that they're going to keep as much computation on our client machines as possible. So, if we're all shooting siege weapons at a wall, the computations for damage, collision etc will be kept on our machines with just the result being sent to the server. The server then decides when to trigger collapses and stuff and just sends that small message back to the client, which then determines how the wall collapses. 

I guess will depend on a lot on collision detection / despawning of voxels. If we destroy a wall, will the rubble be left on the ground? If so, will we have to jump over it or can we walk through it?


The downside is that by pushing the computation onto our machines, it does make the game more susceptible to hacking, but I'm sure they've thought of that. 

The other possibility is that by creating their own engine, they may be able to make better use of multi-threading and scalable cloud computing. It may actually be fairly cheap for them to just have 100s of processors for their server setup, its just a case of having an engine that can take advantage of all that power. 

gervaise1

waynejr2 said:

Well, I am going to wait and see how it turns out.   A the proof is in the pudding kind of thing.

This.

Stuff was said about DAoC as well; stuff and reality sometimes didn't match up 100%.

PurplePoloPlayer

MJ and the gang started a new game engine from scratch to tackle this very issue. Having large scale battles is one of the core designs for CU. They are having great success with large, mostly bot run battles so far. 

Also they are implementing wonderful security features that will prevent hacking that other MMO's are unable to address because of fundamental design flaws. Traditional wall hacking simply isn't possible with this engine. 

Excession

I seem to recall they had an engine tech demo showing thousand's of character's moving around, If my memory is correct, they were only very basic, very low poly model's though.

If they can do the same with actual finished game player model's remain's to be seen.

Gavyne

Excession said:

I seem to recall they had an engine tech demo showing thousand's of character's moving around, If my memory is correct, they were only very basic, very low poly model's though.

If they can do the same with actual finished game player model's remain's to be seen.

Demos with bots pre-alpha really don't tell anything.  I know they did it to test the engine and see if things would crash.  But these bots running around before they have proper combat & skills system in place really don't test anything other than server loads.

The things MJ wants to do with spells, like they want you to be able to counter/block a spell while it's in the air, and turn it into something.  Then have someone else cast something else, and completely change the spells...making the spells do something completely different.  These ideas are what will really increase server-side lag.  Think about the amount of tracking the server has to do just for this one idea to happen.

Until proper combat/skills systems are in place, they won't be able to test this.  Honestly for myself I would settle for a stable combat system with 100-150 players.  I don't see how they can realistically make this happen with 500 players while doing all the things MJ said they want to do with combat.

meddyck

cameltosis said:

Yup, it is certainly going to be a challenge and I don't know enough about engines to be able to predict well enough. But, this is why they've created their own engine, so that it can handle large battles. 

My suspicion is that they're going to keep as much computation on our client machines as possible. So, if we're all shooting siege weapons at a wall, the computations for damage, collision etc will be kept on our machines with just the result being sent to the server. The server then decides when to trigger collapses and stuff and just sends that small message back to the client, which then determines how the wall collapses. 

I guess will depend on a lot on collision detection / despawning of voxels. If we destroy a wall, will the rubble be left on the ground? If so, will we have to jump over it or can we walk through it?


The downside is that by pushing the computation onto our machines, it does make the game more susceptible to hacking, but I'm sure they've thought of that. 

The other possibility is that by creating their own engine, they may be able to make better use of multi-threading and scalable cloud computing. It may actually be fairly cheap for them to just have 100s of processors for their server setup, its just a case of having an engine that can take advantage of all that power. 

No. The calculations in response to players moving or using abilities are done on the server to limit the opportunities for hacking as much as possible.

meddyck

Torval said:

meddyck said:

cameltosis said:

Yup, it is certainly going to be a challenge and I don't know enough about engines to be able to predict well enough. But, this is why they've created their own engine, so that it can handle large battles. 

My suspicion is that they're going to keep as much computation on our client machines as possible. So, if we're all shooting siege weapons at a wall, the computations for damage, collision etc will be kept on our machines with just the result being sent to the server. The server then decides when to trigger collapses and stuff and just sends that small message back to the client, which then determines how the wall collapses. 

I guess will depend on a lot on collision detection / despawning of voxels. If we destroy a wall, will the rubble be left on the ground? If so, will we have to jump over it or can we walk through it?


The downside is that by pushing the computation onto our machines, it does make the game more susceptible to hacking, but I'm sure they've thought of that. 

The other possibility is that by creating their own engine, they may be able to make better use of multi-threading and scalable cloud computing. It may actually be fairly cheap for them to just have 100s of processors for their server setup, its just a case of having an engine that can take advantage of all that power. 

No. The calculations in response to players moving or using abilities are done on the server to limit the opportunities for hacking as much as possible.

Well, that will be interesting because the reason many studios choose client side calculations is explicitly for performance reasons. All server side calcs (which are great in theory) means that the data has to make several round trips in real time. Considering the scope of 500+ players this is what raises concerns and a lot of good questions.

Getting all of that working and at good framerates on the client is a big part of why the game is still in Alpha 4 years after funding. But from what I've seen I think they are on track to meet that goal.

Also keep in mind that after launch on actual servers there probably won't be 500+ player battles all that frequently, so the servers and clients will be able to handle the typical, smaller battles extremely well.

Phry

It should not be difficult, games like Planetside and Planetside 2 have been doing it for years, if anything the hardware available today is far superior to what was available when those games first came out, particularly the original Planetside, games that can't handle 100's of players fighting each other, tend to have very specific game engine problems, the best example of that would be Hero Engine and SW:TOR, but its not the only game engine that doesn't scale well, quite a number of games these days seem to have that issue.

Erevus

DMKano said:

It's possible - how FUN that ends up being is an entirely different point.

Also how often that happens due to playerbase numbers is another issue.

Well as a past loyal player of WAR, I can assure you that is a HELL OF A FUN participating in such a huge battles (Not 500 but close to 200)
As of how often I can again assure you that we were doing that 4 or 5 times per day.

Now to the part of if that's possible, Idk but I just hope.

francis_baud

DMKano said:

This is all pretty spot on - but I do want to point out one different thing with voxel based games - and that is voxel data stream overhead.

So 1 client sends actions to server - server sends back that client's action to the other 499 clients (in a 500 player battle) - yep minimal traffic.

Now add in destructible/buildable world geometry in Voxel MMOs - suddenly you start to choke at around 50 players due to huge voxel overhead.

CU uses voxels - to what extent I don't know - maybe they can pull it off

Afaik CU doesn't use voxels.

collekt

Torval said:

erevus said:

DMKano said:

It's possible - how FUN that ends up being is an entirely different point.

Also how often that happens due to playerbase numbers is another issue.

Well as a past loyal player of WAR, I can assure you that is a HELL OF A FUN participating in such a huge battles (Not 500 but close to 200)
As of how often I can again assure you that we were doing that 4 or 5 times per day.

Now to the part of if that's possible, Idk but I just hope.

Less than 200 is not what we're talking about. We're talking triple, quadruple, or more players on the screen at once. The floor is 500 people. There were numbers thrown around like 1,000 to 2,000 people on screen at once.

What happens to that fun when you get ten or twenty times that many people on the screen. Is your contribution lost in a sea of chaos? That would be realistic, but will it be fun? What will it look like and what will it mean? I think those are pretty important questions to consider.

I'm keeping an open mind. I've seen some amazing software. I'm also waiting to see how it works in reality in the open wild.

Pretty sure they said they'd like to be able to handle 1000+ player battles, but not that they were designing the game in such a way that they would be happening often. Just because they can support that many doesn't mean you're typically going to see it.

MarkJacobs

Just jumping in here for a quick post because it's getting late and it was a looong day. So I'll keept it fairly short (for me at least) and sweet.

1) Is 500 player battles possible - Of course it is. As noted above, Planetside 1 & 2 had larger battles than that. Yes, they pushed a lot of stuff to the client to achieve that (we don't) so it's not totally on point but it is a useful data point.

2) As of our last internal video we had 1500 Bots running around without any major lag and my FPS was well over 30 at the time (3 year old normal PC, 1 GTX970). Now, because it was just Bots and not Bots + VFX + abilities that too is a data point but not proof of 1K battles yet. I was running around in the video through the moshpit of 1500 Bots and performance was great. Perfect? Nope, but my FPS always stayed high enough to play normally and the UI was responsive.

3) There have been some good explanations (not all were totally correct) about the issues with large-scale battles in games. It really boils down to 3 major points: how hard your PC has to work, how hard your video card has to work, and how hard our servers have to work per frame (FPS). We need to be able to pass a ton of data through the network to your PC and back again and then both ends of the equation have to process that data and make magic happen on your screens so the game looks/feels/plays right. Really simplistic explanation I know (it's late, I'm tired) but it is accurate albeit missing tech talk/buzzwords/etc.  highlights one advantage that we have over most other game engines and that is that our engine is fully multithreaded on both the client/server side. Multithreading allows us to take advantage of all the power of the client PC/servers that run the game. This gets us more power to play with than the average game. As many of us know, multithreaded engines are a pain in the ass to create, a major PITA. This is one of the things that slowed us down a bit more than we hoped it would. OTOH, now with both AMD and Intel embracing the "moar cores plz!" philosophy, our engine is well-positioned for the continuing growth of the average PC but also for this game.

4) Bots, while not players, once they begin firing off abilities again, will then be closer to an actual player during an actual battle in terms of the network bandwidth they eat (they currently eat more than a player who is just moving around), and in terms of the VFX that are fired off that eat up the processing power of the card/PC. This will be the next major test of our engine and we're getting close (maybe this week) to having Bots back to their old selves again. In older tests we did have more than 1K Bots firing off abilities under the old ability system and the old animation system.

5) FYI, we aren't cheating in any way to get our numbers. Nobody here said it or hinted at it, I just wanted to make that clear. We are still using our server-side physics and non-friendly collision systems. 

6) There are ways to further optimize performance on the client side that we haven't begin to work with yet. We expect to some additional savings there once we do.

7) We couldn't have done this without building our own engine according to the key senior engineers at CSE and Andrew (who knows a wee bit about engines).  

So, that's a quick summary. Do I think we'll get to 500 people in a battle and have 30 FPS on less than a 10xx series card? Yes. Will we have it when you are sieging a castle and the castle is collapsing around you and lots of siege engines are firing? Yes. Can we get to 1K or more with lots of crazy VFX going off while multiple parts of the castle are collapsing? I think so but I can't attest to that yet because we're still implementing the next stage of our VFX system. Once we are in Beta we will move to the second stage of implementation for that system that will get us other speed increases. Plus, I know we can get a lot of savings with the VFX system once we start playing with optimizations. But, as you folks know, I'm not big on hype in terms of CU (I made that promise with this game and I've stuck to it) so until I actually see the new animation + ability + VFX/SFX systems working in complete harmony, I'll stick to the facts, and nothing but the facts.

Now, can things still go wrong? Absolutely. OTOH, our Bots do a really good job of beating on the important network/server based systems. And one of the things I'm budgeting for Beta 1 is setting up a local server cluster that runs the maximum number of Bots 24x7 with the latest builds of the game. That will shake out a lot of issues. We didn't have that tech nor cloud services for Dark Age nor WAR, though I wanted our own version of Bots for the later. If we had that capability then, things would have been different. I'm fortunate that we can do so for Camelot Unchained and that my co-founder Andrew, and the rest of the team, were totally behind it. Our Bot system will be pretty useful and unique when we are done with it,

Great question and some really great posts/analysis here, well-done all!

-Mark

P.S. Night all!

Gdemami

Torval said:

If the claim is true that no one else has done it before

Such claim is false.

cameltosis

MarkJacobs said:

Just jumping in here for a quick post because it's getting late and it was a looong day. So I'll keept it fairly short (for me at least) and sweet.

1) Is 500 player battles possible - Of course it is. As noted above, Planetside 1 & 2 had larger battles than that. Yes, they pushed a lot of stuff to the client to achieve that (we don't) so it's not totally on point but it is a useful data point.

2) As of our last internal video we had 1500 Bots running around without any major lag and my FPS was well over 30 at the time (3 year old normal PC, 1 GTX970). Now, because it was just Bots and not Bots + VFX + abilities that too is a data point but not proof of 1K battles yet. I was running around in the video through the moshpit of 1500 Bots and performance was great. Perfect? Nope, but my FPS always stayed high enough to play normally and the UI was responsive.

3) There have been some good explanations (not all were totally correct) about the issues with large-scale battles in games. It really boils down to 3 major points: how hard your PC has to work, how hard your video card has to work, and how hard our servers have to work per frame (FPS). We need to be able to pass a ton of data through the network to your PC and back again and then both ends of the equation have to process that data and make magic happen on your screens so the game looks/feels/plays right. Really simplistic explanation I know (it's late, I'm tired) but it is accurate albeit missing tech talk/buzzwords/etc.  highlights one advantage that we have over most other game engines and that is that our engine is fully multithreaded on both the client/server side. Multithreading allows us to take advantage of all the power of the client PC/servers that run the game. This gets us more power to play with than the average game. As many of us know, multithreaded engines are a pain in the ass to create, a major PITA. This is one of the things that slowed us down a bit more than we hoped it would. OTOH, now with both AMD and Intel embracing the "moar cores plz!" philosophy, our engine is well-positioned for the continuing growth of the average PC but also for this game.

4) Bots, while not players, once they begin firing off abilities again, will then be closer to an actual player during an actual battle in terms of the network bandwidth they eat (they currently eat more than a player who is just moving around), and in terms of the VFX that are fired off that eat up the processing power of the card/PC. This will be the next major test of our engine and we're getting close (maybe this week) to having Bots back to their old selves again. In older tests we did have more than 1K Bots firing off abilities under the old ability system and the old animation system.

5) FYI, we aren't cheating in any way to get our numbers. Nobody here said it or hinted at it, I just wanted to make that clear. We are still using our server-side physics and non-friendly collision systems. 

6) There are ways to further optimize performance on the client side that we haven't begin to work with yet. We expect to some additional savings there once we do.

7) We couldn't have done this without building our own engine according to the key senior engineers at CSE and Andrew (who knows a wee bit about engines).  

So, that's a quick summary. Do I think we'll get to 500 people in a battle and have 30 FPS on less than a 10xx series card? Yes. Will we have it when you are sieging a castle and the castle is collapsing around you and lots of siege engines are firing? Yes. Can we get to 1K or more with lots of crazy VFX going off while multiple parts of the castle are collapsing? I think so but I can't attest to that yet because we're still implementing the next stage of our VFX system. Once we are in Beta we will move to the second stage of implementation for that system that will get us other speed increases. Plus, I know we can get a lot of savings with the VFX system once we start playing with optimizations. But, as you folks know, I'm not big on hype in terms of CU (I made that promise with this game and I've stuck to it) so until I actually see the new animation + ability + VFX/SFX systems working in complete harmony, I'll stick to the facts, and nothing but the facts.

Now, can things still go wrong? Absolutely. OTOH, our Bots do a really good job of beating on the important network/server based systems. And one of the things I'm budgeting for Beta 1 is setting up a local server cluster that runs the maximum number of Bots 24x7 with the latest builds of the game. That will shake out a lot of issues. We didn't have that tech nor cloud services for Dark Age nor WAR, though I wanted our own version of Bots for the later. If we had that capability then, things would have been different. I'm fortunate that we can do so for Camelot Unchained and that my co-founder Andrew, and the rest of the team, were totally behind it. Our Bot system will be pretty useful and unique when we are done with it,

Great question and some really great posts/analysis here, well-done all!

-Mark

P.S. Night all!

Thanks for stopping in Mark, much appreciated!

The big things I pulled out of your post:

1) You're trying to keep most of the computation server side. Definitely a good thing in terms of preventing hacking. Sounds like your bot system is close to simulating the amount of processing needed so sounds like you're nearly there. 

2) You've solved the bandwidth issue. From what you were saying, you can already simulate the amount of data transfer needed for 1500+ player battles, so that is one of the potential issues already solved. 

3) You personally use a GTX970... same card as me! Makes me feel more confident that the game will run well on "average" machines. 

4) Visual effects sounds like the last big thing to test and solve, but also sounds like you've got a lot more room for client-side optimisation which is where the burden of processing VFX is going to lie. 



Definitely feeling good about the future of this game

SEANMCAD

here is a little game created by a few people..no biggy kinda studio

Erevus

MarkJacobs said:

Just jumping in here for a quick post because it's getting late and it was a looong day. So I'll keept it fairly short (for me at least) and sweet.

1) Is 500 player battles possible - Of course it is. As noted above, Planetside 1 & 2 had larger battles than that. Yes, they pushed a lot of stuff to the client to achieve that (we don't) so it's not totally on point but it is a useful data point.

2) As of our last internal video we had 1500 Bots running around without any major lag and my FPS was well over 30 at the time (3 year old normal PC, 1 GTX970). Now, because it was just Bots and not Bots + VFX + abilities that too is a data point but not proof of 1K battles yet. I was running around in the video through the moshpit of 1500 Bots and performance was great. Perfect? Nope, but my FPS always stayed high enough to play normally and the UI was responsive.

3) There have been some good explanations (not all were totally correct) about the issues with large-scale battles in games. It really boils down to 3 major points: how hard your PC has to work, how hard your video card has to work, and how hard our servers have to work per frame (FPS). We need to be able to pass a ton of data through the network to your PC and back again and then both ends of the equation have to process that data and make magic happen on your screens so the game looks/feels/plays right. Really simplistic explanation I know (it's late, I'm tired) but it is accurate albeit missing tech talk/buzzwords/etc.  highlights one advantage that we have over most other game engines and that is that our engine is fully multithreaded on both the client/server side. Multithreading allows us to take advantage of all the power of the client PC/servers that run the game. This gets us more power to play with than the average game. As many of us know, multithreaded engines are a pain in the ass to create, a major PITA. This is one of the things that slowed us down a bit more than we hoped it would. OTOH, now with both AMD and Intel embracing the "moar cores plz!" philosophy, our engine is well-positioned for the continuing growth of the average PC but also for this game.

4) Bots, while not players, once they begin firing off abilities again, will then be closer to an actual player during an actual battle in terms of the network bandwidth they eat (they currently eat more than a player who is just moving around), and in terms of the VFX that are fired off that eat up the processing power of the card/PC. This will be the next major test of our engine and we're getting close (maybe this week) to having Bots back to their old selves again. In older tests we did have more than 1K Bots firing off abilities under the old ability system and the old animation system.

5) FYI, we aren't cheating in any way to get our numbers. Nobody here said it or hinted at it, I just wanted to make that clear. We are still using our server-side physics and non-friendly collision systems. 

6) There are ways to further optimize performance on the client side that we haven't begin to work with yet. We expect to some additional savings there once we do.

7) We couldn't have done this without building our own engine according to the key senior engineers at CSE and Andrew (who knows a wee bit about engines).  

So, that's a quick summary. Do I think we'll get to 500 people in a battle and have 30 FPS on less than a 10xx series card? Yes. Will we have it when you are sieging a castle and the castle is collapsing around you and lots of siege engines are firing? Yes. Can we get to 1K or more with lots of crazy VFX going off while multiple parts of the castle are collapsing? I think so but I can't attest to that yet because we're still implementing the next stage of our VFX system. Once we are in Beta we will move to the second stage of implementation for that system that will get us other speed increases. Plus, I know we can get a lot of savings with the VFX system once we start playing with optimizations. But, as you folks know, I'm not big on hype in terms of CU (I made that promise with this game and I've stuck to it) so until I actually see the new animation + ability + VFX/SFX systems working in complete harmony, I'll stick to the facts, and nothing but the facts.

Now, can things still go wrong? Absolutely. OTOH, our Bots do a really good job of beating on the important network/server based systems. And one of the things I'm budgeting for Beta 1 is setting up a local server cluster that runs the maximum number of Bots 24x7 with the latest builds of the game. That will shake out a lot of issues. We didn't have that tech nor cloud services for Dark Age nor WAR, though I wanted our own version of Bots for the later. If we had that capability then, things would have been different. I'm fortunate that we can do so for Camelot Unchained and that my co-founder Andrew, and the rest of the team, were totally behind it. Our Bot system will be pretty useful and unique when we are done with it,

Great question and some really great posts/analysis here, well-done all!

-Mark

P.S. Night all!

Thx For stepping in Mark and share some info.

Bring back RvR.

Gavyne

I still have my doubts about smooth 500 players gameplay, especially when we factor in the things CU wants to do with combat & spell abilities.  CU's combat isn't aiming to have normal melee sword swings or single fireball casts that travel unhinged to its destination.  But like my sig says, CU is the game I'm waiting for, the only game I have funded on kickstarter.  

I would totally settle for enjoyable 150 player battles.  Because right now, most if not all of the popular MMO titles on the market can not handle more than 100 players without server-side lag, player warping, spell/combat cast delays, etc..  There are some fun games out there, but I don't play them as much these days because things become not fun when you get above 80-100 players.

Gdemami

Gavyne said:

I still have my doubts about smooth 500 players gameplay

Why?

https://www.youtube.com/watch?v=PvgjEMPfkYs

Like I said before, it is a question whether it is worthy because it comes with a price - graphics, animations, combat system etc.

Regardless, this is mostly a design problem.