Howdy, Stranger!
It looks like you're new here. If you want to get involved, click one of these buttons!
Quick Links
Camelot Unchained - 500 Player Battles - Possible?
MightyUnclean
Member EpicPosts: 3,531
I'm pretty tech-ignorant, so I'm looking for some expert opinions. The Kickstarter for Camelot Unchained states that their goal is to "Maintain an absolute minimum of 30 FPS in battles of up to 500 people." That's why they're building a custom engine: no one else, apparently, has built an available engine that can do this. My question is, is this even feasible with modern technology? For a small team working with a relatively small budget? Have others, especially AAA developers, tried and failed to achieve such a lofty goal? I'm wondering if problems achieving this base programming goal are part of the reason that the game is still in a pre-alpha state after four years. Again, I'm very ignorant of how this all works, so I'm looking for the opinions of programmers or network pros that know a lot about this.
Comments
Epic Music: https://www.youtube.com/watch?v=vAigCvelkhQ&list=PLo9FRw1AkDuQLEz7Gvvaz3ideB2NpFtT1
https://archive.org/details/softwarelibrary_msdos?&sort=-downloads&page=1
Kyleran: "Now there's the real trick, learning to accept and enjoy a game for what it offers rather than pass on what might be a great playing experience because it lacks a few features you prefer."
John Henry Newman: "A man would do nothing if he waited until he could do it so well that no one could find fault."
FreddyNoNose: "A good game needs no defense; a bad game has no defense." "Easily digested content is just as easily forgotten."
LacedOpium: "So the question that begs to be asked is, if you are not interested in the game mechanics that define the MMORPG genre, then why are you playing an MMORPG?"
The question is, whether it is worthy endeavor to pursue.
I have also worked in the games industry, albeit only for 6 months in QA, so I have a small amount of insight into games development and used to chat to our dev team quite a bit.
That said, I'm in no way an expert in game engines - I did build a physics engine for my university dissertation, but it sucked!
From my understanding of MMO games engines, there are 3 main areas of difficulty when scaling up the large amounts of players.
1) Data Transfer
Every time I perform an action in game, my PC has to transmit my actions to the server, the server has to acknowledge receipt, and then the server has to send my action plus any further outcomes to everyone who needs to know. In a 500 player battle, my 1 action has to be transmitted to 499 other players. This needs to happen quickly, otherwise the game feels unresponsive.
This is probably the easiest hurdle to overcome. The amount of data being transferred is actually small, it is just very frequent. You can optimise the data to keep it small (you're only transferring text messages essentially), you can probably compress it too and I imagine for security purposes, it is probably encrypted. Internet speeds are fast enough that bandwidth is unlikely to be an issue.
So, on data transfer, yes, 500+ is easily doable.
2) Data Processing
Lets say I use an AoE skill. Something has to process what my base damage is (so, need to process stats, skill modifiers etc), then calculate who is in range of me, then calculate the damage I do to those people. The more people involved in a battle, the more potential number crunching that needs to happen.
The primary challenge here is where does the processing take place and keeping it synchronised. If all the processing gets done server side, then it means you have greater control over the synchronisation (you can ensure all processing happens within the same GCD) and thus everyone gets the same results at the same time, but this does require some pretty beefy servers.
The alternative is to move some of that processing to the client. For example, when using my AoE skill, my client could process the base damage and then transmit that to the server, which would then determine who it hits. The server could then transfer the hits to my enemies clients, which could then calculate the actual damage caused.
This spreads the processing load out, however, it makes synchronising everything harder. If the processing happens on the server, then all calculations are handled at the point of arrival on the server, but by distributing the calculations, I get the first one on my machine, a second one on the server, a third on my enemies machine, plus the wait time for data to transfer from my pc -> server -> enemy pc -> back to server -> back to my pc.
That said, this is all achievable.
Computers are just big logic machines and combat mechanics are all logic based. A standard processor can process millions of calculations a second, so if you've properly coded your engine and have a suitable network of servers, there should be nothing standing in your way. It is still a challenge - optimisation of code is always a challenge - but I see nothing that would stand in their way.
3) Graphics
This, in my opinion, is the toughest nut to crack. When we experience lag in MMOs, it is hardly ever the server or network speeds that cause it, it is almost always graphics lag.
As I understand, when rendering a frame your computer has to first build a complete 3D representation of everything on your screen. This means calculating the relative positions of everything nearby by loading in the models, calculating positions and then sorting out things like overlaps / intersections etc. It then has to load in the textures for each face that is visible. Then, it will decide what you can actually see, then render that image. Then it applies post-processing effects like AA etc.
With small scale games, the developer remains in almost complete control of what needs to be rendered. The devs know all of the environment and can make accurate guessing as to the number of people and gear you will need to render at any given moment. This makes optimising easier. It is also the reason why so many games are getting smaller in terms of level design etc - by keeping it small, they can increase the graphics quality as it means processing similar amounts of data.
With an MMO, the developer is no longer in control. They can still control the environment, but the number of players is variable. It is far easier for your graphics card to process and render 1 person than it is 500+. This sort of graphics processing is also restricted to taking place on your PC - a server can't really help you out.
Now, there are various techniques that can be used to improve graphics performance. Sharing textures is one way - you don't need so much RAM to store textures if textures are shared, or using algorithms to process models, rather than loading a model itself (e.g. using tessellation). The easiest way is to simply reduce the polycount so there is less to process.
Admittedly, my knowledge of graphics is pretty old, so everything I've just written may be wrong or simply out of date. But, I do believe that graphics is going to be the hardest thing to maintain when it comes to large scale battles. It is not a problem that CSE can solve simply by throwing hardware at the issue, because the hardware is our own PC hardware and thus out of their control.
CSE also have an added complication with their CUBE system. This is going to allow us to design and build our own structures in the real world, so not only will CSE not control the number of players, they also won't control all of the environment.
I still remain confident that they will achieve this. Working in their favour is the relative simplicity of their environments and lack of NPCs. With trees, forests, hills, streams etc, they can super-optimise the way those types of environments are processed, freeing up more time for processing players and player structures.
I would definitely be curious to hear from CSE as to how their engine will handle the graphics of large scale fights.
Also, not sure if you're aware but it's not uncommon for MMOs to be in development for 5+ years. It just feels like a long time to you because companies don't normally announce their game on day one, before development has begun at all. They normally announce a few years after they start the development, so it feels like a shorter time before they are in beta/release.
Yes making their own engine is the big reason aside from difficulty hiring programmers until they opened the Seattle office for how long development has taken.
Redoing the ability system took like 6 months. That's a big part of why Beta 1 still hasn't started rather than having started late last year, but it doesn't explain the previous 2 1/2 years.
DAOC Live (inactive): R11 Cleric R11 Druid R11 Minstrel R9 Eldritch R6 Sorc R6 Scout R6 Healer
For GW2 in 3-way large scale WvW combat, things start to become a slideshow when you have more than 150-200 players in the same area. But FPS isn't your only problem when you go large scale, it's the response time. Calculations get choked up and you end up having server-side lag that delays your casts/melee combat. So you would cast a standard spell, you press it, and it doesn't fire off for at least 10 seconds. This server-side cast delay is worse than FPS issue.
Problem with CU is their ambition to create chained & reactionary combat. If you have been following CU, you know what they're trying to do with their spells casting. The first time I heard it, all I could think of is the bottleneck in computation during large scale combat. There will most definitely be server-side lag if they want to achieve spell on spell combat. Like they want you to be able to counter a spell that's traveling in the air, and make something else happen. All that creates an issue with server-side computation because the server will have to track every lil movement of the spells.
So while I'm a founder for CU and have been supporting them, I don't believe they'll be able to achieve 500 players at 30 fps. And like I said FPS isn't the only thing to worry about when you have that many players fighting, action delays become a huge problem, players warping becomes a problem. You see this issue in GW2 and BDO today when you cross a certain threshold of number of players.
I understand why they want to set the ceiling high, like MJ said before if they weren't ambitious then there would be no reason for them to do it. But I fear sometimes these smaller studios with small funding get too ambitious, it ends up hurting the game development as a result.
Played: EQ1-AC1-DAOC-FFXI-L2-EQ2-WoW-LOTR-VG-WAR-GW2-ESO-BDO
Waiting For: CU & Vanilla WoW
My suspicion is that they're going to keep as much computation on our client machines as possible. So, if we're all shooting siege weapons at a wall, the computations for damage, collision etc will be kept on our machines with just the result being sent to the server. The server then decides when to trigger collapses and stuff and just sends that small message back to the client, which then determines how the wall collapses.
I guess will depend on a lot on collision detection / despawning of voxels. If we destroy a wall, will the rubble be left on the ground? If so, will we have to jump over it or can we walk through it?
The downside is that by pushing the computation onto our machines, it does make the game more susceptible to hacking, but I'm sure they've thought of that.
The other possibility is that by creating their own engine, they may be able to make better use of multi-threading and scalable cloud computing. It may actually be fairly cheap for them to just have 100s of processors for their server setup, its just a case of having an engine that can take advantage of all that power.
This.
Stuff was said about DAoC as well; stuff and reality sometimes didn't match up 100%.
Also they are implementing wonderful security features that will prevent hacking that other MMO's are unable to address because of fundamental design flaws. Traditional wall hacking simply isn't possible with this engine.
If they can do the same with actual finished game player model's remain's to be seen.
A creative person is motivated by the desire to achieve, not the desire to beat others.
The things MJ wants to do with spells, like they want you to be able to counter/block a spell while it's in the air, and turn it into something. Then have someone else cast something else, and completely change the spells...making the spells do something completely different. These ideas are what will really increase server-side lag. Think about the amount of tracking the server has to do just for this one idea to happen.
Until proper combat/skills systems are in place, they won't be able to test this. Honestly for myself I would settle for a stable combat system with 100-150 players. I don't see how they can realistically make this happen with 500 players while doing all the things MJ said they want to do with combat.
Played: EQ1-AC1-DAOC-FFXI-L2-EQ2-WoW-LOTR-VG-WAR-GW2-ESO-BDO
Waiting For: CU & Vanilla WoW
DAOC Live (inactive): R11 Cleric R11 Druid R11 Minstrel R9 Eldritch R6 Sorc R6 Scout R6 Healer
Also keep in mind that after launch on actual servers there probably won't be 500+ player battles all that frequently, so the servers and clients will be able to handle the typical, smaller battles extremely well.
DAOC Live (inactive): R11 Cleric R11 Druid R11 Minstrel R9 Eldritch R6 Sorc R6 Scout R6 Healer
Well as a past loyal player of WAR, I can assure you that is a HELL OF A FUN participating in such a huge battles (Not 500 but close to 200)
As of how often I can again assure you that we were doing that 4 or 5 times per day.
Now to the part of if that's possible, Idk but I just hope.
― Terry Pratchett,
1) Is 500 player battles possible - Of course it is. As noted above, Planetside 1 & 2 had larger battles than that. Yes, they pushed a lot of stuff to the client to achieve that (we don't) so it's not totally on point but it is a useful data point.
2) As of our last internal video we had 1500 Bots running around without any major lag and my FPS was well over 30 at the time (3 year old normal PC, 1 GTX970). Now, because it was just Bots and not Bots + VFX + abilities that too is a data point but not proof of 1K battles yet. I was running around in the video through the moshpit of 1500 Bots and performance was great. Perfect? Nope, but my FPS always stayed high enough to play normally and the UI was responsive.
3) There have been some good explanations (not all were totally correct) about the issues with large-scale battles in games. It really boils down to 3 major points: how hard your PC has to work, how hard your video card has to work, and how hard our servers have to work per frame (FPS). We need to be able to pass a ton of data through the network to your PC and back again and then both ends of the equation have to process that data and make magic happen on your screens so the game looks/feels/plays right. Really simplistic explanation I know (it's late, I'm tired) but it is accurate albeit missing tech talk/buzzwords/etc. highlights one advantage that we have over most other game engines and that is that our engine is fully multithreaded on both the client/server side. Multithreading allows us to take advantage of all the power of the client PC/servers that run the game. This gets us more power to play with than the average game. As many of us know, multithreaded engines are a pain in the ass to create, a major PITA. This is one of the things that slowed us down a bit more than we hoped it would. OTOH, now with both AMD and Intel embracing the "moar cores plz!" philosophy, our engine is well-positioned for the continuing growth of the average PC but also for this game.
4) Bots, while not players, once they begin firing off abilities again, will then be closer to an actual player during an actual battle in terms of the network bandwidth they eat (they currently eat more than a player who is just moving around), and in terms of the VFX that are fired off that eat up the processing power of the card/PC. This will be the next major test of our engine and we're getting close (maybe this week) to having Bots back to their old selves again. In older tests we did have more than 1K Bots firing off abilities under the old ability system and the old animation system.
5) FYI, we aren't cheating in any way to get our numbers. Nobody here said it or hinted at it, I just wanted to make that clear. We are still using our server-side physics and non-friendly collision systems.
6) There are ways to further optimize performance on the client side that we haven't begin to work with yet. We expect to some additional savings there once we do.
7) We couldn't have done this without building our own engine according to the key senior engineers at CSE and Andrew (who knows a wee bit about engines).
So, that's a quick summary. Do I think we'll get to 500 people in a battle and have 30 FPS on less than a 10xx series card? Yes. Will we have it when you are sieging a castle and the castle is collapsing around you and lots of siege engines are firing? Yes. Can we get to 1K or more with lots of crazy VFX going off while multiple parts of the castle are collapsing? I think so but I can't attest to that yet because we're still implementing the next stage of our VFX system. Once we are in Beta we will move to the second stage of implementation for that system that will get us other speed increases. Plus, I know we can get a lot of savings with the VFX system once we start playing with optimizations. But, as you folks know, I'm not big on hype in terms of CU (I made that promise with this game and I've stuck to it) so until I actually see the new animation + ability + VFX/SFX systems working in complete harmony, I'll stick to the facts, and nothing but the facts.
Now, can things still go wrong? Absolutely. OTOH, our Bots do a really good job of beating on the important network/server based systems. And one of the things I'm budgeting for Beta 1 is setting up a local server cluster that runs the maximum number of Bots 24x7 with the latest builds of the game. That will shake out a lot of issues. We didn't have that tech nor cloud services for Dark Age nor WAR, though I wanted our own version of Bots for the later. If we had that capability then, things would have been different. I'm fortunate that we can do so for Camelot Unchained and that my co-founder Andrew, and the rest of the team, were totally behind it. Our Bot system will be pretty useful and unique when we are done with it,
Great question and some really great posts/analysis here, well-done all!
-Mark
P.S. Night all!
Mark Jacobs
CEO, City State Entertainment
The big things I pulled out of your post:
1) You're trying to keep most of the computation server side. Definitely a good thing in terms of preventing hacking. Sounds like your bot system is close to simulating the amount of processing needed so sounds like you're nearly there.
2) You've solved the bandwidth issue. From what you were saying, you can already simulate the amount of data transfer needed for 1500+ player battles, so that is one of the potential issues already solved.
3) You personally use a GTX970... same card as me! Makes me feel more confident that the game will run well on "average" machines.
4) Visual effects sounds like the last big thing to test and solve, but also sounds like you've got a lot more room for client-side optimisation which is where the burden of processing VFX is going to lie.
Definitely feeling good about the future of this game
Please do not respond to me, even if I ask you a question, its rhetorical.
Please do not respond to me
Thx For stepping in Mark and share some info.
Bring back RvR.
― Terry Pratchett,
I would totally settle for enjoyable 150 player battles. Because right now, most if not all of the popular MMO titles on the market can not handle more than 100 players without server-side lag, player warping, spell/combat cast delays, etc.. There are some fun games out there, but I don't play them as much these days because things become not fun when you get above 80-100 players.
Played: EQ1-AC1-DAOC-FFXI-L2-EQ2-WoW-LOTR-VG-WAR-GW2-ESO-BDO
Waiting For: CU & Vanilla WoW
https://www.youtube.com/watch?v=PvgjEMPfkYs
Like I said before, it is a question whether it is worthy because it comes with a price - graphics, animations, combat system etc.
Regardless, this is mostly a design problem.