The planning fallacy or why indie mmorpg developers always run late

Shaigh

Planning fallacy is best described for plans and forecasts that

• are unrealistically close to best-case scenarios
• could be improved by consulting the statistics of similar cases

On these boards we constantly hear about unrealistic expectations, recruiting was a lot harder than we thought, we lacked key personnel that made things go much slower than anticipated, it was a lot harder to adapt the game engine for our game, we had to revamp the combat model, we had to do a lot more testing than we anticipated. All of these things are caused by developers that thought that all tasks involved in the project would run under the best-case scenario.

Its not just optimism that cause projects to run late, at the start developers think of the bare minimum that needs to be added to the game and forget that mmorpg require far more parts than what anyone could think of at the start. Developers will also think of new things they want added to the game to make it as good as possible and feature creep always make things run late.

In many cases developers aren't innocent on the reasons why projects run late, unrealistic plans are driven by the desire to get the plan approved knowing that projects rarely gets abandoned unfinished.

Its a common pattern that people who have the information rarely feel the need to know the statistics. When exposed to the outside view they choose to ignore it because an insider knows more about the project than an outsider will ever know. When an outside observer predicts tasks completion they show a pessimistic bias, overestimating the time needed.


I came across it by chance yesterday and it immediately made me think of mmorpg kickstarters.

cameltosis

Planning for software development is a nightmare. I manage a team of software developers, my company builds websites so fairly straight forwards. It is my job to provide quotes to new customers and then schedule stuff in. 

My general rule of thumb is get the devs to quote, double it, then allow an additional 25% for overrun. 

Because you're right, the developers always consider the best case scenario and there are always unforeseen problems. In my company, we have tried moving away from time estimates with little success. So, instead of saying "the login page will take 4 hours to implement", we instead assign points to features. The easiest feature is given a value of 1, and all other features are compared to that. 

So, we might say the easiest feature is ingame mail, and give that a value of 1. We might then give crafting a value of 500, and combat 1000, with cosmetics a value of 100 or something. We then estimate how many points we could complete in a week / month, then extrapolate to find total time. It is supposed to result in more accurate quotes, because you are comparing complexity and size of features to one another, rather than estimating each item in a vacuum, but we've not noticed much improvement. 



That said, there are somethings I can easily forgive developers for. Recruitment is one of them - game programmers are a rare breed to begin with, so finding the right one for your business is going to be hard. Its such a small market so availability fluctuates a lot and this is completely out of the developer's hands. Producers, artists etc are more common, but actual programmers are rare. 

I can also forgive the desire to redo big features. No matter how much research and planning you do, sometimes a feature just isn't fun. Hell, Blizzard cancelled an entire MMO because it wasn't fun. These sorts of things can normally be caught in the design stages, but sometimes you need to get hands on to evaluate properly. I fully respect devs that can admit to something not being fun and then working to fix it. 


The other stuff, yeh, that should really have been caught during the planning stages. The one that annoys me most is feature creep. I can't stand delays due to feature creep.....if you want to keep developing new features, at least finish the base game and launch it first, then add the new features in patches later on. 

SpottyGekko

Shaigh said:

... When exposed to the outside view they choose to ignore it because an insider knows more about the project than an outsider will ever know. When an outside observer predicts tasks completion they show a pessimistic bias, overestimating the time needed.

...

A seasoned professional may quite likely know more about the inside of a project than any outside observer.

However, in the case of many indie MMO projects, the people involved in the project often have little to no experience of overseeing such a project. And when they promise to achieve what others (with experience) have not been able to achieve, they will naturally be greeted with suspicion...

CrazKanuk

Ok, so first of all, I don't know what you're meaning when you say "planning fallacy". Are you saying that there is a problem with the people planning the projects in general? Are you saying that the issue is with the community not understanding project management? 


1) You can't say that someone else's plan is too close to best case scenarios. A plan consists of a collection of tasks that YOU simply aren't privy to. So when people state that someone is unrealistic, it's based on a collection of knowledge of past development efforts. However, the actual USE of this information is almost never based in logic, but based in supporting one's own agenda. 

For instance, Star Citizen initially quoted 2014 as a release date. Compared against the CURRENT scope of the project, that date was unrealistic. HOWEVER, it will be used as ammunition when it suits the agenda of the person commenting on it. Similarly, people on the opposite side of the fence can argue that Star Citizen is still alright based on how it's tracking against comparable projects. Again, neither is really based in logic. Both are based on factual evidence, which people ALSO like to throw around, but neither is accurate based on the context of the project itself. 

However, that's not to say that the person isn't basing their estimates ON best case scenarios, which would be a mistake in and of itself, because shit rarely goes the way you think it will. Not evaluating risk factors, knowledge gaps, etc. is just asking for trouble. FURTHERMORE, I don't even know HOW you could ever create a schedule without the appropriate staff to comment on the time it will take them to do the work. It would be like me telling you how long it will take to build a custom hot rod, and I've never built a hot rod before ever, and then going out and trying to find someone who can build a hot rod in the time that I said. I just don't understand that. 

2) Schedules slipping is a reality. We've had a couple people chime in and basically confirm that. One actually told us that he is REWARDED for being late (as long as he's no more than 20% over time). Let me walk you through how that math works. 

a) Developers are asked for their opinion on a particular task. The build in 15% risk for accuracy purposes and deliver that to their lead/supervisor/scrum master
b) That person then brings that number to the PM and put another 15% buffer on top of that. 
c) The PM then takes that number up to their directors and puts another 15% on top of that. 

So by the time that a project actually reaches the directors, it's effective 50% inflated. On top of that, the PM will get a bonus if they deliver no more than 20% over their budget). This represents the buffer that the organization is placing on the projects at a high level. So we're up to 70% buffer. On top of that, there only really needs to be ANY sort of explanation given if they go 30% past budget, so you really have 80% buffer overall. 

SO!! when all is said and done, the "realistic" expectation of the organization is that the project will deliver at almost double the time compared to what the developer thought that the ACTUAL work was going to be. Now try to imagine that you're the one developing the schedule and you have no previous project management experience. If your mind wasn't just blown by that math then it probably means you're a PM and you're doing that on the daily. If you're a greener, though, and you don't throw necessary buffer in, then you're effectively fucked. Never mind that this is probably the most simplistic calculation of buffer that there is. It doesn't take into consideration ANYTHING that it should, such as risks, knowledge gaps, talent, resources, etc. Realistically, if someone hasn't run a project before, and they are dropped into something like an MMORPG, it won't end well. However, we already know that, don't we? If we're seeing veterans in the industry struggling to bring a project to market on time, then why we'd expect that someone who is effectively a noob would do any better is beyond me. Maybe someone, some time, will deliver a crowdfunded MMO on time, but until that happens I'll continue to be amazed at people who are amazed that these developers are missing schedules. 

Shaigh

CrazKanuk said:

Ok, so first of all, I don't know what you're meaning when you say "planning fallacy". Are you saying that there is a problem with the people planning the projects in general? Are you saying that the issue is with the community not understanding project management? 

snip 

Planning fallacy was first proposed back in 1979 so its a common problem with all sorts of projects and there are empirical studies on the subject. The second part of the title is that without an attention grabbing title no-one reads it.

I would make a longer response to your post but its all I have time with for now.

CrazKanuk

Shaigh said:

CrazKanuk said:

Ok, so first of all, I don't know what you're meaning when you say "planning fallacy". Are you saying that there is a problem with the people planning the projects in general? Are you saying that the issue is with the community not understanding project management? 

snip 

Planning fallacy was first proposed back in 1979 so its a common problem with all sorts of projects and there are empirical studies on the subject. The second part of the title is that without an attention grabbing title no-one reads it.

I would make a longer response to your post but its all I have time with for now.

Thanks! So LITERALLY planning fallacy. Sorry, it didn't read right in my head. Cool, thanks. 

kjempff

I usually go by this rule of thumb when predicting software projects (including games).. Find out out how fast it can be done given the information at hand and best case scenario, then multiply by PI. Very smooth projects with few unknowns and experienced people can be done with a 2.0 multiplier, but don't count on it unless you got a tried&tested team.

Tamanous

One clear reason why is because they have to build their team from near scratch. Hiring is a bitch and if you do not base your office in areas already strong in the industry (eg. US west coast and key places in Canada) you will have troubles building your team.

A few companies also seem to base their ease of development around the engine they choose. The issue tends to be the scaling of the tech to mmorpg size. This often impacts them. There is a very good reason why mmos have shrunk over the years: they use engines built for smaller games without the scaling and networking concerns.

Some of this is very realistic and why I am only confident of a very few indie developers will release viable products. Outside of mmorpgs, there are many crowd funded games that have released and done very well.

CrazKanuk

Tamanous said:

One clear reason why is because they have to build their team from near scratch. Hiring is a bitch and if you do not base your office in areas already strong in the industry (eg. US west coast and key places in Canada) you will have troubles building your team.

A few companies also seem to base their ease of development around the engine they choose. The issue tends to be the scaling of the tech to mmorpg size. This often impacts them. There is a very good reason why mmos have shrunk over the years: they use engines built for smaller games without the scaling and networking concerns.

Some of this is very realistic and why I am only confident of a very few indie developers will release viable products. Outside of mmorpgs, there are many crowd funded games that have released and done very well.

Hiring also becomes problematic because the industry isn't a cool and full of wonder as it used to be. So you actually find few people who you would class as a senior developer still in the industry. Fact is that there is plenty of work that is less stressful, pays better, and allows for a great deal more creativity. 

Shaigh

@CrazKanuk
#4. Its worse than just inexperience and there are plenty of projects where your original budget determines if you get the project or not. What makes it worse is that we might have the information told to us by someone we know but we still choose to ignore it. 

Imagine that you are 3 months in on your project which you now have estimated to take 2 years. In a conversation with your friend that is also a project manager you get information that others have done similar project with only 60% completing the project and it took them an average of 7 years. As icing on the cake you hear that your team isn't as experienced compared to the average team.

Given that your first 3 months have been running smooth you ignore the information you heard. In the end the project took 8 years and ended up running over budget by a huge margin. If you had reacted on the information you might have saved yourself the trouble, checked up on the information you got and acted on it. Problem is that since our project isn't in trouble yet we are very quick to disregard that sort of of information.

Octagon7711

If you cranked out anything on a regular basis it gets easier to gauge dates. For most, it's a first time venture and everyone's very optimistic.  Plus very few people tack on extra time to allow for the unexpected problems at zero hour.  Then there's the pressure to get it out as soon as possible.  As someone said in another thread.  People get sick, leave, you find out at the last min. that the guy with the great portfolio doesn't work well with others, etc. 

delete5230

Shaigh said:

Planning fallacy is best described for plans and forecasts that

• are unrealistically close to best-case scenarios
• could be improved by consulting the statistics of similar cases

On these boards we constantly hear about unrealistic expectations, recruiting was a lot harder than we thought, we lacked key personnel that made things go much slower than anticipated, it was a lot harder to adapt the game engine for our game, we had to revamp the combat model, we had to do a lot more testing than we anticipated. All of these things are caused by developers that thought that all tasks involved in the project would run under the best-case scenario.

Its not just optimism that cause projects to run late, at the start developers think of the bare minimum that needs to be added to the game and forget that mmorpg require far more parts than what anyone could think of at the start. Developers will also think of new things they want added to the game to make it as good as possible and feature creep always make things run late.

In many cases developers aren't innocent on the reasons why projects run late, unrealistic plans are driven by the desire to get the plan approved knowing that projects rarely gets abandoned unfinished.

Its a common pattern that people who have the information rarely feel the need to know the statistics. When exposed to the outside view they choose to ignore it because an insider knows more about the project than an outsider will ever know. When an outside observer predicts tasks completion they show a pessimistic bias, overestimating the time needed.


I came across it by chance yesterday and it immediately made me think of mmorpg kickstarters.

This is an easy one !!

I've been in manufacturing all my life, and the public doesn't need to know the process no matter how complicated and with all it's set backs.

The problem with video games are they are advertised WAY to early !!!!!!

The reason is simple Marketing !!!!!!....... Way to early doesn't help anything !!!!!!

 

However, "Early Access" changed this to a degree because the public needs to know if they are to help fund the project.  But even still, pre-pre Alpha ?..... Come on !

Here's your answer.

waynejr2

Shaigh said:

CrazKanuk said:

Ok, so first of all, I don't know what you're meaning when you say "planning fallacy". Are you saying that there is a problem with the people planning the projects in general? Are you saying that the issue is with the community not understanding project management? 

snip 

Planning fallacy was first proposed back in 1979 so its a common problem with all sorts of projects and there are empirical studies on the subject. The second part of the title is that without an attention grabbing title no-one reads it.

I would make a longer response to your post but its all I have time with for now.

And since you know this has been around since 1979, are you surprised it happens to mmoRPGs?  Or are you just trying to pass of this information to show off?

VengeSunsoar

It's difficult to get investors unless you sure there is an interest. So they must advertiser early.

 It's like that with many things. Want to build an apartment building. You can't get the investors till you have said contacts in place for about half the units already.

delete5230

VengeSunsoar said:

It's difficult to get investors unless you sure there is an interest. So they must advertiser early.

 It's like that with many things. Want to build an apartment building. You can't get the investors till you have said contacts in place for about half the units already.

You don't go out and build an apartment building if you don't have the money in the first place, at least a large chunk ! 

You don't build a car manufacturing facility, just because you have a good idea !   


Did you ever think, maybe a lot of video games shouldn't have been started ?

Shaigh

delete5230 said:

Shaigh said:

Planning fallacy is best described for plans and forecasts that

• are unrealistically close to best-case scenarios
• could be improved by consulting the statistics of similar cases

On these boards we constantly hear about unrealistic expectations, recruiting was a lot harder than we thought, we lacked key personnel that made things go much slower than anticipated, it was a lot harder to adapt the game engine for our game, we had to revamp the combat model, we had to do a lot more testing than we anticipated. All of these things are caused by developers that thought that all tasks involved in the project would run under the best-case scenario.

Its not just optimism that cause projects to run late, at the start developers think of the bare minimum that needs to be added to the game and forget that mmorpg require far more parts than what anyone could think of at the start. Developers will also think of new things they want added to the game to make it as good as possible and feature creep always make things run late.

In many cases developers aren't innocent on the reasons why projects run late, unrealistic plans are driven by the desire to get the plan approved knowing that projects rarely gets abandoned unfinished.

Its a common pattern that people who have the information rarely feel the need to know the statistics. When exposed to the outside view they choose to ignore it because an insider knows more about the project than an outsider will ever know. When an outside observer predicts tasks completion they show a pessimistic bias, overestimating the time needed.


I came across it by chance yesterday and it immediately made me think of mmorpg kickstarters.

This is an easy one !!

I've been in manufacturing all my life, and the public doesn't need to know the process no matter how complicated and with all it's set backs.

The problem with video games are they are advertised WAY to early !!!!!!

The reason is simple Marketing !!!!!!....... Way to early doesn't help anything !!!!!!

 

However, "Early Access" changed this to a degree because the public needs to know if they are to help fund the project.  But even still, pre-pre Alpha ?..... Come on !

Here's your answer.

The only reason we are at a forum like this is that we actually want to hear about games in production, we want to be excited about the new game in development, talk about how good the screenshots look, how the described game mechanics reminds us of the old mmorpg we played and loved.

When game developers come around here to interact with fans and talk about game mechanics we feel that they care about us, its an interaction that's beneficial for both developers and us.

The problem is that the public wants information that are semi-accurate but when indie developers speak of dates they have a tendency to throw around dates that are off by a big factor. When hearing those numbers its obvious that lots of indie developers have no idea how big and complex their project really is.

Gdemami

Shaigh said:

The problem is that the public wants information that are semi-accurate but when indie developers speak of dates they have a tendency to throw around dates that are off by a big factor. When hearing those numbers its obvious that lots of indie developers have no idea how big and complex their project really is.

Yep, because 'public' knows much better about game development than those who are actually working on the game.

The problem you talk about has a name - forums armchair developers.

Shaigh

Gdemami said:

Shaigh said:

The problem is that the public wants information that are semi-accurate but when indie developers speak of dates they have a tendency to throw around dates that are off by a big factor. When hearing those numbers its obvious that lots of indie developers have no idea how big and complex their project really is.

Yep, because 'public' knows much better about game development than those who are actually working on the game.

The problem you talk about has a name - forums armchair developers.

I can't tell you when a game will be finished but I can smell bullshit miles away.

Gaendric

In order to get any traction for their crowdfunding campaigns, projects nowadays have to overpromise to a high degree, only to then get a tiny fraction of the funds needed to actually turn those crazy promises into reality from the crowdfunding push.
At this point the projects are already in a tough spot, before development even started.

Sure, they can then try to get actual investors based on "look we did well in crowdfunding", but that puts them back to square one, defying the actual purpose of crowdfunding, which is being free of the beancounters and investors that push the game down the usual paths.

Gdemami

Shaigh said:

I can't tell you when a game will be finished but I can smell bullshit miles away.

No doubt, ignorance accompanied by lack of critical thinking provide no restraints to hold you back from w/e bullshit, embrace your airmchair !

Nanfoodle

I don't see a difference between triple A and crowd funded MMOs. They are always late and make big promises and only deliver half of them. Gamers always end up waiting and when a MMO releases, gamers cry fowl. Nothing to see here. 

VengeSunsoar

delete5230 said:

VengeSunsoar said:

It's difficult to get investors unless you sure there is an interest. So they must advertiser early.

 It's like that with many things. Want to build an apartment building. You can't get the investors till you have said contacts in place for about half the units already.

You don't go out and build an apartment building if you don't have the money in the first place, at least a large chunk ! 

You don't build a car manufacturing facility, just because you have a good idea !   


Did you ever think, maybe a lot of video games shouldn't have been started ?

The banks will not lend you the money until you have a certain percentage sold already. Just like other investments you need to prove the interest first.

That's why they do it. They have enough for a demo now they need to prove the interest to get get the investment.

Whether we think they should be built or not is irrelevant. The devs want it built. They need the money for it to be built. Therefore they need to show there is interest in it. Therefore they advertise.

Vrika

It's not really a fallacy. The plan is to give us too optimistic estimates to get more sales.

Experience has shown time and again that MMO players don't punish the devs by ceasing to buy until they've completed their missed deadline. Instead we buy more and more and more as long as the devs keep teasing us with things that are coming just around the corner.

At this point anyone half-competent running a crowdfunded MMO is able to tell that giving us real time estimates is wrong. Unless the estimates are missed by at least 100% they've been doing bad business by not giving us optimistic enough estimates.