Howdy, Stranger!
It looks like you're new here. If you want to get involved, click one of these buttons!
Quick Links
[Column] General: Prioritizing Bugs
SBFord
Former Associate EditorMember LegendaryPosts: 33,129
Every beta tester should love to help the development team squash bugs. When the next patch comes, they eagerly look to see if theirs made the cut. In today's column, Matt Miller explains the prioritization process for bug squashing. See what you think before heading to the comments to chat.
When players see a new patch for their favorite MMO, the first thing many do is scour the bug list to see if their pet-peeve bug has finally been addressed. Especially if the patch is big, and there are a ton of fixes. More often than not, their “game-breaking” bug goes ignored for yet another cycle.
Read more of Matt Miller's Prioritizing Bugs.
)
Comments
All time classic MY NEW FAVORITE POST! (Keep laying those bricks)
"I should point out that no other company has shipped out a beta on a disc before this." - Official Mortal Online Lead Community Moderator
Proudly wearing the Harbinger badge since Dec 23, 2017.
Coined the phrase "Role-Playing a Development Team" January 2018
"Oddly Slap is the main reason I stay in these forums." - Mystichaze April 9th 2018
I would add that fixing bugs in a live-world is a hard task which in some cases may bring new issues or new bugs so, another reason to omit those low-priority bugs or just leave their solution for a later major update which can be tested with more resources.
Very nice reading, thanks for the article.
Usually stuff was either a day to fix or a week or more. The week long bugs were scary. It usually involved rewriting a core functionality in order to fix a bug, and that was likely to cause more bugs in the process. The long and the short is that usually time-to-fix never entered the picture.
Problem with game developers today, they do not spend much time on testing. Using the general public to beta test a game is like inviting the sharks in to the pool of food fish. They spend most of their time finding loop holes to take advantage of, instead of actually reporting problems.
If a developer creates test code as they write the code, bugs are far and few between. Believe it or not the extra time for this more than pays for itself by cutting down on fixing bugs in the code.
Game developers use some of the industry's worst practices because of the threat of the money running out.
Matt you should have talked about the pros and cons of giving incentives to players who find bugs.
I remember when you guys in COH gave out the Bug Hunter badge and you had people like Beefcake writing 5 page cry letters saying how he deserved it and you guys owe him it and he cried and cried until you finally gave him one. I think you even made a joke about it in your comic book. I remember when people would talk about something they found like a glitch or a trick and it was always "Don't tell so-and-so they will sell you out for a badge." Seemed like a lot of positives and negatives from that.
Nowadays all Priority A bugs are anything that affects the Cash Shop or your ability to buy from the Cash Shop.
Game developers who run out of money do not fix any bugs at all.
Older games (like CoH) often had sections of poorly documented code that no one presently on staff had any understanding of. If the bugs were rooted in that area, they were approached with some trepidation.
If you are holding out for the perfect game, the only game you play will be the waiting one.
"Because of this prioritization becomes a necessary tool, so before you get out the pitchforks as to why bug A was fixed over bug B in the latest patch, just ask yourself how would you prioritize it, looking at the big picture of the entire game as a whole?"
My preference would be to say, why are you wasting your time creating new content that won't work right when the old content also doesn't work right?
I can understand declining to fix bugs that are intractible to fix or that the developers can't reproduce. I'm a lot less understanding when simple things go unfixed for months, even after being reported in ample detail that reproducing and fixing the bug should be a trivial manner. Admittedly, sometimes it can be hard to tell how hard some things would be to fix, but sometimes even simple typos go unfixed for months.
One problem with developers testing things is that you can test your code and have it pass the test, but only glitch in situations that you didn't test. The only practical way to get all of the arcane situations tested is to let a bunch of people play your game and wait for the bug reports to come in.
A while ago, I found a problem in my game where crossing one particular line in the world in one particular direction would warp you into a wall off to the side. Crossing the same line in the other direction worked fine. Moving anywhere else in the game world that I checked worked fine.
When I finally tracked down the bug, it was doing some computations in the wrong coordinate system entirely, and I was completely mystified as to how this only caused problems in one particular spot in the game world and nowhere else. I had tested the code when I wrote it, and it worked fine with all of the tests that I did--but only bugged in one particular spot.
Another problem is that shaders are much harder to test than CPU-side code. For code running on a CPU, you can stop at any arbitrary step and output whatever data you want, to see if it's working the way you think it ought to. You can't easily do that with shaders. You input some data, then a bunch of steps happen on the video card and you can't stop it to see what is going on at any particular step. There are several programmable stages and several fixed function stages that result in a final image on the screen.
You can try to hack it with testing code to output something that is obviously wrong, just to see what it will do. For example, I once changed geometry shaders to draw every triangle as covering most of the screen, just to see if any triangles made it that far or if everything was being killed by something earlier in the pipeline.
Fortunately, graphical bugs tend not to be subtle, so if something doesn't work, it will probably be obvious that it doesn't work. But it can be a pain to track down why it doesn't work.
Epic Music: https://www.youtube.com/watch?v=vAigCvelkhQ&list=PLo9FRw1AkDuQLEz7Gvvaz3ideB2NpFtT1
https://archive.org/details/softwarelibrary_msdos?&sort=-downloads&page=1
Kyleran: "Now there's the real trick, learning to accept and enjoy a game for what it offers rather than pass on what might be a great playing experience because it lacks a few features you prefer."
John Henry Newman: "A man would do nothing if he waited until he could do it so well that no one could find fault."
FreddyNoNose: "A good game needs no defense; a bad game has no defense." "Easily digested content is just as easily forgotten."
LacedOpium: "So the question that begs to be asked is, if you are not interested in the game mechanics that define the MMORPG genre, then why are you playing an MMORPG?"
This happens in many fields. It is why authors don't count on being able to proofread their writing themselves....
If you are holding out for the perfect game, the only game you play will be the waiting one.
"Since some bugs only affect a small number of players, and making new content affects a large number of players, those bugs lose the priority war and go another round of patching being unfixed."
Anyway, nice seeing you around, Posi.
What used to bug me was in a meeting you point out a flaw only to have it dismissed as me being to retentive only to have the problem rear it's ugly head later.
Epic Music: https://www.youtube.com/watch?v=vAigCvelkhQ&list=PLo9FRw1AkDuQLEz7Gvvaz3ideB2NpFtT1
https://archive.org/details/softwarelibrary_msdos?&sort=-downloads&page=1
Kyleran: "Now there's the real trick, learning to accept and enjoy a game for what it offers rather than pass on what might be a great playing experience because it lacks a few features you prefer."
John Henry Newman: "A man would do nothing if he waited until he could do it so well that no one could find fault."
FreddyNoNose: "A good game needs no defense; a bad game has no defense." "Easily digested content is just as easily forgotten."
LacedOpium: "So the question that begs to be asked is, if you are not interested in the game mechanics that define the MMORPG genre, then why are you playing an MMORPG?"
I played WoW up until WotLK, played RoM for 2 years and now Rift.
I am F2P player. I support games when I feel they deserve my money and I want the items enough.
I don't troll, and I don't take kindly to trolls.
It's a simple case that if you made the mistake the first time and look at it later, you're likely to make the same mistake the second time. Rather than seeing what you actually wrote, you can easily see what you meant to write. If you want to proofread your own writing and mostly avoid that, you have to set it aside for a year or so and then come back when you've forgotten why you did things the way you did the first time. For many writing situations, that's not practical--but getting someone else to proofread it for you is.
Returning to programming, of course game developers will only test the cases that they considered when writing the code. If you think of a case as something to test, then you'd think of it as something to handle properly in your code. Sometimes there is no substitute for bringing in a ton of players and seeing if they break your code.
Typos that don't obscure intent are very low priority to fix. And can be tricky if it's "talk" from someone with an "accent".
And another thing about content before bugfixes - the people who create don't always have the right skills/mindset to fix bugs. And while some bugs can be fixed by the people in charge of a given section of the game, sometimes it requires actual programer time to fix.
I blame games that used "beta test" as a "PR move"
i agree with this, and i wish beta testing would go back to beta testing like the old days, where beta testers where their to tair the game apart rather then expect to have a good time and complain that things dont work right in a negitive way, instead of helping find whats wrong
F2P may be the way of the future, but ya know they dont make them like they used to
Proper Grammer & spelling are extra, corrections will be LOL at.
Another good topic,
How large was the COH bug/tester team? I know nothing is unlimited but during periods in the COH, the MMO it seemed harder to maintain than others. Several questions I have about this:
1. Was COH proactive with fix bugs? I know you had the test server but in the early days of the incarnate game, I had problems staying connected during raids. It took months to fix. I became so angry I left the game and waited for you to fix that before I came back.
2. What is the best way for players to submit bugs to the tester development team? I used the forums and trouble-ticket system but it still took months for my issues to get fixed even with detailed notes and pictures. I hated that the support team didn't even know the game well enough to really understand some of the problems I was experiencing. Sometimes, no support is better than poor support.
3. How much time did you guys spend with 3rd parties making their products work with the game? COHtitan comes to mind. I used some of their products with the game and I wondered if their products might be causing some of the problems.
In part, it's a case, again, of a developer that runs out of money fixes no bugs. If you want to keep people interested, you have to have more/new content.
Reminds me of COH when they found out how the "unique" enhancment system treated an enhancement being made unqiue and people already had many slotted.
DELETE!
And they only found out because of a bug where an item was marked unqiue in oneo fthe two places that mattered, and people noticed they could slot more. I recall laughing at the people who where upset when it was fixed, because how where they supposed to know the [UNIQUE] Tag ment it was supposed to be only one per build like every other IO tagged [UNIQUE]
Bugs are hard to catch...we all know this. Even in software engineering school now they don't really address testing...everything is about getting products to market.
COH had problems with bugs but they weren't any worse than other games. I left COH twice for bugs and came back because the other games were worse. Have you played SWTOR? Have you never run around naked or got your lightsaber stuck? How about STO, ever had problems with the upstairs NPCs in the Starbase? They have never fixed these issues...even now!! All of these were well-funded ventures. CoH even went back and redressed some of the content to satisfy users eventually.
Just remember what they taught you in school...opportunity costs. Trade-offs are a part of life...even in a mmorpg! Matt demonstrated this with his explanation. My issue with COH was they should have just fired the technical GMs and their support teams. I just felt they added no value. They didn't know the game well enough to understand and report issues and I found contacting them was such a waste of time. I would have been better posting in the forum and moving on.