Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Fuzzy Avatars Solved! Please re-upload your avatar if it was fuzzy!

[Column] General: Prioritizing Bugs

SBFordSBFord Associate Editor - News ManagerThe Land of AZPosts: 16,614MMORPG.COM Staff Uncommon

Every beta tester should love to help the development team squash bugs. When the next patch comes, they eagerly look to see if theirs made the cut. In today's column, Matt Miller explains the prioritization process for bug squashing. See what you think before heading to the comments to chat.

When players see a new patch for their favorite MMO, the first thing many do is scour the bug list to see if their pet-peeve bug has finally been addressed. Especially if the patch is big, and there are a ton of fixes. More often than not, their “game-breaking” bug goes ignored for yet another cycle.

Read more of Matt Miller's Prioritizing Bugs.

image

Associate Editor: MMORPG.com
Follow me on Twitter: @MMORPGMom

image

Comments

  • Slapshot1188Slapshot1188 Boca Raton, FLPosts: 4,507Member Uncommon
    Where does complexity enter the picture?  And how about a FIFO process?  It seems to me that the complexity of a bug (and time to fix) should be a major determinant in the order they are approached. 

    "I should point out that no other company has shipped out a beta on a disc before this." - Official Mortal Online Lead Community Moderator

    Starvault's reponse to criticism related to having a handful of players as the official "test" team for a supposed MMO: "We've just have another 10ish folk kind enough to voulenteer added tot the test team" (SIC) This explains much about the state of the game :-)

  • ramdyramdy GuadalajaraPosts: 47Member

    I would add that fixing bugs in a live-world is a hard task which in some cases may bring new issues or new bugs so, another reason to omit those low-priority bugs or just leave their solution for a later major update which can be tested with more resources.

    Very nice reading, thanks for the article.

  • MMODesignerMMODesigner Guest Columnist Foster City, CAPosts: 7Member
    Originally posted by Slapshot1188
    Where does complexity enter the picture?  And how about a FIFO process?  It seems to me that the complexity of a bug (and time to fix) should be a major determinant in the order they are approached. 

    Usually stuff was either a day to fix or a week or more. The week long bugs were scary. It usually involved rewriting a core functionality in order to fix a bug, and that was likely to cause more bugs in the process. The long and the short is that usually time-to-fix never entered the picture.

  • maplestonemaplestone Ottawa, ONPosts: 3,099Member
    I like to imagine bugs sneaking around in the code trying not to aggro the devs.
  • OzmodanOzmodan Hilliard, OHPosts: 7,187Member Uncommon

    Problem with game developers today, they do not spend much time on testing.  Using the general public to beta test a game is like inviting the sharks in to the pool of food fish.  They spend most of their time finding loop holes to take advantage of, instead of actually reporting problems.   

    If a developer creates test code as they write the code, bugs are far and few between.  Believe it or not the extra time for this more than pays for itself by cutting down on fixing bugs in the code.

    Game developers use some of the industry's worst practices because of the threat of the money running out.  

  • DashiDMVDashiDMV Not Saying, QCPosts: 362Member

    Matt you should have talked about the pros and cons of giving incentives to players who find bugs.

     

    I remember when you guys in COH gave out the Bug Hunter badge and you had people like Beefcake writing 5 page cry letters saying how he deserved it and you guys owe him it and he cried and cried until you finally gave him one. I think you even made a joke about it in your comic book. I remember when people would talk about something they found like a glitch or a trick and it was always "Don't tell so-and-so they will sell you out for a badge." Seemed like a lot of positives and negatives from that.

    Nowadays all Priority A bugs are anything that affects the Cash Shop or your ability to buy from the Cash Shop.

     

  • ArglebargleArglebargle Austin, TXPosts: 1,416Member Uncommon
    Originally posted by Ozmodan

    ....

    Game developers use some of the industry's worst practices because of the threat of the money running out.  

    Game developers who run out of money do not fix any bugs at all.

     

    Older games (like CoH) often had sections of poorly documented code that no one presently on staff had any understanding of.  If the bugs were rooted in that area, they were approached with some trepidation.

    If you are holding out for the perfect game, the only game you play will be the waiting one.

  • QuizzicalQuizzical Posts: 14,783Member Uncommon

    "Because of this prioritization becomes a necessary tool, so before you get out the pitchforks as to why bug A was fixed over bug B in the latest patch, just ask yourself how would you prioritize it, looking at the big picture of the entire game as a whole?"

    My preference would be to say, why are you wasting your time creating new content that won't work right when the old content also doesn't work right?

    I can understand declining to fix bugs that are intractible to fix or that the developers can't reproduce.  I'm a lot less understanding when simple things go unfixed for months, even after being reported in ample detail that reproducing and fixing the bug should be a trivial manner.  Admittedly, sometimes it can be hard to tell how hard some things would be to fix, but sometimes even simple typos go unfixed for months.

  • QuizzicalQuizzical Posts: 14,783Member Uncommon
    Originally posted by Ozmodan

    Problem with game developers today, they do not spend much time on testing.  Using the general public to beta test a game is like inviting the sharks in to the pool of food fish.  They spend most of their time finding loop holes to take advantage of, instead of actually reporting problems.   

    If a developer creates test code as they write the code, bugs are far and few between.  Believe it or not the extra time for this more than pays for itself by cutting down on fixing bugs in the code.

    Game developers use some of the industry's worst practices because of the threat of the money running out.  

    One problem with developers testing things is that you can test your code and have it pass the test, but only glitch in situations that you didn't test.  The only practical way to get all of the arcane situations tested is to let a bunch of people play your game and wait for the bug reports to come in.

    A while ago, I found a problem in my game where crossing one particular line in the world in one particular direction would warp you into a wall off to the side.  Crossing the same line in the other direction worked fine.  Moving anywhere else in the game world that I checked worked fine.

    When I finally tracked down the bug, it was doing some computations in the wrong coordinate system entirely, and I was completely mystified as to how this only caused problems in one particular spot in the game world and nowhere else.  I had tested the code when I wrote it, and it worked fine with all of the tests that I did--but only bugged in one particular spot.

    Another problem is that shaders are much harder to test than CPU-side code.  For code running on a CPU, you can stop at any arbitrary step and output whatever data you want, to see if it's working the way you think it ought to.  You can't easily do that with shaders.  You input some data, then a bunch of steps happen on the video card and you can't stop it to see what is going on at any particular step.  There are several programmable stages and several fixed function stages that result in a final image on the screen.

    You can try to hack it with testing code to output something that is obviously wrong, just to see what it will do.  For example, I once changed geometry shaders to draw every triangle as covering most of the screen, just to see if any triangles made it that far or if everything was being killed by something earlier in the pipeline.

    Fortunately, graphical bugs tend not to be subtle, so if something doesn't work, it will probably be obvious that it doesn't work.  But it can be a pain to track down why it doesn't work.

  • waynejr2waynejr2 West Toluca Lake, CAPosts: 4,474Member Uncommon
    It's amazing how often devs miss bugs because they test the coding knowing how it's suppose to work only to have a not too bright end user stumble upon bugs/design flaws.  Dev blinders I call  it.
  • ArglebargleArglebargle Austin, TXPosts: 1,416Member Uncommon
    Originally posted by waynejr2
    It's amazing how often devs miss bugs because they test the coding knowing how it's suppose to work only to have a not too bright end user stumble upon bugs/design flaws.  Dev blinders I call  it.

    This happens in many fields.  It is why authors don't count on being able to proofread their  writing themselves....

    If you are holding out for the perfect game, the only game you play will be the waiting one.

  • ET3DET3D Posts: 227Member Uncommon
    Originally posted by Quizzical

    My preference would be to say, why are you wasting your time creating new content that won't work right when the old content also doesn't work right?

    "Since some bugs only affect a small number of players, and making new content affects a large number of players, those bugs lose the priority war and go another round of patching being unfixed."

     

     

    Anyway, nice seeing you around, Posi.

  • waynejr2waynejr2 West Toluca Lake, CAPosts: 4,474Member Uncommon
    Originally posted by Arglebargle
    Originally posted by waynejr2
    It's amazing how often devs miss bugs because they test the coding knowing how it's suppose to work only to have a not too bright end user stumble upon bugs/design flaws.  Dev blinders I call  it.

    This happens in many fields.  It is why authors don't count on being able to proofread their  writing themselves....

     What used to bug me was in a meeting you point out a flaw only to have it dismissed as me being to retentive only to have the problem rear it's ugly head later. 

  • RossbossRossboss Runes of Magic, TXPosts: 240Member
    I like the article. I think most beta "testers" aren't even there to help with the testing process. When I find a bug or a glitch that doesn't get fixed, I just tack it onto the next bug I find and keep adding to my list. I usually end up handing over a couple of lists that get longer as time progresses, and often times I toss ones that don't really effect my gameplay. Adding all information about how the bug occured and how the QA team can reproduce it is essential in my mind, although sometimes I wonder if they actually make use of it.

    I played WoW up until WotLK, played RoM for 2 years and now Rift.
    I am F2P player. I support games when I feel they deserve my money and I want the items enough.
    I don't troll, and I don't take kindly to trolls.

  • QuizzicalQuizzical Posts: 14,783Member Uncommon
    Originally posted by Arglebargle
    Originally posted by waynejr2
    It's amazing how often devs miss bugs because they test the coding knowing how it's suppose to work only to have a not too bright end user stumble upon bugs/design flaws.  Dev blinders I call  it.

    This happens in many fields.  It is why authors don't count on being able to proofread their  writing themselves....

    It's a simple case that if you made the mistake the first time and look at it later, you're likely to make the same mistake the second time.  Rather than seeing what you actually wrote, you can easily see what you meant to write.  If you want to proofread your own writing and mostly avoid that, you have to set it aside for a year or so and then come back when you've forgotten why you did things the way you did the first time.  For many writing situations, that's not practical--but getting someone else to proofread it for you is.

    Returning to programming, of course game developers will only test the cases that they considered when writing the code.  If you think of a case as something to test, then you'd think of it as something to handle properly in your code.  Sometimes there is no substitute for bringing in a ton of players and seeing if they break your code.

  • Originally posted by Quizzical

      Admittedly, sometimes it can be hard to tell how hard some things would be to fix, but sometimes even simple typos go unfixed for months.

    Typos that don't obscure intent are very low priority to fix. And can be tricky if it's "talk" from someone with an "accent".

    And another thing about content before bugfixes - the people who create don't always have the right skills/mindset to fix bugs. And while some bugs can be fixed by the people in charge of a given section of the game, sometimes it requires actual programer time to fix.

    Originally posted by Rossboss
    I like the article. I think most beta "testers" aren't even there to help with the testing process. 

    I blame games that used "beta test" as a "PR move"

  • Squeak69Squeak69 Colorado Springs, COPosts: 956Member
    Originally posted by Rajani_Isa

     

    Originally posted by Rossboss
    I like the article. I think most beta "testers" aren't even there to help with the testing process. 

    I blame games that used "beta test" as a "PR move"

    i agree with this, and i wish beta testing would go back to beta testing like the old days, where beta testers where their to tair the game apart rather then expect to have a good time and complain that things dont work right in a negitive way, instead of helping find whats wrong

    F2P may be the way of the future, but ya know they dont make them like they used toimage
    Proper Grammer & spelling are extra, corrections will be LOL at.

  • residentxresidentx Columbia, MDPosts: 88Member

    Another good topic,

     

    How large was the COH bug/tester team? I know nothing is unlimited but during periods in the COH, the MMO it seemed harder to maintain than others. Several questions I have about this:

     

    1. Was COH proactive with fix bugs? I know you had the test server but in the early days of the incarnate game, I had problems staying connected during raids. It took months to fix. I became so angry I left the game and waited for you to fix that before I came back.  

     

    2. What is the best way for players to submit bugs to the tester development team? I used the forums and trouble-ticket system but it still took months for my issues to get fixed even with detailed notes and pictures. I hated that the support team didn't even know the game well enough to really understand some of the problems I was experiencing. Sometimes, no support is better than poor support.

     

    3. How much time did you guys spend with 3rd parties making their products work with the game? COHtitan comes to mind. I used some of their products with the game and I wondered if their products might be causing some of the problems.

  • Originally posted by Quizzical

    My preference would be to say, why are you wasting your time creating new content that won't work right when the old content also doesn't work right?

    I can understand declining to fix bugs that are intractible to fix or that the developers can't reproduce.  I'm a lot less understanding when simple things go unfixed for months, even after being reported in ample detail that reproducing and fixing the bug should be a trivial manner.  Admittedly, sometimes it can be hard to tell how hard some things would be to fix, but sometimes even simple typos go unfixed for months.

    In part, it's a case, again, of a developer that runs out of money fixes no bugs. If you want to keep people interested, you have to have more/new content.

  • Originally posted by Quizzical
    Originally posted by Ozmodan

    Problem with game developers today, they do not spend much time on testing.  Using the general public to beta test a game is like inviting the sharks in to the pool of food fish.  They spend most of their time finding loop holes to take advantage of, instead of actually reporting problems.   

    If a developer creates test code as they write the code, bugs are far and few between.  Believe it or not the extra time for this more than pays for itself by cutting down on fixing bugs in the code.

    Game developers use some of the industry's worst practices because of the threat of the money running out.  

    One problem with developers testing things is that you can test your code and have it pass the test, but only glitch in situations that you didn't test.  The only practical way to get all of the arcane situations tested is to let a bunch of people play your game and wait for the bug reports to come in.

    Reminds me of COH when they found out how the "unique" enhancment system treated an enhancement being made unqiue and people already had many slotted.

    DELETE!

    And they only found out because of a bug where an item was marked unqiue in oneo fthe two places that mattered, and people noticed they could slot more. I recall laughing at the people who where upset when it was fixed, because how where they supposed to know the [UNIQUE] Tag ment it was supposed to be only one per build like every other IO tagged [UNIQUE]

  • residentxresidentx Columbia, MDPosts: 88Member

    Bugs are hard to catch...we all know this. Even in software engineering school now they don't really address testing...everything is about getting products to market.

     

    COH had problems with bugs but they weren't any worse than other games. I left COH twice for bugs and came back because the other games were worse. Have you played SWTOR? Have you never run around naked or got your lightsaber stuck? How about STO, ever had problems with the upstairs NPCs in the Starbase? They have never fixed these issues...even now!!  All of these were well-funded ventures. CoH even went back and redressed some of the content to satisfy users eventually. 

    Just remember what they taught you in school...opportunity costs. Trade-offs are a part of life...even in a mmorpg!  Matt demonstrated this with his explanation. My issue with COH was they should have just fired the technical GMs and their support teams. I just felt they added no value. They didn't know the game well enough to understand and  report issues and I found contacting them was such a waste of time. I would have been better posting in the forum and moving on.  

Sign In or Register to comment.