Apr 22

Tentonhammer did an article on Beta Testing’s Past, Present and Future last week and asked me for an opinion.  I mostly commented on the budgetary side of it, after being asked if I thought big betas were going away any time soon, with the questioner having noted the change of tenor of betas over the past few years — Away from gameplay/assumption testing, and how they’ve become more of pre-launch marketing events.

Clip from the article:

 For the AAA, eight- and nine- figure budget extravaganzas, big betas aren’t going away any time soon.  What companies get out of them has shifted over time, but they remain an important part of getting a game out the door.

As product cost and complexity have increased, the emphasis of beta has indeed shifted toward toward marketing and load testing both your gameplay and operational systems.  However, those are still critical activities in the high-budget, launch-big-or-die model.   (That model has many weaknesses, but that’s an entire topic in itself.)

The reason this happened is simple - It’s about the money.  Let’s say you’re a AAA game with 3-4 years of time and money invested, enough money to support a large team having worked on it for that long.  Games like this frequently need to go for years before enough pieces come together before you can start making decisions about what’s fun and what isn’t.

By the time beta begins, you’ve made decision after decision that have compounded on each other.  Your assumptions’ assumptions’ have assumptions about what your game is.  The whole product, systems, content, operations, marketing, PR, community ramp, you name it — is built upon them.  Changing core assumptions about the product itself is unlikely to be possible without significant delays, costing progressively more money per month.  (Remember, the months toward the end of the dev cycle are the most expensive ones by far.)

The game is, for the most part, what it is.  You’re capable of making shifts, but the more complex the game, the more minor the shifts you can make with any confidence.  If assumptions that you made years ago turn out to be wrong, you’re left to scramble, or in most cases, do your best to ameliorate the now-certain fallout.

If you haven’t verified your gameplay at the point of having a beta, you’ve already left your fate to chance.  (This is, of course, all presuming that your game has passed the technical bar in terms of stability, which is all too often not the case.  And, again, is another flaw with the launch-big-or-die model.)

As budgets go up and schedules get longer, the model is growing more and more analogous to movies.  If anything, people can see what goes on with blockbuster movie releases and draw certain comparisons.

No big beta?  With a quality product at this stage in the industry’s evolution the negatives almost never outweigh the positives.

Unlike movies, seldom are there a half dozen launches competing for attention in the same month, much less the same week, where movies might have some competitive advantage to keeping secrets this late in the game.  MMOs differ from movies in that they’re a long term time investment.  The pattern of hype generation is different.

The way MMOs are most similar to movies, exploding costs aside, is that if you don’t see an advance reviewer screening for a movie:  Something Has Gone Terribly Wrong.  Bad news is being kept out of the market in hopes of keeping day-one sales high.

The same can be said for lack of betas, repeatedly late betas, or overly-restrictive betas for MMOs.

The company knows that early sales are now where the bulk of the money is going to come from, instead of huge usage numbers over time, and it’s doing what it needs to — preserving those precious day one revenues, since it could well need that money to survive.

The biggest underlying assumption that’s changed over time, largely thanks to World of Warcraft is that the dominant thinking used to be that “Getting too many people in before launch is going to hurt sales and subscriptions.”

After all - People made characters, levelled them up, got some loot, had their fun…just to get wiped at the end of beta. “The more people we do that to, the less who’ll want to buy the game, then come back to start all over at launch” — right?

WoW proved this to be a fallacy.  For a good game, it turns out that there’s not just a higher tolerance for starting over than anyone imagined, but many people are actively interested in repeating their progress once a beta is over.  Counterintuitive at first, but it does make a lot of sense.  To a significant part of the core MMO audience…

Starting over feels like cheating.

One of the best things you can do in a game is to give people opportunities to feel like they’re cheating, or at least getting away with something, in a way that doesn’t make them feel guilty for doing it.  Feeling like you’re pulling one over on the system is a good motivator.

It’s smart of developers to give people “safe” ways to derive that feeling from playing the game.  Better they’re “cheating” by zooming through something they know than by becoming destructive cheaters - botters, hackers, and the like.

Given that, an outside observer can treat the size of the final beta as a referendum on the developer/publisher’s confidence in the game itself.  That’s what I meant in the clip above referring to movie analogies.

It’s not a 100% correlation - But in general, the bigger the beta, the more confidence.  The later and smaller the beta, the less confidence, and the higher internal pressure (usually driven by the cost) to get something, anything out the door, as a hail mary.

By now most of the core audience realizes…

The Miracle Patch doesn’t exist, and it never has.

Smart developers know that (enough of) their audience knows this, and are planning their beta’s progress accordingly.  If you’re operating in the launch-big-or-die model, and you put an un-fun or unstable beta out on promises of a future patch coming out to Make Everything Awesome, people will see right through it, and you’ve just shot yourself in the foot.

If developers are so smart, why do un-fun betas still happen?

This is a tangent I wasn’t planning on getting into.  Akil Hooper, who I worked with at SOE for a good many years, described one reason really well:

I think that one of the problems that MMOs have kind of inherent to the system is the length of development of tech. So many MMOs develop on new or untested/unproven tech that a lot of time and money is spent building a foundation that could very well be faulty.

By the time that beta comes around the meat and potatoes of the game hasn’t had enough time to marinate in the juices of fun, but the stock took so long to cool that you can’t throw it away. (Man, do I love metaphors)

Lots of non-MMO teams (maybe some MMO teams too for all I know) are working toward rapid iteration styles of development, instead of standard long pre-production and short final production cycles. This allows for them to taste the soup earlier and still have some time to change some of the basic flavors without ruining the broth too much.

Expressed another way, one of the biggest reasons for un-fun betas is that there’s traditionally been far too much effort required from many other people until “fun” is even able to be evaluated at its most basic level.

The classic problem is compounded by the fact that designers are forced to continue progressively building more and more (on paper) upon unproven hypotheses (also on paper), until they end up with a 1,000+ page document of “Here’s the game we’re going to make once the architecture is in place.”

The risk of potential wasted effort increases geometrically the longer it takes to get to iteration.

Looking at who’s doing what in making a game…

Artists - Get to do some useful concepting, color keys, planning, before their production and (hopefully) continuous iteration, unless they’re being asked to make far too many assets in too little time.  Most of their time spent is useful in the end.

Engineers - Also very likely that their earliest work will be useful, and will be the practical foundation for the product.  Ends up being what a lot of the iteration point depends on.

Designers - Get “the time at the end” to find out if anything they’ve been planning is useful, usable, or fun.  Frequently, “the end” is a fixed date on a calendar.  Not to understate, but: This is a problem.

Smart teams making games all over realize this and are doing everything they can to push the iteration start point as early as possible in a game’s development cycle.

We’ve still got some distance to go until we’re entirely out of the Bad Old Days, but as Akil points out, it is getting better.

9 Comments so far »


    Nick McLaren said

    April 22 2009 @ 9:45 am

    Well said!


    Ferrel said

    April 22 2009 @ 10:34 am

    Great write up Scott and really on point. It isn’t every day that someone, more or less, airs all the dirty laundry. There are, most certainly, problems with how MMOs are developed and how users perceive that.

    Glad you did it and I’m happy to see an article from you!


    Brian 'Psychochild' Green said

    April 22 2009 @ 11:05 am

    One thing to consider is, “What is the purpose of the Beta?”

    The original purpose from the software field was to have the application battle-tested by real users under something approaching the actual situations it would be used. For a game, that meant letting real gamers in to play the game and see how the game fares.

    In most cases, however, if there’s a problem it’s too late to actually do anything about it. A launch date has been set and boxes are shipped to the store, so even something like server instability at higher populations wasn’t something that could stop the machinery once it was set in motion. Therefore, the last Beta period became more about marketing than actual testing, because any bugs found were probably going to be in the final version anyway.

    As far as betas ruining people from playing the game, there is some effect that I’ve seen first hand. First, if someone really goes in and tests a game throughly, there is the possibility of burning out on the game before it launches. Testing is necessary, but it’s not really all that fun. So, being asked to repeatedly test something over and over again can burn out even the most enthusiastic supporter; I saw this happen when we relaunched Meridian 59. The other thing that can happen is that people decide they want to play the game, so they don’t play the beta anymore because they don’t want to “waste time”. This can mean you don’t get good testing, or good word-of mouth, or possibly even enough people to make the world feel truly populated.

    Anyway, there’s my take on things. Nice to see you updating again, finally. ;)


    Wiqd said

    April 22 2009 @ 11:05 am

    Very well said. Glad you finally updated this blog :P


    Makaze said

    April 22 2009 @ 1:29 pm

    Well said.

    To expand on issues within the Engineer role a bit, I’ve often found that to hit milestones we’ve got to get things up and running NOW (or at least look like they’re running). Of course all of that undocumented and poorly laid out code ends up being refactored and cleaned up “someday”. So we end up heading into production with code that was never meant to be anything more than a prototype or tech test. That usually bites us in the ass…

    The lesson I think, get something up and running right now with a small team both from an engineering and a design standpoint. Doesn’t have to look good, doesn’t have to be easy to use but it should embody the core principles of your game. Then budget both from a time and resources standpoint assuming that you’ve have to completely start over. You’re not actually starting over mind you, you’ve learned some valuable lessons. But ditch the code and revisit all your code and design decisions to see what worked and what didn’t in a wholesale manner. Don’t just keep something crappy because 4 other systems depend on it. That applies to engineering and design, now is the time to change things.

    And that’s the sound of the beancounters dropping dead from heart attacks. The thing is it doesn’t really cost you all that much more in the long run since you’ve got less gristing and wasted work taking place at the end of development where your burn rate is 10+ times what it was. And you’ll often end up with a superior product.


    Jennifer Stavros said

    April 23 2009 @ 2:39 pm

    It’s good to hear from someone who actually has first hands experience talking about it to clear up common misconceptions. It’s a beta, it’s not always going to be this glorious thing… it’s a form of the test cycle not necessarily a promotion for the game. People tend to forget that and/or get greedy.

    Finding out what works, where you can improve, and if your product is worth investing the time (both financial ROI and potential for virility) to make it gold.

    Excellent post. You really should make some more often.


    Lovi said

    April 24 2009 @ 11:03 pm

    Having participated in a fair number of betas as a player and as a community volunteer, one of the things that seems missing from this discussion on beta testing’s past, present and future, is the question of whether or not betas will render better data and better pre-release commentary if the beta has been planned and designed to yield the best results from a group of beta testers, each member of which will be participating out of a variety of complex motives and testing history.

    I’ve yet to see a beta that had a comprehensive, consistent and fully used plan to coordinate the information needs of the developers and the recognition and response needs of the testers. Those two sets of needs, after all, are mutually entangled.

    While some individual developers can and do maintain excellent communication channels with testers, the betas I have participated in have usually suffered from an us/them dynamic that has closed channels of communication and discounted community understanding, intelligence and viable feedback (from the company side) and has evoked frustration, anger, bad behavior and negativity from the community side. The result? The company does not get the information it needs when it needs it. The community of testers becomes confused as to what is needed from them, at best, or clearly antagonistic and destructive, at worst.

    In the ten years since EQ1 was beta’d, there has been ample opportunity to study the results or lack of results from both beta testing and playtesting (which too often seem to be conjoined in such a way as to make the feedback and bugs coming in less useable) and to use those results to better plan and design beta tests. However, honestly, I don’t see any real step forward in game developer understanding of how to structure a beta test for the specific results needed by the game company and the responses needed to keep a community of testers involved and progressing, and reasonably positive.

    I’ve also not experienced a beta where there was a specific person at a high enough responsibility level to manage the beta from start to finish. Many betas in the past decade have suffered either from territorial politics within the company that kept information from getting to where it needed to go or from inconsistent communication from the game company to the testers and back. A person who manages the beta needs to be a neutral presence in the company (not someone beholden directly to designers or artists or engineers or marketers). However, a beta manager should also never be a doorkeeper. There would be too many ego and abuse of power issues involved. A beta manager position should also not be under the auspices of customer service or community management. An effective beta test should have a coordinator who knows the game inside and out and is able to understand the issues raised and correctly direct information to and from developers and testers.

    It might be fun to have a beta design competition for a game that is not real. The designer would have to not only plan for the information needed for the artists, engineers, designers, marketing and community development needs, but also would need to come up with a target group of beta testers with specific enough characteristics to try to integrate.

    I’d really like to be part of beta and/or playtest where I felt confident that the developers were asking the right questions, were clearly letting me know what goals needed to be met at a particular stage of the beta, and were recognizing fairly and respectfully the contributions of the testers.

    Just a comment on the “fun” factor. When EQ1 went into beta 1, there was almost no content. You could drown in Qeynos. You could kill some rats. But it was fun. Why was it fun? First, because the world itself, experienced from a very small available playing area, evoked a psychological response from players that the world was “real” and inviting. That is, players knew from the beginning that their imagination and intellect were going to be securely engaged in play. Second, because enough communication channels were in place that the free dynamic of the community added to the fixed dynamic of the coded game created an explosion of immersion and involvement by players even though the full experience of Norrath was months away.

    There have been a fair number of games recently that large enough numbers of people have not found fun, however, that perhaps adding in an evaluation of early immersion and community communication in beta could assist in assessment of whether or not a game is fun even in its most primitive form. Is it possible to tell very early on whether the world can successfully imprint itself on players so that they will, in turn, make it come alive and, in the end, have fun? It’d be worth finding out.


    John "Githil" Ruehs said

    May 1 2010 @ 5:59 am

    Very nice write up. I also think what you have stated shows one other thing that needs improved upon in beta’s. Having a huge beta helps developers with seeing how many people are interested in the game. It also helps developers see what is or isn’t working. The other “thing” I’m referring too is the gall to see what isn’t working and if it is going to take time to fix it to actually take the time to fix it.

    The Launch is the first impression most people will have with the game. If people know you pushed back the release date because you wanted to fix something that beta testers found not fun I’m sure they would be more understanding. I know it takes time and money. Investors need to understand that what you are fixing will make a better Launch and will make it so people will stick around longer in the game.


    Game By Night » The Beta Philosophy Behind Rift said

    December 3 2010 @ 7:34 am

    […] current HOURS away from Rift’s first big beta event, I found this article most interesting: MMO Betas: Tying Budgets to Beta Size to Production to Fun. In it, he touches on many aspects of the beta scene as we know it (marketing push), how it got […]

Comment RSS · TrackBack URI

Leave a comment

Name: (Required)

eMail: (Required)