Tentonhammer did an article on Beta Testing’s Past, Present and Future last week and asked me for an opinion. I mostly commented on the budgetary side of it, after being asked if I thought big betas were going away any time soon, with the questioner having noted the change of tenor of betas over the past few years — Away from gameplay/assumption testing, and how they’ve become more of pre-launch marketing events.
For the AAA, eight- and nine- figure budget extravaganzas, big betas aren’t going away any time soon. What companies get out of them has shifted over time, but they remain an important part of getting a game out the door.
As product cost and complexity have increased, the emphasis of beta has indeed shifted toward toward marketing and load testing both your gameplay and operational systems. However, those are still critical activities in the high-budget, launch-big-or-die model. (That model has many weaknesses, but that’s an entire topic in itself.)
The reason this happened is simple - It’s about the money. Let’s say you’re a AAA game with 3-4 years of time and money invested, enough money to support a large team having worked on it for that long. Games like this frequently need to go for years before enough pieces come together before you can start making decisions about what’s fun and what isn’t.
By the time beta begins, you’ve made decision after decision that have compounded on each other. Your assumptions’ assumptions’ have assumptions about what your game is. The whole product, systems, content, operations, marketing, PR, community ramp, you name it — is built upon them. Changing core assumptions about the product itself is unlikely to be possible without significant delays, costing progressively more money per month. (Remember, the months toward the end of the dev cycle are the most expensive ones by far.)
The game is, for the most part, what it is. You’re capable of making shifts, but the more complex the game, the more minor the shifts you can make with any confidence. If assumptions that you made years ago turn out to be wrong, you’re left to scramble, or in most cases, do your best to ameliorate the now-certain fallout.
If you haven’t verified your gameplay at the point of having a beta, you’ve already left your fate to chance. (This is, of course, all presuming that your game has passed the technical bar in terms of stability, which is all too often not the case. And, again, is another flaw with the launch-big-or-die model.)
As budgets go up and schedules get longer, the model is growing more and more analogous to movies. If anything, people can see what goes on with blockbuster movie releases and draw certain comparisons.
No big beta? With a quality product at this stage in the industry’s evolution the negatives almost never outweigh the positives.
Unlike movies, seldom are there a half dozen launches competing for attention in the same month, much less the same week, where movies might have some competitive advantage to keeping secrets this late in the game. MMOs differ from movies in that they’re a long term time investment. The pattern of hype generation is different.
The way MMOs are most similar to movies, exploding costs aside, is that if you don’t see an advance reviewer screening for a movie: Something Has Gone Terribly Wrong. Bad news is being kept out of the market in hopes of keeping day-one sales high.
The same can be said for lack of betas, repeatedly late betas, or overly-restrictive betas for MMOs.
The company knows that early sales are now where the bulk of the money is going to come from, instead of huge usage numbers over time, and it’s doing what it needs to — preserving those precious day one revenues, since it could well need that money to survive.
The biggest underlying assumption that’s changed over time, largely thanks to World of Warcraft is that the dominant thinking used to be that “Getting too many people in before launch is going to hurt sales and subscriptions.”
After all - People made characters, levelled them up, got some loot, had their fun…just to get wiped at the end of beta. “The more people we do that to, the less who’ll want to buy the game, then come back to start all over at launch” — right?
WoW proved this to be a fallacy. For a good game, it turns out that there’s not just a higher tolerance for starting over than anyone imagined, but many people are actively interested in repeating their progress once a beta is over. Counterintuitive at first, but it does make a lot of sense. To a significant part of the core MMO audience…
Starting over feels like cheating.
One of the best things you can do in a game is to give people opportunities to feel like they’re cheating, or at least getting away with something, in a way that doesn’t make them feel guilty for doing it. Feeling like you’re pulling one over on the system is a good motivator.
It’s smart of developers to give people “safe” ways to derive that feeling from playing the game. Better they’re “cheating” by zooming through something they know than by becoming destructive cheaters - botters, hackers, and the like.
Given that, an outside observer can treat the size of the final beta as a referendum on the developer/publisher’s confidence in the game itself. That’s what I meant in the clip above referring to movie analogies.
It’s not a 100% correlation - But in general, the bigger the beta, the more confidence. The later and smaller the beta, the less confidence, and the higher internal pressure (usually driven by the cost) to get something, anything out the door, as a hail mary.
By now most of the core audience realizes…
The Miracle Patch doesn’t exist, and it never has.
Smart developers know that (enough of) their audience knows this, and are planning their beta’s progress accordingly. If you’re operating in the launch-big-or-die model, and you put an un-fun or unstable beta out on promises of a future patch coming out to Make Everything Awesome, people will see right through it, and you’ve just shot yourself in the foot.
If developers are so smart, why do un-fun betas still happen?
This is a tangent I wasn’t planning on getting into. Akil Hooper, who I worked with at SOE for a good many years, described one reason really well:
I think that one of the problems that MMOs have kind of inherent to the system is the length of development of tech. So many MMOs develop on new or untested/unproven tech that a lot of time and money is spent building a foundation that could very well be faulty.
By the time that beta comes around the meat and potatoes of the game hasn’t had enough time to marinate in the juices of fun, but the stock took so long to cool that you can’t throw it away. (Man, do I love metaphors)
Lots of non-MMO teams (maybe some MMO teams too for all I know) are working toward rapid iteration styles of development, instead of standard long pre-production and short final production cycles. This allows for them to taste the soup earlier and still have some time to change some of the basic flavors without ruining the broth too much.…
Expressed another way, one of the biggest reasons for un-fun betas is that there’s traditionally been far too much effort required from many other people until “fun” is even able to be evaluated at its most basic level.
The classic problem is compounded by the fact that designers are forced to continue progressively building more and more (on paper) upon unproven hypotheses (also on paper), until they end up with a 1,000+ page document of “Here’s the game we’re going to make once the architecture is in place.”
The risk of potential wasted effort increases geometrically the longer it takes to get to iteration.
Looking at who’s doing what in making a game…
Artists - Get to do some useful concepting, color keys, planning, before their production and (hopefully) continuous iteration, unless they’re being asked to make far too many assets in too little time. Most of their time spent is useful in the end.
Engineers - Also very likely that their earliest work will be useful, and will be the practical foundation for the product. Ends up being what a lot of the iteration point depends on.
Designers - Get “the time at the end” to find out if anything they’ve been planning is useful, usable, or fun. Frequently, “the end” is a fixed date on a calendar. Not to understate, but: This is a problem.
Smart teams making games all over realize this and are doing everything they can to push the iteration start point as early as possible in a game’s development cycle.
We’ve still got some distance to go until we’re entirely out of the Bad Old Days, but as Akil points out, it is getting better.
