Raph’s Website » Ultra-fast release cycles





By N2H

blog · games · writing · music · art · shop · about Raph · contact

Ultra-fast release cycles

March 16th, 2006 (Visited 15644 times) Tags:

Thought this post over at Creating Passionate Users was interesting. What if we did patch nearly daily in our MMOs? Granted, there’s such a higher level of interdependence in virtual world code than in even MySpace’s code that testing would be highly problematic, but perhaps if things are only in the game very briefly, you could minimize that.

Would MMOs be better run at warp speed, sort of like how riding a bicycle is easier the faster you are pedaling? Hmm.

You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.

26 Responses to “Ultra-fast release cycles”

Jump to reader comments » | Leave a reply »

Trackbacks & Pingbacks

ben barren - rss'ing down under wrote on March 18th, 2006 at 8:50 am:

[IMG ] a virgin Originally uploaded by benbarren. An interesting mother daughter interaction about MySpace and the implications for2 week development cycles which our tech team are into. Kathy Sierra asks MySpacing daughter “why didn’t she fall in love with LiveJournal? Her answer is a lesson for software developers (especially Web 2.0-ers), and was a theme of SXSW: “myspace keeps doing what everybody

Creating Passionate Users wrote on March 17th, 2006 at 6:29 pm:

So, quick release cycles and a new plane of existence. I have to think about this some more… I’d love to hear your thoughts about any of this! [UPDATE: there's a lively discussion about how this relates to game development, onRaph Koster's blog (an excellent regular read for those interested in game design/dev)] [FYI -- I'm still not current with my emails. I'm working on it, and I'm home now for the next six weeks, so I WILL catch up.]

Mischiefblog » Blog Archive » Open Kimono MMO wrote on March 17th, 2006 at 10:23 am:

[...] Finally, consider rapid iterations and spend world building time when it’s paid for and immediately evaluated by subscribers rather than years before it will be used. Or, consider procedural world generation–it’ll have less character unless tweaked by hand, but can tremendous time (and money) when developing your world. [...]

#noisier.net: <punk> if i had a rocket-powered panda i could ride it to the moon. (2006-03-17 15:03) wrote on March 17th, 2006 at 3:28 pm:

[...] Raph’s Website » Ultra-fast release cycles http://www.raphkoster.com/2006/03/16/ultra-fast-rele... posted by majcher-w at 2006-03-17 15:17# [...]

www.kooxoo.com wrote on April 15th, 2006 at 7:47 am:

不错,推荐一个: 酷讯火车票、出租房、二手房、二手车生活信息搜索引擎

kooxoo.com

soso�� wrote on May 24th, 2006 at 11:22 pm:

[...] Raph��s Website �0�3 Ultra-fast release cycle [...]

Reader Comments

Wolfe said on March 17th, 2006 at 12:19 am:
If built with this in mind yes. Getting to a point where you are allowed to spend budget on building this way is probably harder than actually doing it.
Jim said on March 17th, 2006 at 12:20 am:
Pandering to your customer base’s every whim works in a toolset, but wouldn’t work so well in a game. They’re based on capability growth on the part of the player, not on the part of the game.

That said, quick turnaround would serve MMOs well in many ways. Bug fixes, for one. Assign someone to typos, wiping out or fixing broken quests as soon as they’re found, and producing a constant stream of new low-risk content, and you’d probably have much happier customers.

As far as changing major systems daily– well, my gut reaction is that that would be a test and integration nightmare. I don’t know enough about MySpace to say if their systems are interrelated enough, or deep enough, to make those sorts of risks inherent.

Also– things in games develop meaning as they endure over time, and with familiarity. Changing things every day — rendering the past meaningless — would destroy a lot of the appeal of durable-world games.

On the other hand, if for you meaning only exists in the moment, as it often does for children and teenagers (and anyone else who’d say if you weren’t jacked in, you didn’t exist) constant breaking with the past is immaterial. It’s in fact good, because it’s the only way you fulfill desires for what you didn’t have in the last instant.
Damien Neil said on March 17th, 2006 at 12:32 am:
In my experience, players are very forgiving of bugs that have been fixed. Nobody cares that something was broken yesterday; they’re focused on today.

When you have a patch process that leaves bugs in for weeks before the fix passes through the QA process, the players hate you for every bug. When you have a process that fixes bugs ten minutes after they’re discovered, nobody really minds.

So, yes, I think (most) MMOs would be better with a patch process that gets new and nifty features out quickly at the expense of testing, if it also lets the inevitable bugs be fixed instantly. A Tale in the Desert works much this way; just about every major feature added into the game was initially buggy, and yet most of the bugs were fixed within minutes of players finding them.
Andrew Krausnick said on March 17th, 2006 at 12:51 am:
I couldn’t agree more, if it were feasible, it would give a significant edge to whatever virtual world could pull it off (given that you’d need what amounts to a larger than average live team, by my understanding).

In my mind, the money quote is right here, “Skyler said she’s seen things break on myspace, but nobody seems to care much since they know it’ll probably be fixed tomorrow.”

If new content were patched in daily, it would enhance the idea of a living and breathing world without requiring the difficult balancing act of emergent behavior.

For a solid example, look at WoW. Blizzard has been constantly streamlining their patching process since release, mostly due to very negative community feedback when the initial patch took a matter of months. The patch cycle/content generation is still their biggest weakness, but it is something they’ve been continuously making faster and faster.
Nick Lamb said on March 17th, 2006 at 4:11 am:
A Tale in the Desert is both a good and a bad example here…

Good, because ATITD really does stream changes into the game in real time. You find a bug, a developer teleports in to look at your bug, the game maybe stutters for a moment, and they say “How about now?” and it’s gone. They can do that because they knew they needed to when working on the very earliest versions, and once you have it you can’t live without it, so any features that would be forbidden by live streaming updates just don’t happen.

Bad, because ATITD’s development style is so haphazard that it annoys players even despite the fact that they know they won’t usually have to wait a month for a bug fix. First hand accounts of Andrew Tepper’s development style suggest that the idea of testing as part of the development process is in fact alien to him… So it becomes a joke. First people to build something will always see the magenta placeholder graphic instead of their fabulous building. First person to achieve any goal will crash out and be unable to rejoin for minutes, hours or days…
chabuhi said on March 17th, 2006 at 6:52 am:
As a player, I’m all for micropatching, but I’m not sure if I totally agree that players will be forgiving of bugs that might be introduced due to accelerated (shortcut) testing. SL had a new build out the other day and it seemed to introduce some lagginess that didn’t have the community well pleased.

In the end, by speeding things up this way you might wind up forcing MORE patches because you have to fix bugs more frequently. In a more moderate release cycle you will have had time to test out bugs (as much as possible, we hope) so you are more likely to avoid a “patch of a patch”.
Adrian Faulkner said on March 17th, 2006 at 7:25 am:
You know, if I am a Dwarf Hunter and there’s a bug where undead warlocks are unable to equip hats, I’m really not gonna care. But you can bet that if I am an undead warlock that I will be cursing to high heaven. All bugs are relative to the player. There are also different levels of bugs. Do they stop me playing the game or do they stop me accessing one part of a game? For example, I might not be able to sit on the couch, but I can still go and fight

There is a tendancy with larger companies to keep quiet about bugs, but you know if someone popped onto the forums and said “You know what, this Undead Warlock bug is actually quite complex so it’ll take us a month to fix” most people would moan a little but basically be OK. Too often companies are afraid of being honest with their customers with bugs, thinking people will moan. Well people might still moan but at least expectations have been set. There’s this common thought that secrecy is the best policy, but all that really does is give the impression that the company is either unaware of the bug or not bothering to fix it or worse doesn’t care. It might mean less moans on a forum but does nothing for customer satisfaction. They might not be moaning at the forums, but you can bet they’ll be moaning to all those people they know that they might otherwise recruit into a game. Companies think that if they remain silent no-one can complain, but in reality the opposite is true. Online communities foster 2-way communication, and if the devs aren’t playing an active part in that then it creates a weird “them and us” situation.

With the exception of exploits, there should be more transparency on bugs. Internally you prioritise these, assign them to a dev for a set number of hours. So internally you have an idea of when a bug should be fixed, so why can’t this be communicated. If timelines slip, you have to explain these to your management, so why can’t these be communicated to the players. Sure there will be those that moan, but people are genuinely reasonable. Online communities might deal in extremes but I do think they run as a temperature gauge of the feelings of your community. So if a bug slips and the community is in uproar, have your community team recognise this, and respond by raising the priority of this.

I see these community teams online posting just a couple of posts a day. As a player, I see pages and pages of complaints about a bug, and questions of when it will be fixed, and yet it’s weeks after this that a member of a community team responds saying “I don’t know, I will try and find out”. Now I’m sure they are hard workers but they are part of the bugs equation. They should know each ETA, each slippage, each bug and be able to be extremely responsive to the community. Again the companies have the tendancy to just keep silent, but you know in the best online communities, the devs play an active part in their communities. Yeah, sometimes this takes balls of steel, cos you can’t please everyone, but if you’re honest and open, then really, is there any comeback? They’ll be those that try and turn the words to suit their perception or personal feuds against devs, but you know, other players will see this. If you treat your community openly and fairly, the majority will treat you the same way, and the minority will be seen for the idiots they genuinely are.

As a player, if you are making a massive change to crafting in a patch, I can accept that there might be a bug in say the armor stats or the auction house system, or anything that I as a player would think logically connected. But you know, if patch after patch you keep breaking the same associated things, I’m gonna start questioning why your test plan does think to check these related items.

Likewise if you issue a patch for crafting, and as a result the emotes break, I’m gonna question why?

I have this idea that MMOs should be like lots of LEGO bricks with each resembling a game system (I think the phrase is modularized), so that if you change that brick, so long as you don’t change any of the connectors, things will still work

I’m really surprised by comments I’ve seen made by MMO devs about how hard it is to add content into a game. I’m not a developer (done some IT work but not in the gaming industry) but I would have thought that when building the support tools (i.e. the GM, CSR, Billing, etc mechanics) tools would be built to add content, after all it’s just a set of logic rules, isn’t it (OK, maybe some exception but I’m just talking your basic quests here). It’s been said that players play content quicker than devs can develop it, but if the tools were there, that could theoretically become untrue (1 dev working 8 hours x 5 days Vs a player (x the number of subscribers playing for Y hours a week)
Raph said on March 17th, 2006 at 9:08 am:

Well, it’s definitely interesting, for example, that by and large MMO companies aren’t willing post lists of known bugs. Some of this is for fear that they will be exploited in a multiplayer setting, of course, but many of them aren’t exploits, they are simply non-functioning features. There’s always a concern that showing off your dirty laundry will look bad.

On the other hand, it’s also possible that even though it may turn off new customers to see a laundry list of problems, it might enhance the experience of the long-term customers, who then at least know that their daily experience is something that developers are actually aware of.
Amaranthar said on March 17th, 2006 at 9:34 am:
As far as bugs, yeah, speed is very good.

Content is something I’ve been hitting on for a long time. But then, I’ve also been espousing that games need to get away from focusing on new quests and like features that revolve around grinding.

What if a game were built around living in the game space, and the new contect required could be new discoveries, GM events, surprise ambushes, and that kind of thing.

If this game had a well developed set of tools for the GMs (Seers, EMs, whatever) to use to quickly build live content in this “living” world, then it solves alot of the problems of testing and time consumption, because it’s already done. The GMs could build new stuff, give it direction, and it could remain invisible to the game world untill he hits the “Git’er Done” button.

New quests could be built, dungeons could be altered, new caves and contents added, whatever the GMs have in mind. And each thing would only take minutes to a few hours.

Think how much a single GM could add to a game in one eight hour shift. He’d be more limited by his ability to conceptualize new content than anything else.
Dan Lewis said on March 17th, 2006 at 10:01 am:
Via Bruce Schneier, a faster patch cycle would probably mean that MMO data miners like this guy from PlayOn would have degraded capabilities.

While the NSA has been capturing and analyzing international phone calls and electronic communications, with far less press coverage I’ve spent much of the last year collecting and helping to analyze data scraped from World of Warcraft as part of the largest quantitative study of virtual worlds to date. …

We deal with a semi-supported part of the game. The best documentation for the WoW API has been created by the modding community, but is still spotty in places, and wrong in others. Someone writing scraping code can easily become the expert — outside Blizzard — on some arcane portion of the game.

By far the most difficult problem we’ve faced is that the game itself is not static, but presents a moving target. Even our interest in and understanding of certain data changes over time. Patch day is a mad scramble to get the scraping software working again. Maybe the login code changed. Or, for example in the most recent release, information on whether a character was grouped or not disappeared from the API, breaking our software in the process.

If every day is patch day, could they keep up?

I don’t know how valuable this kind of information is to MMO companies, so I don’t know if the gains from a faster patch cycle would outweigh the losses in the ability to monitor players.
JesDer said on March 17th, 2006 at 12:53 pm:
As a customer I would actually like to see a laundry list of bugs to a game I was thinking about. Mainly because it would show some willingness for communication from the company to the players.

As far as patching, I feel that both styles have their own issues. However, I feel that a fast dev cycle can normally be worse when dealing with a MMO. As a gamer nothing upsets me more than a poorly designed patch. I dont care how long that patch took to make, but when it breaks core gameplay, you bet I am going to be upset. Sometimes the problem is not even a new bug, but a misguided effort at balance. At least if we were talking about a non-MMO game, I could just not patch my game untill whatever bug/issue is fixed.

Saga of Ryzom is the best example i cant think of about what can happen when a poorly designed patch is set loose on the players. The gameplay changed overnight and after a total outcry from the community for a revert, nothing was done. I remember loosing 80% of my guildmates within 2 weeks of this first major patch. This was not due to “bugs”, it was due to a total re-adjustment in both the fighting and craft parts of the game. It destroyed the risk vs reward balance and made the game impossible to play. I do not think there was any real gameplay QA done with the patch. Sure they might have tested to make sure they didnt add bugs, but I dont see how they could have done any actual gameplay testing with the changes they made else they would have seen the issue. It was a shame really, that game had a huge potential to become a great game, and it lost most of it’s small player base due to a few early patches.
Craig Huber said on March 17th, 2006 at 1:08 pm:
Question: is the quick-turn-around focus here on bug-fixes, or changes to the game?

Playing the devil’s advocate for a moment: “Patch Day” in my years of experience as a player is when one or more groups of individuals declare loud and long that the game has been utterly destroyed because Power A takes .25 seconds longer to fire, or Weapon B does 5% less damage, or Component C is now dropping only 25% of the time from Mob D (as opposed to 33% before), and so on.

Note that these are (usually) not “bugs” (except to the complaining parties who had apparently tailored their entire experience to this one facet of the game), they are changes.

Even acknowledging a distinction between changing the “physics” of the setting vs. changing details of the setting: do the vast majority of players truly want a ever-changing experience? How many howls of outrage or discouragement do short-term events regularly generate because they were “out of town and couldn’t participate”, etc.? And further, is it a simple matter of setting different expectations in the players up front, or is there a deeper issue at work with this kind of thing?

I personally like the idea of a more responsive update schedule, but given previous experience, I’m not 100% sure that desire is truly shared by the “masses”.
Michael Chui said on March 17th, 2006 at 1:55 pm:
http://headrush.typepad.com/creating_passionate_users/2006/03/be_surprising_y.html

Kathy Sierra wrote a follow-up, pointing that the MySpace devs she references are responding to what’s being asked for, but instead anticipating what they’ll want. This isn’t merely bug fixing or rapidly iterating new stuff. This is saying, “They’re going to want that to happen, so let’s do that.”

The rest of her post can be summed up in “Play your own game.”
Over00 said on March 17th, 2006 at 2:47 pm:
I’m really surprised by comments I’ve seen made by MMO devs about how hard it is to add content into a game.

I sometime gets the same comment from the people I work for (web applications, core business applications).

When you start a new project, you always tell yourself “This is gonna be the best software I’ll build, got to keep it clean”. Then you got deadlines, budget constraint, new critical features asked in late development process and more… It’s at that point that you somehow lost focus on your primary objective and that you end up with something “harder to improve”.

I still haven’t to met someone that is coding to put food on the table that hasn’t gone that path, at least in major development.
Jim said on March 17th, 2006 at 8:14 pm:
Anticipating an angle Raph may have originally meant –

If you can manage players’ expectations to the point where they expect gameplay changes on a regular basis, do you think they’d be more receptive to them?
Amaranthar said on March 17th, 2006 at 9:09 pm:
If you can manage players’ expectations to the point where they expect gameplay changes on a regular basis, do you think they’d be more receptive to them?

People only like changes if they, well, like them.

This bothers me a little, this talk of managing players. While I understand it to a degree, it’s important to remember that people don’t like to be managed. And they are keen to sense it, even if they don’t quite understand the details of what it is they don’t like. Of course there’s that whole social consent thing.

I think developers would be far better off to work with the goal of serving the players with a good game, instead of the players serving them to make a good game.
SBrodie said on March 17th, 2006 at 11:58 pm:
If you can manage players’ expectations to the point where they expect gameplay changes on a regular basis, do you think they’d be more receptive to them?

While not an MMO, GalCiv2 bases a lot of its success off of daily interaction with users, and constant content updates. I’ve seen similiar situtaions like those described in the article where bugs really become a non-issue because the game players are aware that a patch will be coming shortly. In fact, the content updates are used as a form of anti-piracy (loose installation requirements in exchange for the fact that users will want to use a serial to get all of the new content they would be missing out on otherwise). I think that this works for them because their audience isn’t on the scale of something like WoW (niche, if you will), and can cater to many more individual requests (protecting long term customers and keeping word of mouth positive). It should also be noted that their creators have a lot of experience in non-game software as well.

I am skeptical that a model like this could work well with a large community due to magnification of “just 1% complaining or leaving”, but would be very interested in seeing someone try to make it work. Great food for thought, anyway.
Max Battcher said on March 19th, 2006 at 1:13 am:
First of all, one of the MMOs that is silently making updates just about every day is the one you might least expect as it has no monthly charge: Guild Wars. (Proof: http://www.guildwars.com/aboutgw/gameupdates/default.php) Most players don’t notice game updates and there are fewer “NERFED!” yells after patches.

there’s such a higher level of interdependence

Isn’t that the real problem here? Not “what if we built MMOs with daily patching”, but “what if we built MMOs suitably decoupled to allow for daily patching”? The answer to the second is something that I tend to rant about, and its hard not to point to Guild Wars for at least having some very pretty technology. As an outside observer (and I would love to be inside someday), I am just amazed at how much poor design is “obvious” in the end results, and that get in the way when/should I try to “play”. Admittedly, I realize that not all design can be “perfect world”, but there are some fairly obvious things that can be controlled. For instance, one of my big gripes is in the use of these seperate RTPatch-based pre-game Patchers. Why should you, on every patch, apply deltas across all several GBs of a game directory? Why isn’t there better asset management than that? Not all of the art assets are going to be loaded into memory at once, on game start, are they? (…and if they are, you have even bigger problems. Bad asset management extends just as equally to RAM needs as well…)

I could go on, but I fear I might not get the point across: It’s very possible to have good release cycles if you have good coding standards, good asset management, and a good team, which in the end are keys to most Software Engineering efforts.
Raph said on March 19th, 2006 at 9:16 am:

I wasn’t referring to the technical challenges there, though your points are interesting ones. I was more referring to the fact that the design has a higher level of interdependence. The more systems you add, the more they tend to web together in some fashion, leading to emergent behaviors when something changes.

You could build a design that was more decoupled, and then your game will emphasize the “playing alone together” effect much more strongly; you would be there with other players, sharing a space, but not necessarily any gameplay.
Brask Mumei said on March 19th, 2006 at 9:55 am:
I’m for micro-patches, not so much for the benefit of the end user, but for the benefit of the programmers. It’s part and parcel of the “Every day is release day” mindset that is critical to avoid the slip into ball of unmaintainable code.

Of course, it is also anti-thetical to the game developer’s mindset which has been long focussed on “going gold” and shipping a fixed and final product. Which is why I give more credit to Live teams than to the initial developers. Anyone can build something. It takes true dedication and determination to successfully maintain something. This is also why I don’t like the mind set of having a different Live team from development team. They should be the same. There should be no “Crunch time” leading into the final release of an MMORPG, because MMORPGs are not sprints, they are marathons.

Interdependent systems don’t preclude micro-patches. Indeed, the best way to tackle an interdependent system is with micropatches. Learning from the effects of small changes is the best way to discover how those systems are interdependent. Because the changes are small, it is a lot less difficult to roll them back if they prove wrongheaded. On the other hand, if you try and change the interdependent system in one massive patch, it is almost guaranteed you’ll screw something up. The patch itself will be so interdependent you’ll have a difficult time figuring out what part of it screwed things up.

Finally, one key advantage of micro patches is that you can correct typos and similar stupid bugs *quickly*. This is essential if you want a game that seems polished to the end user. This may seem backwards to priority-based bug approaches, for the lowest priority bugs will likely be fixed fastest, but has a significant impact on the feel of the package as a whole.

Recent Trackbacks

Twitter @raphkoster

@cuppy It could be worse -- could be called Wang Chung. in reply to cuppy 13 hrs ago
@cuppy I see piles of people with active LinkedIn status update streams... someone must be using it that way! in reply to cuppy 4 days ago
Connected LinkedIn to Twitter. Let's see if it's annoying or not. :) 4 days ago
More updates...