Data entry, risk management and tacos: Inside Halo 4’s playtest labs

Schuh's road to this point began in 1992. Enrolled in the University of Washington, studying to be a social psychologist, he "almost but not quite" got his PHD when a friend turned him onto a job opening at Microsoft.

Since 2002, he's been in the Xbox group, touching most games that have come through. Collecting and analyzing data, managing risk and limiting player confusion and frustration.

Schuh's done well for himself. After working on Halo: ODST, Schuh graduated into a management role to oversee many of the projects in the User Research department. But when Halo 4 came around — then known under the codename "Midnight" — he stepped down from that role to focus on it.

That might seem counter intuitive to an outsider, but as he explains it, for someone who works in data collection and user research analysis, it's tough to find a better gig than Halo 4.

"The allure of not eating Top Ramen was pretty high," he says.

On the surface, it's Microsoft's most popular game. In an interview with Polygon, Microsoft Studios head Phil Spencer says it's a three billion dollar franchise and the "most important entertainment product in the company." Asked if it's the most expensive game Microsoft has made, he says, "Absolutely. Nothing's even close."

But it's also a franchise that has grown up with the User Research department, leaving behind 10 years of legacy data. It's a franchise that spills into other media, like comics and short films, each providing their own data for the group to analyze. It's a game made down the street, so development team members can easily visit the labs. And perhaps most importantly, it's the first game from a new development studio looking to mimic much of a blueprint established by another team over the past 10 years.

One of the most important tasks on Schuh's plate has been helping smooth over that transition. Gathering player data on what feels like old Halo, and what feels different. Seeing what players think of the changes. Assisting with making the new Halo feel enough like the old Halo, and enough like something new.

Today, he says, he's just watching for final tweaks and hoping it all works. Though really, he already knows. He's seen the close-to-final survey results, and knows how Halo 4's aggregate user scores compare to those of previous Halo games. He just won't tell us.

At 10:00 a.m. on July 15th, Schuh sets up in "the lounge," a control room where the User Research team and 343 developers watch live tests, for the second day of this two-day weekend test.

Down the hall in the labs, you can hear a hair drop on what looks like daily-shampooed carpet. The walls run neutral colors from floor to ceiling, everyone wears headphones, and the cubicle dividers are large enough so no one sees each other. It's a doctor's office.

The lounge is a frat house. Unevenly glued posters stick to the walls. Stacks of fighting game joysticks spill out from a storage closet in the back. In front sit two 50-inch TVs, each showing feeds from eight Xboxes running the game. Behind sit developers watching how players react. Back at the 343 office across town, more team members tune into live feeds.

Sit in the lounge for more than a few minutes and it feels like watching horse racing or NCAA basketball in a casino gambling pit, with Microsoft employees calling out players by their station initials.

"C04 knows what she's doing."

"B06, what are you doing? Oh my god no ... no! What are you doing?!"

"Don't go over there! Yes, keep doing that! That's what that blip means."

"Just turn to the left! To the left! To the left! ... or just fall off that cliff."

The 343 team members in the room claim they don't place bets on players, impossible as it may seem with all the data that gets thrown around. They're impressed with the diversity in today's test because four of the 18 participants are female. They note that, on average, a game's overall survey score goes up two-tenths to three-tenths of a point when the developers add music.

And they cheer for specific players — in today's test, B03 is the fastest, running towards enemies, using lots of quick melee attacks and at one point flying past enemies in a Banshee [one of the game's aerial vehicles] to save time.

"He's our rabbit," says Schuh.

At one point, a 343 designer notices B03 miss a weapon he wanted him to pick up, so he plans to move its location to make it more obvious. Most tweaks the team calls out are similarly straightforward.

"A lot of the stuff has been ironed out already," says Schuh. "This is sort of the final cruise. But yeah, we're definitely looking at a bunch of specific small things in each of the missions [today], and we're also just looking at how people are interacting with the Promethians, just to make sure that those are working perfectly."

"One of the things we're going to be looking at is a mission that's fairly late in the game, and it's had some issues with people understanding what the goals are, what they're trying to achieve," says Schuh, keeping things vague to avoid late-game spoilers. "It's one where it's not completely clear. We didn't have completely working waypoint markers to drive people to the right place. And so last time, we saw a lot of people wandering around being lost. We're trying to see if — we put some wav points in; we added audio — to see if that actually solved the problem."

In general, Holmes says, the Halo team values polish and consistency over features that may be exciting but only work 99% of the time.

"I think for Halo in particular, because there's a competitive aspect to it in multiplayer, it's really important to us to keep that consistency," he says. "So players can learn a skill, pick it up, and know that they can apply that skill without the game sort of overriding their expectation. And so philosophically, that is something that we really held very dear."

That same philosophy came into play when the team set out to balance the game's first mission. Internally known as M10 (where each level follows the convention: M10, M20, etc.), the mission initially threw more at players than they expected from a Halo game.

"We had all these different beats and events that were occurring simultaneously," says Holmes. "So you'd raise the blast shields, you'd see this mysterious planet, but you'd also see a Covenant fleet, and then the planet would open and start pulling you in, but at the same time the Covenant fleet would start boarding the ship. It was super exciting."

But players weren't able to follow what was happening.

"We thought we were having this amazing bombastic moment, and it just turned into noise," says Wolfkill.

So the team broke the scene into pieces, and spread them across different parts of the level, and the survey data came back higher in both 'comprehension' and 'user enjoyment.'

From Schuh's perspective, a job well done.

He says he tries to forget people are watching when he's in the room but feels pressure to play quickly so he can finish the game before the test ends. "You wouldn't go into a movie and leave three-fourths in and be like, 'I'll wait 'til it's out on DVD,'" he says.

He seems relatively understanding of the time he got stuck in the spoiler section. "There was distinctly one point where, from as far as I knew, that's where I was supposed to go. And it took me a few tries before I realized it was just this little small thing, and eventually I got it, and I was just like, 'Huh, well maybe someone should have pointed that out for me.'"

What does he think of the game?

"I think it's much more like Halo 1 than, say, Reach or 3, and to me that's a good thing," he says. "That was my favorite one ... I felt there were more open areas. Granted, there were a lot more closed-in ones, but the open areas have always been what I enjoyed out of the Halo games."

For Schuh and the rest of the team hoping to make the game feel like a natural transition from previous Halo games, this is the kind of thing they want to hear. Something that references old Halo, without being too familiar.

With that, Microsoft gives Peluso his four gratuity items (a year of Xbox Live, an Xbox Live points card, Forza 4 and Alan Wake) and sends him on his way home.