Online Matters Transcending the single Player Experience.

Monthly Archives: September 2011

Dungeon Defenders: Case Study, PAX Dev Videos, Monk Trailer

Lots of news to share this week about our partners over at Trendy Entertainment, creators of the indie hit Dungeon Defenders (which uses GameSpy Technology’s multiplayer matchmaking services to enable cross-platform co-op play on iOS, Android, PC, and PS3).

First up, for those looking for some deeper information about what Trendy did — and why we feel it was so important — we’ve published a case study on Dungeon Defenders here on the GameSpy Technology Web site. It covers: the basics of what Trendy implemented; how GameSpy Technology’s services manifest in the game; why Trendy felt it was crucial to adopt a cross-platform (rather than just a multi-platform) strategy for the game; and what players can expect from future versions of Dungeon Defenders.

For those who want to go SUPER in-depth on Trendy’s implementation of our tech — and beyond — we’ve also uploaded “shaky cam” video footage to our YouTube channel from our PAX Dev 2011 panel discussion with Trendy’s Jeremy Stieglitz and Philip Asher (moderated by GameSpy Technology Director of Marketing Sean Flinn). Why “shaky cam?” PAX Dev is blacked out to the media and provides no taping services of its own; so, it was up to us to record video ourselves. The room wasn’t exactly lit for filming (and we have no pretensions to greatness as film makers), so the visual quality is a bit dim. Still, tons of useful information in here for developers of all levels — particularly indies and those working in the mobile space. You can watch part 1 of 5 right here:

Finally, Trendy today launched the final “character class” video for Dungeon Defenders, this one focusing on the “Monk” class. “How powerful can a class be that has no physical blockades or traps?” Trendy asks. Watch the trailer to find out — then prepare yourself for the game’s launch on PC and Xbox on October 19 (with the PS3 version appearing shortly thereafter).

Growing Pains: They Hurt So Good

First, housekeeping: there will be free beer at the end of this discussion. Now that I have your attention, let’s get started.

For those of you following our exploits, our Principal Systems Engineer Resident Cthulhu Cultist Ryan Creasey mentioned in a previous post – “Arming a Bee with a Machine Gun – that we’ve been actively working on logging all of the data that runs through our systems; and we track a lot of data (8 billion requests per month). The goal here: provide real-time analytics to our customers. Simple, right?

Let me step back and give some context – since the launch of GameSpy Open we’ve been on a path to continually improve developer experience with our products and services. We’ve been actively working on improving platform integration, simplifying the process for integrating our SDKs, and of course, bringing more awesome to our Developer Dashboard which is the impetus for this blog post.

The beauty of data visualization (

Now, back to the data. The long and short is that we outgrew the analytics system that we initially chose for handling all the data we collect for the games that use our services – and we outgrew it MUCH faster than we’d ever anticipated. Shortly after the launch of Open, it became abundantly clear that our analytics data would be too big for its britches.

With this goal in mind, we set out to create a superior analytics system that could keep up with the heavy load over the long haul.

We Need More Powah…

Our first pass at building this new analytics system leveraged an existing webservice infrastructure written in C# using .NET 4.0/WCF. Hardware: two Windows 2K3 quad-core boxes with 2GB RAM to get things started. Our initial concern was the data bottleneck, the issue behind the aforementioned growing pains. Of course, NoSQL has already addressed this problem – TL;DR (as this will be discussed in a future post), we settled on MongoDB for our data store.

We then wrangled up some really pissed-off bees, which proceeded to annihilate our systems, Max-Payne-style. What we found is that our data store was no longer the bottleneck – Mongo was kicking ass and taking names.

Instead, we began hitting limits on our web tier. Our initial tests never even matched the identified goal of handling 3K reqs/sec (roughly what we see from production load across all of our current titles) – the two boxes we spun up for the PoC hit their cap at ~2,400 reqs/sec. Worse, these requests took 3-5 seconds to complete, on average; insufficient, to say the least, and unacceptable for a real-time system.

Next Song!

We knew that we needed something different; better, faster, stronger – if you will. Our first thought was to evaluate something from the Java world – which would be an easy paradigm-shift from C# and could leverage frameworks such as Play/Netty.

Diving a bit further, we stumbled upon an interesting article around Goliath/EventMachine that piqued our interest. This framework leverage Ruby 1.9 Fibers to schedule cooperative lightweight, thread-like structures that can pause and resume on a whim to create a highly efficient web server framework for handling concurrent load.

I find your lack of faith (in other languages) disturbing

You might ask yourself: Ruby… really? Believe me, we were skeptical as well. On the one hand, Java is tried and true and has reliable (albeit bloated) libraries to choose from. Ruby, on the other glove hand, provided the agility we needed to get a smaller, modular piece of our infrastructure done quickly. And, with the increasing number of Ruby projects available on GitHub, it also provides a great deal of extensibility. With this, we unleashed Goliath.

Drum Roll Please…

Now on the Linux front, we used two equivalent quad-core, 2GB RAM XenServer VMs running CentOS5 to keep our tests as even as possible. Since Goliath creates individual reactors to handle requests – we put HAProxy in front to balance the load. Pro Tip: ten reactors per machine gave us optimal results, but experiment to see what works best for you.

Speaking of which, those results were astounding (especially for some of our harder-core Ruby skeptics). Our load tests showed that our new Goliath endpoint was handling 4,500 reqs/sec with a 200ms avg. response time. Holy crap! This knocked the socks off our previous go ’round by an order of magnitude – even running on equivalent hardware. Now granted… 200 ms is still not ideal, but it’s a huge step in the right direction.

Papa Loves Mongo

But we weren’t out of the woods yet. If you haven’t played with MongoDB yet, forewarning: Mongo is not like other NoSQL offerings. If configured improperly, it will bite you. Hard. We’ve had our fair share of S&M, but this was not the time nor the place, as we were stone cold sober and had a deadline to meet. After poring over the Mongo wiki, Google group, and some online validation we came up with an approach that worked for us and could scale horizontally while maintaining a redundant backup.

We settled on a configuration of six nodes for the main cluster: three replica sets with two nodes apiece, each other node of the set is part of a sharted (sharded) cluster; e.g. (1-2-3) | (4-5-6). A final seventh node acts as the hidden backup of each replica set, runs three mongod instances and a configsvr to backup all data in case of zombie apocalypse (only a matter of time). The important part to note is that this third instance of the replica set means there is an odd number of nodes in the set, so an arbiter is unnecessary. Furthermore, the configsvr can live on this backup node to protect any failures from the primary cluster serving traffic. Check out our MongoDB Chef Cookbook if you want to replicate this setup.

You Talk Too Much

Thanks! The results of our efforts are now an updated Developer Dashboard (currently in the last stages of QA), with working metrics that are “super speedy”! Yes, it’s a technical term. We also have a lightweight, modular, highly-optimized and horizontally-scalable analytics webservice and a distributed load-test harness to boot. Not too shabby. We’re not done of course! There’s still more to come. Join us next time for a deeper dive into the infrastructure behind this system and our plans for the future. Plus, we’ll have more beer.

Comments Off

Categories: Engineering

“Building Iconic Online Games” Tutorial Day at GDC Online 2011

On Monday, October 10 from 9:30 a.m. – 6:30 p.m., GameSpy Technology will return to the Game Developers Conference Online in Austin, TX to host a full-day tutorial on using online features to “Build Iconic Online Games.” Attendance is open to all who have registered for GDC Online (except for the Games Career Seminar). We had a phenomenal turnout for last year’s event, and we invite you to join us again this year!

Our full agenda for the event is below. Speakers for the keynote and the panels to be announced soon. Stay tuned!

9:30 – 10:00 – Opening Remarks from GameSpy Technology
GameSpy Technology will provide its perspective on the state of online gaming, deliver information on the wildly successful growth of its indie developer focused “GameSpy Open” initiative, and preview several prominent indie titles – from Warm Gun by Emotional Robots to Dungeon Defenders: Second Wave by Trendy Entertainment – that use GameSpy’s Technology to push rich online features to new audiences in the mobile space. Particular focus will be given to Halfbrick’s Fruit Ninja — a hit casual game that used GameSpy Technology’s multiplayer services to connect its players on Android devices. GameSpy Technology will discuss the studio’s achievements while asserting that rich online features have become essential for indie developers to reach “icon” status.

10:00 – 10:30 – Keynote: Ernest Cline, Author: Ready Player One
Ernest Cline’s new novel, Ready Player One, is a set within a global MMO of epic proportions, filled with references to and appearances by video games both classic and current, and explores the nature of life both in and out of a game world that has essentially replaced the internet. Mr. Cline will join GameSpy Technology’ Tutorial Day to provide insight into his rationale for setting the novel in an online world, providing a argument for why game-based entertainment at every level should embrace rich online as a key element in “finding the fun.”

11:00 – 12:00 – Session #1: The Database Is the Next Game Engine
Tracking LOTS of game data is critical for all game developers in the era of online ubiquity. From leaderboards to stats-rich user-profiles to telemetry, games on every platform lean on big data to make games more fun. But tracking data at scale can intimidate even the largest developer. At this session, GameSpy Technology will discuss how it scaled its stats-tracking services so that any studio can leverage “big data” to produce big fun. We will discuss our move from a “SQL for everything” approach to a heterogeneous, partially NoSQL environment built for the data needs of game developers big and small. {Presenter: Mike Ruangutai, Technical Director, GameSpy Technology}

12:00 – 1:30 – Networking Lunch
Eat a meal on our dime and connect directly with fellow online game pioneers, alongside GameSpy Tech engineers, account managers and product staff (we promise to chew with our mouths closed).

1:30 – 2:30 — Panel #1: Beyond the Couch – State of Online Games in 2011
Before we know where online games should go, it’s important to take stock of where we’re at. This panel will convene players from various online-enabled spaces within gaming (mobile, console, social, and free-to-play) to provide a cohesive vision of where online gaming stands – and point to its future at the dawn of the mobile / html5 revolution. {Kris Graft, Editor-in-Chief, Gamasutra; Panelists: Bernie Yee – Bungie Aerospace; Tom DuBois – OnLive; Dallas Dickerson – BioWare / The Old Republic; Brian Cho – Booyah; Lorin Jameson – SOE / DC Universe Online}

2:30 – 2:45 – Live Demo: Dungeon Defenders – Cross-Platform Co-Op Multiplayer
Trendy Entertainment’s Philip Asher presents a live demo of the studio’s innovative tower defense / RPG, which uses GameSpy Technology’s multiplayer services to deliver cross-platform co-op play across iOS, Android, PC and PS3. The game’s Xbox, PC and PSN versions release on October 19.

3:00 – 4:00 – Session #3: Tutorial – Building Limitless Leaderboards – and More – on iOS, Android, and Beyond
You can argue about genres, gameplay mechanics and story all day long, but the great uniter among gamers is a love of stats. From high scores to leaderboards to stats rich user profiles, everyone loves to measure their performance. In this tutorial, GameSpy Technology will show you, step-by-step, how to use its freely available, cross-platform Player Metrics & Rankings services in the Unity engine to create the stats-driven features that gamers love and demand from games on every platform – from iOS to Android and beyond. {Presenter: Dan Maas, Technical Product Manager, GameSpy Technology}

4:00 – 4:15 – Live Demo: Super Comboman & IGN Indie Open House
Todd Northcutt, VP of GameSpy Technology present an overview of GameSpy Technology and IGN’s Indie Open House residency program, which provides free office space, technology, and consultation to independent game developers. Justin Woodward of Interabang — on of the studios participating in the program — will deliver his studio’s game, Super Comboman, currently in development at IOH.

4:30 – 5:30 – Panel #2: Don’t Look Back – Why I Went Indie
Some game developers became indies by circumstance; others were just born that way. This panel will bring together developers who chose the independent route from the start, those who found independence thrust upon them, and others who chose indie development because of its appeal relative to their experiences with big studio development. They will discuss their path to indie-dom, explore its advantages and disadvantages over AAA studio life, and share their secrets (or just tightly gripped hopes) for achieving success with only themselves to call “boss.” {Moderator: Ben Kuchera – Ars Technica | Panelists: Philip Asher – Trendy Entertainment / Dungeon Defenders; Justin Woodward – Interabang / Super ComboMan; Richard Wykoff – Reverge Games / SkullGirls; Max Hoberman – Certain Affinity / Age of Booty & Crimson Alliance}

5:30 – 6:30 – Multiplatform Cocktails
Have one on us and toast the launch of your next successful online game!

New GameSpy Technology SDK Sample Applications Available

In our ongoing efforts to make GameSpy Technology’s Open SDKs better than ever, we’ve introduced several new C-based sample applications. Where our previous sample apps called all GameSpy Open functions, these new sample apps demonstrate specific features and use-cases so that you can immediately begin using our services for:

  • Player Authentication
  • Paging through a Leaderboard
  • Stats Reporting
  • Storing and Retrieving Saved Games
  • Uploading and Downloading Screenshots

You can download these samples easily by visiting your Developer Dashboard, scrolling down to the “Links” pane, and downloading the GameSpy Technology SDK package; you’ll find them in the “Samples” directory of the SDK package.

Keep your eyes peeled for future updates with even more sample applications!

Download GameSpy SDKs from your Developer Dashboard

Categories: Implementation, Open, SDKs

GameSpy Unity Connector Screencast

Here at GameSpy Technology we’ve been hard at work making our SDKs available to Unity developers. We’re pleased to provide a Screencast showing the step-by-step process of incorporating our ATLAS Stats Reporting service into your Unity projects through our C# connector.

We invite you to pull up a chair, crack open a cold one, and take a look at how easy it is to use GameSpy client services with Unity:

Categories: Implementation, Open, SDKs, Unity / Tags: , ,