In the Amazon

Posted: September 20th, 2016 | Author: | Filed under: Rapture | No Comments »

Today we’re releasing the Amazon version of Rapture World Conquest. You can find it here – UK or US

If you have a Kindle and haven’t already played the game, then please give the Amazon version a go.

And if you haven’t seen it already, it uses our new Rapture Logo:

amazonstore

We’ll be working on new updates to the game in the coming months – please email through your ideas. We’re also looking into the Steam version  – look out for out Greenlight campaign soon.

 


To Android and Beyond!!!

Posted: July 4th, 2016 | Author: | Filed under: Rapture | No Comments »

Rapture World Conquest is free to download here and on Android here.

To Android and Beyond!!!

Introduction

This is the third in the series of articles discussing the experience of working on our title, Rapture – World Conquest. Hopefully you’re still along for the ride! The other articles can be found here (looking at the main release in November 2014) and here (looking at going from paid to free to play). This article will delve into life since October 2015 after we released the game for Android.

Tundra consists of myself (Dan Collier) and Andy Weinkove. We’re game industry veterans, having done our time in the game dev trenches. A few years ago we decided to throw our hats in the indie ring by starting our own company and creating our own game. Rapture – World Conquest is on iOS here and on Android here. It’s a mix of strategy and arcade – where you play a jealous god directing your armies to wipe out the opposing armies – all before the Rapture descends on the planet. The game was influenced by Sid Meier’s Civilization – as we attempted to make a lightning fast, 5 minute version of the classic world-conquering title.

We released the game for iOS as a paid title in November 2014 to good reviews. The game did fairly well, and better when we got featured. However sales dropped off quickly. We then decided to try our hand at the free to play market. In May 2015 we released the free version with adverts and in-app purchases. It didn’t get much attention, but we saw reasonable downloads and decided to have one last roll of the dice by porting the game to Android. In November 2015 we released on the Play store.

Release of Android

It took around 7 months to convert the game to Android. Mainly because both of us still work full time, so development goes on in the evenings and weekends. Also we had to get up to speed with the Android development environment and the new tech involved. Rapture is built on top of our own Mojo platform. This allows us a lot of flexibility and control over the tech – it also means the game runs fast even on older devices – we don’t have the blackbox of other 3rd party engines to contend with. However the downside is that to add a new platform we need to do the leg work and add it to Mojo. The conversion was long work, especially implementing the Android specific functionality and attempting to support the variety of devices out there. However both of us are PC born and bred, so it was nice to have a break from Apple macs. We used Tegra Nsight to develop the game in Visual Studio – again a piece of software we are both comfortable with.


RevenueOverall

Once the game was ready for release – we tried to make a similar splash with the game as the iOS version, but it’s difficult to get attention for a game that’s been out for a year already on another platform. Initial downloads were fairly quiet, but we saw a slow and steady rise.

Using the Google Dev Console and being on the Play Store has so far has been a good experience. The review system on Google is particularly wonderful, Android users seem to be more inclined to review games than iOS users so we get a large amount of feedback each day and can get a better view of what players enjoy or criticize, which helps decide on what we focus on. For example we had larger than expected downloads from Russia and had lots of players asking for the game to be translated into their language. It made sense to get it translated and we got a lot of positive reviews in response (with many Russian players good enough to raise 1 star reviews to 5). We can also ask players who see bugs in the game to email directly with further details and screenshots to help track problems down. So it’s been great to have that direct channel with people actually playing the game.

Stats

So how have we done? Here’s the initial 3 weeks on Android:

FirstMonthDownloads

23,795 in that first few weeks. After the initial surge, downloads levelled off to about 1000 to 1500 a day. Here’s the downloads we had from November to February – a fairly regular pattern:

NovToFebDownloads

In February we were featured by Google.

GooglePlayFeatured2

When we initially saw the positioning of our title in the feature (#36 in the “New and Updated” list in the US) we worried it might not have much of an impact, but we were very wrong! Just as the iOS feature had been, it was a massive boost for us – a real milestone for the game and a chance to be seen by a lot more players worldwide. Over the week we were featured we received 260K downloads with a record 44K downloads on the 3rd day. Interestingly, our iOS downloads doubled during this period, an extra 100 to 150 per day, indicating word-of-mouth in action. One thing we did notice with the feature is that our average review score and revenue per user dropped slightly during this period presumably because the users who find a game via a feature are less likely to be engaged by a title than the ones who seek it out themselves.

FebFeatureDownloads

We did have a little misstep towards the end of the feature. Our payment account was suddenly suspended, which turned out to be due to a missed email about accepting an agreement on European Tax law. We had a frantic few days where our in-app purchases weren’t working, but Google were quick to respond and help sort out the problem. Note to self – never skim read an email from Google again!

The feature gave us fantastic downloads, but didn’t push us up massively in the charts, we managed 15th the US strategy charts, but only pushing 300 in the overall games chart. Afterwards our base level was pushed up by a little, but generally there was no lasting effect.

ChartPositionFeature

Then, at the start of March something significant happened – we received a sudden bump in downloads, going up from our steady 1000 per day to 8000 per day. Over the following weeks it gradually dropped and levelled off at 4000 downloads per day. Our chart position for Strategy has climbed and stayed in the top 100 for many key regions. We have searched for a reason for this sudden jump in downloads across all regions but have drawn a blank, we had no new reviews in the press or coverage from large YouTubers, our Google Play keywords strengthened but that came after the jump, so we can’t explain the leap upwards. We seem to have hit a sweet spot in terms of organic search results – so, currently at least, we’re no longer invisible! Our best guess is that perhaps we crossed some threshold in terms of downloads, reviews or user engagement such that we’re now being placed in Google Play’s “Recommended for you” section more often.

FebToNowDownloads

Overall our downloads have been very good for an Android title, even if you include the iOS feature, downloads have been more than 8 times higher than iOS 747,623 compared to 88,494. (Although we went in initially as a paid app to iOS, and free with Android, so perhaps that’s to be expected). The numbers that play the Android version has meant that multiplayer games are easier to find (though it still takes a little time to find a match).

So how has the revenue been? Well generally it’s been good. Compared to iOS, Android generates reasonable Ad and in-app purchase money as we have the greater numbers. So far we have taken in $23K:

RevenueOverallUSD

This is split between adverts (static and videos) $17K and in-app purchases $6K. Android players seem generally to be more open to purchasing the VIP pack in the game (which removes the ads) – but then we have many more players and the game has always been free. The surprise lift in the last few months has been very positive – if it maintains the same level, then we could finally get a reasonable return on the time we originally put into the game and hopefully on ongoing income which we can look to use in whatever we do next.

How has iOS been doing?

The iOS version of the game has been ticking over quietly in the background. The F2P version has 100 and 200 downloads a day and $7 to $21 per day in revenue.

IOSLast90Days

This is obviously a lot less than the Android version, conventional wisdom has it that Android versions will get more downloads because of the larger market but iOS versions will generate more revenue overall because iOS users are so much more inclined to spend money. Our title seems to be something of an outlier, our current Android downloads are about 30 times higher than iOS which is way more than the normal ratio and as a result our Android revenue is about 10-15 times higher. This might be due to the way we released the iOS version originally as paid and transitioned to F2P. The free iOS version didn’t get any sort of initial push that might have kept it up the charts long enough to solidify its place there.

After the change to free we also released a premium version, this was the free version with the VIP pack automatically unlocked and no ads or in-app purchases. We decided to put the premium version on the store just to see what it would do in terms of downloads. Originally we received criticism for our paid version having in-app purchases, so we were interested to see if a completely IAP free game might generate some interest. However the premium version got very few downloads, possibly again because it got no initial push. So we decided to take it off the store.

Latest update

Over the last few months we’ve been working on a big update to the game. Previously we’ve released updates with a few new items and options. This time we decided to try creating a much larger package to see if we could generate some renewed interest. We’ve also focused on the feedback we’ve had through the Play store and emails.

There are several new items to unlock – spacescapes, civs and planet shapes. Here’s our Dice Planet with a new rainbow sea and Inca civilization:

Dice Rainbow Incas

We have also added 2 new options that have been a common request by players. The first is an option to select the length of a game. We have always pushed back against adding this, however right from the start players have asked for longer games – so the length can now be select and we’ll be very interested to see the reaction. The second related change is the ability to slow down the game – there is now a speed button in the top left during the game and players can now slow things down if they wish. It should also give beginner players an easy time when learning.

Our hope is that the update will be big enough to attract the attention of Google and/or Apple and snag a “Best New Update” feature. We’re interested to see whether this happens. We are currently trying to decide on the best way to move forward with Rapture.

The Future

When we initially released Rapture as a paid title, after a strong launch it felt that there was always constant gravity pulling our revenue downwards week after week. When we switched to free on iOS, it felt like we had established a small and steady ongoing revenue stream but with no sign of that downwards pull. On Android, the situation feels even better – after a slow launch it felt like our title was able to establish a foothold and gradually grow in terms of downloads and revenue. Now with the burst of downloads from the feature and the big unexplained jump upwards in March, we’re looking forward to hitting the one million downloads milestone on Google Play within the next few weeks and we’re very pleased to have got to a place where our title is earning a decent steady income which we plan to invest in our next title.

Both of us are keen on a sequel to Rapture. There are many potential features that might work well, but they may not necessarily fit into the current game as updates. A sequel would allow a clean slate and fresh ideas. However the alternative would be to implement all the new features into the current game and keep our existing players. It’s a tough call at the moment and the next update will help make that decision. The other possibility is to work on something different; we’ve been working on Rapture now for 5 years, so it would be interesting to start something new. But we’ll see how the latest update is received.

We’ll be at the Develop conference in the summer. Any developers out there fancy meeting up for a drink, please get in touch – dan.c@tundragames.com

Update – We managed to snag a Best New Update from Google, which gave us a decent boost. More details on that in a future blog.

About Tundra

Tundra Games is an independent game studio based in Oxford & Colchester UK. It was founded in November 2013 by industry veterans Andrew Weinkove and Daniel Collier. We have worked on many AAA titles including Skate It, Sims 3, FIFA, Madden, Angry Birds Star Wars, Angry Birds Go and Need for Speed. We have worked for companies such as Exient, First Touch Games and Rebellion.

Please contact Dan for further details at dan.c@tundragames.com.

Our game trailer can be found here:  with further details and screenshots via our website –www.tundragames.com.

Play the game for free on iOS here and on Android here.


One Million Downloads!

Posted: June 17th, 2016 | Author: | Filed under: Rapture | No Comments »

Been a fantastic week with the Google feature – loads of new players have been in touch with feedback on the update and ideas for the next.

We’re currently working on a few new translations – then we’ll look into the next update – so please keep your ideas coming.

We have just passed the 1 million mark which is a huge landmark for our game – so thank you to everyone that has downloaded Rapture.

1 Million

 

Dr_evil_one_million_dollars


Welcome to the Update

Posted: June 6th, 2016 | Author: | Filed under: Rapture | No Comments »

We have just released a big update for Rapture on iOS and Android – update your game and take a look at the new features. We’ve had lots of feedback from players (thanks for the emails!) and added a lot of the suggestions.

Download for Android and iOS

This screenshot shows the new “Colossal” planet with the Holodeck background and the new Incans civ. You can also see the new game speed button in the top left.

Collosal Planet

Here’s some further details of what’s new:

A new game length option – Plenty of players have been asking for this – you can now select how long the game should be in the Play Options screen

Game speed – You can also change the play speed with a new button in game – especially useful as you are just getting to grips with the game. (The button can be hidden via the Play Options and will be turned off at higher difficulty levels)

New planet size option – There is now a colossal option which creates an enormous planet – please note that this option requires a device with reasonable power – older devices might experience slow down

New planet landmass “Very High” and “Full” options – These cover the world with even more land – again older devices might experience slow down with these options. Also note that full coverage will still leave a few small inland seas – as they are needed for gameplay

Play Options – We’ve moved the Play Options screen to appear after the missions screen – there’s a Quick Play button on the Missions screen if you want to get straight into game

New Planets – There are 5 new planet shapes to choose from – X and Y shape, disk, heart and die shaped

New sea – A beautiful rainbow colour sea option

New spacescapes – 2 new “room” backgrounds – holodeck and padded room

7 new civilizations – Atlantis, Brazil, Zulu, Inca, Persian, Portuguese and Dutch

New roman flag – A few people have mentioned that the yellow roman flag is actually the flag of the Holy Roman Empire – so we’ve changed the flag to try and be more historically accurate

Help button – A new button on the main menu will take you through to our website where we’ve added help on how to play the game and what the miracles and domestic policies do

Let us know what you think at feedback@tundragames.com and if you have any new ideas you’d like to see in the game we’d be interested to hear from you.

Please like and share this post!


Hitting 500K

Posted: April 13th, 2016 | Author: | Filed under: Rapture | No Comments »

Been a fun few months at Tundra – the android feature back in February brought in lots of new players. We have now tipped over the 500K downloads, which is fantastic.

500K Downloads

We’ve also had lots of feedback and ideas for updates, which always gets added to the big list that we keep, with everything we want to do to the game. Please keep them coming in the comments or at feedback@tundragames.com

The Russian update has been well received. We’ve had lots of updated reviews to 5 stars. We had a particularly detailed email, stretching over three A4 pages with thoughts on the translation and we hope to get all those in the next update.

Work on the update is going well – it should be out in the next few weeks (cross-fingers). We have a number of new planet shapes, spacescapes and civilizations. The main change will be the game options to select your game length and a button in game to select play speed. It’s probably the most asked for feature. Here’s a photo of the heart shaped planet to whet your appetite:

Heart Shaped P

You might also be able to guess a few of the new civs in the screen shot. The hope is to get the game featured on the Play or App store as Best New Update – which again would be a great boost.

Strangely the last few weeks we’ve had an unusual jump in downloads on Android. We haven’t been featured again and we can’t find any reviews that might have pushed us up the charts, but it’s been very welcome, gives us hope that the game won’t disappear off the radar. If you have any ideas of where the boost might have come from, let us know. Here’s the downloads for the past couple of months, you can see the Feb feature and then the unexpected peak the last couple of weeks. There will be a new article to go with the update with some details of how the game has been going since the Android release in November – with lots of lovely statistics.

downloads

Lastly we’ve been trying to track down a bug that a number of players have mentioned – Blue Screens. It seems to be connected with videos, occasionally the screen will just go completely blue. If you are getting blue screens and especially if you can get them consistently please email us. A video of it happening would be fantastic.


IAP problems sorted out

Posted: February 28th, 2016 | Author: | Filed under: Uncategorized | No Comments »

Hi all, there has been a few problems with google payments over the last week or so and players haven’t been able to buy the VIP Pack. We appreciate anyone wanting to support us by getting the VIP Pack and those that wrote to us to let us know there was a problem.

It should all now be working and if you still fancied getting rid of ads (and getting the other goodies the VIP pack gives) please give it another try in game.


Multiplayer update

Posted: February 19th, 2016 | Author: | Filed under: Rapture | No Comments »

We’ve put up a new version of the game with a potential fix to the multiplayer issues we’ve been having the last couple of days. Please update and let us know if you still have problems. There’s also a new version of Google Play Services available and it might be worth grabbing that as well.

In the update is also a potential fix for the blue screens that a couple of people have been having.

If you still get either of these problems, or any others, please get in touch at feedback@tundragames.com


Multiplayer issues

Posted: February 19th, 2016 | Author: | Filed under: Uncategorized | No Comments »

Just a quick post to say Rapture is currently having multiplayer troubles. Looks like Google have changed something and it’s affecting online play so players can’t connected to each other.

We’re putting together a fix at the moment and hoping to have it up very soon.

Thanks for your patience!


New Planet Shapes update

Posted: February 6th, 2016 | Author: | Filed under: Rapture | No Comments »

In case anyone missed the latest update for Android – there are now two new planet shapes to play on – flat earth (for all you deniers out there) and the torus/donut shape. Just update your game and unlock them in the shop to play on them. There’s also an extra “Pangaea” option in the Play Options page (found off the Missions screen) – which makes all the regions appear as one big landmass.

This slideshow requires JavaScript.

Also for you techies out there – we have updated the game to work with the latest “MarshMallow” SDK. Let us know what you think of the new shapes and send us any thoughts on other shapes you’d like added.

This isn’t for iOS yet – we’re working on getting it on Apple devices very soon.

If you haven’t already please leave us a review on the Google Play store or the Apple App Store


Minimizing the Pain of Lockstep Multiplayer

Posted: November 23rd, 2015 | Author: | Filed under: Rapture | 2 Comments »

Written by Tundra Games CTO Andrew Weinkove

Rapture World Conquest is free to download here for iOS and here for Android

Our previous blog about going free can be found here

Introduction

Rapture – World Conquest was released on iOS in September 2014 as a single player only game. Multiplayer was implemented later and was added in an update in March 2015. The game was released for Android at the end of October 2015.

Any programmer who has ever grafted multiplayer functionality onto a single player game will tell tales of just how difficult the process can be. For Rapture, the process went relatively smoothly because we planned ahead for lockstep multiplayer and built in a number of systems from the start.

To coincide with the recent release of our Android version we have written this article to talk about the systems and processes we put in place that minimized the pain of lockstep.

Choosing Lockstep

The idea behind lockstep multiplayer is simple, if every player shares their input with every other player, the simulation can be run on everyone’s machines with identical input to produce identical output.[1]

Lockstep multiplayer has become somewhat unfashionable these days as server/client approaches offer many advantages in a lot of genres [2][3]. The key advantage of lockstep is its low bandwidth demands, the key disadvantage is high lag as the simulation can’t step forward until a machine has received the input from all of the players. With audiences generally having access to decent broadband, the attractiveness of low bandwidth requirements typically looks insignificant against the usability cost of high lag.

However, the real-time strategy (RTS) genre is something of an exception. Latency in an RTS typically doesn’t affect usability anywhere near as much as it might in other genres like first person shooters. Meanwhile, massive numbers of units mean that alternatives to lockstep can easily use prohibitive amounts of bandwidth even on good connections.

Even in the RTS genre, lockstep is not the only option[4], however lockstep is especially attractive to mobile developers because cellular and bluetooth connections can be extremely poor relative to the broadband connections that PC and console developers can generally rely upon.

For these reasons, we decided early on that Rapture – World Conquest would use lockstep multiplayer, and we planned for it from the start.

Desynchronisation

Although the idea of lockstep is very simple, in practise it can be very challenging to keep the simulation running in perfect synchronisation. The code that drives your game logic must be fully deterministic across all the machines that will play against each other. That is, the machines must run the exact same set of calculations based on the exact same set of inputs and produce the exact same results. Any tiny variation between machines will cause knock-on effects so that after a short period of time the game state on different machines is totally different with no mechanism to bring them back into sync.

When variations occur, it is known as a desynchronisation bug, and in common with most lockstep games we share checksums of the game state between machines to detect desynchronisation and treat checksum mismatches similarly to network errors by stopping the game and displaying an appropriate message.

Desynchronisation bugs can be extremely difficult to track down, often requiring hours or days of debugging[5], so any steps that can be taken to catch these bugs early are very valuable. In the following paragraphs we discuss the strategies that we used to address these bugs as early as possible during development.

Code Organization

One of the simplest but often forgotten steps that we took was to organize our code to make it obvious which systems needed to be fully deterministic for lockstep. We refer to the section of our code that must run in lockstep as ‘the simulation’ and tried to make a clear distinction between simulation code and ordinary code.

organisedcode

All the simulation code source files were put under a directory called ‘Sim’ and have the word ‘Sim’ in their filename. We made sure coders knew to be extra careful when making modifications to the simulation, and regularly reviewed checkins on this directory for potential issues. We also compiled all our code with a high warning level, which can help highlight some programming errors like uninitialized variables which can lead to desyncs.

Fixed Timestep Updates

When we talk about all machines having to run from the same input, the immediate thought is of button presses or commands produced through the UI. In fact, the timestep is another important input into the simulation.

diving-gif

Right from the start we designed our simulation to run at a fixed timestep with interpolation[6]. In common with lots of other RTS games, we run our simulation at quite a low tick rate, 10hz in our case, and interpolation allows the rendering to run at a fast smooth framerate.

This decoupling of the simulation loop from the render loop not only achieves the fixed timestep necessary for lockstep, but it also reduces CPU load by letting the complex simulation run at a lower rate.

At the same time, we also engineered our code to run the simulation on a separate thread. This has little to do with lockstep, but was great for performance as it can conceal the large CPU spikes when hundreds of units pathfind on the same frame, and also takes advantage of the multicore processors that most modern mobile devices have.

Random Numbers

Another area that can produce desyncs is random number generation. We needed to ensure that every machine is generating the same set of pseudo random numbers otherwise multiplayer matches would quickly go out of sync.

Dilbert0001

Right from the start we made sure that our simulation used an independent random number generator from the rest of the game. This is because the game might be running at different framerates on different machines, so any random number generation in our rendering code (a particle emitter for example) will be called a different number of times across machines. Obviously, the random number seed used by the simulator needs to be agreed upon by all machines and we did this by generating a seed from a checksum of the shared launch settings.

Deciding to Prohibit Floating Point Math

Floating point numbers can present something of a problem. The challenge is that while the IEEE standards set out maximum errors for floating point calculations, individual implementations (either hardware implementations of maths operations or software implementations of functions like sine) have scope for producing slightly different results. These results won’t be very different, they’ll vary only occasionally and usually only by a very small amount. However, that is enough to upset the delicate balancing act of a lockstep game.

floatingpoint

Now, to be clear, floating point math is not vague and indeterminate. We would always get the same result when running the same executable on the same hardware. The problems arise when running on different hardware, using a different compiler, or using different compiler settings.

It probably isn’t very productive to delve too deeply into issues surrounding cross-hardware floating point determinism as it seems to be a topic that can cause some passionate disagreement and many good articles cover this ground [7][8][9]. Suffice to say that while games exist which use floats and run without desync across a variety of PC architectures, no-one has made the claim that it is an easy path to tread.

We decided that attempting to use floats in areas of our code that need to be deterministic would be a world of pain that is best avoided. Further to that, we were unsure if all the different FPUs found in mobile devices can actually be coerced to produce identical floating point results in the same way that PC FPUs can.

Finally, we’d set ourselves a target of supporting cross platform play during development. Having decided to support 4 player matches, it makes testing and debugging a lot easier if we can mix and match PC, OSX, Android and iOS builds with differing optimisation levels. Achieving flexible cross platform play like this just would not have been possible with floating point maths in our simulation.

Enforcing the Prohibition of Floating Point Math

Having decided not to use floating point math, we naturally decided to use fixed point math in its place. The idea is simple enough, we have a class that represents fixed point numbers, it’s simply a 32-bit integer where the bottom 12 bits represent the fractional part. So, the number 4096 represents 1.0, 2048 represents 0.5. Multiplying two fixed point numbers together would look something like ((FixedA * FixedB) >> 12), although all of our actual functions have a lot of checks to avoid preventable overflows and assert about unavoidable ones. We looked around for an existing fixed point math library that could be plugged in easily, but didn’t find anything suitable so we rolled our own.

Deciding not to use floating point maths is only half the story though. Ideally we would have a mechanism in place to catch any accidental use of them. Our solution was to use the floating point inexact exception.

Bruce Dawson’s article on floating point exceptions is a highly recommended introduction to the subject (his entire series of articles on floating point numbers is excellent), it includes some simple classes to help control which floating point exceptions are enabled [10]. One of the floating point exceptions that you can enable is called the inexact exception and it fires if a floating point operation produces a result which requires rounding. As pointed out in the article this is usually not useful to game developers because floating point operations are very frequently inexact, and we are all used to writing code that allows for this fact. However, for this very specific purpose, the inexact exception is perfect as it will fire on almost any piece of code that uses floats. We do not attempt to handle or catch the exception, we simply let the debugger catch it and treat it the same way we would treat a crash or assert, as a serious bug that needs to be fixed.

On our debug builds, we enable the inexact exception (along with the rest for good measure) whenever we enter our simulation, and the exception will fire if we absent-mindedly use floats. The exception is disabled when we leave the simulation, and we use floating point maths extensively in other areas of our code-base such as rendering.

In practice we found that we rarely used floats in our simulation directly, more often we called an innocuous looking function that had some floating point operations buried deep within it. It is possible to use floating point maths without firing the inexact exception, but any floating point maths that is exact will probably exhibit consistent behaviour across different hardware anyway.

Serialization

From the outset we knew from experience where desync bugs were likely to come from. However, for all the care we took to use the correct random number generator, to avoid uninitialised variables and to tightly control input to the simulation, we knew that we would make mistakes. We decided to put systems into place to catch them in our single-player mode long before we wrote a line of actual networking code.

The main tool we used was that every time we step our simulation forward, we load the previous state, and step forward again. We then compared the two new states we produced and if there are any differences then it indicates a problem in our determinism.

In order to achieve this we wrote code to serialize and deserialize the entire simulation state. Implementing and maintaining serialization code was a moderate amount of work, however we found that being able to save and load the entire game state was useful for other reasons. We used the same serialization code to support our in-game rewind feature, to generate the checksums that detect desyncs and for some debug systems (we found it handy being able to rewind and single-step the game in debug builds).

An simpler approach to ours would be to permanently maintain a main game state and a debug state, and by stepping both forward together and comparing the result, most of the same desync bugs can be caught. The reason we chose to load the previous state each time is that our approach lets us catch bugs in our serialization code at the same time as catching our desyncs.

When we detect a difference between the two states, we found that it’s not that useful to just assert there and then, as it leaves a lot of work to be done to track down where things went wrong. To help find the cause quicker, we added code to work out which is the first out-of-sync byte in the serialised data stream. We then loaded the serialized state again, but this time we fire an assert when we get to the first desynchronized byte. That way our callstack indicated which instance of which class had gotten out of sync, this helped us track down the cause a lot faster.

Overnight CPU matches

Putting systems in place to catch desyncs fast is all very well, and most bugs that these systems are capable of catching will be caught almost immediately. However, there’s always a handful of very rare problems.

robot fighting

We found it was incredibly valuable to invest the time on code to let the computer run automated matches overnight. Particularly as we lacked a QA budget, it was a huge deal to know that tens of thousands of games had been run without problems. With the mechanisms we’d put in place to catch non-deterministic behaviour, our overnight single-player tests caught lots of rare event desync bugs. In addition to helping us catch determinism issues, automated testing also helped us catch other bugs throughout our codebase and we have also used it for game balancing by examining how well AI perform against each other when forced to use only specific miracles or follow specific policies.

When we finally implemented multiplayer, we extended our automated testing to our multiplayer mode, and had two machines playing matches against each other continuously overnight.

Logging/Local Player/Undefined behaviour

Despite all our care there were a small handful of desync bugs that slipped through the net. It is invaluable during development to have extensive debug logging, so that if a rare desync occurs you can pinpoint the cause without necessarily needing to repro.

Our multiplayer logging in Rapture involved serialising and writing the entire state to a logfile every frame, along with the launch data structure and any input messages. We put code in place so that it was easy to re-run the step where the machines diverged.

What slipped through

Despite our forward planning, when we finally came to implement multiplayer, there were a few desync bugs to deal with. Mostly they were caused by bugs in the networking code as it was being implemented (i.e. a failure to share the input correctly, or the accidental processing of steps before input was received).

There were a couple of instances of code in the simulation being dependent on the index of the local player, which were simple to find and fix.

We had an interesting bug caused by ambiguous sequencing that looked something like:

MyFunction(myRNG.GetRand(), myRNG.GetRand());

There is no sequence point [11] between the two calls to GetRand, so different compilers are free to do the two GetRand calls in whatever order they like. MSVC and XCode obliged by picking different orderings.

The other bug we had was a qsort callback we had implemented which sometimes returned zero. As qsort is an unstable sort, we got inconsistent results across compilers. In most of our deterministic code, if we were sorting an array then we would always have a ‘tie break’ condition (usually based on the object’s unique ID) to ensure the sort is consistent. In this one case we had omitted to do so.

After fixing these few bugs, our overnight multiplayer tests began to run smoothly, bar the odd hitch where our extensive logging led us to run out of hard drive space! After a few nights of successful overnight runs, we became pretty confident in our multiplayer code.

Conclusion

By planning ahead and implementing a number of systems in advance to ensure our code was deterministic, it meant that the vast majority of multiplayer breaking bugs were found and fixed long before a single line of multiplayer code was written.

To anyone beginning a project which will eventually utilise lockstep multiplayer, we would strongly recommend investing the time upfront to build in systems to catch common desynchronisation errors. We are confident that by implementing these systems, we saved ourselves a lot of time.

If you have any questions or comments regarding multiplayer implementation please get in touch at feedback@tundragames.com

Further Reading

[1] Lockstep as favoured by many RTS games: http://www.gamasutra.com/view/feature/131503/1500_archers_on_a_288_network_.php

[2] Server/Client with prediction as favoured by many action games:   https://developer.valvesoftware.com/wiki/Latency_Compensating_Methods_in_Client/Server_In-game_Protocol_Design_and_Optimization

[3] Analysis of networking options for physics based gameplay: http://gafferongames.com/networked-physics/introduction-to-networked-physics/

[4] Planetary Annihilation, uses a modified server/client approach suitable for RTS: http://forrestthewoods.com/the-tech-of-planetary-annihilation-chronocam/

[5] Article on tracking desyncs: http://forrestthewoods.com/synchronous-rts-engines-and-a-tale-of-desyncs/

[6] Article on fixed timesteps: http://gafferongames.com/game-physics/fix-your-timestep/

[7] Floating point determinism: http://gafferongames.com/networking-for-game-programmers/floating-point-determinism/

[8] More on floating point determinism: http://randomascii.wordpress.com/2013/07/16/floating-point-determinism/

[9] More on floating point determinism: http://yosefk.com/blog/consistency-how-to-defeat-the-purpose-of-ieee-floating-point.html

[10] Article on floating point exceptions: https://randomascii.wordpress.com/2012/04/21/exceptional-floating-point/

[11] Article on sequence points in C and C++: https://en.wikipedia.org/wiki/Sequence_point

About Tundra

Tundra Games is an independent game studio based in Oxford & Colchester UK. It was founded in November 2013 by industry veterans Andrew Weinkove and Daniel Collier. We have worked on many AAA titles including Skate It, Sims 3, FIFA, Madden, Angry Birds Star Wars, Angry Birds Go and Need for Speed. We have worked for companies such as Exient, First Touch Games and Rebellion. Please contact Dan for further details at dan.c@tundragames.com.

Our game trailer can be found here, with further details and screenshots via our website.

Play the game for free on iOS here and on Android here.