With all the work that’s been continuing at Stainless Games to improve Carmageddon: Reincarnation since its release, we thought it was time to also resurrect our blog. We wanted to give you a clearer idea of the challenges we’ve encountered along the way and explain why it’s not as easy as some of our critics seem to think to increase fps or improve the look of the game. As a small indie dev team everyone has been pulling together magnificently to fix reported issues and help as many players as possible to enjoy the game. Our first blog comes from Simon Lacey, as he explains how we worked to increase the fps significantly between release and our first update. Simon is pictured below, testing the motion blur code…
Hi, I'm Simon, the lead programmer on Carmageddon: Reincarnation. I've been asked to write a blog entry describing a bit of the story behind the poorly received performance of the 'FULL RELEASE DAY!' version of the game (Build 7039) and the big performance step in 'Post Launch Update #1' (Build 7470).
It can be quite tricky developing on PC. You never know exactly what you are going to be running on. The best you can do is to run the game on as many different machines as possible, and hope the coverage is broad enough to minimise how many people are left with a crappy experience.
We have a range of machines at work; we have a couple of dozen developers regularly playing the game, but as developers, quite a lot have pretty decent machines, especially the artists that have to work with large uncompressed assets. In addition we have a selection of machines specifically chosen to cover the range from minimum spec (or indeed a little below) up to Titan powered machines. There's a good mixture of AMD, NVidia, and Intel hardware, and these machines are regular used for benchmarking.
We chose quite early on to have a minimum frame rate of 30 FPS, and following the optimisation in the final early access update and the subsequent final release, benchmarking was showing that we were typically exceeding this and by a pretty decent margin. Unfortunately this didn't match up with your expectations, and you let us know! It was always our intention to optimise further as part of our continuing development of the game, but the release feedback made it our number one focus.
Whilst we were busy on our release branch for launch, Rosario was busy modernising our renderer. It was too risky to include for our initial release but it gave us a nice head start on our optimisation work, and also had the benefit of improved lighting. From there it was just a case of doing the standard optimisation things, essentially various combinations of doing less and/or doing it more efficiently. Profiling, looking for bottlenecks, and recoding the hell out of them, aiming not just to improve minimum frame rate but to also increase the consistency of that frame rate.
In professional development with the need to include time for planning, review, testing, etc., three months quickly becomes two. Patrick spent most of this optimising the damage code, then tuning the new damage code and fixing the resultant bugs. This work combined with caps on expensive features like pinball mode lead to greatly minimised performance lows.
A lot of the other work involved knocking off half a millisecond here and there. Things like the UI only processing stuff that has actually changed, improving unexpectedly expensive string building, reducing data transferred between C++ and LUA, rationalising some vehicle construction, among an awful lot of other things.
This got us to a point where the game was running significantly faster, and a little smoother than our full launch, but lead to an issue on one specific min spec machine. Frame rate had improved significantly on this machine, but had become horrendously, unplayably inconsistent. Unfortunately our user friendly performance graphs were hiding this behaviour, and it wasn't until we saw it with our own eyes that we started to understand more of the issues that had previously been reported. So with just 2 weeks to our intended launch date for the update we had a massive new optimisation issue to get to the bottom of. I can't express enough how 'cool' it is to have the machine with the problem actually sat in front of you. Trying to debug/diagnose issues remotely through conversation and limited logs is often frustrating and frequently futile (for all parties). We still try, but I really wish every time someone reported a problem, we could just pop around their house with a laptop and sort it out. Although I suspect the ones that would be happy with me doing that, are the ones my mum warned me about.
Anyway, hardware with a problem, on my desk, go fix! Turns out there were two issues. The first issue that was identified was that we'd exceeded the VRAM on a 1GB card. Not typically a massive issue, drivers tend to do a decent job of swapping stuff out to main RAM for you, and as long as you're not too much over budget you should just get the occasional micro-stutter. Still, with limited main RAM as well as VRAM, it seemed like an ideal candidate for optimisation. We were able to come up with a mostly win-win for this, reducing our vertex size saved a significant amount of VRAM and gave a minor performance improvement.
The second issue wasn't quite so straightforward to track. Profiling was just showing the machine as doing a lot of work, but mostly because running a game is quite lot of work. After some investigation, the issue disappeared completely with virtual textures disabled. It turned out the VT system was trying to load/decompress too many images simultaneously, and whilst each task was relatively self-contained and quick to process, the sum total of them put too much of a drain on the system as a whole. Especially on lower end systems. A drop to the number of concurrent tasks gave us a functional VT system, removed the horrendous stutter, and had an equally significant impact on frame rate consistency and performance across the full spectrum of PCs. There was a slight increase in texture pop-in, but the benefit vastly outweighed this.
This left us with 3 days to test and get it live on Steam. Fortunately, for the most part, it seems to have worked out well. And we all hope that the next update (currently in its final QA/test phase) will be received with greater enthusiasm still.
Discuss this in the forums