Rant ^ 3 (Debug vs Release)

I’ve encountered a very strange problem (and the roots actually lie in the decision to continue with the upgraded physics system with Leadwerks – and possibly with the upgraded BlitzMax version). I explain this in as non-techy way as I can – and I go in the very, very beginner basics.

First the basics: when one creates games, they compile the code which is like “from human to computers translation”. The .exe files are understood by the computers, and the code files are understood by human. Now, in order for humans to find bugs and errors a “debug mode” was invented, so that humans can see what’s going on in the game while playing it. This error mode means slower performance (for example, a game might run at 50% speed), but it makes it easier to locate bugs and correct them.

After errors have been fixed (most of them anyway…), the game can be compiled in “release” mode.

Release mode is the one that you customers use, the debug mode is for developers only.

Now my “small problem”
About a week ago, I spotted a problem: my release mode wouldn’t work right. The collisions were “broken” and acted really weird. I hadn’t spotted this earlier, since I hadn’t compiled release mode for some time. Since I never had any problems with debug versus release modes, I simply hadn’t felt need to do that (game runs fast in debug mode in my computer, so it wasn’t necessary – I was concentrating on other issues).

So basically:
- Debug (slow) mode works
- Release (fast) mode is broken.

I have zero errors. Zero warnings. All my code is 100% the same for both debug and release modes. No threads.

I’ve upgraded to the latest Leadwerks (LE2.24) and I’m using BlitzMax (1.33 RC5). In earlier versions (LE2.18 for example) there wasn’t this problem, but I had to upgrade to the latest version to ensure compatibility (that’s another long story which was discussed some time ago here on my blog).

Anyway.

It’s pretty hard thing to tackle. I’ve never faced anything like this, and the bad thing is that there is no errors. Nothing about uninitialized variables. The only error is that the collisions between the player controller versus everything else acts weird. Other collisions (like boxes against ground or boxes to boxes) all work perfectly in both modes.

I’ve used the past week to locate this error and trying to isolate what the problem really is, but so far no luck.

This actually didn’t turn into Rant ^ 3, but I can now say that this is quite horrible problem that comes in horrible time (this has now turned into the biggest task left to be able to release the game. I have some other tasks on my list, and was hoping to release the thing already – but now my energy goes into solving this nice little thing).

I know that this will be solved, one way or another – but it can be… slightly annoying not to be able to release your game since the release version compiles differently from debug version. (I know that of course the problem can lie in my code – and I’d hope it would be there – but still it’s really strange that same (collision) code is treated differently depending on the compile mode)

Oh well, I’ll just take a deep breath and continue…

14 thoughts on “Rant ^ 3 (Debug vs Release)

  1. The only difference between Debug and Release mode in BlitzMax I am aware of is that BlitzMax calculates Floats with different accuracy in Debug and Release mode.
    To fix this bug, don’t use Floats, but Doubles (remember to put each constant number also as double, for example 9:Double, or 12.0:Double). They are also faster, as my various tests have shown.

  2. Seriously, any programmer who’s worth something has to know how to debug code running in release mode. Whether that means reading -O3 assembly dumps or using printfs to their full capacity (beware of stack layout changes!) – one should do it.

    Regardless, if your code works in debug mode but not in release mode, first, try to build in release mode without optimizations, and if it works then, somebody is playing not nice with memory, and that somebody is you.

    Since you are on Windows, your only real option is CrtMemoryDebug or BoundsChecker.

  3. FYI, in release the machine code is optimize big time. This is why the code is completely different from the debug version. Like everybody said, it is 95% a certainty that you are doing something bad with memory. It could range from unitializade variable (debug set the memory to zero, release don’t) to writing out of bounds. I’m surprised you never seen this kind of problems before, it’s pretty commun.

    BTW, once (about 10 years ago) I had a bug in release mode that I couldn’t identify in debug. Surprisingly it was the compiler (MSDEV) that was just creating bugged code in release. I was very lucky that it was happening in a very simple and short function, I was able to track it down in asm code. That was really surprising and scary… but it only happen once in 25 years programming.

    JC

  4. hi,

    not so long ago i had your problem with a HUGE game (about 40 projects in visual studio, each project a considerable amount of code).
    No warnings, no errors. Game just crashed or worked weird in release and perfect in debug.
    The biggest part of the problem is to identify the source.
    From experience i can tell you that visual studio is somewhat more permissive with what it lets you to do with the memory in debug. For ex
    in debug u can do this: memcpy(dest, source, size_of_source+1).
    And it won’t crash in debug. EVER :)
    In release it will crash from time to time, depending where source+1 memory points at :).
    So check your memory operations.
    If you use multithreading use one of intel’s tool to find problems (i believe that this is not the case with your problem ). Most defenetly something that you do with the memory is bad :). That is my view about it at this moment now.

  5. Can you get the source for the module and compile that with Print commands in it instead? (it’s what I do with my framework, which is not a module yet)

  6. Printf… (of course one problem with this is the fact that collision stuff also happens “inside” Leadwerks modules, so I have to do trial/error for various parts in my code).

    @Marco: and to add… naturally I do stuff in chunks… but I always have tested only “does it work without errors/warnings in debug mode”. I guess that needs to change too…

  7. Thanks guys for the support. Psycho’s statement was especially inspiring: “Let us know where the problem was. So we can laugh too… ;)”

    :D

    @Marco: To be honest, I had never known that debug & release modes would work differently. (Oddly enough, I’ve been doing debug/releases for like half a decade and had never heard anyone having problem like this)

    Btw – I have no threads.

    (And btw II, I’m using Leadwerks with BlitzMax / Blide)

  8. long live printf debugging!

  9. The way to find this type of problem is to go old-school: “printf debugging”. It’s ugly, but it often works to narrow down the problem. Go to your collision code and have it spit out values relating to collision. Figure out where your bad values are, then trace back to the source of the problem. It’ll probably be an uninitialized variable or out-of-bounds array reference.

    If adding print statements seems to fix the problem, then you probably have a timing or threading issue, as people have mentioned above.

    Hope that helps. Sometimes the ugly method works. :)

  10. Unitialized variable, obviously.

    The good news is that it always comes out, however hopeless you may feel.
    I’ve been through hundreds of these situations, but never it has remained hidden when I tried hard to find it.

    Let us know where the problem was. So we can laugh too… ;)

  11. Probably it is a problem due to memory management / leaks. If you are using visual studio, you may want to raise the warning level.
    Project properties -> C/C++ -> General -> Warning Level
    You can set it at Level 4, it may provide some useful hints.

    A further strategy can be to comment out chunks of code, until the game compiles in release mode. Then, incrementally re-insert the code (one chunk at time) until the game does not compile; in this way you will have some hint on where is the problem.

  12. Look out for code that is meant to Do Something but is located inside an ASSERT ;)

  13. Another potential issue is timing. As you said, debug code isn’t as fast as release code. If you’re using multiple threads, it’s possible something isn’t quite thread safe, but the threads run slowly enough in Debug mode that they don’t end up stepping on each other’s toes, but Release mode is fast enough that they do find issues.

    Memory and pointer issues are probably more likely though.

  14. Pretty hard to tackle, very true. One of the things to know is that memory layout is quite different between debug and release mode. So what can happen here (and this might be your problem, somewhere) is that you have some index access out of array bounds, which goes undetected in debug mode, since the memory layout leaves some space behind the end of an allocated memory block. In release, such security measures are not in place, so this could result in overwriting some bytes in other memory chunk…

    Just a guess, this is just one of the more typical things that happen in this context. Good luck finding and fixing this!