This is the story of a debugging case with a happy ending. TL;DR This is a bug in the glibc malloc(). It mainly affects 64 bits multi-threading applications. With a special mention to Java applications because the JVM seems to always trigger the worst case. The write up is quite long, skip to third section glibc malloc() if you just want the ending.
Source: Major Bug in glibc is Killing Applications With a Memory Limit – The HFT Guy