Qovery's Engine-Gateway Experiences Out-of-Memory Crashes
/ 1 min read
🪄 Qovery’s engine-gateway faces mysterious out-of-memory crashes. The engine-gateway service at Qovery, responsible for client connections and data transmission, experienced unexpected out-of-memory (OOM) crashes despite stable memory usage readings. Initial investigations revealed no spikes or leaks, but further analysis using the jemalloc allocator identified that excessive memory allocation occurred when logging errors with backtraces enabled in the anyhow library. This issue, triggered by a specific code path, led to significant memory consumption and ultimately caused service outages. The resolution involved disabling backtrace capture during normal operations, highlighting the importance of understanding documentation and monitoring limitations in software development.
