Researchers have disclosed a denial-of-service vulnerability in OpenSSL, dubbed HollowByte, that allows unauthenticated attackers to inflate a server’s memory usage using a malicious payload as small as 11 bytes. OpenSSL has fixed the issue without assigning a CVE identifier, but has backported the patch to several supported release branches.

The flaw was detailed in an advisory from Okta’s Red Team, which explained the underlying mechanics. Every TLS handshake message opens with a 4-byte header, including a three-byte field declaring the size of the data that will follow. Vulnerable versions of OpenSSL allocate memory based on that declared length before the actual payload arrives or its size is verified.

How the attack works

An attacker opens a TLS connection and sends a small payload (11 bytes) with a header falsely claiming a much larger message body is coming. The server allocates memory for the promised data and the worker thread then waits indefinitely for a payload that never arrives. Repeating this across many connections with randomized claimed sizes forces the server to reserve substantial memory for minimal attacker bandwidth.

According to Okta, the problem is compounded by how glibc, the GNU C Library, manages memory. Rather than immediately returning freed small-to-medium allocations to the operating system, glibc retains them for potential reuse. When an attacker varies the claimed sizes across connection waves, the allocator cannot efficiently reuse the freed chunks, causing heavy heap fragmentation. The server’s Resident Set Size climbs continuously and remains inflated even after the attacker disconnects. Restarting the process is the only way to fully reclaim the memory.

Scope and impact

OpenSSL underpins TLS encryption and certificate handling across much of the internet, embedded in web servers like NGINX and Apache, language runtimes including Node.js, Python, Ruby, and PHP, and databases such as MySQL and PostgreSQL. It also ships pre-installed on most Linux distributions.

In testing against NGINX, Okta found that low-capacity environments can be exhausted of memory quickly, while higher-spec servers can lose up to 25% of memory even as the attack traffic stays under typical bandwidth alerting thresholds, making detection difficult.

Remediation

OpenSSL has resolved HollowByte in version 4.0.1, with backported fixes in 3.6.3, 3.5.7, 3.4.6, and 3.0.21. The fixed versions grow memory buffers only as data actually arrives, ignoring unverified header claims. Although classified by OpenSSL as a hardening fix rather than a formal security vulnerability, Okta is urging organizations to upgrade their distribution’s OpenSSL packages immediately given the library’s foundational role in secure communications.