Unauthenticated TLS records can exhaust memory or CPU during a handshake

Summary

Reseau 1.1.0 through 1.3.1 contain two resource-exhaustion conditions in the native TLS 1.2 and TLS 1.3 record layers. An unauthenticated network peer can send records that consume memory without a bound or keep a handshake task processing non-progressing input.

### Impact

A peer can retain approximately 16 KiB of plaintext per record until the process exhausts memory, or stream empty handshake records to keep a connection task consuming CPU. Multiple connections can amplify either condition and impair service availability.

### Technical details

This advisory consolidates two handshake-time input-validation defects:

1. After obtaining handshake traffic keys but before completing the handshake, a peer could send application-data records. The record layers appended the decrypted bytes to an unbounded plaintext buffer instead of rejecting the record. Because each record was nonempty, it also reset the useless-record counter. 2. Zero-length handshake records produced no handshake bytes but were not rejected or charged against the useless-record limit. The handshake reader could therefore loop forever while continuing to receive non-progressing records.

Both defects were introduced with Reseau's native TLS stack in version 1.1.0. Releases 1.0.x used the previous OpenSSL TLS engine and are not affected.

### Patches

Upgrade to Reseau 1.3.2 or later. The fixed record layers reject both application data received before handshake completion and empty handshake records with a fatal `unexpected_message` alert.

### Workarounds

There is no complete in-process workaround. Set a finite `handshake_timeout_ns` and enforce connection rate and concurrency limits to bound each handshake's duration and reduce repeated attacks; an attacker can still consume substantial resources within that window. Terminating TLS in a patched proxy before forwarding traffic to a vulnerable Reseau service avoids the vulnerable record parser.

Reported to the JuliaLang security team through Anthropic's Coordinated Vulnerability Disclosure program.