Closed Bug 2052703 (CVE-2026-16402) Opened 3 months ago Closed 3 months ago

Integer overflow in nsJXLDecoder CMYK K-buffer sizing -> heap out-of-bounds read

Categories

(Core :: Graphics: ImageLib, defect, P3)

defect

Tracking

()

RESOLVED FIXED
154 Branch
Tracking Status
firefox-esr115 --- unaffected
firefox-esr140 --- unaffected
firefox152 --- wontfix
firefox153 + fixed
firefox154 + fixed

People

(Reporter: kg3n.research, Assigned: tnikkel)

References

(Regression)

Details

(4 keywords, Whiteboard: [adv-main153+])

Attachments

(3 files, 1 obsolete file)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36

Steps to reproduce:

Affected code: image/decoders/nsJXLDecoder.cpp, nsJXLDecoder::AllocateFrameBuffers().

The CMYK "K" (black) plane buffer is sized with an unchecked 32-bit multiplication, while the pixel
buffer immediately above it correctly uses mozilla::CheckedInt:

// pixel buffer: CHECKED (64-bit)
CheckedInt<size_t> bufferSize =
    CheckedInt<size_t>(size.width) * size.height * BytesPerPixel();
... mPixelBuffer.resize(bufferSize.value()) ...

// K buffer: UNCHECKED  (size is OrientedIntSize -> IntSize = BaseSize<int32_t>)
if (mPixelFormat.value() == PixelFormat::Cmyk8 &&
    !mKBuffer.resize(size.width * size.height))    // int32 * int32 -> int (32-bit)
    return NS_ERROR_FAILURE;

size.width/size.height are int32_t. A CMYK image whose width*height exceeds INT32_MAX overflows
this product (signed overflow), so mKBuffer is allocated far too small. WritePixelRowsToPipe()
then reads the K plane per row with only a null check, not a bounds check:

const uint8_t* kRow = mKBuffer.begin() + y * size.width;
... out[x*4+3] = kRow ? (255 - kRow[x]) : 0;   // reads mKBuffer[y*width + x]

Repro dimensions: width=65536, height=65537. Both pass the per-dimension INT32_MAX check, but
65536*65537 = 4,295,032,832 wraps (mod 2^32) to 65536, so mKBuffer is 65536 bytes while the loop reads
65537 rows x 65536 bytes.

Attached poc_m1_kbuffer.cpp is a self-contained ASan+UBSan harness faithfully reproducing the two
vulnerable statements plus the consumer loop. Build:
clang++ -std=c++17 -fsanitize=address,undefined -fno-sanitize-recover=address -g poc_m1_kbuffer.cpp -o poc_m1

Actual results:

UBSan: signed integer overflow: 65536 * 65537 cannot be represented in type 'int32_t' (poc line 41)
ASan: heap-buffer-overflow READ of size 1 at the kRow[x] read (poc line 57)
"located 0 bytes after 65536-byte region" (the under-allocated mKBuffer)

Full log attached (`asan_m1.txt`).

Honest scope: end to end, the pixel buffer (`CheckedInt<size_t>`, 64-bit) computes the true ~4.3 GB
size and must allocate first, so on typical RAM the decode OOMs before reaching the K-buffer line ->
the bug is OOM-adjacent ("OOM-triggered memory corruption" / Moderate), reachable on high-memory /
memory-overcommit systems. Not a cross-origin leak (canvas CORS tainting). The attached PoC is a
harness-level proof of the mechanism; I can build an end-to-end image/jxl testcase against a
JXL-enabled ASan build on request. Dedup: no public Bugzilla match (checked 2026-07-05).


Expected results:

Size the K buffer with overflow protection, exactly like the pixel buffer above it:
```cpp
CheckedInt<size_t> kSize = CheckedInt<size_t>(size.width) * size.height;
if (mPixelFormat.value() == PixelFormat::Cmyk8 &&
    (!kSize.isValid() || !mKBuffer.resize(kSize.value())))
    return NS_ERROR_FAILURE;

so an oversized/overflowing CMYK image is rejected rather than under-allocating mKBuffer. Optionally
also bounds-check kRow indexing in WritePixelRowsToPipe as defense in depth.

Group: core-security → gfx-core-security
Duplicate of this bug: 2052704
Severity: -- → S3
Status: UNCONFIRMED → NEW
Ever confirmed: true
Priority: -- → P3
Assignee: nobody → tnikkel
Keywords: regression
Regressed by: 2034984
Attached file (secure) —

Set release status flags based on info from the regressing bug 2034984

Comment on attachment 9604610 [details]
(secure)

Security Approval Request

  • How easily could an exploit be constructed based on the patch?: not too hard
  • Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?: Yes
  • Which branches (beta, release, and/or ESR) are affected by this flaw, and do the release status flags reflect this affected/unaffected state correctly?: 152+
  • If not all supported branches, which bug introduced the flaw?: Bug 2034984
  • Do you have backports for the affected branches?: No
  • If not, how different, hard to create, and risky will they be?: trivial
  • How likely is this patch to cause regressions; how much testing does it need?: very unlikely
  • Is the patch ready to land after security approval is given?: Yes
  • Is Android affected?: Yes
Attachment #9604610 - Flags: sec-approval?

Note that jxl is enabled by default on nightly only. On beta and release we surface the option to enable it in Firefox Labs, which shows up in the Firefox preferences, and jxl is getting a lot of external attention, so more exposed then your typical nightly only feature. I will ask for uplift because the fix is so trivial and the risk is so low, most people will have this code disabled (so zero risk to them), but the security gain is probably worth it for the small percent that enabled jxl.

Attached file (secure) —
Attachment #9604689 - Flags: approval-mozilla-beta?
Attached file (secure) (obsolete) —
Attachment #9604690 - Flags: approval-mozilla-release?

firefox-release Uplift Approval Request

  • User impact if declined/Reason for urgency: sec bug
  • Code covered by automated testing?: yes
  • Fix verified in Nightly?: no
  • Needs manual QE testing?: no
  • Steps to reproduce for manual QE testing:
  • Risk associated with taking this patch: low
  • Explanation of risk level: does multiplication in correctly sized int, code is disabled by default (jxl) so no risk to most users, but code can be enabled via firefox labs so more users have it enabled then most nightly only features
  • String changes made/needed?: none
  • Is Android affected?: yes

firefox-beta Uplift Approval Request

  • User impact if declined/Reason for urgency: sec bug
  • Code covered by automated testing?: yes
  • Fix verified in Nightly?: no
  • Needs manual QE testing?: no
  • Steps to reproduce for manual QE testing:
  • Risk associated with taking this patch: low
  • Explanation of risk level: does multiplication in correctly sized int, code is disabled by default (jxl) so no risk to most users, but code can be enabled via firefox labs so more users have it enabled then most nightly only features
  • String changes made/needed?: none
  • Is Android affected?: yes

The bug is marked as tracked for firefox153 (beta) and tracked for firefox154 (nightly). However, the bug still has low priority and has low severity.

:bhood, could you please increase the priority and increase the severity for this tracked bug? Given that it is a regression and we know the cause, we could also simply backout the regressor. If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Flags: needinfo?(bhood)
Flags: needinfo?(bhood)
Attachment #9604610 - Flags: sec-approval? → sec-approval+

The rating assigned by our bug rating tool (currently under development) is:

Rating: sec-moderate

Explanation: Valid heap OOB read confirmed in nsJXLDecoder.cpp: mKBuffer is sized at line 421 with size.widthsize.height, a 32-bit int32int32 product that can overflow, unlike the pixel buffer above which uses CheckedInt<size_t>. The CMYK consumer loop (lines 626-645) then reads kRow[x] out of bounds. However, to overflow the 32-bit product width*height must exceed ~2.1 billion, which forces the true-size pixel buffer (CMYK = 4 bytes/px, computed as CheckedInt<size_t>) to be >8.6 GB and allocate FIRST; on typical systems that allocation OOMs and the function returns before reaching the vulnerable K-buffer line (reporter concedes this). It is therefore a memory-safety issue constrained by OOM conditions with no precise triggering shown -> sec-moderate. Further mitigations: it is an OOB read (no write; huge OOB extent makes a crash far more likely than a controlled adjacent-heap disclosure), same-origin (not cross-origin), and JXL is opt-in (nightly default / Firefox Labs on beta/release).

tnikkel: Please ni? me if you disagree with this rating.

Keywords: sec-moderate
Attachment #9604689 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Flags: sec-bounty?
Group: gfx-core-security → core-security-release
Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → 154 Branch
QA Whiteboard: [sec] [qa-triage-done-c154/b153]
Attachment #9604690 - Attachment is obsolete: true
Attachment #9604690 - Flags: approval-mozilla-release?
Whiteboard: [adv-main153+]
Alias: CVE-2026-16402
Flags: sec-bounty? → sec-bounty+
Group: core-security-release
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: