CVE-2024-35956

In the Linux kernel, the following vulnerability has been resolved:

btrfs: qgroup: fix qgroup prealloc rsv leak in subvolume operations

Create subvolume, create snapshot and delete subvolume all use btrfssubvolumereservemetadata() to reserve metadata for the changes done to the parent subvolume's fs tree, which cannot be mediated in the normal way via starttransaction. When quota groups (squota or qgroups) are enabled, this reserves qgroup metadata of type PREALLOC. Once the operation is associated to a transaction, we convert PREALLOC to PERTRANS, which gets cleared in bulk at the end of the transaction.

However, the error paths of these three operations were not implementing this lifecycle correctly. They unconditionally converted the PREALLOC to PERTRANS in a generic cleanup step regardless of errors or whether the operation was fully associated to a transaction or not. This resulted in error paths occasionally converting this rsv to PERTRANS without calling recordrootintrans successfully, which meant that unless that root got recorded in the transaction by some other thread, the end of the transaction would not free that root's PERTRANS, leaking it. Ultimately, this resulted in hitting a WARN in CONFIGBTRFS_DEBUG builds at unmount for the leaked reservation.

The fix is to ensure that every qgroup PREALLOC reservation observes the following properties:

1. any failure before recordrootin_trans is called successfully results in freeing the PREALLOC reservation.
2. after recordrootin_trans, we convert to PERTRANS, and now the transaction owns freeing the reservation.

This patch enforces those properties on the three operations. Without it, generic/269 with squotas enabled at mkfs time would fail in ~5-10 runs on my system. With this patch, it ran successfully 1000 times in a row.