The async implementation of CollectiveReduceV2
suffers from a memory leak and a use after free:
import tensorflow as tf
tf.raw_ops.CollectiveReduceV2(
input=[],
group_size=[-10, -10, -10],
group_key=[-10, -10],
instance_key=[-10],
ordering_token=[],
merge_op='Mul',
final_op='Div')
This occurs due to the asynchronous computation and the fact that objects that have been std::move()
d from are still accessed:
auto done_with_cleanup = [col_params, done = std::move(done)]() {
done();
col_params->Unref();
};
OP_REQUIRES_OK_ASYNC(c,
FillCollectiveParams(col_params, REDUCTION_COLLECTIVE,
/*group_size*/ c->input(1),
/*group_key*/ c->input(2),
/*instance_key*/ c->input(3)),
done);
Here, done
is already moved from by the time OP_REQUIRES_OK_ASYNC
macro needs to invoke it in case of errors. In this case, we get an undefined behavior, which can manifest via crashes, std::bad_alloc
throws or just memory leaks.
We have patched the issue in GitHub commit ca38dab9d3ee66c5de06f11af9a4b1200da5ef75.
The fix will be included in TensorFlow 2.7.0. We will also cherrypick this commit on TensorFlow 2.6.1, as this version is the only one that is also affected.
Please consult our security guide for more information regarding the security model and how to contact us with issues and questions.
This vulnerability has been reported by members of the Aivul Team from Qihoo 360.
{ "nvd_published_at": "2021-11-05T23:15:00Z", "cwe_ids": [ "CWE-416" ], "severity": "HIGH", "github_reviewed": true, "github_reviewed_at": "2021-11-08T22:07:11Z" }