Prior to Spark 2.3.3, in certain situations Spark would write user data to local disk unencrypted, even if spark.io.encryption.enabled=true. This includes cached blocks that are fetched to disk (controlled by spark.maxRemoteBlockSizeFetchToMem); in SparkR, using parallelize; in Pyspark, using broadcast and parallelize; and use of python udfs.
{
"versions": [
{
"introduced": "1.0.2"
},
{
"last_affected": "1.6.3"
},
{
"introduced": "2.0.0"
},
{
"last_affected": "2.0.2"
},
{
"introduced": "2.1.0"
},
{
"last_affected": "2.1.3"
},
{
"introduced": "2.2.0"
},
{
"last_affected": "2.2.2"
},
{
"introduced": "2.3.0"
},
{
"fixed": "2.3.2"
}
]
}