spark in the dark

Spark in the Dark: Silent Cross-Tenant Log Exfiltration in Amazon EMR

Dan Gansel November 02, 2026

Spark in the Dark: Silent Cross-Tenant Log Exfiltration in Amazon EMR

Book A Demo
See how Upwind uses runtime context to turn cloud security fingdings into real-time protection.
Book a Demo

This is the third post in our series on S3 bucket squatting through half-patched AWS patterns. Previously, we covered cross-tenant RCE in SageMaker via a broken ownership check and silent query result exfiltration in Athena through an SDK that was never patched alongside the service. This time, we cover an actively exploitable vulnerability in EMR.


TL;DR

We found that the aws-sdk-pandas library constructs a predictable EMR logging path and passes it to the RunJobFlow API without ever touching S3. The API accepts it without validation. AWS hardened the EMR console against this exact attack – the console checks bucket ownership before starting a cluster. The API does not. Any client that calls the API directly, including aws-sdk-pandas, bypasses the console’s protection entirely.

Key Findings

  • aws-sdk-pandas constructs a default logging path s3://aws-logs-{account_id}-{region}/elasticmapreduce/ – a predictable name in a globally unique namespace
  • The library never calls CreateBucket, HeadBucket, or any S3 API – it builds a string and passes it as LogUri to the EMR RunJobFlow API
  • The RunJobFlow API accepts any LogUri without validating bucket existence or ownership
  • AWS hardened the EMR console (which calls CreateBucket and blocks squatted buckets). The API and SDK path is unprotected.
  • Late bucket creation works: EMR’s log pusher buffers and retries, so an attacker can squat the bucket after the cluster starts and still capture logs retroactively
  • Captured data includes Spark secrets (PYSPARK_GATEWAY_SECRET), application output with PII, step source code, and full provisioning logs
  • We identified this vulnerability actively exploitable in customer environments across a couple of organizations
  • Unpatched at the API level. We submitted a fix to aws-sdk-pandas in September 2026: it was accepted in October 2026, AWS released v3.18.0 which includes the PR.

Background

Amazon EMR is AWS’s managed Hadoop/Spark platform. When an EMR cluster starts, it writes operational logs to an S3 location specified by the LogUri parameter – step stdout/stderr, YARN and HDFS daemon logs, bootstrap actions, provisioning reports, and Spark application output.

image-12
Creating an EMR cluster in the AWS Console

The aws-sdk-pandas library (formerly awswrangler) wraps the EMR RunJobFlow API via wr.emr.create_cluster(). When the caller doesn’t specify logging_s3_path, the library resolves it to a default:

def _get_default_logging_path(subnet_id=None, account_id=None, region=None, ...):

    return f"s3://aws-logs-{_account_id}-{_region}/elasticmapreduce/"

This path is passed to _build_cluster_args():

if pars.get("logging_s3_path") is None:

    pars["logging_s3_path"] = _get_default_logging_path(...)

...

cluster_args["LogUri"] = pars["logging_s3_path"]

What makes this vulnerability mechanically distinct from the SageMaker and Athena findings in this series: the library never interacts with S3 at all. There is no CreateBucket call, no HeadBucket call, no ownership check. The entire vulnerability surface is a string constructed from two predictable values – the account ID and the region – and passed directly to the EMR API.

The Vulnerability

The API Accepts Any Bucket

The RunJobFlow API accepts any S3 URI as LogUri and starts the cluster. It does not check whether the bucket exists. It does not check whether the caller owns it. It does not check whether it belongs to a different account entirely. The API trusts whatever string it receives and begins writing logs to it.

This is a problem independent of any client library. Any caller – the AWS CLI, Terraform, a custom script, or aws-sdk-pandas – can pass an attacker-controlled bucket as LogUri and the API will comply. LogUri is treated as a trusted input when it should be validated against the caller’s account.

aws-sdk-pandas makes exploitation trivial by constructing a predictable default, but even if the library didn’t exist, any RunJobFlow call with an unverified LogUri is vulnerable.

image-10
image-11
EMR cluster configured with a non-existent bucket

No Bucket Interaction

The code constructs a URI and hands it off. That is the entire vulnerability surface – two lines of code that build a predictable string and one that passes it to RunJobFlow.

The library does not create the bucket. It does not check if it exists. It does not verify who owns it. It does not call any S3 API.

Console vs API: The Mitigation Gap

AWS has already mitigated this attack in the EMR console. When creating a cluster through the console, a CreateBucket call is made for the default logging bucket. If the bucket is owned by another account, S3 probably returns BucketAlreadyExists and the console blocks cluster creation with an error:

image-17
EMR Cluster creation through the console

The RunJobFlow API has no equivalent protection. It accepts any LogUri value and starts the cluster regardless of whether the bucket exists, whether it belongs to the caller, or whether it’s owned by a different account entirely.

This means the console mitigation protects only console users. Any automation, SDK, CLI call, or library that calls RunJobFlow directly – including aws-sdk-pandas – bypasses the protection.

Late Bucket Creation

The attacker doesn’t need to squat the bucket before the victim creates the cluster. EMR’s log pusher service buffers logs locally and retries S3 writes at approximately 5-minute intervals. If the attacker creates the bucket while the cluster is already running, the log pusher flushes all buffered logs – including logs generated before the bucket existed – to the newly available bucket.

This is unique among the three vulnerabilities in this series. The attacker can observe that a target account is running EMR clusters and squat the logging bucket retroactively.

Independently, Yahav Festinger later demonstrated a related vector: if a logging bucket is deleted, an attacker can recreate it under their own account and EMR’s log pusher will resume writing to it. Our finding extends this – the bucket never existed in the first place – both exploit the same underlying behavior: the log pusher routes to a name, not an owner.

Impact: What Gets Captured

From a single test cluster running a PySpark step, we captured 49 log files across multiple categories:

Step stdout/stderr – complete output from EMR steps, including any data printed by the application. Example output:

=== Employee PII Records ===

|email            |name         |salary|ssn        |

|[email protected]|Alice Johnson|95000 |123-45-6789|

|[email protected]  |Bob Smith    |120000|987-65-4321|

|[email protected]|Carol Davis  |87000 |456-78-9012|

Any data that passes through df.show(), print(), or stdout during ETL steps ends up in the attacker’s bucket. This commonly includes PII, financial records, and business-critical data.

Step controller logs – full step definitions including inline script source code. The attacker can read the victim’s processing logic.

Spark internal secrets – PYSPARK_GATEWAY_SECRET, an auto-generated authentication token for Spark’s internal Py4J gateway, exposed via environment variable logging.

Provisioning logs – instance state transitions, Puppet configuration reports, bootstrap action output, installed packages.

YARN/HDFS daemon logs – container manager, resource manager, namenode/datanode logs with full cluster configuration including HDFS paths, YARN memory settings, and resource manager hostnames.

Attack Flow

Found in the Wild

This is not a theoretical risk. Through Upwind’s cloud security platform, we identified EMR clusters across a couple organizations using the vulnerable default logging pattern – s3://aws-logs-{account_id}-{region}/elasticmapreduce/ – spanning a few accounts and regions.

In every case, the default logging bucket did not exist. Any attacker who pre-created it would have silently captured all cluster logs.

These organizations were using aws-sdk-pandas or following the same default naming convention that the library establishes. The pattern propagates through documentation, example code, and infrastructure templates.

We helped an affected organization assess their exposure by squatting the default bucket. Roughly 20,000 log files landed in it:

image-13-1024x253
20k Exfiltrated files: stderr, stdout, scripts, etc…

Detection Challenges

  • No errors or warnings. The cluster starts normally, steps complete successfully, and the victim sees no indication that logs are being written to a bucket they don’t own.
  • No CloudTrail signal in the victim’s account. S3 write events for the attacker-owned bucket are logged in the attacker’s account, not the victim’s. The victim has no audit trail of the exfiltration.
  • Logs appear to work. From the victim’s perspective, logging is configured and the cluster ran. They would only discover the problem by checking whether the logging bucket actually exists in their account – and most don’t.
  • No validation failure to catch. Unlike a misconfigured bucket policy or a permissions error, this attack produces zero observable failures.

Disclosure and Fix Status

This vulnerability was reported to AWS via HackerOne in April 2026. The report was closed as informative. We are not the only researchers to have identified this issue.

No fix has been released for the RunJobFlow API. No security advisory or customer notification has been published. The EMR console already includes a mitigation for this exact attack pattern, which demonstrates AWS’s awareness of the risk. That mitigation has not been extended to the API layer.

In September 2026, we submitted a pull request to aws-sdk-pandas that adds bucket existence and ownership verification before passing LogUri to RunJobFlow. The PR has been accepted, it should close the library-side exposure – but the underlying API gap will remain open for any other caller.

In October 2026, the pull request has been accepted and the HackerOne report has been reopened. Later that month, AWS released v3.18.0, which should include our PR with the fix.

image-12-1024x626
image-13-1024x43



A note on that fix by AWS:

“That check needs s3:ListBucket on the bucket, and the call raises an error if the bucket is missing or owned by another account. A user whose role lacks that permission could see the new error after upgrading”

Recommendations

If you use EMR today:

  1. Audit every EMR logging bucket, not just the default. The RunJobFlow API does not validate any LogUri – custom paths are equally at risk if the bucket isn’t owned by your account. List all active and recent clusters (aws emr list-clusters), check the LogUri on each, and verify ownership with aws s3api head-bucket --bucket <bucket-name> --expected-bucket-owner <your-account-id>.
  2. Always specify logging_s3_path explicitly when calling wr.emr.create_cluster() or the RunJobFlow API. Do not rely on the default.
  3. Create and verify the logging bucket yourself before launching any cluster. Run aws s3api head-bucket --bucket <your-logging-bucket> --expected-bucket-owner <your-account-id> to confirm ownership.
  4. If the default bucket doesn’t exist, create it now – aws s3 mb s3://aws-logs-{account_id}-{region} – before an attacker does.

What should change:

  • The RunJobFlow API should validate that LogUri points to a bucket owned by the caller’s account – bringing the API in line with the existing console behavior
  • AWS should extend the console’s CreateBucket mitigation to the API layer so that all callers are protected, not just console users

Disclosure Timeline

DateEvent
Late April 2026Vulnerability identified in aws-sdk-pandas EMR integration
Late April 2026Report submitted to AWS via HackerOne
Late April 2026Report closed as informative
May-June 2026Follow-ups
Mid September 2026Fix submitted as PR to aws-sdk-pandas
Mid October 2026RunJobFlow API remains unpatched. No advisory published.
Fix PR accepted and the library should be protected.
HackerOne Report reopened.
Late October 2026Fix PR should be included in v3.18.0 of the SDK, CVE in review

Conclusion

This is the simplest vulnerability in this series – mechanically, it’s a string concatenation that produces a predictable S3 URI and an API that accepts it without question. There is no broken ownership check to analyze, no logic bug to trace through. The library builds a name, and the API trusts it.

The console was fixed. The API was not. The gap between the two is where every SDK, CLI, and automation tool operates – and where the vulnerability lives.

If you use EMR with default logging paths, check your bucket ownership today. A library-level fix has been accepted, but the API-level gap remains open.


In This Series

  1. Training on Enemy Ground: Cross-Tenant RCE in Amazon SageMaker 
  2. Athena’s Trojan Horse: Cross-Tenant Data Poisoning and Exfiltration via AWS Athena
  3. Spark in the Dark: Silent Cross-Tenant Log Exfiltration in Amazon EMR (this post)
  4. Kick the Bucket: S3 Bucket Squatting Through a Series of Half-Patched AWS Patterns

Demo the Upwind Platform

See how Upwind is reducing risk and strengthening cloud security for global enterprises.
Get a Demo
Contents
Add the Upwind RSS Feed to Slack
Connect the Upwind RSS Feed to your Slack.
Follow the how-to here.
Threat RSS
Add the Upwind RSS Feed to Slack
Connect the Upwind RSS Feed to your Slack.
Follow the how-to here.
Main RSS