Training on Enemy Ground: Cross-Tenant RCE in Amazon SageMaker

Dan Gansel November 02, 2026

Training on Enemy Ground: Cross-Tenant RCE in Amazon SageMaker

Book A Demo
See how Upwind uses runtime context to turn cloud security fingdings into real-time protection.
Book a Demo

This is the first in a four-part series on S3 bucket squatting vulnerabilities we found across AWS services, each in a service that had already been “fixed.” We start with the most severe: cross-tenant remote code execution in Amazon SageMaker.


TL;DR

We found that AWS SageMaker’s default S3 bucket – used by virtually every SageMaker tutorial and notebook – follows a predictable naming pattern that allows an attacker to pre-register it in their own account. Once squatted, the attacker silently intercepts training data, poisons model artifacts, and achieves cross-tenant remote code execution in the victim’s environment.

We share our findings on the SageMaker Python SDK and the implications of bucket name squatting across SageMaker’s ML workflows – from data exfiltration to persistent cross-tenant code execution on live inference endpoints. Following our responsible disclosure report to AWS, a patch was released in version 3.10.0 of the SageMaker Python SDK.

Key Findings

  • The SageMaker Python SDK creates a default S3 bucket named sagemaker-{region}-{account_id} – a fully predictable name in a globally unique namespace
  • AWS shipped an ownership check (ExpectedBucketOwner) in October 2023. It contains a logic bug: the check only runs when the bucket is already in the victim’s account listing – exactly the case where it isn’t needed
  • Multiple S3 upload and download operations across the SDK never pass ExpectedBucketOwner at all, bypassing even a corrected session-level check
  • We demonstrated two independent RCE vectors: processing job code injection and persistent endpoint serving code injection, both with live IAM credential exfiltration
  • The attack produces no errors, no warnings, and no S3 CloudTrail events in the victim’s account

Background

AWS S3 bucket names must be unique across all AWS accounts in all regions within a partition. A common pattern in AWS services is to automatically create S3 buckets using a deterministic naming convention that includes the region and account ID. As prior research has shown, this pattern creates a window for bucket name squatting: an attacker who can predict the bucket name can create it first.

image-1-1024x74
AWS Docs: General purpose bucket naming rules

Amazon SageMaker is AWS’s managed machine learning platform. Nearly every SageMaker workflow – training jobs, processing jobs, model deployments – stores artifacts in S3. The SageMaker Python SDK provides a default_bucket() helper that constructs and returns a bucket with this name:

sagemaker-{region}-{account_id}

If you’ve used SageMaker, you’ve almost certainly seen this:

import sagemaker

session = sagemaker.Session()

bucket = session.default_bucket()

The aws/amazon-sagemaker-examples repository – 11k+ stars, 200+ notebooks – uses this pattern throughout. Every official AWS SageMaker workshop, quickstart, and blog post follows the same convention.

image-5-1024x336
Example notebook from aws/amazon-sagemaker-examples

The Vulnerability: Squatting the Bucket

The attack starts with a single piece of information: the victim’s 12-digit AWS account ID. This isn’t a secret – it leaks through public AMIs, error messages, IAM role ARNs in shared resources, CloudFormation exports, and more.

image-2-1024x241
AWS Docs: View AWS account identifiers

With the account ID and a target region, the attacker knows the bucket name. They create it in their own account and apply a bucket policy granting the victim read/write access:

bucket_name = f"sagemaker-{region}-{victim_account_id}"

policy = {

    "Version": "2012-10-17",

    "Statement": [{

        "Effect": "Allow",

        "Principal": {"AWS": f"arn:aws:iam::{victim_account_id}:root"},

        "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject",

                    "s3:ListBucket", "s3:GetBucketLocation"],

        "Resource": [f"arn:aws:s3:::{bucket_name}",

                     f"arn:aws:s3:::{bucket_name}/*"]

    }]

}

When the victim’s SDK calls default_bucket(), the call succeeds – the bucket exists and the victim has access. The SDK proceeds without error.

The victim has no indication that the bucket belongs to another account.

The Broken Fix

AWS added an ExpectedBucketOwner parameter to the session helper in v2.190.0 (October 2023). This is the standard S3 mechanism for verifying bucket ownership – it causes S3 to return 403 if the bucket owner doesn’t match the expected account ID.

image-4-1024x780
Mock of the code in aws/sagemaker-python-sdk

However, the fix has a logic bug. The SDK iterates over the caller’s bucket listing and checks if bucket.creation_date is not None.

If the bucket’s creation date is none the expected owner check doesn’t run which seems weird as there is no reason not to run the check anyways. I checked the two cases and to my surprise:

image-3
Bucket creation_date on an owned bucket

image-6
Bucket creation_date on a bucket I don’t own

If the bucket is in the listing (meaning the caller owns it), the ownership check runs. If it isn’t in the listing – which is exactly what happens when an attacker pre-creates the bucket – creation_date is None, and the code falls through to the unprotected branch.

The fix protects the case that doesn’t need protection (victim already owns the bucket) and skips the case that does (someone else owns it).

The vulnerability remained fully exploitable through April 2026 – over two years after the fix was introduced – when we reported it to AWS. The actual fix shipped in v3.10.0 via PRs #5801 and #5803, which add ExpectedBucketOwner to S3 data-plane calls across the SDK.

Beyond the session-level logic bug, we identified several S3 upload and download operations across the SDK that never pass ExpectedBucketOwner at all. Even if the session-level check had been correct, these code paths bypassed it entirely.

Impact: From Data Theft to Remote Code Execution

What makes the SageMaker case especially concerning is what flows through the bucket. SageMaker doesn’t just store static files – it stores executable code that runs in containers with the victim’s IAM role. This turns bucket squatting into a direct path to remote code execution.

We demonstrated this through two independent attack vectors affecting different SageMaker features.

Processing Job Code Injection

When a data scientist runs a SageMaker processing job with a local Python script, the SDK uploads the script to:

s3://{default_bucket}/{job_name}/input/code/preprocessing.py

The attacker deploys an S3 event-triggered Lambda function on the squatted bucket. The timing works overwhelmingly in the attacker’s favor:

  1. The victim’s SDK uploads the script to S3
  2. S3 fires an ObjectCreated event – the attacker’s Lambda triggers within milliseconds
  3. The Lambda reads the victim’s original script, appends an exfiltration payload, and overwrites the S3 object
  4. SageMaker provisions a processing container – this takes 2 to 5 minutes
  5. The container downloads the now-modified script and executes it
  6. The victim’s code runs normally (the job succeeds), then the attacker’s payload runs

The payload harvests IAM credentials, environment variables, and all input data from the processing container, writing the results back to the attacker’s bucket.

Result – live IAM credentials exfiltrated from the victim’s SageMaker execution role:

{

  "role_arn": "arn:aws:sts::VICTIM_ACCOUNT:assumed-role/SageMakerExecutionRole/...",

  "credentials_harvested": true,

  "access_key_id": "ASIAWN26...[REDACTED]",

  "secret_access_key": "7KppHfX9...[REDACTED]",

  "session_token": "IQoJb3JpZ2luX2VjEEsa...[TRUNCATED]"

}

Endpoint Serving Code Injection (Persistent)

This is the more severe vector.

When a victim deploys a model to a SageMaker endpoint, the SDK packages the serving code into sourcedir.tar.gz and uploads it to the squatted bucket. The SDK encrypts the archive using the default AWS-managed S3 KMS key – which belongs to the bucket owner. That’s the attacker. The attacker’s Lambda can decrypt, unpack, inject into, repack, and rewrite the serving code.

The injected payload does two things:

  • On container startup and health checks: exfiltrates IAM credentials
  • On every inference request: captures the complete request body and exfiltrates it

Unlike the processing job vector (which executes once), a deployed SageMaker endpoint can run for days, weeks, or months. For the entire lifetime of the endpoint, every inference input is silently forwarded to the attacker.

The victim’s model loads correctly. Predictions return accurate results. The attack produces no errors, no warnings, and no log anomalies.

Attack Flow

Beyond RCE

Code injection is the most severe outcome, but the squatted bucket exposes the full surface of the victim’s ML workflow:

  • Training data exfiltration. All training data uploaded to the default bucket is readable by the attacker – proprietary datasets, customer data, labeled examples.
  • Training data poisoning. The attacker can modify training inputs before jobs consume them, silently degrading or biasing the resulting models.
  • Model artifact theft. Trained model weights and architectures stored in the bucket are the attacker’s to copy – direct intellectual property theft.
  • Model poisoning. The attacker can swap or tamper with saved model artifacts, so the victim deploys a model that was never the one they trained.

These impacts don’t require code injection or precise timing. They follow directly from the attacker’s read/write access to the bucket that underpins the victim’s entire ML pipeline.

Detection Challenges

  • No S3 data events for the squatted bucket in the victim’s account. S3 data events for operations on the attacker-owned bucket are logged in the attacker’s account, not the victim’s. The victim has no audit trail.
  • No SDK errors. S3 actions succeed. From the SDK’s perspective, everything is working as intended.
  • The victim’s code still works. The attacker’s payload wraps the original code – jobs succeed, models serve correct predictions, pipelines complete.
  • No “wrong account” signal. The victim can read and write to the bucket. There is no visible difference between a bucket they own and one that’s been squatted with a permissive policy.

Scale

The default_bucket() pattern is not a shortcut or a workaround. It is the documented, recommended way to use SageMaker:

  • 200+ notebooks in aws/amazon-sagemaker-examples (11k+ stars) use default_bucket()
  • Every official AWS SageMaker workshop and quickstart uses this pattern
  • The SageMaker Python SDK (aws/sagemaker-python-sdk – 2.5k Stars) has been downloaded millions of times
  • SageMaker operates in 30+ AWS regions – each a separate squatting target per account

An attacker does not need to target a specific victim. They can pre-register sagemaker-{region}-{account_id} for batches of account IDs across all SageMaker regions and wait. When any of those accounts first uses SageMaker, the SDK silently adopts the attacker’s bucket.

Recommendations

If you use SageMaker today, take these steps now:

  1. Check your bucket ownership. Run aws s3api head-bucket –bucket sagemaker-{region}-{account_id} –expected-bucket-owner <account-id>  and verify the owner matches your account.
  2. Create the bucket yourself before first SDK use if it doesn’t exist yet – aws s3 mb s3://sagemaker-{region}-{account_id} from your own account.
  3. Stop using default_bucket(). Pass an explicit, uniquely-named bucket to SageMaker APIs.
  4. Use SCPs to restrict S3 operations to buckets owned by accounts in your organization.
  5. Upgrade to SageMaker Python SDK v3.10.0 or later.

Disclosure Timeline

Official references published by AWS:

DateEvent
Early April 2026Vulnerability identified in SageMaker Python SDK
Early April 2026Report submitted to AWS Security via HackerOne
Mid April 2026Triaged by AWS
Early May 2026Fix released in SageMaker Python SDK v3.10.0 (PRs #5801, #5803)
Mid September 2026AWS CNA confirmed that this finding is in scope for CVE assignment
Mid October 2026AWS CNA assigned CVE-2026-112408

Conclusion

Amazon S3 serves as the backbone of SageMaker’s ML workflows. When the SDK automatically creates a bucket with a predictable name and treats its existence as proof of ownership, it opens a window that an attacker can step through before the victim arrives.

What makes the SageMaker case particularly impactful is the nature of what flows through the bucket. This isn’t static configuration or template files – it’s executable code running with privileged IAM roles, and live production data passing through inference endpoints. A single squatted bucket gives the attacker access to an organization’s training data, model intellectual property, and real-time customer data, with remote code execution as a bonus.

The fix is conceptually straightforward: verify ownership, not just existence. Even better, start by creating the S3 bucket yourself and confirming its ownership before letting any SDK or service operate against it.

If you use SageMaker, check your bucket ownership today.

In This Series

  1. Training on Enemy Ground: Cross-Tenant RCE in Amazon SageMaker (this post)
  2. Athena’s Trojan Horse: Cross-Tenant Data Poisoning and Exfiltration via AWS Athena
  3. Spark in the Dark: Silent Cross-Tenant Log Exfiltration in Amazon EMR
  4. Never Skip Leg Day: S3 Bucket Squatting Through a Series of Half-Patched AWS Patterns

Demo the Upwind Platform

See how Upwind is reducing risk and strengthening cloud security for global enterprises.
Get a Demo
Contents
Add the Upwind RSS Feed to Slack
Connect the Upwind RSS Feed to your Slack.
Follow the how-to here.
Threat RSS
Add the Upwind RSS Feed to Slack
Connect the Upwind RSS Feed to your Slack.
Follow the how-to here.
Main RSS