notyet: Open-Source Tool to Test AWS IAM Credential Revocation Gaps

Key Takeaways

AWS IAM credential revocation is not instant. IAM eventual consistency creates a roughly 4-second propagation window where disabled or deleted credentials still work.
Attackers can automate persistence within that window by rotating to new users, roles, and credentials before revocation takes effect.
notyet is an open-source tool that simulates this attack, giving incident response (IR) teams a way to validate whether their containment playbooks actually stop an adversary that fights back.
The same consistency gap exists in Azure, GCP, and other cloud providers, making this a cloud-wide IR concern.

When an incident responder disables an access key or deletes a compromised IAM user, they expect the attacker to lose access immediately. Well, not yet. In AWS (or any other major cloud provider for that matter), that's not what happens.

AWS IAM is an eventually consistent system. Changes like key deletion, policy detachment, or role removal take up to several seconds to propagate globally. During that window, the old credentials still work. An attacker who knows this can use those seconds to create new users, assume new roles, and re-establish access before the revocation takes effect.

notyet is an open-source tool that exploits this propagation window to maintain persistent access after defenders revoke credentials. It was built to help IR teams test their containment playbooks against a real, automated adversary that fights back, and to make the results visible in real time.

What Is AWS IAM Eventual Consistency?

IAM eventual consistency is not a bug. It is an architectural property of how AWS distributes IAM state across regions. When you disable an access key, the change propagates over roughly 4 seconds. During that time, the disabled key can still be used to make IAM API calls: creating users, attaching policies, assuming roles.

This means that a defender who disables a key and moves on has given the attacker a window to rotate to fresh credentials. If the attacker automates this, the defender is playing whack-a-mole against a script that reacts faster than they can click.

OFFENSAI's research team was the first to discover and weaponize this propagation window, publishing both the offensive technique and a defensive countermeasure:

Exploiting AWS IAM Eventual Consistency for Persistence details how disabled access keys can be used during the ~4-second window to create new IAM users, assume roles, and re-establish full access before revocation propagates.
AWS Incident Response: IAM Containment That Survives Eventual Consistency introduces an SCP-enforced quarantine technique that makes containment policies irremovable by the attacker, closing the gap that standard IR playbooks leave open.

Independent research by Edwin Chua later validated that the same eventual consistency issue extends beyond AWS to Azure, GCP, and other cloud providers:

Exploring Eventual Consistency Amongst Major Cloud Service Providers

How notyet Works

notyet works with both temporary and persistent AWS access credentials, either supplied directly or via an AWS CLI profile. It starts two concurrent monitoring loops:

Health check loop (every 5 seconds): calls S3 ListBuckets to detect whether access has been revoked
Policy monitor loop (every 0.5 seconds): checks the identity's state and reacts to defender actions

When the monitor detects a defensive action, it responds within the consistency window:

notyet CLI terminal showing AWS IAM credential revocation detected and automated attacker response within the eventual consistency window — notyet detecting a defender action and responding within the consistency window

Persistence Techniques

The tool handles two credential types, persistent access keys (AKIA) and temporary role sessions (ASIA), and implements several persistence mechanisms:

Credential rotation. When an access key is disabled or deleted, notyet creates a temporary IAM role, assumes it, and uses the role's credentials to create a brand new user with a random name. The defender doesn't know the new username, so they can't pre-target it.

Role session rotation. When a role is deleted, notyet creates a new role with a same-account trust policy and assumes it. For session expiry, it proactively rotates 5 minutes before the session expires.

Policy persistence. notyet attaches an admin inline policy and continuously validates its content. It checks not just that the policy exists, but that it hasn't been modified (e.g., changing Allow to Deny or adding a Condition that short-circuits evaluation). If tampered with, the policy is immediately overwritten.

Defensive action stripping. The tool monitors for and removes:

Managed policies with deny effects
Other inline policies added by defenders
Permission boundaries
Group memberships with deny policies

The Web UI

notyet includes a real-time web interface designed as a training and awareness tool. The dashboard shows a split view: defender actions appear in red on the right, attacker responses in green, with a central event log showing the full timeline.

notyet web UI dashboard with three-column layout showing real-time AWS IAM credential revocation events for both defender and attacker — notyet web UI dashboard showing real-time defender and attacker events

This makes the race condition tangible. Instead of reading about eventual consistency in documentation, an IR team can watch it happen: disable a key in one terminal and see the tool detect the change and rotate to new credentials in under a second.

The UI includes:

Control panel: start, stop, and configure sessions from the browser
Event log: searchable, filterable real-time log with color-coded event types
Defender/Attacker panels: dedicated views for each side of the interaction
Session management: switch between past sessions for comparison

notyet defender panel showing sequence of AWS IAM containment actions detected during incident response testing — Defender panel showing a sequence of detected containment actions

Getting Started

Install dependencies

notyet requires Python 3.11+ and uv.

Clone and install notyet

Clone the repository and install dependencies with uv.

git clone https://github.com/OffensAI/notyet.git
cd notyet
uv sync

Run notyet from the CLI

notyet supports three credential input methods: an AWS CLI profile, explicit access keys, or temporary role credentials.

# Using an AWS profile
uv run notyet --profile attacker --output-profile attacker-tmp
 
# Using explicit credentials
uv run notyet --access-key-id AKIA... --secret-access-key ... --output-profile attacker-tmp
 
# With temporary role credentials
uv run notyet --access-key-id ASIA... --secret-access-key ... --session-token ... --output-profile attacker-tmp

Launch the web UI

Start the backend server and open http://localhost:8000 to access the real-time dashboard.

uv run --with-requirements backend/requirements.txt uvicorn backend.app:app --host 0.0.0.0 --port 8000

Clean up test resources

After testing, remove all notyet-* IAM users, roles, and policies from the account.

uv run notyet cleanup --profile admin

Red-Blue Collaboration

notyet was hardened through collaboration with Nigel Sood, Cloud Privilege Threat Researcher at Sonrai Security. Nigel tested the tool from the perspective of an incident responder attempting to contain a compromised identity, systematically trying every containment action available in AWS and reporting what worked and what didn't.

This feedback loop (researcher tests containment, finds a gap, tool gets updated, researcher tests again) drove significant improvements to notyet's detection and rotation capabilities, including group membership monitoring, policy content validation, and role session refresh.

Nigel's companion piece on the containment strategies and their effectiveness will be linked here when published.

Why This Matters for Cloud Incident Response

AWS IAM eventual consistency is a known architectural property, but its impact on incident response is underappreciated. Most IR playbooks treat credential revocation as an atomic operation: disable the key, problem solved. notyet demonstrates that this assumption doesn't hold, and gives IR teams a concrete way to test their containment procedures against an adversary that exploits the gap.

If your IR runbooks do not account for the propagation delay, an automated attacker can re-establish persistence faster than a human operator can respond. The only reliable defense is SCP-enforced containment that blocks all actions at the organization level, bypassing the IAM propagation window entirely.

notyet is open source and available on GitHub: github.com/OffensAI/notyet

Frequently Asked Questions

Why doesn't disabling an AWS access key revoke access immediately?

AWS IAM is an eventually consistent system. When you disable or delete an access key, the change must propagate across all AWS regions and service endpoints. This takes roughly 4 seconds. During that window, the old credentials remain valid and can be used to make API calls, including creating new users, assuming roles, and attaching policies.

What is notyet and who is it for?

notyet is an open-source adversary simulation tool designed by OFFENSAI's research team to exploit the AWS IAM eventual consistency window and maintain persistent access after credential revocation. It is designed for incident response teams, red teams, and cloud security engineers who want to validate whether their containment playbooks actually work against an attacker that automates credential rotation within the propagation delay.

Does IAM eventual consistency affect Azure, GCP, and other cloud providers?

Yes. OFFENSAI first discovered and published research on how IAM propagation delays enable persistence and why credential revocation alone fails as a containment strategy, then developed SCP-enforced containment as a defense. Independent research later validated that the same gap exists in Azure, GCP, and other cloud providers, making this a cross-cloud IR concern.

How do you defend against IAM eventual consistency attacks?

For AWS IAM eventual consistency attacks, the most reliable defense is SCP-enforced containment using AWS Organizations Service Control Policies. SCPs bypass the IAM propagation delay because they are evaluated at the Organizations level, not the account level. OFFENSAI published a detailed SCP-based containment technique that makes quarantine policies irremovable by compromised identities, and teams can use notyet to continuously validate that these containment controls hold under automated attacker behavior.

Is notyet safe to run in a production AWS account?

notyet is a security testing tool that creates IAM users, roles, and policies in the target account. It should only be run in isolated test accounts with proper authorization. The tool includes a cleanup command (notyet cleanup) that removes all notyet-* resources after testing.