OffensAI Logo

Exploring the Integration of AWS Lake Formation and Amazon EMR on EKS

Discover how to leverage AWS Lake Formation with Amazon EMR on EKS for efficient big data analytics. Simplify cluster management and optimize your data workflows.

Dragos Stanescu
Dragos Stanescu - Chief Product Officer
Exploring the Integration of AWS Lake Formation and Amazon EMR on EKS

Introduction

The integration of AWS Lake Formation with Amazon EMR on EKS represents a significant advancement in simplifying data lake security and governance while enhancing the flexibility of big data processing on Kubernetes. This integration, announced in February 2025, introduces fine-grained access control (FGAC) capabilities that allow organizations to enforce security policies at the database, table, column, row, and even cell levels. By leveraging Lake Formation's governance features and EMR on EKS's scalability, businesses can efficiently manage and secure their data lakes while running analytics workloads on Apache Spark and other frameworks.

This blog post outlines the benefits and challenges of this integration, providing insights into its practical applications and potential pitfalls. Additionally, it includes examples of flawed and proper AWS policy templates to illustrate best practices for implementing secure and efficient configurations.

Benefits of AWS Lake Formation and EMR on EKS Integration

  • Enhanced Data Security and Governance

    • The integration enables fine-grained access control (FGAC), allowing administrators to define granular permissions at the row, column, and cell levels. This ensures that sensitive data is protected while maintaining accessibility for authorized users.
    • Policies defined in Lake Formation are automatically enforced across compatible engines, including Spark on EMR on EKS, Athena, and Redshift Spectrum.
    • All access requests are logged in AWS CloudTrail, providing robust auditing capabilities for compliance and security monitoring.
    • For more details, see the official announcement on AWS Lake Formation FGAC with EMR on EKS.

  • Simplified Data Lake Management

    • Lake Formation provides a centralized platform to catalog, cleanse, and secure data in Amazon S3, reducing the complexity of managing large-scale data lakes.
    • The integration supports Glue Data Catalog views, enabling Spark jobs to create reusable views that can be queried by multiple engines without exposing underlying data.

  • Improved Resource Utilization

    • By running EMR workloads on Amazon EKS, organizations can consolidate big data processing with other containerized applications, improving resource utilization and reducing infrastructure costs.
    • The shared EKS cluster supports multiple versions of open-source frameworks, simplifying infrastructure management and enabling efficient collaboration across teams.

  • Flexibility and Scalability

    • EMR on EKS allows businesses to run analytics workloads on a common pool of resources without provisioning dedicated infrastructure.
    • The integration supports serverless options like AWS Fargate, enabling dynamic scaling based on workload demands.

  • Cross-Account Data Sharing

    • Lake Formation facilitates secure cross-account data sharing through resource links and named grants, ensuring that only authorized users in external accounts can access shared datasets.

  • Fine-Grained Access Control (FGAC)

    • AWS Lake Formation provides robust fine-grained access control (FGAC) capabilities when integrated with Amazon EMR on EKS. With this integration, organizations can enforce access policies at the table, row, column, and even cell levels within their data lakes. This feature is particularly beneficial for ensuring compliance with regulatory requirements such as GDPR or HIPAA. For example, administrators can configure Lake Formation to allow Spark jobs on EMR to access only specific rows or columns of sensitive data. This ensures that data access is limited to authorized users and processes. (AWS announcement)

  • Support for Open-Table Formats

    • This integration supports open-table formats such as Apache Hudi, Delta Lake, and Iceberg, enabling seamless data sharing and interoperability across different analytics engines. Organizations can use Glue Data Catalog views to create virtual tables that can be queried by Spark jobs on EMR without exposing the underlying data. This feature simplifies multi-engine data workflows and enhances the flexibility of data processing pipelines. (AWS documentation)

  • Enhanced Security Features

    • By combining the security features of Lake Formation and EMR on EKS, organizations can enforce encryption at multiple levels. For example, inter-worker encryption is enabled by default when using Lake Formation with EMR Serverless. Additionally, administrators can configure client-side encryption (CSE) for data stored in Amazon S3, ensuring that sensitive data remains protected even during processing. This layered security approach minimizes the risk of data breaches.

Challenges of AWS Lake Formation and EMR on EKS Integration

Complexity in Configuration

Integrating Lake Formation with EMR on EKS requires a multi-step setup process, including configuring IAM roles, creating security configurations, and enabling fine-grained access controls. For example, administrators must set the LakeFormationAuthorizedCaller session tag and configure EKS role-based access control (RBAC) permissions. This complexity can be a barrier for organizations with limited expertise in AWS services. (AWS documentation)

Limited Third-Party Integration

While Lake Formation integrates seamlessly with AWS services, its support for third-party tools and platforms is limited. This can be a challenge for organizations that rely on non-AWS tools for data processing or analytics. For instance, integrating Lake Formation with external data visualization tools may require additional configurations or custom development.

Performance Overheads

Enforcing fine-grained access controls can introduce performance overheads, particularly for large-scale Spark jobs. For example, applying row-level or cell-level permissions may require additional processing time, which can impact query performance. Additionally, operations such as INSERT INTO and INSERT OVERWRITE on non-governed Apache Spark tables may not benefit from Lake Formation optimizations, further affecting performance. (AWS documentation)

Troubleshooting and Debugging

Debugging issues in an integrated Lake Formation and EMR on EKS environment can be challenging due to the complexity of permissions and configurations. For instance, access errors such as "Insufficient Lake Formation permission(s)" often require detailed investigation of IAM policies, Lake Formation permissions, and Spark job configurations. This can be time-consuming and may require specialized knowledge. (AWS re:Post)

Cost Considerations

While the integration offers cost efficiencies through dynamic scaling, it can also incur additional costs for organizations that require extensive fine-grained access controls. For example, using EMR Serverless with Lake Formation incurs additional charges, as noted in the AWS documentation. Organizations must carefully evaluate their budget and workload requirements to determine the cost-effectiveness of this integration. (AWS documentation)

Complex Configuration Requirements**

Setting up FGAC policies requires careful configuration of IAM roles, KMS key policies, and Lake Formation permissions. Misconfigurations can lead to unauthorized access or operational failures. For example, administrators must explicitly enable external engines to filter data in S3 locations registered with Lake Formation.

Example of a Flawed AWS Policy Template

A flawed IAM policy for integrating EMR on EKS with Lake Formation might look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    }
  ]
}

Issues with This Policy:

  • Overly Broad Permissions: The policy grants unrestricted access to all AWS services and resources, violating the principle of least privilege.
  • Security Risks: Such a policy increases the risk of accidental or malicious actions, including data breaches or unauthorized access to sensitive data.
  • Non-Compliant with Best Practices: AWS recommends using managed policies or creating custom policies with specific permissions tailored to the use case.

Proper AWS Policy Template for Lake Formation and EMR on EKS Integration

A well-constructed IAM policy for this integration would follow the principle of least privilege and include specific permissions for required actions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "lakeformation:GetDataAccess",
        "glue:GetTable",
        "glue:GetDatabase",
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket",
        "kms:Decrypt"
      ],
      "Resource": [
        "arn:aws:lakeformation:region:account-id:table/*",
        "arn:aws:lakeformation:region:account-id:database/*",
        "arn:aws:s3:::bucket-name/*",
        "arn:aws:kms:region:account-id:key/key-id"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "emr-containers:StartJobRun",
      "Resource": "arn:aws:emr-containers:region:account-id:virtualcluster/*"
    }
  ]
}

Key Features of This Policy:

  • Specific Actions: The policy includes only the actions required for accessing Lake Formation, Glue Data Catalog, S3, and KMS resources.
  • Resource-Level Permissions: Permissions are scoped to specific resources, such as tables, databases, and S3 buckets.
  • Compliance with Best Practices: By limiting permissions to necessary actions and resources, this policy aligns with AWS security best practices. (AWS documentation)

Examples of Flawed and Proper AWS Policy Templates for Lake Formation and EMR on EKS Integration

Common Pitfalls in AWS Policy Design for Lake Formation and EMR on EKS

Overly Broad Permissions

As mentioned previosuly, one of the most significant flaws in AWS policy design is granting overly broad permissions. For instance, a policy that allows all actions on all resources ("Action": "*", "Resource": "*") violates the principle of least privilege and increases the risk of unauthorized access. This type of policy is often created for convenience but poses severe security risks, especially in sensitive integrations like AWS Lake Formation and EMR on EKS.

A flawed example of such a policy might look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    }
  ]
}

This policy grants unrestricted access to all AWS services and resources, which is unnecessary and dangerous for most use cases. It fails to limit access to only the required actions and resources, leaving the environment vulnerable to accidental or malicious misuse.

Lack of Resource-Level Permissions

Another common issue is the absence of resource-level permissions. For example, if a policy allows access to all S3 buckets instead of specific buckets required for the job, it increases the attack surface and violates AWS security best practices. This is particularly problematic when integrating EMR on EKS with Lake Formation, as sensitive data stored in S3 buckets could be exposed.

For instance, a flawed policy might include:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::*"
    }
  ]
}

This policy allows unrestricted access to all S3 buckets, rather than scoping permissions to specific buckets required for the integration.

Ignoring Fine-Grained Access Control (FGAC)

AWS Lake Formation supports fine-grained access control (FGAC) at the database, table, column, row, and cell levels. However, some policies fail to leverage this feature, granting broader access than necessary. For example, a policy that allows access to an entire database when only specific tables or columns are needed introduces unnecessary risk.

A flawed policy might look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "lakeformation:GetDataAccess",
      "Resource": "arn:aws:lakeformation:us-east-1:123456789012:database/*"
    }
  ]
}

This policy grants access to all databases in the account, rather than limiting access to specific databases, tables, or columns required for the job.

Proper AWS Policy Templates for Lake Formation and EMR on EKS Integration

Principle of Least Privilege

A proper policy adheres to the principle of least privilege, granting only the permissions necessary for the specific use case. For example, a policy for integrating EMR on EKS with Lake Formation should include permissions for accessing only the required S3 buckets, Glue Data Catalog resources, and Lake Formation tables.

A well-designed policy might look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-data-lake-bucket",
        "arn:aws:s3:::my-data-lake-bucket/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "lakeformation:GetDataAccess",
        "glue:GetTable",
        "glue:GetDatabase"
      ],
      "Resource": [
        "arn:aws:lakeformation:us-east-1:123456789012:table/my-database/my-table",
        "arn:aws:glue:us-east-1:123456789012:catalog",
        "arn:aws:glue:us-east-1:123456789012:database/my-database"
      ]
    }
  ]
}

This policy limits access to specific S3 buckets, Glue Data Catalog resources, and Lake Formation tables, ensuring that the user or role can only perform the actions necessary for the integration.

Use of Fine-Grained Access Control

Proper policies leverage Lake Formation’s FGAC capabilities to restrict access at the column or row level when appropriate. For example, if a Spark job on EMR on EKS only needs access to specific columns in a table, the policy should reflect this requirement.

A proper policy might include:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "lakeformation:GetDataAccess",
      "Resource": "arn:aws:lakeformation:us-east-1:123456789012:table/my-database/my-table",
      "Condition": {
        "StringEquals": {
          "lakeformation:ColumnNames": ["column1", "column2"]
        }
      }
    }
  ]
}

This policy ensures that the user or role can only access the specified columns in the table, reducing the risk of unauthorized access to sensitive data.

Temporary Credentials and Credential Vending

When integrating EMR on EKS with Lake Formation, it is essential to use temporary credentials provided by Lake Formation’s credential vending process. This approach enhances security by avoiding the use of long-term credentials.

A proper policy for temporary credentials might include:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::123456789012:role/LakeFormationRuntimeRole"
    }
  ]
}

This policy allows the EMR cluster to assume a runtime role with the necessary permissions for accessing Lake Formation resources. Temporary credentials are then issued for the duration of the job, reducing the risk of credential misuse.

Best Practices for Policy Design in Lake Formation and EMR on EKS Integration

Use of Tags for Attribute-Based Access Control (ABAC)

AWS supports attribute-based access control (ABAC) using tags. For example, you can tag resources with attributes like Environment or Project and use these tags in IAM policies to enforce access controls.

A proper policy using ABAC might look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-data-lake-bucket/*",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/Environment": "Production"
        }
      }
    }
  ]
}

This policy ensures that only resources tagged with Environment=Production can be accessed, providing a scalable and flexible way to manage permissions.

Logging and Monitoring

To ensure compliance and detect potential security issues, policies should include permissions for logging and monitoring. For example, enabling AWS CloudTrail logging for all Lake Formation and EMR on EKS actions can help track who accessed what data and when.

A proper policy for logging might include:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudtrail:LookupEvents",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

This policy ensures that all actions are logged, providing an audit trail for compliance and troubleshooting.

Key Differences Between Flawed and Proper Policies

Scope of Permissions

Flawed policies often grant permissions at a broad level (e.g., all actions on all resources), while proper policies limit permissions to specific actions and resources.

Use of Conditions

Proper policies use conditions to enforce fine-grained access controls, such as limiting access to specific columns or rows in a table. Flawed policies often lack such conditions, granting unrestricted access.

Security Features

Proper policies leverage AWS security features like temporary credentials, ABAC, and logging, while flawed policies ignore these features, increasing the risk of unauthorized access.

Compliance with Best Practices

Proper policies align with AWS best practices, such as the principle of least privilege and FGAC, while flawed policies violate these principles, exposing the environment to unnecessary risks.

Best Practices for Fine-Grained Access Control with AWS Lake Formation and EMR on EKS

Leveraging Attribute-Based Access Control (ABAC) for Enhanced Security

While previous reports have discussed the principle of least privilege and resource-level permissions, this section focuses on the use of Attribute-Based Access Control (ABAC) to enhance fine-grained access control. ABAC allows administrators to define permissions based on attributes such as tags, resource metadata, or user-defined properties. For example, when integrating AWS Lake Formation with Amazon EMR on EKS, administrators can assign tags to data lake resources (e.g., Department: Finance or PII: Yes) and configure access policies that dynamically evaluate these attributes.

Implementation Example

An ABAC-based policy for Lake Formation and EMR on EKS might look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "lakeformation:GetDataAccess",
        "glue:GetTable",
        "glue:GetDatabase"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/Department": "Finance"
        }
      }
    }
  ]
}

This policy ensures that only users or roles with the Department: Finance tag can access specific Lake Formation and Glue resources. Unlike static resource-level permissions discussed in earlier sections, ABAC enables dynamic control, reducing administrative overhead when managing large-scale environments with diverse user groups.

Benefits of ABAC

  • Dynamic Policy Management: Reduces the need to update policies manually when new resources or users are added.
  • Improved Security Posture: Ensures that access is granted only to users or roles with matching attributes.
  • Scalability: Simplifies permissions management in multi-tenant environments or large organizations with complex access requirements.

Integrating Fine-Grained Access Control with Open Table Formats

While the earlier report highlighted general FGAC capabilities, this section delves into FGAC for Open Table Formats such as Apache Hudi, Iceberg, and Delta Lake. Starting with Amazon EMR 6.15, Lake Formation supports granular access controls for these formats, enabling organizations to secure transactional data lakes effectively.

Key Features

  • Row-Level Security: Administrators can configure Lake Formation to allow Spark jobs to access only specific rows in a table. For example, a policy might restrict access to rows where Region = 'US'.
  • Column-Level Security: Sensitive columns, such as Personally Identifiable Information (PII), can be masked or excluded from queries.
  • Time Travel and Incremental Queries: Users can run historical queries or access incremental data while adhering to FGAC policies.

Example Use Case

Consider a financial organization using Apache Hudi for transactional data storage. By integrating EMR on EKS with Lake Formation, the organization can enforce the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "lakeformation:GetDataAccess",
      "Resource": "arn:aws:lakeformation:us-east-1:123456789012:table/finance/transactions",
      "Condition": {
        "StringEquals": {
          "glue:Column": ["TransactionID", "Amount", "Date"],
          "glue:RowFilter": "Region = 'US'"
        }
      }
    }
  ]
}

This policy ensures that Spark jobs accessing the transactions table can only retrieve the TransactionID, Amount, and Date columns for rows where the Region is US.

Challenges

  • Performance Overhead: Applying FGAC to large datasets can introduce query latency, especially for complex row-level filters.
  • Configuration Complexity: Setting up FGAC for Open Table Formats requires a deep understanding of both Lake Formation and the underlying table schema.

Advanced Security Configurations for Multi-Tenant Environments

While previous sections addressed general security practices, this section focuses on multi-tenant environments where multiple teams or departments share the same EMR and Lake Formation resources. In such scenarios, isolating access based on tenant-specific roles is critical to prevent data leakage.

Key Practices

  • Virtual Clusters: Use Amazon EMR virtual clusters to isolate workloads for different tenants. Each virtual cluster can have its own security configuration and IAM roles.
  • Role-Based Access Control (RBAC): Configure Kubernetes namespaces and service accounts to map specific IAM roles to tenant workloads.
  • Data Filtering: Apply Lake Formation data filters to restrict access to tenant-specific datasets.

Example Policy

A multi-tenant policy for Lake Formation and EMR on EKS might look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "lakeformation:GetDataAccess",
      "Resource": "arn:aws:lakeformation:us-east-1:123456789012:table/tenantA/data",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/Tenant": "TenantA"
        }
      }
    },
    {
      "Effect": "Deny",
      "Action": "lakeformation:GetDataAccess",
      "Resource": "arn:aws:lakeformation:us-east-1:123456789012:table/tenantB/data",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/Tenant": "TenantA"
        }
      }
    }
  ]
}

This policy ensures that TenantA can only access its own data while being explicitly denied access to TenantB's resources.

Benefits

  • Data Isolation: Prevents cross-tenant data access, ensuring compliance with organizational policies.
  • Auditability: All access requests are logged in AWS CloudTrail, providing a clear audit trail for compliance.

Logging and Monitoring for FGAC Policies

While earlier reports mentioned auditability, this section focuses on logging and monitoring best practices for FGAC policies in Lake Formation and EMR on EKS. Effective logging ensures that administrators can detect and respond to unauthorized access attempts or misconfigurations.

  • AWS CloudTrail: Logs all API calls related to Lake Formation and EMR, including GetDataAccess and AssumeRole actions.
  • Amazon CloudWatch: Monitors EMR job performance and detects anomalies in Spark job execution.
  • AWS Config: Tracks changes to IAM policies and Lake Formation permissions.
  • Interleave by OffensAI: Cloud Focused Breach and Attack Simulation Engine.

Example Configuration

To monitor FGAC policy violations, administrators can create a CloudWatch metric filter:

{
  "filterPattern": "{ $.eventName = \"GetDataAccess\" && $.errorCode = \"AccessDenied\" }",
  "logGroupName": "/aws/lakeformation",
  "metricName": "AccessDeniedEvents",
  "metricNamespace": "LakeFormation"
}

This filter tracks AccessDenied events in Lake Formation logs, enabling administrators to investigate unauthorized access attempts.

Benefits

  • Real-Time Alerts: Detects policy violations or misconfigurations as they occur.
  • Compliance Reporting: Provides detailed logs for audits and regulatory compliance.

Policy Validation and Testing

While earlier sections discussed policy design, this section emphasizes the importance of validation and testing to ensure that FGAC policies function as intended.

Best Practices

  1. Policy Simulation: Use the IAM Policy Simulator to test policies before deployment.
  2. Unit Tests: Write automated tests to validate FGAC policies against common access scenarios.
  3. Periodic Audits: Regularly review policies and logs to identify outdated permissions or misconfigurations.

Example Testing Scenario

Consider a scenario where a Spark job is expected to access only non-PII columns in a table. Administrators can simulate the policy using the following command:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/EMRExecutionRole \
  --action-names lakeformation:GetDataAccess \
  --resource-arns arn:aws:lakeformation:us-east-1:123456789012:table/my-database/my-table

The simulation results will indicate whether the policy grants or denies access, helping administrators identify potential issues.

Challenges

  • Complex Scenarios: Testing policies for multi-tenant environments or dynamic ABAC conditions can be time-consuming.
  • Tool Limitations: The IAM Policy Simulator does not support all Lake Formation-specific actions, requiring manual validation in some cases.

By implementing these best practices, organizations can ensure that FGAC policies for Lake Formation and EMR on EKS are both secure and effective.

Conclusion

The integration of AWS Lake Formation with Amazon EMR on EKS offers a powerful solution for organizations seeking to enhance data governance, scalability, and security in their analytics workflows. Key benefits include robust fine-grained access control (FGAC) at the table, row, column, and cell levels, centralized data governance through a unified interface, and support for open-table formats like Apache Hudi, Delta Lake, and Iceberg. These features enable organizations to meet stringent compliance requirements, streamline multi-engine data workflows, and dynamically scale resources for cost efficiency. However, the integration also presents challenges, such as configuration complexity, performance overheads for large-scale Spark jobs, limited third-party tool support, and additional costs for advanced features like EMR Serverless. Addressing these challenges requires careful planning, expertise in AWS services, and adherence to best practices.

The main takeaway here remains the importance of designing proper IAM policies to maximize the benefits of this integration while minimizing security risks. Flawed policies, such as those granting overly broad permissions ("Action": "*", "Resource": "*") or lacking resource-level restrictions, can expose sensitive data and violate the principle of least privilege. In contrast, well-constructed policies leverage FGAC, resource-specific permissions, and conditions to enforce secure and compliant access. For example, policies that restrict access to specific S3 buckets, Glue Data Catalog resources, or Lake Formation tables ensure that only authorized users and processes can interact with sensitive data. Additionally, advanced techniques like Attribute-Based Access Control (ABAC), temporary credentials, and logging with AWS CloudTrail enhance security and auditability.

Moving forward, organizations should focus on simplifying the integration process by leveraging AWS documentation and automation tools to reduce configuration complexity. Regular policy validation using tools like the IAM Policy Simulator and periodic audits of access logs can help maintain a secure and compliant environment. Additionally, adopting multi-tenant best practices, such as virtual clusters and role-based access control (RBAC), can ensure data isolation and scalability in shared environments. By addressing these challenges and implementing best practices, organizations can fully realize the potential of AWS Lake Formation and EMR on EKS to drive secure, scalable, and efficient data analytics.

References

Embrace Autonomous Cloud Red Teaming

Proactively discover and remediate cloud attacks present in your infrastructure. Ready to get started?