Exploring Amazon Security Lake
At Salesforce, Trust is our #1 value. Nothing is more important than the security and privacy of customer data. Our Customer 360 Platform connects data from marketing, sales, commerce, service, and IT teams around every customer so they can work together while boosting productivity, increasing efficiency, and decreasing costs.
To support the growth of our global customer base, we now deliver the Customer 360 on both first-party infrastructure and through Hyperforce, a new infrastructure architecture that allows Salesforce to scale rapidly and securely using public cloud partners — including AWS and its latest security innovations.
Solving Large-Scale Log Collection on Hyperforce
The Detection and Response (DnR) team is critical to securing Salesforce’s infrastructure, detecting malicious threat activities, and providing timely responses to security events. We do this by collecting and inspecting petabytes of security logs across dozens of organizations, some with thousands of accounts.
As Hyperforce instances, accounts, and services grow, the DnR team is challenged to improve the efficiency and scalability of our data pipeline and data lake. Time-consuming tasks like reducing log ingestion latency, improving log onboarding efficiency, making pipeline scalable, rationalizing cost-to-serve, and managing log storage effectively must be accounted for.
DnR designed and deployed an existing pipeline to collect security logs from global Hyperforce instances on AWS. Due to the variety and heterogeneity of security log types, we applied a divide-and-conquer approach to building distinct log collection mechanisms for different log types. For example, we leverage CloudWatch-based log collection, S3 storage-based log collection, Lambda-based log collection, Kinesis-based log collection, MSK-based log collection, etc. As log volume increases exponentially, the complexity and efficiency of our log collection mechanisms can be challenging to handle.
To solve a large-scale log collection on Hyperforce, we want to enable a convergent architecture that supports multiple log sources, enables scalable ETL, activates accurate schema management, helps log analytics/queries and covers end-to-end observability of the entire pipeline.
Security Events at Scale with Amazon Security Lake
Amazon Security Lake, a service that automatically centralizes an organization’s security data into a purpose-built data lake in a customer’s AWS account, allows enterprise customers like Salesforce to centrally aggregate, manage, and use security-related log and event data at scale. It easily consolidates security logs and events from AWS, on-premise, and other cloud providers.
It does so by automating the collection of security-related log and event data from integrated AWS services and third-party sources, managing the lifecycle of that data with customizable retention settings and roll-up to preferred AWS Regions, and transforming that data into a standard open-source format called Open Cybersecurity Schema Framework (OCSF). We can then use the security data that is stored and accessed in Amazon Security Lake for incident response and security data analytics.
Salesforce DnR had the opportunity to experiment with the beta version of Amazon Security Lake and in doing so, identified several advantages. Amazon Security Lake is good at collecting and transforming AWS native logs (e.g. CloudTrail log, VPC flow log, Route53 Resolver log). After transformation, the log is directly consumable in the OCSF schema and Parquet data format.
- One-Stop Log Collection Management for Accounts Under an AWS Organization: Amazon Security Lake offers centralized management of security log collection across AWS accounts. It is easy and straightforward to enable log collection by reducing previous days and weeks of work to a few hours.
- Abundant Security Log Sources: Amazon Security Lake supports a list of AWS native logs such as CloudTrail log, VPC flow log, Route53 Resolver log, etc. In addition, it adds support for OCSF-compliant vendor logs.
- Automatic ETL to Transform Log With OCSF Schema: Amazon Security Lake runs automatic ETL jobs to transform log data to specific OCSF schema and makes Parquet format data in Amazon Security Lake S3 bucket easily consumable.
- Effective Log Partitioning and Regional Rollup: Amazon Security Lake provides log data rollup to designated regions that can help manage log data in our global infrastructure. Log data is well partitioned by log source, region, account id, and event hour.
- Custom Log Source Ingestion Support: Customer log data can be ingested into Amazon Security Lake security data lake as long as customer log data is OCSF-compliant. This creates a unified data lake to manage AWS's native security logs and Salesforce’s own log data.
- Integration Support for Other Services: Amazon Security Lake has good support for integration with other services, such as Athena, Splunk, etc. We can easily search and run analytics jobs against Amazon Security Lake log data with several query engines.
Innovation Meets Trust
Ultimately, our Amazon Security Lake adoption really helps us to solve some heavy-duty work for log collection, transformation, aggregation, search, and management. It fits well with Salesforce Hyperforce on AWS and complements DnR's own data pipeline. Moreover, Amazon Security Lake log aggregation and regional roll-up fully align with DnR's own global, decentralized, and hybrid data lake infrastructure.
We are excited about Amazon Security Lake’s potential to integrate with third-party OCSF consortium members and bring all security log data under one roof. We anticipate that Amazon Security Lake will help offload 30% - 50% of the traffic of our own data pipeline, significantly reduce our log onboarding time, and increase log coverage, and we will further uplift the security posture of Salesforce, reinforcing our focus on trust and innovation.
This post was authored by members of Salesforce's Detection and Response (DnR) Engineering Team:
Lei Ye, Software Engineering Architect, Detection & Response (DnR) Engineering, focuses on innovating data processing pipelines, data lake, and query engine for DnR and other infrastructure security-related problems. He is the tech lead driving Salesforce's Amazon Security Lake project. He architected the blueprint of DnR's next-generation data lake infrastructure and drove the collaboration across organizations and teams.
Ajith Jayamohan, Vice President of Detection & Response (DnR) Engineering, leads the Machine Learning, and AI-driven data platform teams at Salesforce focused on detecting and responding to internal and external adversaries or threat actors to protect Salesforce and its customers.