Salesforce Developers, profile picture
Uploaded bySalesforce Developers
PPTX, PDF8,256 views

Large Data Management Strategies

The document discusses various aspects of managing large data volumes in Salesforce, addressing risks, strategies, and operational challenges associated with data loads, extracts, and cleanups. It emphasizes the use of Salesforce's Bulk API for efficient data handling and outlines best practices for data loading, validation, and cleanup processes. Additionally, it provides recommendations for optimizing performance and managing the complexity of large-scale data operations.

Embed presentation

Downloaded 285 times
Managing Large Data Volumes Suchin Rengan, Director, Salesforce Services @Sacrengan Mahanthi Gangadhar, Senior Solutions Technical Architect, Salesforce Services
Safe harbor Safe harbor statement under the Private Securities Litigation Reform Act of 1995: This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services. The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site. Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.
We do a lot of things with data… Data Searches/ Reporting/ List Views Data Creation (Loads/ Manual) Data Archival Am I using the platform’s features optimally? How do we ensure we keep up with performance? What factors do we need to consider across each topic? How can I ensure I have a scalable process? Data Integration (Out and In) Data Extracts
SFDC Cloud Computing for the Enterprise Infrastructure Services Network Storage Operation System Database App Server Web Server Data Center Application Services Security Sharing Integration Customization Web Services API Multi-Language Operations Services Authentication Availability Monitoring Patch Mgmt Upgrades Backup / NOC Customer Avoids: Tuning OS, Capacity Management Tuning Web Servers, Certificates, Log file Mgmt, etc Tuning App Servers, threads, Java stack, memory/log mgmt Tuning DB Servers, memory mgmt, disk distribution Network management, bandwidth Innovation Development Data Model Business Logic User Interface
Lets understand the underlying Platform Your System Apex API User Custom Object Custom Object Account …
Data Loads Suchin Rengan Mahanthi Gangadhar
Considerations at every layer and level Apex Triggers File Storage VF Workflow Rules Sandbox Apex Data Storage Data Objects Sharing Tables Indexes Skinny Tables Validation Rules API Logic / Application Layer Storage Layer
SOAP/REST API SOAP API Real Time Relatively slow for large volumes Loads for up to 250K records Batch side Time out / failure API Bulk API Batches per day Parallel mode Larger than 250K records Rolling 24 clock for available batches Time to download results
Bulk API – Asynchronous Process Data streamed to temporary storage Job updated in Admin Setup Client Send all data Processing Processing Processing Servers Thread Thread Data Batches Check Status Retrieve Results Dataset processed in parallel Dequeue batch Insert/ update Results Save results
Bulk API The “go-to” option for tens of thousands of records and up Up to 10,000 records in a batch file Asynchronous loading, tracked in Setup’s “Monitor Bulk Data Load Jobs” section Walkthrough time! Example: American Insurance Co. ? 230 million records processed in 33 hours, 14 hours ahead of schedule
Some tips for loading •Difficult to extrapolate performance in a Sandbox • At Par or Better in Production •Sharing Calculations •Indexing •File Storage •Triggers – Act Judiciously •Upserts – Avoid Them! •Parent References
Sequence of Events/Logic Parent Rollup Summary System Validation Custom Validation Assignment Rules Auto-response Rules Workflow Rules Escalation Rules APEX Layer Before Trigger After Trigger Database Layer Begin Transaction End Transaction
Initial Load – Incremental or Big Bang Obj 1 Obj2 Obj3 Option 1 All Objects Option 2 Legend Pre-Implementation Activities Initial Load Validation Catch up and Ongoing Sync User Activation
Audience Question! Scenario: Data Load on …
Data Extraction Mahanthi Gangadhar @twittername
Data Extraction – Bulk Query Current Limitations Bulk query works exactly like data loads  Create a job (Job Id)  Each query is a batch (Batch Id)  Close the Job and fetch results when job is complete Limitations  Query optimizer has 100mins processing time (timeout issue)  Informatica currently does not support Bulk Query  Other tools like data loader can submit only one query  Currently requires a custom client for submitting multiple queries for a given job
Data Extraction - Chunking Auto Number Chunking  Query smaller chunks of the entire data  Use Auto number and formula field for internal indexing  Find chunk boundaries (25K) and issue queries for each chunk PK Chunking  Use primary key to chunk (ID)  Usually better performance when entire object is extracted  Find chunk boundaries (250K) and issue queries for each chunk Auto Chunking (“safe harbor”)
Data Extraction - Chunking 1 Q1 Job Id (1234) Q2 Q1 -> Batch Id (123) Q2 -> Batch Id (234) Q3 ….. Q4 Qn -> Batch Id (789) Close Job Id (1234) Q5 Find Chunk Boundaries -> Create Job -> Submit each batch/query -> Get Results -> Close Job Qn 150 M
Data Cleanup - General Considerations Deletion is one of the most resource intensive data operations in Salesforce and can perform even slower than data load in some cases (objects with complex sharing, with Master/Detail relationships, with Rollup Summary fields etc.) even in bulk hard delete mode. Custom objects can be cleaned up by truncation. Note that truncation cannot be performed on OOB standard objects. Data in standard objects can be deleted only using Delete API, faster performing Bulk Hard Delete option is available Records from User object cannot be deleted, only deactivated.
Data Cleanup - Truncation and Hard Delete Where Truncation is not Possible?  Are referenced by another non empty object through a lookup field  Are referenced in an analytic snapshot  Contains a geo location custom field  Have a custom index or an external ID field Hard Delete  Hard Delete option is disabled by default and must be enabled by an administrator  Observed about 4.5M/hr on Task object delete, versus 18M/hr for load time, indicative of how slow of a process it is…
Data Cleanup - General Recommendations General Recommendations  When removing large amounts of data from a sandbox org consider Refresh option first. This is the fastest way to clean up an org. Additional benefit of Refresh option is that data in User object is removed also.  To remove data from a custom object use truncate function. Truncation deletes an object entirely and places it into the recycle bin. Use Erase function to physically remove data from the system it frees up storage space. Note it might take several hours to erase data from a large custom object.  To remove data from standard object use bulk hard delete option. Note that performance of bulk hard delete is quite low so plan sufficient time to remove large amounts of data using this option. Recent test on Task object in Shadow production org demonstrated performance of ~2M records deleted per hour. On Account object the rate might be even lower.  When planning large scale tests in production environment consider possibility of getting 2 orgs that can be used in “round robin” mode: when a test is performed in one org, cleanup is being performed in another.
General Guidelines – Tips / Partner Tools  Tips  Test early and often and as big data sets as possible  Split initial load set to smaller subsets (we used 10MM records). This allows for greater flexibility in load monitoring and control  Queries: for aggregate data validation queries (since bulk query option is not available for them) consider use of Work Bench with special asynchronous background processing configuration that prevents early timeout on client side (more info: http://wiki.developerforce.com/page/Workbench). Publicly available WB app https://workbench.developerforce.com/login.php can be utilize  ETL Tool  Timeout settings  Bulk query support  Handling of Success and Error files  Monitoring of Job Status
Audience Question! Scenario: Data Load on …
Other Areas Suchin Rengan @twittername
General Guidelines for other areas  Searching and Reporting and List Views • • Filtered Queries • Skinny Tables • Data Scope • Roles and Visibility •  Indexes Report Folders Data Governance • • Data Management and Administration •  Data Model Security and Access Controls Data Archival and Retention • Storage
Speaker Name Speaker Name Speaker Name Speaker Name Speaker Title, @twittername Speaker Title, @twittername Speaker Title, @twittername Speaker Title, @twittername
Initial Load - Parent References (* goes into notes) General LDV recommendations  Avoid reference to parent record via External Id if possible, use SFDC Id instead.  XLDV considerations  Consider preparing source data sets with native SFDC id for parent reference. This might be achieved by querying key mapping data from parent objects and performing joins on client side to retrieve SFDC Id for parent reference vs. using External Id reference. Additional benefit – this is a good data validation step. Particularly important on large objects with multiple parent references  Note that querying data from extra large objects might be a challenge by itself and often takes hours. Consider alternative approaches, for example collecting mapping keys from initial upload log files.  Referencing parent via native SFDC GUID will also allow the use of FAST LOAD option when available (next release, “safe harbor”)
Initial Load - Data Validation General LDV recommendations  The following post load data validation checks are generally can be considered:  Aggregate values/subtotals comparison (numeric and currency values) Data validation on XLDV is a challenging task. Queries on tables with hundreds of millions of records are slow and often time out (even bulk queries). Consider options for splitting (chunking ) your data validation queries by filtering on indexed attributes.  Special chunking techniques: auto number chunking, PK chunking  Do not underestimate and plan for enough time for performing post load data validation on XLDV. Total counts comparison   Attribute to attribute comparison  XLDV considerations  Spot checking  Negative validations
Incremental synch’s General LDV recommendations  For SOAP API’s use the largest batch size (200).  During incremental syncs on objects with large amount of processing the biggest batches can fail due to time out (10 min per batch). In this case batch size might need to be reduced.  It is possible to “batch” standard SOAP API calls by submitting several jobs in parallel. When loading data into the same object using several SOAP API batches in parallel consider to group child records by the parent in the same batch to minimize DB locking conflicts. XLDV considerations  For incremental synchs of larger data sets (over 50K-100K) use bulk API’s.  Consider configuring client application to programmatically set load mode (SOAP vs. Bulk) based on the size of incremental data set in each load session.
Data Loads - Current Volumes - REMOVE Object User Role User Account (+Contact) Agent Role Relationship Household Household Member Opportunity Task Remark Note Attachment Life Event External Agreement Agent Marketing Inf. Policy Policy Role Agent Agreement Role Billing Account Billing Account Role Total • Initial volume close to 3.5 B • Worked with customer to reduce volumes • Split the implementation into two phases approx. 1A # of rows approx. 1B # of rows 60,000 200,000 186,000,000 155,000,000 11,000,000 96,000,000 155,000,000 100,000,000 104,500,000 16,500,000 15,000,000 6,000,000 845,413,000 120,000,000 18,000,000 900,000 33,540,000 26,000,000 149,000,000 165,000,000 172,000,000 23,220,000 23,220,000 730,880,000
Data Loads - Org/Environment preparation and tuning For XLDV it is recommended to perform test loads in Production like environments  To get a true representation of the performance in Production  To test loads in real production environment for environment specific settings  Allows for more accurate planning of data load in terms of timing and dependencies Concerns: Large scale deletion / environment cleanup after tests. Deletion is an resource intensive data operation and the slowest (even using bulk hard delete)  Cannot predict in advance all possible issues and consequences related to multiple mass data deletions in actual production environment  Users cannot be deleted from Production environment, only deactivated. Multiple User load tests can produce a “garbage pile” of inactive User records that would clutter Production environment.
Data Loads - Org/Environment preparation and tuning (Contd..) General best practice recommendation:  Request a dedicated production test environment to avoid testing in actual Production org.  Work with SFDC Operation on plans for re-provisioning test production environment because erasing XLDV data from an org for subsequent test might not be feasible Coordinate with SFDC Operations on any large scale test activities in Prod environment (systems can be shut down by SFDC Operations as a suspected DDOS attack) Plan accordingly and factor in the time required for setting up and configuring test production environments when multiple production tests rounds are required (production org cannot be refreshed as sandbox org so it needs to be re-provisioned and configured from scratch)
Data Loads – Org/Environment preparation and tuning General LDV recommendations  For large volume data loads it is possible to request additional temporary special changes on SFDC side to expedite load performance XLDV considerations  Request increase of batch API limit  Request Increase of file storage  Notify SFDC Operations about upcoming loads with approximate data volumes  Request turning off Oracle Flashback  Use FASTLOAD option (future option, “safe harbor”) Note that some settings/perms can technically be tweaked only in Production environments, not in Sandbox environments
Initial Load - Use Bulk APIs for massive Data Loads General LDV recommendations SFDC Bulk API are specifically designed to load large volumes of data. It is recommended to use bulk API for LDV initial data load. Use the Salesforce Bulk API in parallel mode when loading more that few hundred thousand records. Main performance gain of bulk API’s is executing batches in parallel. Note that generally bulk API (w/o batching, in sequential mode) might perform slower than SOAP standard API so if number of records is less than certain threshold (that depends on number of factors: object being loaded, processing on SFDC side, number of attributes loaded etc…) using bulk API’s might become counterproductive. XLDV considerations  Use bulk API’s in parallel mode with maximum allowed batch size whenever possible to maximize number of records that can be loaded within 24 hour period (based on bulk API batch limit – standard limit 2K batches per 24 hours)  Use standard SOAP APIs for incremental ongoing data synchs (data sets less than 50-100K) and bulk APIs for larger incremental data sets (over 50K-100K rows)  Using standard SOAP API’s for incremental synchs has a benefit of reducing the risk of database contention during uploads of child objects  Allow enough time for collecting Request and Result log files (takes ~15 min for INFA to extract load logs for 10MM rows loaded)
Initial Load - Planning Bulk Jobs (Parallel vs Single Object) General LDV recommendations When planning bulk jobs consider the following:  When loading large volume of data into a single object it is recommended to submit all the batches in one job. Splitting (partitioning) data across several parallel jobs does not improve performance when each individual job is big enough. Note that batches of all jobs are placed in the same batch pool and queued for processing independently of jobs they belong to.  When loading data into several objects running several jobs for different objects in parallel is recommended when jobs are small enough to not consume all the available system resources. XLDV considerations  For loading extra large volumes of data into single object in Salesforce it is recommended to run one object load at a time. The rationale behind this recommendation is that bulk jobs with hundreds/thousands of batches consume all available system resources and running other jobs in parallel just causes jobs to compete for the limited resources. The other consideration is DB cache. When several objects are loaded in parallel sharing DB cache is causing more frequent swapping that slows down overall performance on the DB layer.
Initial Load - Defer Sharing Calculations General LDV recommendations  Reduce sharing recalculations processing during initial data upload.  Disable sharing rules and/or use the defer sharing calculation perm for custom objects while loading data.  Note that sharing calculations will still need to be performed after data load is complete and it can take significant amount of time but it can reduce loading time by allowing sharing to be calculated in “off” hours. XLDV considerations   Based on the XLDV performance tests general recommendation for deferring sharing calculations might not be applicable for extra large volumes. Post loading sharing calculations might take very long time to execute on bigger volumes (it is not parallel currently, on roadmap for the nearest release)  Load data in the following sequence to enforce main sharing calculations to be performed during data upload:   Create main user groups and group members  Create sharing rules   Create User Role Hierarchy and upload users with correct roles assigned  Sharing tables can alternatively be uploaded directly that allows to avoid sharing recalculation on initial load altogether Set OWD on objects Private where applicable Upload data with correct owner specified. Consider uploading sharing tables on some objects directly to avoid sharing calculations during upload and post load processing.
Initial Load - Triggers, WF Rules, Data Validation Rules General LDV recommendations  Disable when possible triggers, WF rules, data validations on objects being uploaded.  To avoid lengthy post load catch up processing consider performing data transformations/ validations on source data set prior to data upload or  consider batch APEX post-load processing for data transformations that can’t be performed prior to upload.  For each trigger, WF rule and validation rude individual analysis should be performed to define the best strategy. XLDV considerations  General best practice rules apply
Initial Load - Defer Indexing General LDV recommendations  Reduce additional processing on SFDC during initial data upload: turn off search indexes, consider creating custom indexes after data load XLDV considerations  Based on initial performance results index recalculations on extra large volumes of data take long time post load (not parallelized currently). It might be a better approach to configure all required indexes prior to XLDV load.
Initial Load - Use Fastest Operation Possible General LDV recommendations  Use the fastest operation possible. insert is fastest then update, then upset XLDV considerations  If possible load full set of initial data in one go using insert with only small incremental upserts when needed (failed records reprocessing for example).  To avoid loading big volumes data in upsert mode (for example when main load job fails in the middle and remaining data should be added on top of existing data in an object) consider configuring a client load job the way that joins existing records to remaining records (on external Id) outside of SFDC and then loading resulting set in insert mode.
Initial Load - Use Clean Data Set General LDV recommendations  Use as clean data set as possible  Note that errors in batches are causing single row processing on the remainder of the batch that slows down load performance significantly.  XLDV considerations Perform rigorous data validation prior to upload. This will allow: 1. Loading most of the records in the fastest insert operation and avoid slow down due to the preventable errors 2. Avoid slow down when processing goes record by record within a 200 record transactional batch when number of failed record reaches a Threshold
Slide parts
Slide parts
Slide parts
Slide parts
Data Load Considerations (*- will be replaced) Topic Consideration Data Model Normalized/ De-normalized Sharing Model Sharing calculations Target Data Does it really need to exist in SFDC? BP etc. Test Environments Test them before you deploy! Timelines Adequate timeline for testing User data Cannot be deleted API Soap versus Bulk Batches Limit of 2K batches for Bulk API a day Deletes Oh No.. Extrapolation Not really possible What do I need to do? SF Support to increase batches It is at par or better, if Prod can be used even better

More Related Content

PDF
Data modeling for the business
byChristopher Bradley
 
PPTX
Enterprise Data Architect Job Description
byLars E Martinsson
 
PDF
Sample - Data Warehouse Requirements
byDavid Walker
 
PDF
Get Intelligent with Metabase
byAnant Corporation
 
PDF
15 Tips on Salesforce Data Migration - Naveen Gabrani & Jonathan Osgood
bySalesforce Admins
 
PPTX
9 Principles for Salesforce Application Architecture
bySteven Herod
 
PDF
Data Warehouse - Incremental Migration to the Cloud
byMichael Rainey
 
PPT
Master Data Management
bySung Kuan
 
Data modeling for the business
byChristopher Bradley
 
Enterprise Data Architect Job Description
byLars E Martinsson
 
Sample - Data Warehouse Requirements
byDavid Walker
 
Get Intelligent with Metabase
byAnant Corporation
 
15 Tips on Salesforce Data Migration - Naveen Gabrani & Jonathan Osgood
bySalesforce Admins
 
9 Principles for Salesforce Application Architecture
bySteven Herod
 
Data Warehouse - Incremental Migration to the Cloud
byMichael Rainey
 
Master Data Management
bySung Kuan
 

What's hot

PDF
Dell Boomi AtomSphere - A presentation by RapidValue Solutions
byRapidValue
 
PDF
Deloitte & Mulesoft : The Right Mix
byDavid Graham
 
PPTX
Salesforce Cross-Cloud Architecture
byThierry TROUIN ☁
 
PPTX
Managed vs Unmanaged Solution- Dynamics 365 CE (CRM)
byAli Sharifi
 
PPT
Metabase
byWagner Alves
 
PDF
Data Mesh for Dinner
byKent Graziano
 
PDF
Data Quality in the Banking Industry: Turning Regulatory Compliance into Busi...
byPrecisely
 
PDF
Microsoft role-based-certification-roadmap
byNetCom Learning
 
PDF
Case Study: Migrating Hyperic from EJB to Spring from JBoss to Apache Tomcat
byVMware Hyperic
 
PDF
From Data Warehouse to Lakehouse
byModern Data Stack France
 
PDF
Best Practices with Apex in 2022.pdf
byMohith Shrivastava
 
PPTX
Why Do Banks Need A Customer Data Platform?
byLemnisk
 
PDF
Introduction to Spark (Intern Event Presentation)
byDatabricks
 
PDF
DDD and Microservices: Like Peanut Butter and Jelly - Matt Stine
byVMware Tanzu
 
PPTX
‏‏Chapter 8: Reference and Master Data Management
byAhmed Alorage
 
PPTX
Domain Driven Design Introduction
bywojtek_s
 
PDF
Data Profiling, Data Catalogs and Metadata Harmonisation
byAlan McSweeney
 
PDF
Master Data Management - Practical Strategies for Integrating into Your Data ...
byDATAVERSITY
 
PDF
Space Camp - API Contract Testing
byPostman
 
PPTX
A Business Intelligence requirement gathering checklist
byMadhumita Mantri
 
Dell Boomi AtomSphere - A presentation by RapidValue Solutions
byRapidValue
 
Deloitte & Mulesoft : The Right Mix
byDavid Graham
 
Salesforce Cross-Cloud Architecture
byThierry TROUIN ☁
 
Managed vs Unmanaged Solution- Dynamics 365 CE (CRM)
byAli Sharifi
 
Metabase
byWagner Alves
 
Data Mesh for Dinner
byKent Graziano
 
Data Quality in the Banking Industry: Turning Regulatory Compliance into Busi...
byPrecisely
 
Microsoft role-based-certification-roadmap
byNetCom Learning
 
Case Study: Migrating Hyperic from EJB to Spring from JBoss to Apache Tomcat
byVMware Hyperic
 
From Data Warehouse to Lakehouse
byModern Data Stack France
 
Best Practices with Apex in 2022.pdf
byMohith Shrivastava
 
Why Do Banks Need A Customer Data Platform?
byLemnisk
 
Introduction to Spark (Intern Event Presentation)
byDatabricks
 
DDD and Microservices: Like Peanut Butter and Jelly - Matt Stine
byVMware Tanzu
 
‏‏Chapter 8: Reference and Master Data Management
byAhmed Alorage
 
Domain Driven Design Introduction
bywojtek_s
 
Data Profiling, Data Catalogs and Metadata Harmonisation
byAlan McSweeney
 
Master Data Management - Practical Strategies for Integrating into Your Data ...
byDATAVERSITY
 
Space Camp - API Contract Testing
byPostman
 
A Business Intelligence requirement gathering checklist
byMadhumita Mantri
 

Viewers also liked

PDF
Extreme Salesforce Data Volumes Webinar
bySalesforce Developers
 
PDF
Handling of Large Data by Salesforce
byThinqloud
 
PDF
Extreme Salesforce Data Volumes Webinar (with Speaker Notes)
bySalesforce Developers
 
PDF
Inside the Force.com Query Optimizer Webinar
bySalesforce Developers
 
PDF
Salesforce API Series: Fast Parallel Data Loading with the Bulk API Webinar
bySalesforce Developers
 
PDF
Top 5 ETL Tools for Salesforce Data Migration
byIntellipaat
 
PDF
Follow the evidence: Troubleshooting Performance Issues
bySalesforce Developers
 
PPTX
The definitive guide to salesforce sandbox flosum
byFlosum
 
PPT
Under the Hood of Sandbox Templates
bySalesforce Developers
 
PDF
How Salesforce built a Scalable, World-Class, Performance Engineering Team
bySalesforce Developers
 
PDF
Real-time personal trainer on the SMACK stack
byAnirvan Chakraborty
 
PDF
Demystifying salesforce for developers
byHeitor Souza
 
PPTX
Kafka Lambda architecture with mirroring
byAnant Rustagi
 
PDF
Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...
byMartin Zapletal
 
PPTX
How Apache Kafka is transforming Hadoop, Spark and Storm
byEdureka!
 
PPTX
Trailhead Live Developer Workshop - Salesforce App Cloud
bySam Garforth
 
PPTX
7 Step Data Cleanse: Salesforce Hygiene
byCloudFixer
 
PDF
How to Setup Continuous Integration With Git, Jenkins, and Force.com
bySalesforce Developers
 
PDF
Salesforce Application Lifecycle Management presented to EA Forum by Sam Garf...
bySam Garforth
 
PPTX
Big Data Day LA 2015 - Event Driven Architecture for Web Analytics by Peyman ...
byData Con LA
 
Extreme Salesforce Data Volumes Webinar
bySalesforce Developers
 
Handling of Large Data by Salesforce
byThinqloud
 
Extreme Salesforce Data Volumes Webinar (with Speaker Notes)
bySalesforce Developers
 
Inside the Force.com Query Optimizer Webinar
bySalesforce Developers
 
Salesforce API Series: Fast Parallel Data Loading with the Bulk API Webinar
bySalesforce Developers
 
Top 5 ETL Tools for Salesforce Data Migration
byIntellipaat
 
Follow the evidence: Troubleshooting Performance Issues
bySalesforce Developers
 
The definitive guide to salesforce sandbox flosum
byFlosum
 
Under the Hood of Sandbox Templates
bySalesforce Developers
 
How Salesforce built a Scalable, World-Class, Performance Engineering Team
bySalesforce Developers
 
Real-time personal trainer on the SMACK stack
byAnirvan Chakraborty
 
Demystifying salesforce for developers
byHeitor Souza
 
Kafka Lambda architecture with mirroring
byAnant Rustagi
 
Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...
byMartin Zapletal
 
How Apache Kafka is transforming Hadoop, Spark and Storm
byEdureka!
 
Trailhead Live Developer Workshop - Salesforce App Cloud
bySam Garforth
 
7 Step Data Cleanse: Salesforce Hygiene
byCloudFixer
 
How to Setup Continuous Integration With Git, Jenkins, and Force.com
bySalesforce Developers
 
Salesforce Application Lifecycle Management presented to EA Forum by Sam Garf...
bySam Garforth
 
Big Data Day LA 2015 - Event Driven Architecture for Web Analytics by Peyman ...
byData Con LA
 

Similar to Large Data Management Strategies

PPTX
Solving Complex Data Load Challenges
bySunand P
 
PDF
Performance Tuning for Visualforce and Apex
bySalesforce Developers
 
PDF
Data Design Tips for Developing Robust Apps on Force.com
bySalesforce Developers
 
PPT
Df12 Performance Tuning
byStuart Bernstein
 
PPTX
Large Data Volume Salesforce experiences
byCidar Mendizabal
 
PPTX
LDV.pptx
byShams Pirzada
 
PDF
Performance Tuning for Visualforce and Apex
bySalesforce Developers
 
PPTX
Replicating One Billion Records with Minimal API Usage
bySalesforce Developers
 
PPTX
Fast parallel data loading with the bulk API
bySalesforce Developers
 
PDF
The Need for Speed: Building Reports That Fly
bySalesforce Developers
 
PPTX
Spring ’15 Release Preview - Platform Feature Highlights
bySalesforce Developers
 
PDF
Batch Jobs: Beyond the Basics
bySalesforce Developers
 
PDF
Avoid Growing Pains: Scale Your App for the Enterprise (October 14, 2014)
bySalesforce Partners
 
PPTX
LDV-v2.pptx
byShams Pirzada
 
PDF
Salesforce Spring'15 release overview
byRakesh Gupta
 
PPTX
Salesforce Winter '23 Release Highlights.pptx
byOm Prakash
 
PPTX
Dive Deep into Apex: Advanced Apex!
bySalesforce Developers
 
PDF
Bulkify Your Org
bySalesforce Developers
 
PPT
Building Reports That Fly
bySalesforce Developers
 
PPTX
Advanced Apex Webinar
bypbattisson
 
Solving Complex Data Load Challenges
bySunand P
 
Performance Tuning for Visualforce and Apex
bySalesforce Developers
 
Data Design Tips for Developing Robust Apps on Force.com
bySalesforce Developers
 
Df12 Performance Tuning
byStuart Bernstein
 
Large Data Volume Salesforce experiences
byCidar Mendizabal
 
LDV.pptx
byShams Pirzada
 
Performance Tuning for Visualforce and Apex
bySalesforce Developers
 
Replicating One Billion Records with Minimal API Usage
bySalesforce Developers
 
Fast parallel data loading with the bulk API
bySalesforce Developers
 
The Need for Speed: Building Reports That Fly
bySalesforce Developers
 
Spring ’15 Release Preview - Platform Feature Highlights
bySalesforce Developers
 
Batch Jobs: Beyond the Basics
bySalesforce Developers
 
Avoid Growing Pains: Scale Your App for the Enterprise (October 14, 2014)
bySalesforce Partners
 
LDV-v2.pptx
byShams Pirzada
 
Salesforce Spring'15 release overview
byRakesh Gupta
 
Salesforce Winter '23 Release Highlights.pptx
byOm Prakash
 
Dive Deep into Apex: Advanced Apex!
bySalesforce Developers
 
Bulkify Your Org
bySalesforce Developers
 
Building Reports That Fly
bySalesforce Developers
 
Advanced Apex Webinar
bypbattisson
 

More from Salesforce Developers

PDF
Sample Gallery: Reference Code and Best Practices for Salesforce Developers
bySalesforce Developers
 
PDF
Maximizing Salesforce Lightning Experience and Lightning Component Performance
bySalesforce Developers
 
PDF
Local development with Open Source Base Components
bySalesforce Developers
 
PPTX
TrailheaDX India : Developer Highlights
bySalesforce Developers
 
PDF
Why developers shouldn’t miss TrailheaDX India
bySalesforce Developers
 
PPTX
CodeLive: Build Lightning Web Components faster with Local Development
bySalesforce Developers
 
PPTX
CodeLive: Converting Aura Components to Lightning Web Components
bySalesforce Developers
 
PPTX
Enterprise-grade UI with open source Lightning Web Components
bySalesforce Developers
 
PPTX
TrailheaDX and Summer '19: Developer Highlights
bySalesforce Developers
 
PDF
Live coding with LWC
bySalesforce Developers
 
PDF
Lightning web components - Episode 4 : Security and Testing
bySalesforce Developers
 
PDF
LWC Episode 3- Component Communication and Aura Interoperability
bySalesforce Developers
 
PDF
Lightning web components episode 2- work with salesforce data
bySalesforce Developers
 
PDF
Lightning web components - Episode 1 - An Introduction
bySalesforce Developers
 
PDF
Migrating CPQ to Advanced Calculator and JSQCP
bySalesforce Developers
 
PDF
Scale with Large Data Volumes and Big Objects in Salesforce
bySalesforce Developers
 
PDF
Replicate Salesforce Data in Real Time with Change Data Capture
bySalesforce Developers
 
PDF
Modern Development with Salesforce DX
bySalesforce Developers
 
PDF
Get Into Lightning Flow Development
bySalesforce Developers
 
PDF
Integrate CMS Content Into Lightning Communities with CMS Connect
bySalesforce Developers
 
Sample Gallery: Reference Code and Best Practices for Salesforce Developers
bySalesforce Developers
 
Maximizing Salesforce Lightning Experience and Lightning Component Performance
bySalesforce Developers
 
Local development with Open Source Base Components
bySalesforce Developers
 
TrailheaDX India : Developer Highlights
bySalesforce Developers
 
Why developers shouldn’t miss TrailheaDX India
bySalesforce Developers
 
CodeLive: Build Lightning Web Components faster with Local Development
bySalesforce Developers
 
CodeLive: Converting Aura Components to Lightning Web Components
bySalesforce Developers
 
Enterprise-grade UI with open source Lightning Web Components
bySalesforce Developers
 
TrailheaDX and Summer '19: Developer Highlights
bySalesforce Developers
 
Live coding with LWC
bySalesforce Developers
 
Lightning web components - Episode 4 : Security and Testing
bySalesforce Developers
 
LWC Episode 3- Component Communication and Aura Interoperability
bySalesforce Developers
 
Lightning web components episode 2- work with salesforce data
bySalesforce Developers
 
Lightning web components - Episode 1 - An Introduction
bySalesforce Developers
 
Migrating CPQ to Advanced Calculator and JSQCP
bySalesforce Developers
 
Scale with Large Data Volumes and Big Objects in Salesforce
bySalesforce Developers
 
Replicate Salesforce Data in Real Time with Change Data Capture
bySalesforce Developers
 
Modern Development with Salesforce DX
bySalesforce Developers
 
Get Into Lightning Flow Development
bySalesforce Developers
 
Integrate CMS Content Into Lightning Communities with CMS Connect
bySalesforce Developers
 

Recently uploaded

PDF
IDSECCONF2025 - Aan Wahyu - Maze–Malware Analysis Platform.pdf
byidsecconf
 
PDF
Flatpak CLI Application Management & Building on RHEL.pdf
byLinuxCert Guru
 
PDF
FastTrack CISSP Reference By Jobyer Ahmed
bysocial72
 
PPTX
Opening Plenary - Esri UK Welsh Conference 2025
byEsri UK
 
PDF
Agile Trailblazers (Scrum Masters, Product Owners, Coaches, Project Managers,...
byScott M. Graffius
 
PDF
November Patch Tuesday
byIvanti
 
PDF
Inclusive India_Samir_Dash_10 NOV_FINAL 2025.pdf
bySamir Dash
 
PDF
Webinar: Introduction to LF Energy SEAPATH
byDanBrown980551
 
PDF
UnityNet AI Induced Harm & Youth Protection Position Statement 2025-11-15
byLHelferty
 
PPTX
Standardising and simplifying systems and data management - Esri UK Welsh Con...
byEsri UK
 
PPTX
Cyber Security Quiz 1st Year CSE CS Students
bynirmalamca
 
PDF
#MakeAIMatter for HR Professionals | AI Transformation Workshop by Tekdi Tech...
byParth Lawate
 
PDF
IDSECCONF2025 - Ali - DursGo–Web Security Scanner with AI Analysis.pdf
byidsecconf
 
PDF
Upskill to Agentic Automation - Accelerating Your Job Search using AI
byDianaGray10
 
PDF
Build Agentic AI Applications with Oracle AI Database Private Agent Factory
bySandesh Rao
 
PDF
Reset RHEL Root User Password - RHCSA.pdf
byLinuxCert Guru
 
PDF
Building Powerful Web Apps to Improve Productivity and Engagement - Esri UK W...
byEsri UK
 
PPTX
Understanding the digital experience of FE teaching staff in 2024/25
byJisc
 
PDF
Open Source SecurityCon 2025 in Atlanta - Transparency Exchange API: Where To...
byPavel Shukhman
 
PDF
IDSECCONF2025 - Rama Tri Nanda - Hunting Stingray GSM on Budget.pdf
byidsecconf
 
IDSECCONF2025 - Aan Wahyu - Maze–Malware Analysis Platform.pdf
byidsecconf
 
Flatpak CLI Application Management & Building on RHEL.pdf
byLinuxCert Guru
 
FastTrack CISSP Reference By Jobyer Ahmed
bysocial72
 
Opening Plenary - Esri UK Welsh Conference 2025
byEsri UK
 
Agile Trailblazers (Scrum Masters, Product Owners, Coaches, Project Managers,...
byScott M. Graffius
 
November Patch Tuesday
byIvanti
 
Inclusive India_Samir_Dash_10 NOV_FINAL 2025.pdf
bySamir Dash
 
Webinar: Introduction to LF Energy SEAPATH
byDanBrown980551
 
UnityNet AI Induced Harm & Youth Protection Position Statement 2025-11-15
byLHelferty
 
Standardising and simplifying systems and data management - Esri UK Welsh Con...
byEsri UK
 
Cyber Security Quiz 1st Year CSE CS Students
bynirmalamca
 
#MakeAIMatter for HR Professionals | AI Transformation Workshop by Tekdi Tech...
byParth Lawate
 
IDSECCONF2025 - Ali - DursGo–Web Security Scanner with AI Analysis.pdf
byidsecconf
 
Upskill to Agentic Automation - Accelerating Your Job Search using AI
byDianaGray10
 
Build Agentic AI Applications with Oracle AI Database Private Agent Factory
bySandesh Rao
 
Reset RHEL Root User Password - RHCSA.pdf
byLinuxCert Guru
 
Building Powerful Web Apps to Improve Productivity and Engagement - Esri UK W...
byEsri UK
 
Understanding the digital experience of FE teaching staff in 2024/25
byJisc
 
Open Source SecurityCon 2025 in Atlanta - Transparency Exchange API: Where To...
byPavel Shukhman
 
IDSECCONF2025 - Rama Tri Nanda - Hunting Stingray GSM on Budget.pdf
byidsecconf
 

Large Data Management Strategies

  • 1.
    Managing Large DataVolumes Suchin Rengan, Director, Salesforce Services @Sacrengan Mahanthi Gangadhar, Senior Solutions Technical Architect, Salesforce Services
  • 2.
    Safe harbor Safe harborstatement under the Private Securities Litigation Reform Act of 1995: This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services. The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site. Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.
  • 3.
    We do alot of things with data… Data Searches/ Reporting/ List Views Data Creation (Loads/ Manual) Data Archival Am I using the platform’s features optimally? How do we ensure we keep up with performance? What factors do we need to consider across each topic? How can I ensure I have a scalable process? Data Integration (Out and In) Data Extracts
  • 4.
    SFDC Cloud Computingfor the Enterprise Infrastructure Services Network Storage Operation System Database App Server Web Server Data Center Application Services Security Sharing Integration Customization Web Services API Multi-Language Operations Services Authentication Availability Monitoring Patch Mgmt Upgrades Backup / NOC Customer Avoids: Tuning OS, Capacity Management Tuning Web Servers, Certificates, Log file Mgmt, etc Tuning App Servers, threads, Java stack, memory/log mgmt Tuning DB Servers, memory mgmt, disk distribution Network management, bandwidth Innovation Development Data Model Business Logic User Interface
  • 5.
    Lets understand theunderlying Platform Your System Apex API User Custom Object Custom Object Account …
  • 6.
  • 7.
    Considerations at everylayer and level Apex Triggers File Storage VF Workflow Rules Sandbox Apex Data Storage Data Objects Sharing Tables Indexes Skinny Tables Validation Rules API Logic / Application Layer Storage Layer
  • 8.
    SOAP/REST API SOAP API RealTime Relatively slow for large volumes Loads for up to 250K records Batch side Time out / failure API Bulk API Batches per day Parallel mode Larger than 250K records Rolling 24 clock for available batches Time to download results
  • 9.
    Bulk API –Asynchronous Process Data streamed to temporary storage Job updated in Admin Setup Client Send all data Processing Processing Processing Servers Thread Thread Data Batches Check Status Retrieve Results Dataset processed in parallel Dequeue batch Insert/ update Results Save results
  • 10.
    Bulk API The “go-to”option for tens of thousands of records and up Up to 10,000 records in a batch file Asynchronous loading, tracked in Setup’s “Monitor Bulk Data Load Jobs” section Walkthrough time! Example: American Insurance Co. ? 230 million records processed in 33 hours, 14 hours ahead of schedule
  • 11.
    Some tips forloading •Difficult to extrapolate performance in a Sandbox • At Par or Better in Production •Sharing Calculations •Indexing •File Storage •Triggers – Act Judiciously •Upserts – Avoid Them! •Parent References
  • 12.
    Sequence of Events/Logic ParentRollup Summary System Validation Custom Validation Assignment Rules Auto-response Rules Workflow Rules Escalation Rules APEX Layer Before Trigger After Trigger Database Layer Begin Transaction End Transaction
  • 13.
    Initial Load –Incremental or Big Bang Obj 1 Obj2 Obj3 Option 1 All Objects Option 2 Legend Pre-Implementation Activities Initial Load Validation Catch up and Ongoing Sync User Activation
  • 14.
  • 15.
  • 16.
    Data Extraction –Bulk Query Current Limitations Bulk query works exactly like data loads  Create a job (Job Id)  Each query is a batch (Batch Id)  Close the Job and fetch results when job is complete Limitations  Query optimizer has 100mins processing time (timeout issue)  Informatica currently does not support Bulk Query  Other tools like data loader can submit only one query  Currently requires a custom client for submitting multiple queries for a given job
  • 17.
    Data Extraction -Chunking Auto Number Chunking  Query smaller chunks of the entire data  Use Auto number and formula field for internal indexing  Find chunk boundaries (25K) and issue queries for each chunk PK Chunking  Use primary key to chunk (ID)  Usually better performance when entire object is extracted  Find chunk boundaries (250K) and issue queries for each chunk Auto Chunking (“safe harbor”)
  • 18.
    Data Extraction - Chunking 1 Q1 JobId (1234) Q2 Q1 -> Batch Id (123) Q2 -> Batch Id (234) Q3 ….. Q4 Qn -> Batch Id (789) Close Job Id (1234) Q5 Find Chunk Boundaries -> Create Job -> Submit each batch/query -> Get Results -> Close Job Qn 150 M
  • 19.
    Data Cleanup - GeneralConsiderations Deletion is one of the most resource intensive data operations in Salesforce and can perform even slower than data load in some cases (objects with complex sharing, with Master/Detail relationships, with Rollup Summary fields etc.) even in bulk hard delete mode. Custom objects can be cleaned up by truncation. Note that truncation cannot be performed on OOB standard objects. Data in standard objects can be deleted only using Delete API, faster performing Bulk Hard Delete option is available Records from User object cannot be deleted, only deactivated.
  • 20.
    Data Cleanup - Truncationand Hard Delete Where Truncation is not Possible?  Are referenced by another non empty object through a lookup field  Are referenced in an analytic snapshot  Contains a geo location custom field  Have a custom index or an external ID field Hard Delete  Hard Delete option is disabled by default and must be enabled by an administrator  Observed about 4.5M/hr on Task object delete, versus 18M/hr for load time, indicative of how slow of a process it is…
  • 21.
    Data Cleanup - GeneralRecommendations General Recommendations  When removing large amounts of data from a sandbox org consider Refresh option first. This is the fastest way to clean up an org. Additional benefit of Refresh option is that data in User object is removed also.  To remove data from a custom object use truncate function. Truncation deletes an object entirely and places it into the recycle bin. Use Erase function to physically remove data from the system it frees up storage space. Note it might take several hours to erase data from a large custom object.  To remove data from standard object use bulk hard delete option. Note that performance of bulk hard delete is quite low so plan sufficient time to remove large amounts of data using this option. Recent test on Task object in Shadow production org demonstrated performance of ~2M records deleted per hour. On Account object the rate might be even lower.  When planning large scale tests in production environment consider possibility of getting 2 orgs that can be used in “round robin” mode: when a test is performed in one org, cleanup is being performed in another.
  • 22.
    General Guidelines –Tips / Partner Tools  Tips  Test early and often and as big data sets as possible  Split initial load set to smaller subsets (we used 10MM records). This allows for greater flexibility in load monitoring and control  Queries: for aggregate data validation queries (since bulk query option is not available for them) consider use of Work Bench with special asynchronous background processing configuration that prevents early timeout on client side (more info: http://wiki.developerforce.com/page/Workbench). Publicly available WB app https://workbench.developerforce.com/login.php can be utilize  ETL Tool  Timeout settings  Bulk query support  Handling of Success and Error files  Monitoring of Job Status
  • 23.
  • 24.
  • 25.
    General Guidelines forother areas  Searching and Reporting and List Views • • Filtered Queries • Skinny Tables • Data Scope • Roles and Visibility •  Indexes Report Folders Data Governance • • Data Management and Administration •  Data Model Security and Access Controls Data Archival and Retention • Storage
  • 26.
    Speaker Name Speaker Name SpeakerName Speaker Name Speaker Title, @twittername Speaker Title, @twittername Speaker Title, @twittername Speaker Title, @twittername
  • 28.
    Initial Load -Parent References (* goes into notes) General LDV recommendations  Avoid reference to parent record via External Id if possible, use SFDC Id instead.  XLDV considerations  Consider preparing source data sets with native SFDC id for parent reference. This might be achieved by querying key mapping data from parent objects and performing joins on client side to retrieve SFDC Id for parent reference vs. using External Id reference. Additional benefit – this is a good data validation step. Particularly important on large objects with multiple parent references  Note that querying data from extra large objects might be a challenge by itself and often takes hours. Consider alternative approaches, for example collecting mapping keys from initial upload log files.  Referencing parent via native SFDC GUID will also allow the use of FAST LOAD option when available (next release, “safe harbor”)
  • 29.
    Initial Load -Data Validation General LDV recommendations  The following post load data validation checks are generally can be considered:  Aggregate values/subtotals comparison (numeric and currency values) Data validation on XLDV is a challenging task. Queries on tables with hundreds of millions of records are slow and often time out (even bulk queries). Consider options for splitting (chunking ) your data validation queries by filtering on indexed attributes.  Special chunking techniques: auto number chunking, PK chunking  Do not underestimate and plan for enough time for performing post load data validation on XLDV. Total counts comparison   Attribute to attribute comparison  XLDV considerations  Spot checking  Negative validations
  • 30.
    Incremental synch’s General LDVrecommendations  For SOAP API’s use the largest batch size (200).  During incremental syncs on objects with large amount of processing the biggest batches can fail due to time out (10 min per batch). In this case batch size might need to be reduced.  It is possible to “batch” standard SOAP API calls by submitting several jobs in parallel. When loading data into the same object using several SOAP API batches in parallel consider to group child records by the parent in the same batch to minimize DB locking conflicts. XLDV considerations  For incremental synchs of larger data sets (over 50K-100K) use bulk API’s.  Consider configuring client application to programmatically set load mode (SOAP vs. Bulk) based on the size of incremental data set in each load session.
  • 31.
    Data Loads - CurrentVolumes - REMOVE Object User Role User Account (+Contact) Agent Role Relationship Household Household Member Opportunity Task Remark Note Attachment Life Event External Agreement Agent Marketing Inf. Policy Policy Role Agent Agreement Role Billing Account Billing Account Role Total • Initial volume close to 3.5 B • Worked with customer to reduce volumes • Split the implementation into two phases approx. 1A # of rows approx. 1B # of rows 60,000 200,000 186,000,000 155,000,000 11,000,000 96,000,000 155,000,000 100,000,000 104,500,000 16,500,000 15,000,000 6,000,000 845,413,000 120,000,000 18,000,000 900,000 33,540,000 26,000,000 149,000,000 165,000,000 172,000,000 23,220,000 23,220,000 730,880,000
  • 32.
    Data Loads -Org/Environment preparation and tuning For XLDV it is recommended to perform test loads in Production like environments  To get a true representation of the performance in Production  To test loads in real production environment for environment specific settings  Allows for more accurate planning of data load in terms of timing and dependencies Concerns: Large scale deletion / environment cleanup after tests. Deletion is an resource intensive data operation and the slowest (even using bulk hard delete)  Cannot predict in advance all possible issues and consequences related to multiple mass data deletions in actual production environment  Users cannot be deleted from Production environment, only deactivated. Multiple User load tests can produce a “garbage pile” of inactive User records that would clutter Production environment.
  • 33.
    Data Loads -Org/Environment preparation and tuning (Contd..) General best practice recommendation:  Request a dedicated production test environment to avoid testing in actual Production org.  Work with SFDC Operation on plans for re-provisioning test production environment because erasing XLDV data from an org for subsequent test might not be feasible Coordinate with SFDC Operations on any large scale test activities in Prod environment (systems can be shut down by SFDC Operations as a suspected DDOS attack) Plan accordingly and factor in the time required for setting up and configuring test production environments when multiple production tests rounds are required (production org cannot be refreshed as sandbox org so it needs to be re-provisioned and configured from scratch)
  • 34.
    Data Loads –Org/Environment preparation and tuning General LDV recommendations  For large volume data loads it is possible to request additional temporary special changes on SFDC side to expedite load performance XLDV considerations  Request increase of batch API limit  Request Increase of file storage  Notify SFDC Operations about upcoming loads with approximate data volumes  Request turning off Oracle Flashback  Use FASTLOAD option (future option, “safe harbor”) Note that some settings/perms can technically be tweaked only in Production environments, not in Sandbox environments
  • 35.
    Initial Load -Use Bulk APIs for massive Data Loads General LDV recommendations SFDC Bulk API are specifically designed to load large volumes of data. It is recommended to use bulk API for LDV initial data load. Use the Salesforce Bulk API in parallel mode when loading more that few hundred thousand records. Main performance gain of bulk API’s is executing batches in parallel. Note that generally bulk API (w/o batching, in sequential mode) might perform slower than SOAP standard API so if number of records is less than certain threshold (that depends on number of factors: object being loaded, processing on SFDC side, number of attributes loaded etc…) using bulk API’s might become counterproductive. XLDV considerations  Use bulk API’s in parallel mode with maximum allowed batch size whenever possible to maximize number of records that can be loaded within 24 hour period (based on bulk API batch limit – standard limit 2K batches per 24 hours)  Use standard SOAP APIs for incremental ongoing data synchs (data sets less than 50-100K) and bulk APIs for larger incremental data sets (over 50K-100K rows)  Using standard SOAP API’s for incremental synchs has a benefit of reducing the risk of database contention during uploads of child objects  Allow enough time for collecting Request and Result log files (takes ~15 min for INFA to extract load logs for 10MM rows loaded)
  • 36.
    Initial Load -Planning Bulk Jobs (Parallel vs Single Object) General LDV recommendations When planning bulk jobs consider the following:  When loading large volume of data into a single object it is recommended to submit all the batches in one job. Splitting (partitioning) data across several parallel jobs does not improve performance when each individual job is big enough. Note that batches of all jobs are placed in the same batch pool and queued for processing independently of jobs they belong to.  When loading data into several objects running several jobs for different objects in parallel is recommended when jobs are small enough to not consume all the available system resources. XLDV considerations  For loading extra large volumes of data into single object in Salesforce it is recommended to run one object load at a time. The rationale behind this recommendation is that bulk jobs with hundreds/thousands of batches consume all available system resources and running other jobs in parallel just causes jobs to compete for the limited resources. The other consideration is DB cache. When several objects are loaded in parallel sharing DB cache is causing more frequent swapping that slows down overall performance on the DB layer.
  • 37.
    Initial Load -Defer Sharing Calculations General LDV recommendations  Reduce sharing recalculations processing during initial data upload.  Disable sharing rules and/or use the defer sharing calculation perm for custom objects while loading data.  Note that sharing calculations will still need to be performed after data load is complete and it can take significant amount of time but it can reduce loading time by allowing sharing to be calculated in “off” hours. XLDV considerations   Based on the XLDV performance tests general recommendation for deferring sharing calculations might not be applicable for extra large volumes. Post loading sharing calculations might take very long time to execute on bigger volumes (it is not parallel currently, on roadmap for the nearest release)  Load data in the following sequence to enforce main sharing calculations to be performed during data upload:   Create main user groups and group members  Create sharing rules   Create User Role Hierarchy and upload users with correct roles assigned  Sharing tables can alternatively be uploaded directly that allows to avoid sharing recalculation on initial load altogether Set OWD on objects Private where applicable Upload data with correct owner specified. Consider uploading sharing tables on some objects directly to avoid sharing calculations during upload and post load processing.
  • 38.
    Initial Load -Triggers, WF Rules, Data Validation Rules General LDV recommendations  Disable when possible triggers, WF rules, data validations on objects being uploaded.  To avoid lengthy post load catch up processing consider performing data transformations/ validations on source data set prior to data upload or  consider batch APEX post-load processing for data transformations that can’t be performed prior to upload.  For each trigger, WF rule and validation rude individual analysis should be performed to define the best strategy. XLDV considerations  General best practice rules apply
  • 39.
    Initial Load -Defer Indexing General LDV recommendations  Reduce additional processing on SFDC during initial data upload: turn off search indexes, consider creating custom indexes after data load XLDV considerations  Based on initial performance results index recalculations on extra large volumes of data take long time post load (not parallelized currently). It might be a better approach to configure all required indexes prior to XLDV load.
  • 40.
    Initial Load -Use Fastest Operation Possible General LDV recommendations  Use the fastest operation possible. insert is fastest then update, then upset XLDV considerations  If possible load full set of initial data in one go using insert with only small incremental upserts when needed (failed records reprocessing for example).  To avoid loading big volumes data in upsert mode (for example when main load job fails in the middle and remaining data should be added on top of existing data in an object) consider configuring a client load job the way that joins existing records to remaining records (on external Id) outside of SFDC and then loading resulting set in insert mode.
  • 41.
    Initial Load -Use Clean Data Set General LDV recommendations  Use as clean data set as possible  Note that errors in batches are causing single row processing on the remainder of the batch that slows down load performance significantly.  XLDV considerations Perform rigorous data validation prior to upload. This will allow: 1. Loading most of the records in the fastest insert operation and avoid slow down due to the preventable errors 2. Avoid slow down when processing goes record by record within a 200 record transactional batch when number of failed record reaches a Threshold
  • 42.
  • 43.
  • 44.
  • 45.
  • 46.
    Data Load Considerations(*- will be replaced) Topic Consideration Data Model Normalized/ De-normalized Sharing Model Sharing calculations Target Data Does it really need to exist in SFDC? BP etc. Test Environments Test them before you deploy! Timelines Adequate timeline for testing User data Cannot be deleted API Soap versus Bulk Batches Limit of 2K batches for Bulk API a day Deletes Oh No.. Extrapolation Not really possible What do I need to do? SF Support to increase batches It is at par or better, if Prod can be used even better

Editor's Notes

  • #5 Traditional DB performance tuning techniques used to improve data load performance are not applicable to Salesforce multitenant cloud architecture. Standard objects and custom objects have different underlying DB tables and storage mechanisms, and can behave differently.Along with general LDV best practices the XLDV specific recommendations are included based on actual data design, volumes and results of performance testing conducted by SFDC Performance Test Lab team did tests on extra large data sets of several hundred million rows per objects, several billion rows per org.
  • #6 Traditional DB performance tuning techniques used to improve data load performance are not applicable to Salesforce multitenant cloud architecture. Standard objects and custom objects have different underlying DB tables and storage mechanisms, and can behave differently.Along with general LDV best practices the XLDV specific recommendations are included based on actual data design, volumes and results of performance testing conducted by SFDC Performance Test Lab team did tests on extra large data sets of several hundred million rows per objects, several billion rows per org.