Definitions of Data warehouse, Data lake, Data Mart, Operational Data Store
Data warehouse is also known as Enterprise Data Warehouse (EDW). Data warehouse is used as source for Business Intelligent’s reporting and analysis. Data Warehouse system collects data from multiple sources and contains historical data for trend analysis reporting. ETL tool is used mostly to build Data Warehouse and interfaces around it. Data Warehouse acts as Single Version of truth.
2. Operational Data Store (ODS)
Operational Data Store is frequently confused and definition is overlapped with Data Warehouse. Some of my clients had used word ODS instead of Data Warehouse, which got me confused on number of occasion. As per my understanding & research, ODS is used to integrate data from multiple systems and feed it to Data Warehouse. Data Warehouse consist of complete history of data, whereas ODS contains latest or recent data (short window of data). Data load frequency in ODS is mostly hourly whereas data load frequency in Data Warehouse mostly is nightly because of data volume. Most important reason to have ODS in your company is ability to run report realtime, where source system does not have required reporting capabilities.
3. Data Mart
Data warehouse can contain many Data Marts. Mostly Data mart is created per business line or system that needs data from Data Warehouse. Indirectly we can say, Data Mart is access layer used to get data out of Data Warehouse by other systems.
4. Data Lake
Term Data Lake was coined by James Dixon, CTO of Pentaho to compare with Data Mart. As per James, Data Mart have several problems mostly related to data silos. Data Lake is method of storing data from sources in its actual or raw format that could be Relational Data, XML, flat files or even binary files. Other tools like ETL, access Data Lake as per need for reporting or analysis purposes.
Thrilling story of Salesforce Technical Architect on a quest to solve application problems and avoid governor limit errors
This is a work of fiction. Names, characters, businesses, places, events and incidents are either the products of the author’s imagination or used in a fictitious manner. Any resemblance to actual persons, living or dead, or actual events is purely coincidental.
Ha ha ha joking.. its combination of many real stories of my previous projects and few readers might be from my team itself 😉 . Leave comment my brave team mates if you are able to co-relate !!!
I tried to summaries as much as possibilities about governor limit errors and way to get out of it. Most of scenario described below may not be the first choice of Technical Architects however because of many reasons like budget, resource skill, tools available and compliance, bad decision could be taken which causes ripple effect on Salesforce scalibility.
Before start, I would encourage readers to leave comment on post and let everyone know in this suspenseful story, did you catch governor limits and Solution already before its exposed ? In this blog post, answers are mostly in invisible/white font which would not be seen until you highlight it. Its for few readers who don’t want spoilers.
After working on multiple Salesforce implementation project as an Architect, its time to share what I learned from those implementations and would strongly suggest to be considered before designing any “Salesforce Integration”.
Below image shows “integration mind mapping” used by me. I use it to consider some major aspects while discussing integration approaches with enterprise architects in various meetings. This image is very high level however if you think some more points to be considered or have some other thoughts on same, please share.
How to design and architect Salesforce application so that 24 hour Apex email limit error could be resolved and have reporting capabilities on emails sent from Salesforce
Sending email from Apex looks very interesting and powerful feature at first glance however by time your application evolves to more complex structure or more data, chances are very high that you will be hitting many Apex email limit related errors. As Salesforce is not mass email sending tool, it makes sense to have limits on total number of outgoing emails per 24 hours.
At a time of writing this post, mainly there would be two types of error while sending email from Apex as explained below
Transaction based errors
We can send 100 emails per SingleEmailMessage
Per day (24 hour) email limit
We can send 1000 emails per day
One point to note here that we can send unlimited emails to internal users including community users. More details on email limits can be found here (Salesforce governor limits)
I have seen multiple questions on Salesforce developer community on how to resolve error related to sending email per 24 hour in Apex. Below solution has been used by me for a long time and it is working to send thousands of email per day without any problem.
Let’s see how we can resolve this error with below design.