Now lets do some fact check :

• Execute method of Batch Apex can only call 1 Queueable
• Scheduler can call 50 Queueable
• We can have only 100 Schedulers in an Org
• If we run 1 Scheduler every minute = 60 Scheduler
• If each Scheduler executes 50 Queueable every minute, then 60×50 = 3000 jobs / min. Which means 3000 x 24 = 72k jobs / day

Big Question

Can you design an application which can use full potential of Async governor limit – even 250k / day ?

Designing Scalable Hyper Thread Application in Salesforce

I have discussed various approach to answer these questions on below podcast by SalesforceWay

Part 1 – Salesforce Hyper-Threading Road blocks

Part 2 – Salesforce Hyper-Threading Options

Feel free to drop comments and your thoughts on solution !!!

Ha ha ha joking.. its combination of many real stories of my previous projects and few readers might be from my team itself . Leave comment my brave team mates if you are able to co-relate !!!

I tried to summaries as much as possibilities about governor limit errors and way to get out of it. Most of scenario described below may not be the first choice of Technical Architects however because of many reasons like budget, resource skill, tools available and compliance, bad decision could be taken which causes ripple effect on Salesforce scalibility.

Before start, I would encourage readers to leave comment on post and let everyone know in this suspenseful story, did you catch governor limits and Solution already before its exposed ? In this blog post, answers are mostly in invisible/white font which would not be seen until you highlight it. Its for few readers who don’t want spoilers. 

Let’s start a story.

Part 1

There was a huge Salesforce Implementation project going on from last 18 months. I Joined this project almost at end where all major decisions were taken and implementation was done. There was one particular screen where total 14 API calls made. Salesforce team tried their best to convert this to single API and move everything else to ETL tool however it was ruled out, because of many non disclosable limitations and reasons. For sake of simplicity, lets assume Visualforce page is designed in such a way that each section on page needs two API callouts. User can navigate each section one by one or jump to last section. If user directly jumps to last section then all 14 API calls will be made in single transaction. One thing to consider here is that API calls are dependent, You cannot call API number 2 before API 1.

You already started imagining problems ? Well, lets wait for invisible(Select previous invisible text) text.

Problem 1

You need to call two APIs in sequence however save response from first API. Do some processing and use it as request for second API. Guess the problem ? By default, callouts aren’t allowed after DML operations in the same transaction because DML operations result in pending uncommitted work that prevents callouts from executing. (Select previous invisible text for problem)

Solution

Guess a solution ? Create an Apex based REST API which will perform DML operations(Select previous invisible text for solution). This is not a suggested solution however it was the quick way looked feasible at that time.

Problem 2

So, everything was fine, code implemented and tested successfully. However on stress environment, guess what obvious error QA team must have started facing ? Any guess ? Error was Concurrent requests limit exceeded. Check this blog post to know cause of this error message.

Solution

We cannot use Batch, future or Queueable Apex because, response needs to be returned on Visualforce. User needs to change some choices before hitting next section. So best solution found without hitting Concurrent limit request error was to use Continuation Object. You can refer this blog post to read more about it. This blog post explains how to use it in Lightning Component.

Problem 3

Finally we were able to make our system scalable and long running requests were not problem anymore. Ohh boy, we were saved. It was time to go play table tennis, fusse ball and all your favorite games over the weekend.

But wait, you get a call from your technical lead saying sir, new design is not working in one particular scenario. if user directly jumps to last section, we had to make 14 API callouts back to back from Apex. We got an error, guess what it is this time ? You cannot use more than 3 sequential continuation in a single request handling. You used 4 Continuation. (Select previous invisible text for error).

Solution

Was there any solution here ? Its Salesforce governor limit. If we don’t use Continuation Object, we will hit Concurrent request limit error, is catch 22 situation. But, Here is the solution, Chain Unlimited Continuation Objects.

Part 2

We were able to solve some governor limit errors and system was atleast in working state. Problem was, it was taking around 4-6 minutes to complete end to end transactions, which was unacceptable. So, Part 2 for me was improve performance without impacting existing functionalities. So, I thought to check source code. Ohh boy, not sure which design pattern was used but almost 8-9 classes were involved. And there was one class I remember, which had more than 6k lines of code. Anyways, point is, I had very less time to address this as customers were unhappy and got threat that they may abandon System. So, my next quest was analyzing debug log. For one click, around 7 debug log files were generated. After spending hours, I found first Issue in code. Lets repeat Question – System was very slow, and there were two objects with 6 Millions+ records.

Problem 4

Any guess on problem, why Salesforce was slow ? Should not be difficult to guess  It was unindexed SOQL. Problem was not only indexing issue but the way Database was designed. Code was fetching information from 5 level of objects and condition was written on relationship fields. The object on which SOQL was primarily written had 6 Millions + records. Solution to this problem was to convert these 5 objects to one flat object (denormalized table) with indexed field. However application was already in use from last 2 years and there were huge impact. What else could have been done to address the problem asap ?

Solution 4

We searched whole organization using Force.com IDE to identify SOQL on same object and which fields from parents were used in condition. We created actual field on child object and indexed it. Select previous line to know answer.

Problem 5

So, in solution 4, we identified all fields which needs to be created in flat object. We considered leaf object as flat object because it will always have  parent records (5 level). Problem was, how to update these fields without impacting existing code? We didn’t needed these fields in current transaction so decided to use Asynchronous Apex, Queueable Apex. Advantage of this approach was that we didn’t touch any existing trigger or Apex class. Do you smell problem here ? Started getting sporadic errors related to – Unable to lock row – Record currently unavailable, read problem by selecting previous text.

Solution 5

If we move our code in Asynchronous block and if this code takes more than 10 second to complete then other area of application may receive this error. unfortunately, in our case around 7-8k records were going through DML at same time causing lock more than 10 sec. So as a solution, we did what we were trying to avoid, writing trigger on flat object to populate indexed fields.

Problem 6

There were some Apex code where around 7-8k records were upserted in same transaction. This object was vanilla object without any triggers. We added a trigger on this object to populate fields before insert or update of record. Guess the possible governor limit in this approach? QA and UAT testing was successful however our best friend Apex Test classes saved us by giving early alarm of one possible error. “Apex CPU time limit exceeded” (Select previous invisible text for error). Check this blog post for best practices.

Solution 6

When we are inserting / updating around 7k records in single transactions, even small code will cause Apex CPU time limit exceeded error. First of all, we should have avoided this huge DML in single transaction. But implementation was like a spider web with huge dependency with each other. Didn’t had much time left to perform impact analysis as old code was already on production.

So how did we solved it ? We broke upsert into update and insert by introducing extra SOQL. Other than this, we did dirty check before performing any update operation. Select previous line to know answer. Believe me, dirty check reduced around 70% of DML need in update operations.

Few important best practices I would like to share here:

1. We cannot use static variable to avoid recursive execution of trigger if 200+ record gets inserted via Apex. Static variable approach will work with dataloader but in case of Apex, trigger will not execute after 200 until its handled specially.
2. Dirty check of record is very important. In your Apex code, before performing any update DML, check if data is even changed or not ? I have seen code around 90% of time, where DML is done on record but it already has same value in Database. All you have to do is issue 1 extra SOQL and compare. It will save CPU time as well DML governor limit. Its huge game changing approach in LDV (Large Data Volume) orgs.
3. Upsert statements are very slow. Even slower than combined insert and update. It gives lots of advantages however needs to be considered while we are working with LDV. Upsert is even worst in terms of performance when we create parent child relationship using parent records external Id.

Problem 7

This would be last problem for sake of simplicity of this blog post. All above solutions worked but there was a problem. This leaf object was growing exponentially. After stable code, it reached 10 Millions+ records. This object was only used to display around 4-5k records on screen in read only format with the help of server side pagination. We still wanted to show data on screen however reduce total number of records being created in object. Any guess ?

Solution 7

We saved these 4-5k records in single JSON file and saved in Attachment. At controller side, we used wrapper class to deserialize records and show it back on page. Select previous line to know answer.

Hope this blog post will give you a different perspective while designing solutions on Force.com platform. If you reached till this point then am pretty sure you must have got more than half solution by your self.

Let me know Tale of Governor limit in your project and how you handled it in below comment section.

Below image shows “integration mind mapping” used by me. I use it to consider some major aspects while discussing integration approaches with enterprise architects in various meetings. This image is very high level however if you think some more points to be considered or have some other thoughts on same, please share.

1. Data Compliance

Lets start with Compliance part first. It is possible that client has confidential data and doesn’t want to move it to Cloud. In that case, we can expose existing web application of Client using some Salesforce products like “Canvas” or “Lightning Connect”.  For Lightning Connect, data needs to be available on DMZ layer however for Canvas it can exist on Intranet and therefore end users can access data only within office, providing flexibility of cloud and security of intranet.

2. ETL tool

As a best practice for Integration design, loading, transforming and extraction of data should be avoided within Salesforce. Salesforce is multi-tenant architect and therefore has many governor limits. I have seen many projects getting failed because of decision; not to use ETL tool. There may be many reason why ETL may not have been considered, like architect didn’t foresee much in future about its necessity or Client was short on budget. Even though ETL tool adds up some budget but then it opens endless possibilities.

If ETL is considered, then there are some very important question needs to be answered and designed.

a. What will happen if ETL or any System goes down

Software industry is full of possibilities and malfunctioning, no one can be sure about hardware and software design. Business continuity plan is very important to be considered. And this question must be considered in meetings and design “What will happen if ETL goes down” ? Simple answer is declaring clear “Accountability” between systems. Any application can go down or any application can face an error; it could be source System, middle ware or Salesforce. Defining clear accountability of system is very important. Who will be responsible to handle situation if ETL goes down ? Who will be responsible to handle situation if Salesforce goes down, or doesn’t respond in timely manner or some Apex or server error occurs ? We should be having proper error handling and logging mechanism between systems and backup plans as well. For example – If ETL goes down, maintain all request in some “Messaging Queue Server” between Source system and ETL. once ETL comes on-line, start processing queue.

b. Throttling capabilities

When more than two systems are involved in communication, not necessary that both system will perform same. Might be possible that Salesforce could be faster or slower on some unlucky days. We should consider this point as well in our design. Most ETL tool has capability of throttling to make request faster and slower on basis of error or network speed. Personally, I think this is very important in high traffic integration. In one of my project, we had to load (upsert) millions of data in Salesforce on weekly basis. Because of heavy design of Salesforce apex triggers and multiple field update workflows on same object, Salesforce performance was much slower than usual. There were no throttling capabilities and therefore there were lots of loss of data because of various error in Salesforce like “Concurrent Apex limit”, “Unable to lock rows” etc…

c. Priority of incoming and outgoing messages

If we consider “messaging” and “throttling” in our design then most obvious question needs to be considered in “priority of messages”. As message may be sitting in queue, it is possible that some messages should be processed near real time and very important for next system. We should be having logic in “ETL” or “Messaging queue” to decide priority on basis of some message criteria.

3. Fallout reporting capabilities

Either we are going with or without ETL tool, fallout reporting capabilities are very important in integration design. Fallout report may also be referred as “loss in data” report where few message or request has not be processed because of various errors between systems. If we skip this step, validating data accuracy could be very difficult and next to impossible. This may be considered as a part of error and log handling design.

4. API limit considerations

If external system is trying to connect to Salesforce then it is using API limit which is defined for every organization for a 24 hours. We also have a limit on Bulk API per 24 hours. My most of Salesforce implementation project has 3000+ Salesforce users and therefore I have not faced issue with API limit however good to consider in design.

5. Single ETL tool to connect to every applications

This point I know is very debating. I am saying that every application connecting to Salesforce should have single ETL tool where few of you may disagree with me. In one of my project, I had to integrate Salesforce with around 18 Systems. BOOMI was ETL tool of choice and it was used to integrate around 8 systems. Other Systems were integrated using webservices, custom Java tool which implemented BULK API and command line dataloader. We faced lots of issue maintaining and debugging error in all these tools. It would have been very easy if Single ETL tool could be used for all systems. As multiple tools were used, few tool were black box for us and they were using on there Will whenever they wanted to change data inside Salesforce. As discussed above, “Accountability” was main issue. If any tool was failing because of any reason, Salesforce team were to blame in every case and we were on fire .

6. Different user for different System

Lets say, if you are integrating with 10 systems using ETL tool, it is always best practice to have different users for all those system. It has lots of benefit, flexibility and control. We can have different profiles for each user, complete control and easy to debug. Imagine single “integration user” is used to interact with all system, chances of data skew is very high. We cannot switch off user from Salesforce end, hard to configure profile as it may break existing integration and many more.

7. Integration User license

As discussed above, its good practice to have different Salesforce user for different systems. It is not necessary that we need “full Salesforce user license”. If we are not using standard objects then cost can be saved by opting “Force.com app bundle license” or any other suitable Salesforce user license.

8. Data Volume over the time and Data archiving considerations

If you are integrating with multiple systems or not, Data archiving strategy is very important to be considered. There are many advantages of data archiving including improve in performance of reports, list views, global search and saving space of Salesforce. Data archiving can be done using ETL tool or even command line dataloader directly connecting to databases like SQL Server. While designing Data archive strategy, easily getting data back is also very important. Example – using Dataloader, if backup is done in multiple csv files, It would be hard to find record from hundreds of csv files. Even if RDBMS like SQL Server or MySQL is considered, indexing, partition and other performance improvement techniques must be considered. It should not be like one way road where we can archive data but cannot get it back when needed. I have used “DBamp” in many of projects where ETL tools were not used and would recommend to try it out.

9. Single Sign On

This is also very important topic and needs much attention and thoughtful discussion. Salesforce provides lots of option to implement Single Sign On using federated authentication, Delegated authentication or many other Authorization providers. In addition of all APIs available to implement SSO, Salesforce also provides “Identity Connect” which could be considered for SSO implementation. There are many third party applications as well like Ping Identity or Okta. Also, as a best practice, do not enable SSO for System admin users. If something happens with Identity provider or Web services are not responding, end users will be blocked and System admin could still log into system. If we enable SSO for System admin as well then complete Salesforce instance would not be accessible by anyone.

10.  Outbound messages are your best friend

Question is “Are you using or considering Outbound messages ?”. I have seen, many Salesforce implementations where integration could be easily done by using outbound messages instead of Apex callouts. You don’t need to write Apex code, Test classes and polling mechanism if communication fails. Outbound message tries until message is successfully is delivered though there are some design considerations while using Outbound messages.

Please feel free to leave comment and suggestions on this topic so that we can discuss each others experience in integration design and problems.