Bulk API 2 is a new & better way of handling bulk record processing in Salesforce than the previous Bulk API. A few of the advantage of Bulk API 2 is automatic retries of failed records, a governor limit of 24 hours based on the number of records processed instead of batch job count, automatic handling of batch size, etc. More detail of comparison can be found in Salesforce documentation here.
This article & video is assuming that you know the basics of Mulesoft. Coming back to this post, We would be using Mulesoft to insert contacts using Bulk API 2. One of the caveats of bulk API is that Dataloader still does not support Bulk API 2. It takes a little more time than the data loader in Mulesoft to use Bulk API 2 but it’s worth it and better compared to some other tools.
In summary, there are 3 steps that need to happen to use Bulk API 2 to insert records
- Create a request in Salesforce by Sending a CSV file
- Keep Retrying by checking the status of the job
- Retrieve results
In Mulesoft, we are using the below components
- HTTP Request to read CSV file. We can also use a file reader instead of an HTTP request
- Using Salesforce Component – Create Job bulk API v 2 (image Er-2.1)
- Choose Object Type
- sObjects needs CSV content and we can use payload as input
- Line ending – CRLF or LF depending on your configuration
- Column delimiter – COMMA
- To get the status of the job, we are using component – Get job state bulk API v 2
- We are using Until Successful component in Mulesoft to keep retrying
- once we get the status, we use the Choice component. We are raising an error if the state is not JobComplete or Failed. A list of all valid states can be found here
Below image shows how each components are connected & sequenced, it would give you good idea on what needs to be done.
To test the below flow, you can use postman. Make sure you set the Body type as Binary and choose CSV file. You can use this CSV file as a sample to save time. Make sure to replace AccountId with AccountId in your Org.
Below is the Configuration XML
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core" xmlns:salesforce="http://www.mulesoft.org/schema/mule/salesforce"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/salesforce http://www.mulesoft.org/schema/mule/salesforce/current/mule-salesforce.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd">
<http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="b3f7540e-a465-4b2d-ad4d-46fc56cb1637" >
<http:listener-connection host="0.0.0.0" port="8081" />
</http:listener-config>
<salesforce:sfdc-config name="Jit29Org" doc:name="Salesforce Config" doc:id="6f407f5c-ccfa-4db6-befa-3ced16196c0e" >
<salesforce:basic-connection username="youremail@gmail.com" password="somepwd" />
</salesforce:sfdc-config>
<flow name="jit29backupFlow2" doc:id="fa954752-faf4-4e78-bdd6-c44bf54b398b" >
<http:listener doc:name="/bulkInsert" doc:id="73773cb4-9f69-4700-8518-63cea055b444" config-ref="HTTP_Listener_config" path="/bulkInsert"/>
<salesforce:create-job-bulk-api-v2 doc:name="Create job bulk api v 2 - Contact" doc:id="810bb823-d58c-4901-8e5f-9d18ebdc727a" config-ref="Jit29Org" objectType="Contact" operation="insert" lineEnding="CRLF" doc:description='output application/java --- { LastName: payload.LastName, MobilePhone: payload.MobilePhone, Email: payload.Email, AccountId:"0011U000014hIcIQAU" }'>
</salesforce:create-job-bulk-api-v2>
<set-variable value="#[payload.id]" doc:name="jobId" doc:id="2cd419b4-286c-44b0-9ba6-b80ca20a7437" variableName="jobId"/>
<logger level="INFO" doc:name="Logger" doc:id="ebf27c8a-16e4-4a67-a437-369df8ba01e9" message='Job status is #[vars.jobstate] Before making job status request. Job ID - #[vars.jobId]'/>
<until-successful maxRetries="3600" doc:name="Until Successful" doc:id="76d0ffbb-0ff5-4ef8-acfb-43455c2e5af9" millisBetweenRetries="5000">
<salesforce:get-job-state-bulk-api-v2 doc:name="Get job state bulk api v 2" doc:id="b9bff670-b903-4a0b-8258-4d45d428bdb2" config-ref="Jit29Org" id="#[vars.jobId]" />
<set-variable value="#[payload.state]" doc:name="update jobstate" doc:id="4fd26c43-00e0-483b-b8c6-44541b5e110f" variableName="jobstate" />
<choice doc:name="Check Job State" doc:id="7d2d53d8-503c-4960-b685-cdce6ef81795" >
<when expression='#[vars.jobstate == "JobComplete"]'>
<logger level="INFO" doc:name="Logger" doc:id="94810163-b26d-46e2-8f63-2bd75b26b0a6" message="Records Processed Succesfully in Salesforce" />
<salesforce:retrieve-job-failed-results-bulk-v2 doc:name="Retrieve job failed results bulk v 2" doc:id="eff1b41a-20ee-46e6-adbf-b9ca22df655d" config-ref="Jit29Org" id="#[vars.jobId]"/>
</when>
<when expression='#[vars.jobstate == "Failed"]'>
<logger level="INFO" doc:name="Logger" doc:id="b77237de-545d-46e6-aca1-3c171142f005" message="Job Failed . #[payload.errorMessage]. #[payload.state]"/>
<salesforce:retrieve-job-failed-results-bulk-v2 doc:name="Retrieve job failed results bulk v 2" doc:id="5a3cf99b-b1d7-4263-bb5c-5d8bd720dcc2" config-ref="Jit29Org" id="#[vars.jobId]"/>
</when>
<otherwise >
<logger level="INFO" doc:name="Logger" doc:id="6c28b5d8-3486-4c03-96e4-678583e45bf2" message="Going to Raise Exception"/>
<raise-error doc:name="Raise error" doc:id="5abedfd7-eb00-48c9-95b6-12c4a74fb185" type="SF:JOB_IN_PROGRESS" description="Bulk API 2 is still in progress"/>
</otherwise>
</choice>
</until-successful>
<logger level="INFO" doc:name="Logger" doc:id="1d00f918-2c4f-414c-a4c0-3f4ccf6cc687" message="All Done"/>
<logger level="INFO" doc:name="Logger" doc:id="f2e7fabe-d25d-4257-9be0-aa2cf2e89c94" message="#[payload]"/>
</flow>
</mule>
Solving Common errors in Mulesoft while using Bulk API 2
Error : InvalidBatch : Field name not found
There could be 2 reasons for this error
- CSV file header not matching with Actual field API Name OR
- CSV file format is UTF-8. Salesforce does not understand UTF-8 CSV. It would accept plain CSV file
Error : LineEnding is invalid on user data. Current LineEnding setting is CRLF
Make sure the end-of-line setting in the MuleSoft property is set properly either to LF or CRLF as shown in image Er-2.1. To see what a line ending in your file is, open it in CSV and check the bottom right corner as shown in image Er-2.2
Leave a Reply