When Salesforce Goes Down: Ensuring API Availability

How to monitor for disruptive apps, APIs and more

Most experienced Salesforce experts have war stories about a poorly behaving integration that stopped their business processes entirely. In Salesforce, this has much to do with managing your availability by monitoring quotas and much more. This blog covers issues and resolutions to spare you severe Salesforce interruptions.

The Interplay of Security & Availability

In a previous post, I discussed the confidentiality risks associated with unused or untrustworthy SaaS applications having OAuth tokens to your Salesforce environment. Now, let's delve into another crucial aspect of security: availability. Availability refers to maintaining the uptime of your services — in this case, Salesforce. While uptime is often viewed as a quality or business continuity issue, it's also a significant security concern.

Do all inbound Salesforce connections potentially impact the availability of your Salesforce org? Absolutely.

Salesforce API limits operate on a rolling 24-hour window. If you exceed your call quota, you'll have to wait until the number of calls within the past 24 hours decreases. Consequently, exceeding the API limit can disrupt all business processes that rely on inbound API calls. Examples of problems this can cause are:

• Orders or customer tickets aren’t coming through
• Financials or deliveries aren’t synced between ERP and Salesforce
• Marketing automation tool doesn't get data

While your quota will gradually replenish, continued overuse will lead to ongoing disruptions.

Salesforce has documentation of API limits. Here's a simplified overview of the limits for various editions:

What’s interesting is that significant differences exist between user and non-human user license types. Some licenses don't contribute to API calls, while others can add millions.

Beyond total API call limits, there are also some other quotas for API calls:

Bulk API calls have 24-hour quotas as well, and there are limits to how big the batches can be.

Streaming API, on the other hand, has limits as to how many topics, subscribers, and events there can be.

Platform Events have their limits as well.

The Risks of Unused or Unnecessary Applications

Every API call made by third-party applications with credentials consumes your Salesforce 24-hour quotas for API Calls. A best practice is to monitor and be aware of your Salesforce environment's capacity. Adding more app connections and usage can push you toward exceeding these limits, leading to disruptions.

Salesforce Admins should monitor API limits and track which apps are consuming quotas (see the Table below to do this efficiently with Valo). Disconnecting unnecessary or unwanted applications is crucial for preserving quotas and mitigating the risks outlined in my prior blog.

Apps can perform mass updates or many applications can perform a lot of operations at the same time and hence it’s important to monitor API usage. Moreover, all applications have bugs.

One "loop-bug" can eat APIs within a minute, before anyone realizes anything. This can happen with applications that even the Salesforce team is unaware of.

Just as with confidentiality, the availability of your Salesforce org is jeopardized by every connected application. Even seemingly unused SaaS services can make regular API calls with a valid token.

Regular audits and cleanup of your Salesforce landscape are crucial to mitigate unnecessary risks and reduce API load.

What to Do When API Limits Are Exhausted

If you ever end up in a situation where your API limits are exhausted, you can do the following:

API Usage in Setup > System Overview

1. Verify that API limits are the cause of the disruption in Salesforce Setup > System Overview.
2. Immediately contact Salesforce support and request a temporary quota extension. This will provide time to investigate and resolve the root cause.
3. Investigate which applications are causing the excessive load. Use the "API Usage last 7 days" report in Salesforce Classic > Community > Reports > Administrative Reports to identify high-usage users. While the “Client Id” field can hint at the app in question, nowadays most apps seem to leave this optional field blank, so this is not always indicative. Next, check the users’ OAuth Apps and Login History in Setup > Users. If you have Event Monitoring enabled, use the logs for more detailed analysis (though this can be complex without tooling).
   
4. Disable the offending application by revoking its OAuth token or freeze the user account if the issue can't be traced to a single app or if the app uses username-password authentication. Alternatively, remove the user’s "API Enabled" permission by assigning a profile without that system permission.
   
5. If no clear culprit apps exist, or you cannot shut them down, disable less critical apps that consume meaningful API call amounts.
   
6. Optimize in-house app API usage: If internal applications are causing high API consumption, review and optimize their API usage patterns. Consider:
   • Reducing unnecessary API calls.
   • Using Bulk API or Composite REST API.
   • Implementing Change Data Capture or Platform Events instead of frequent polling with REST APIs.
7. Purchase additional API calls: If optimization and disabling apps are insufficient, contact Salesforce to purchase additional API call allocations. Maintain a buffer between peak daily usage and your purchased limit to prevent future disruptions.

How Valo Can Help

Valo provides several features to address and prevent API limit issues, ensuring the smooth operation of your Salesforce integrations:

Problem:

Unidentified or unexpected new applications connecting to your Salesforce org.

Valo’s solution:

Valo sends notifications whenever new applications are authorized to connect to your organization, providing immediate visibility.

Problem:

Knowing when an application is not in use anymore.

Valo’s solution:

Valo alerts you to applications that haven't been used, allowing you to revoke unnecessary authorizations, freeing up API quota and reducing risks.

Problem:

Difficulty in assessing the true value and impact of connected apps.

Valo’s solution:

Valo offers application insights, object usage data, and reputation information, enabling informed decisions about which apps are essential and which can be safely disconnected.

Problem:

Lack of granular API usage tracking.

Valo’s solution:

Valo tracks API usage both per application, per user, per geography, and overall, providing detailed visibility into consumption patterns.

Problem:

Unexpected spikes in API usage.

Valo’s solution:

Valo sends notifications for anomalous API activity, such as sudden usage increases, allowing for quick investigation and response.

Problem:

Managing and revoking OAuth tokens efficiently.

Valo’s solution:

Valo provides easy tools for revoking OAuth tokens and cleaning up obsolete tokens, simplifying the process of managing application authorizations.

Problem:

Navigating complex Salesforce data and logs to find the needle in the haystack.

Valo’s solution:

Valo includes an AI agent to assist with navigating data and performing analysis, making it easier to identify and troubleshoot API issues.

Availability as a Monitored Attribute

Optimized Salesforce operations have much to do with availability, and together with other security attributes, this is important to monitor and manage. Intelligence and efficiency solutions like Valo can make this work less tedious and as importantly, bring your attention when needed before significant disruption occurs.

About Mika

Mika Ståhlberg is the CTO and Co-founder of Valo, bringing 25 years of experience in cybersecurity and R&D to the role. Previously, Mika led security initiatives at Ouraring and served as CTO at F-Secure. He is a Salesforce Certified Integration Architect with extensive experience in leadership and hands-on technology work. Mika is passionate about building usable products that effectively solve customer problems.

Outside of work, Mika enjoys snowboarding, music, and exploring history.

• 

  Mika Stahlberg