Five SRE Best Practices for the Modern Enterprise

While it is certainly possible to implement monitoring capabilities just prior to launch, if you are looking to meet the minimum requirements for observability, it won’t be cheap to move that fast. Still, dollar for dollar, spending on monitoring has some of the greatest ROI when it comes to ensuring you can meet performance and availability goals long-term – to deprioritize them simply because it may be expensive at this point in product development can be a short-sighted mistake. 

On the other hand, if you had considered monitoring requirements at the very beginning of a requirements gathering session with the site reliability engineering team, you may have had more time to implement custom metrics in your code, and use an open-source framework. You would have had the ability to set up a capability that is good and cost-effective, but it requires more investment in your team’s time – something you don’t have if you’ve waited until you’re about to deploy to production. Here’s where engaging with the site reliability engineering team early would have helped.

Product management and site reliability engineering may work on different teams, but when it comes to application goals and performance, they need to work hand-in-hand. It is critical to engage SREs in this process before a single line of code is written because they:  

• Will be responsible for services that contribute to application reliability and performance in production.
• Are concerned with multiple aspects of a service, including architecture and dependencies, metrics and monitoring, incident response, capacity planning, and more. 
• Will prescribe the tools that the organization will use to monitor and measure against goals. 

SREs can forge a solid working relationship with product managers early, and streamline the collaboration process, by providing a production readiness checklist that offers product managers guidance for what they need to think about, including: 

Once the checklist is complete, SREs will know what to monitor and observe to establish the performance thresholds that need to be met. 

When it comes to best practices, what’s old is new again, which is why, as we set out to build and deliver capabilities that meet high-performance goals, the scientific method should guide all of our decision-making. It’s critical that we can form a hypothesis based on our current observations, as it relates to our desired outcome or state, test that hypothesis with an experiment, and then observe the results of that experiment to form our next hypothesis for continuous improvement of our systems and capabilities.
The first step in this process? Take an observation as it relates to the current behavior of your application, or as it’s come to be known, observability. By ensuring we have the ability to observe the inner workings of our systems, we can ensure we are able to make effective decisions and form hypotheses that guide roadmaps and operationalization of our capabilities. But how do we go about observing? 

There are a few different monitoring strategies that can be leveraged to achieve our goals: 

• Instrumenting your code directly, leveraging metrics and timers that are available within your given language or framework, commonly referred to as “observability”
• Using a paid Software-as-a-Service Application Performance Monitoring (APM) solution that allows you to easily install a seamless agent
• Using analytics, real user monitoring (RUM) or synthetic transactions to provide insight into the performance of an application from the user’s perspective

Regardless of the strategy, the end goal is the same: To gain visibility into your application from multiple perspectives to provide you with as much information as possible, as near to real-time as possible. Whichever approach you choose, it’s important to ensure that you can fundamentally state, without question, that you know how your application is performing and what your customer experience is.

When choosing which model to follow, however, there are several factors to consider, primary among them being your desired speed-to-market, your budget, and an honest assessment of the maturity of your organization. In project management, this is the typical “good, cheap, fast” trade-off – if you need something fast and good, it likely won’t be cheap. And if you want something cheap and good, it will not be fast to implement.

The ability to aggregate visibility is also key as you seek to measure specific variables against goals and understand the health of an application. By building dashboards around the key indicators that inform and alert you about operations and dependencies, you can pinpoint problems and identify the dependencies that are impacted – is it a problem with code, the application or a dependency?

Incident management is a just nice way to say what it really is: Crisis management. Here’s where you want to control the chaos. With good monitoring and observability, you should know with relative confidence what is actually happening with applications and resources. But it doesn’t end there, because you also need to assemble a team to respond to the information and events those tools are generating. Teams need to be able to follow a framework and manage the incident successfully, because if incidents aren’t managed correctly, they will wreak havoc for your applications and environment. 

Early in the maturity timeline it’s common for a team member working on an incident to make a gut decision on which action to take. And it may work. But that doesn’t mean it’s the right action and it doesn’t mean that there wasn’t an action that would have been more specific to fix the problem. The difference between a well-managed incident and an unmanaged incident is the difference between making random – gut-influenced – changes that often result in a combination of making things better and worse, versus coordinated activity that follows the scientific method, resulting in specific action designed to follow or test a hypothesis which results in improving the situation. 

It’s critical to follow the scientific method for everything you do in response to an incident. This will help ensure that you’re making the right decisions to fix the problem by controlling the variables and measuring the results of actions taken. Taking actions based upon a hypothesis of an expected outcome, whether it’s successful or not, helps eliminate potential scenarios and moves the process of repair forward.

Once you are able to observe your systems and begin to form hypotheses, you can confidently begin conducting various experiments to validate your hypotheses and prepare to make changes, as necessary, with your application to improve performance, cost, or reliability.

In order to set up a good performance experiment you want to mimic how your customers are going to use your product, to the best of your ability. This will help set you up for success on 95% of your use-cases and customer interactions. However, it’s important to recognize that when it comes to customers, a familiar military adage perfectly describes usage in production: No battlefield plan survives first contact with the enemy. In other words, no matter how hard you try to predict exactly how a customer will use your site and its features, at the end of the day you can’t anticipate every scenario that will happen in production. We’ll see in the next section how to deal with those outliers, but don’t let that fact discourage you – non-production performance testing and experimentation are still critical.

For several years, “performance testing” has been a common step in the operationalization of an application or platform. However, far too often, it is seen as a necessary evil. Teams conduct a test without a goal in mind other than ensuring the application stays up past a certain point. On the other hand, more mature organizations form a specific prediction and hypothesis around how a configuration or application change might affect performance, then test that hypothesis with those specific goals in mind.

When determining how often to conduct performance experimentation, it’s critical to think about what the hypothesis is at a high level. If you consider that any change to a production environment has the potential to introduce risk or cause a performance degradation, then ideally you should conduct at least a basic performance test as part of every release. Integrating performance testing into your CI/CD pipeline as part of each change to your environment helps maintain confidence in your current configuration. Alternatively, if you make a change to the configuration itself, a performance test is also recommended in order to validate the change had its desired effect, when possible.

Experimentation is integral to validating hypotheses and making changes to improve application performance, cost, or reliability. Unfortunately, relying on humans alone to perform experiments isn’t the solution – you can’t put enough human resources toward experimentation to make a real difference. Fortunately, as we’ll see next, modern machine learning and automation capabilities can help.  

Typically, teams configure an environment to hit a target of 99.9% availability based on the number of requests coming in at any given moment. In a Kubernetes environment, this often means sacrificing costs to achieve the level of performance you seek by requesting significantly more CPU or memory than is needed. There is a better alternative, however, and that’s where optimization comes into play.

To start optimizing we need to look once again to experimentation using performance testing – and the tweak and run cycle. Even the best enterprises are only running performance tests once per week – and barely any do even that. At that rate it would take you four years to run 208 performance tests and make 208 configuration changes on a single configuration as you test your application. This is pretty simple math – you can’t optimize at that rate. There aren’t enough bodies or budget to get the job done. 

This is where automation and machine learning come in. Automation can help speed up the tweak and run cycle, running years worth of performance tests over the course of just a few hours, tweaking the configuration with each successive run to see how the application performs and what resource utilization results. Machine learning then provides the ability to analyze all those results and recommend configurations to try next in order to home in on the optimal configuration.

Using experimentation to optimize before deployment is valuable because you can try any scenario you can dream up, and iterate as many times as you’d like. But it’s important to also consider optimization during production because things change and, as we saw in the last section, it isn’t possible to account for every eventuality.

Because we are monitoring and observing our production environment (see Best Practice #2), we can also apply machine learning to that data to tweak and tune our configurations in production. Machine learning can look at actual usage trends to predict and recommend resource settings that will keep our environment running smoothly, minimizing resource usage while still ensuring performance that meets expectations.

By introducing new levels of automation and machine learning in both pre-production and production, teams can efficiently and effectively optimize applications and resources, and meet performance, reliability, and cost goals.