This is the second in a series of posts about the 8 Fallacies of Distributed Computing, a set of common misconceptions or false assumptions that are often made when building distributed systems. Keeping these fallacies in mind when building your cloud services, as well as your front-end clients can drastically improve your user experience and your developer experience.

You can read the rest of the series here

When a user or client interacts with our distributed system we often want them to feel as though they are interacting with it in “real-time”. When we’re developing applications locally on our machines, or running in the cloud from our fast office network (possibly even with a VPN connection) it often feels like everything is responding instantaneously. Then when we deploy the application we find things are not as speedy as we originally thought. Users complain about slow loading times, about videos taking ages to download, or constant timeouts trying to upload large files. Sometimes this is caused by building systems that don’t scale, or that haven’t been scaled correctly, but other times even though your system is running fine the user experience can still be terrible.

Every external request that your system makes adds latency. How much latency is added is dependant on the type of request. Making a request to disk is certainly faster than making a request to a web server, but if you have lots of requests to disk they will still add up. Fortunately for us engineers, the way humans perceive the world is not in “real-time”. In fact, it takes time for our brains to process the world around us. In 1968, Robert Miller published his paper on Response time in man-computer conversational transactions and found three orders of magnitude when a user is interacting with a system.

1. If the system responds within 100ms, they believe it’s instantaneous
2. If the system responds in 1 second or less, the user typically feels they are interacting with the system freely
3. If the system takes more than 10 seconds to respond, the users attention is completely lost

More recently, a study by Microsoft in 2015 determined that the average globally attention space has decreased from 12 seconds to 8 seconds. 8 seconds seems like a long time, but if we remember that the network is not reliable all it takes is for a couple of failures, and subsequent retries, for a user to lose interest.

So with the clock ticking on our request, how do we ensure our users stay focussed?

Finding Bottlenecks

The first step in decreasing latency is to identify the bottlenecks. Every system is different, so you need real data to identify where the issues are, and which ones are the highest priority. When building our systems, we need to make sure they are observable using the Three Pillars of Observability (Metrics, Logs, and Traces). With a Distributed Tracing System, such as OpenTracing or AWS X-Ray, a subset of requests are “sampled” showing the latency as that request moved through the system. By being able to observe where the bottlenecks are we can then get around to fixing them.

Caching

One of the simplest and easiest ways to reduce latency is through caching. Caching can occur at multiple levels, but can introduce it’s own challenges. One of the key things to remember with caching is determining how long data can be cached for, and how to invalidate that data in the cache prematurely if you need to. Most caching systems make use of a Time to Live (TTL) to define how long to store that data, but sometimes things change so we need to be able to delete items from the cache.

The first thing you should cache is all shared assets that are distributed to your users. Images, videos, static HTML/CSS/JS, or any other asset that stays the same for every user download. This level of caching can be done in a Content Delivery Network (CDN) such as Amazon CloudFront. A CDN works by hosting the content closer to where your users are. That way, instead of having to go back to your datacenter to retrieve the data, the system will only need to go to the nearest Point of Presence (PoP), which is typically a much shorter and faster journey.

The next layer to consider is your dynamically generated content. Does every user receive unique, personalised content, or is it grouped by some sort of cohort, such as the subscription the user has or the location they are in? Do your users request the same data several times? When setting up your CDN you define a cache key, a set of parameters in the request that make the request unique, which allows you to cache dynamic content. Again, this allows the data to sit in the CDN and results in a much lower latency when fetching it.

After we’ve cached everything we can at the edge, we start to look internally at requests to see if we can reduce the latency there. Calls to a database or to another service can be costly, as we’re not only introducing network latency, but we’re also introducing latency as the database or other system processes our request. Again, we need to look at these requests and determine if they are dynamically generated, if the response changes quickly between requests, or if we can use stale data. If we’re working in a payment system and we need to know the users balance, then we need the latest data and shouldn’t be using caching, but if the data we’re retrieving only changes once per day then it’s a prime candidate.

To cache internal calls we have two options. The first is to introduce a component purely to cache data, such as Redis or ElastiCache. Keep in mind that while this will undoubtedly reduce latency, it still requires a network call which can fail, or the data may not be in the cache. It also requires you to build a way of populating the cache which can be complicated.

Another option is to introduce Memoization. This is a technique available in many programming languages where the service builds an internal cache of results for function/parameter combinations. Essentially, this means that when a function in your code is called it will first check to see if that function has been called with those arguments before and, if it hasn’t, it will run the function and store the result in a local cache. The next time that function is called with those arguments the cached response is returned. The downside here is that the cache takes up memory in your service, and each service will have it’s own cache, but it’s much simpler to implement. Just remember that you can’t and shouldn’t memoize everything. Sometimes, a cache lookup is more expensive than running the function, but if your function makes external calls or does heavy processing it may be worth looking in to.

Event Driven Architecture

When building a distributed system we can reduce latency by reducing the number of hops required to fulfil a request. This means we need to identify what requests need to happen synchronously, such as ensuring a card payment goes through, or asynchronously, like informing the warehouse they need to package and ship the order. By only making the requests that are required, we’re only introducing latency that is required. If a request can be done asynchronously then instead of making a call directly to another service simply fire off an event and allow that other system to pick it up.

Server Performance

Keep an eye on your servers. Through the observability of the system you should have metrics that let you know how your servers are performing. Ensure that they are scaled correctly, that they can scale both horizontally and vertically, and that they allow for hardware failures and network outages.

Client UX

Lastly, when looking at how latency affects our users, it’s important to remember that it’s all about reducing the time they perceive for the system to respond. This can be difficult, especially if your users are on their phones or tablets on a train with spotty internet. A technique called Optimistic UI can be used to make the user perceive the response as being instantaneous. When a user interacts with your UI the UI optimistically updates itself, assuming that the request to the server will succeed. Then, in the background, the request is sent to the server. If the request does succeed then great, your user has a lightning fast experience and are none the wiser. If the system fails after multiple retries, let the user know, and allow them to try again. Other techniques include pre-emptive loading of assets, using animations to hide request times and page loads, and using contextual elements to show that the user interaction is being processed.

Conclusion

Latency is inevitable, but we can manage that latency through a number of techniques. Ultimately, it’s not just about reducing how long a call takes, but making sure the user feels like the system is responsive. Remember, you have one second before the user starts to get annoyed, so make the most of it.

This is the first in a series of posts about the 8 Fallacies of Distributed Computing, a set of common misconceptions or false assumptions that are often made when building distributed systems. Keeping these fallacies in mind when building your cloud services, as well as your front-end clients can drastically improve your user experience and your developer experience. You can read the rest of the series here

With today’s modern network infrastructure it’s often easy to forget that the Internet, and all of our local networks, are a Best-effort delivery service. The IPv4 protocol used to get datagrams from point A to point B can suffer packet loss, arbitrary delays, corruption, or duplication of packets. TCP builds on top of this and goes some way to solve these issues, but requires trade-offs in latency to achieve. Even with the safeguards like retransmission, slow-start, etc, there is no guarantee that hardware failure won’t result in a failed requests. This issue is even more apparent when working with mobile devices, where a user may be travelling on a train with spotty reception, or working on WiFi with dozens of other clients all competing for bandwidth.

A prime example of this was in December 2021 when Amazon Web Services suffered multiple network device failures simultaneously causing not only their own services to be affected, but also caused issues for Netflix, Disney+, and hundreds of other companies around the world. Similar issues with major providers like AWS, CloudFlare, and GCP have caused outages across the Internet in the past.

Whenever you’re working with code that makes a call to an external system, whether it’s a call to disk, a device running on your local network, another server running within your cloud environment, or your CDN, it’s important to remember one thing; it’s not a question of if a request to an external system will fail, but a question of when, and how often.

So how do we protect against these issues? Where do we start?

Dealing with Failures in Requests

When working with an external service there’s some simple patterns you can use to try and recover from failure when they occur

1. Use Retries, with Exponential Backoff and Jitter to deal with 5XX responses (4XX errors are client errors, and retrying is unlikely to fix them)

2. Wrap every external call (including calls to disk) in a try/catch, and have a plan for dealing with the failure within your app beyond printing an error log

By ensuring that the calls you make to these external systems are able to be caught and retried transient errors within the system can be overcome without requiring the user to know anything ever went wrong (except for a slightly longer response time). By adding Exponential Backoff and Jitter we can prevent errors across all of your clients causing your server to be overwhelmed. One thing to keep in mind with the retries is ensuring an upper limit, so it’s not just trying indefinitely while your user waits.

User Experience for Failures

When working with any client that makes calls to an external resource, be it network or disk, it’s important to design the interactions with failure in mind. It’s not just about making sure you add try/catches around all your external calls (although, again, you definitely should!) but also thinking about your User Experience for Failures.

Above is a fairly good example of a User Experience for Failure. It provides a visually on-brand response that not only explains an action that the user can take (try again in a few minutes) but also gives them the option of using Google Search instead. I’d argue that telling the user a database error occurred is irrelevant, it’s not like the user is going to fix it and the engineers should have metrics and monitoring to tell them something went wrong, but it still provides something more than the default browser error page.

Even before you show the user a failure screen though, there are other ways to ensure a good user experience. If you can, make the request non-blocking i.e. allow the user to continue using your app and then use a push notification with a pop-up to tell them the request succeeded or failed, and if it failed give them a link to go back and sort out the issue.

Architecting for Success

Typically, a lot of these problems can be avoided by looking at your Cloud architecture. One of the simplest ways to reduce the chance of a request failing is to reduce the number of network calls being made by your cloud platform. In practice this generally means preventing daisy-chained requests (i.e. service A calls service B calls service C) within your sytem. When a service is unable to satisfy a request without making requests to other systems it means there is a hard link between those services, one can’t work without the other, meaning that what you have isn’t a decoupled set of microservices, it’s a distributed monolith that brings in the worst of both worlds.

This can’t always be avoided, but architecting an eventually consistent system with data replication greatly reduces the number of requests required to complete a request. This has the trade-off of requiring data duplication (which should be fine, as storage is cheap) but also introduces the much more challenging problem of ensuring your data is correct (especially if you require consistent reads/writes). If you’re finding it difficult to reduce the number of requests you make then it may be time to consider the boundaries of your service.

Conclusion

When working with any software that makes connections over a network of any type, it’s important to remember that at some point those calls will fail, and to build not only your cloud architure, but your clients and user experience to cope with those inevitable failures.