Kickstarter logo - green

Kickstarter helps bring creative projects to life. The platform connects creators with backers who help fund and support projects related to the arts, media, product design, and more. Since 2009, nearly five hundred thousand creators have launched projects on Kickstarter, uploading numerous images and videos to communicate their vision. Fastly enables the Kickstarter infrastructure team to optimize the way that their platform serves content to third-parties, resulting in significant cost savings.
Industry: Financial services
Location: Brooklyn, NY
Customer since: 2013

Favorite features
Custom VCL
Real-time logging
DataDog integration

A few lines of code helps Kickstarter reduce infrastructure costs by 25%

In 2013, Kickstarter adopted Fastly’s edge cloud network to streamline content delivery to the global community, which provided a fast and secure browsing experience for backers and reliability for content creators.

Social media bots spark a surge in traffic
On any given day, Kickstarter servers are optimized to handle a normal traffic pattern with predictable peaks and lulls. Occasionally, big project launches cause huge spikes in demand that can put significant pressure on infrastructure resources.

For creators, starting a Kickstarter campaign is like announcing their plan to launch a product. Many creators have thousands of followers, so they’ll naturally run ads on social media sites to help spread the word. As the ads roll out and people flock to the site, Kickstarter servers get hammered with content requests. Facebook, Twitter, Instagram, and other social media bots all need to scrape public-facing assets from Kickstarter, such as a project’s title and description, to serve their ads across the globe. Because these services are architected as a distributed system, Kickstarter can be hit by tens of thousands of edge computers at once. This could happen at any time of day or night due to the global nature of the Kickstarter community.

To effectively support the surge, the Kickstarter team had to set up their infrastructure to always be ready for scale. Scaling up and down dynamically was difficult, so the team often had expensive resources running that would sit idle during traffic lulls.

"Fastly’s flexibility and the customizability of VCL enabled us to optimize the way that the platform serves content to third-parties and site visitors, resulting in a 30% reduction in requests and 25% reduction in origin costs." Brian Knight
Engineering Manager for Infrastructure

A dedicated page for crawlers cuts traffic by over 33%

When one project in particular caused excessive strain on the platform, the infrastructure team began digging further into the issue. It turned out that the social media bots needed only a few assets from each project page, yet they were causing the entire page to load just to scrape those assets. The team came up with an innovative idea: what if they created a dummy page, just for crawlers?

With the help of custom VCL and the flexibility of the Fastly platform, the team developed a solution that spins up a dedicated crawler page on the back-end for every project. The simplified page is easy to cache and includes text strings, such as project title, description, and image URL, that are embedded in the HTML and use the Open Graph protocol. This allows the social media bots to quickly grab them without forcing the full, public-facing project page to load.

To serve the crawler page correctly, the team queries the social media sites daily to pull their IP addresses (which change frequently) into an access control list (ACL) in Fastly. Using a simple rule and three lines of VCL, Fastly identifies, and then redirects, social media bots to the dedicated crawler page. The impact to the Kickstarter system was immediate.

Real-time data from the edge provides insight into usage trends

Fast-forward to 2019 when the infrastructure team began looking into ways that they could reduce expenses. As they monitored performance data, such as cache hit ratio and trends over time, the team recognized how they might increase their efficiency using Fastly’s real-time log streaming from the edge and an integration with Datadog.

As the Kickstarter business grew over time, their cost to serve video skyrocketed. The team identified a pattern they’d have to innovate to overcome: anime pirates. Some creators who had no intention of launching a project were trying to use Kickstarter to serve videos, often multiple terabytes of anime video for fans in Japan — essentially for free — at Kickstarter’s expense. Fortunately, they were able to put a stop to that with just a few lines of code.

“Fastly’s flexibility and VCL enabled us to build a simple solution that immediately cut our server load by over a third. Our huge, unpredictable spikes have become flat lines.” Brian Knight
Engineering Manager for Infrastructure

Custom code serves video to the right users

Kickstarter’s team wrote a short snippet of custom code using Varnish Configuration Language (VCL) that enabled token validation on the video URL.

Now, when the Kickstarter site serves a project page, it generates a URL for the video on the back-end with an encoded timestamp that’s set to expire in fifteen minutes. When a request comes in to the server to load a video, Fastly checks the URL’s timestamp is valid and if the link is less than fifteen minutes old, Kickstarter assumes that it was triggered by a legitimate visitor, as opposed to an abuser who is trying to serve unauthorized video. The Fastly solution resulted in a dramatic reduction in the cost of serving video.

Dramatically reduced overall infrastructure costs

Kickstarter used to continuously run 60+ application servers and six databases. With their new approach to serving video and social media assets, the team now runs only 40 application servers and three databases — representing a whopping 25% reduction in AWS origin costs. The company no longer faces excessive costs for unused capacity and has gained better control over their infrastructure expenses. Fastly’s flexibility and customization at the edge has been a big part of this result.

This is great news for the business, and also great news for the Kickstarter community, so that everyone enjoys a fast, smooth experience on every visit. Kickstarter is better prepared to handle the success of their creators as creators spread the word about their ventures across their network and beyond.

“Only ten lines of VCL code resulted in a tremendous cost savings for us. We moved all our video over to Fastly and the solution paid for itself within the first couple of months.” Brian Knight
Engineering Manager for Infrastructure