April $232023$ is World Book Day! I would like to introduce you to System Design Interview $$ An insider's guide by Alex Xu$.$ This is a great book for every software engineer who wants to get started with system design.

I decided to write a blog post about this book because I learned the basics and the real$-$world application through this book. I would recommend this to everyone who is interested in getting started with system architecture.

The author, Alex Xu$,$ is making these two chapters open to the public on his website, so please check out his articles as well if you are interested!

In this blog, I would like to introduce you to the system design of YouTube I learned from the book $+$ my own research to dig a little deeper into it$!$ Among all the variety of system design examples in the book, I chose to write about Youtube because I am a big fan of the platform! Furthermore, Youtube system architecture is a great example to go over the basics of system design.

In this blog, we will focus on two features of the Youtube platform:

Uploading a video

Watching a video

Although System Design Interview $$ An insider's guide is to help you prepare for a system design interview, I put emphasis on system design concepts, not the interview prep part.

INDEX

Uploading a Video

What happens when you upload a video on Youtube?

The flow of an actual video

Object storage vs databases

Speeding up the upload to object storage$($S$3)$

The flow of metadata of the video provided by the user

Watching a Video

CDN$($Content Delivery Network$)$

Load Balancer$($LB$)$

Database Replication

Cache

Replicating across geographical locations

How do we recover quickly when the data center fails?

Reference

Uploading a Video

What happens when you upload a video on Youtube?

There are majorly two flows on video upload:

the flow of an actual video

the flow of metadata of the video provided by the user

The flow of an actual video

You have a video file, puppy.mp$4,$ and you want to upload the video to the Youtube platform so that many people can watch your puppy video.

You upload the video through Youtube, and the uploaded video is sent to object storage. An example of object storage is AWS S$3.$ In fact, Youtube does not use S$3($S$3$ is Amazon$$s product; Youtube is under Google$/$alphabet$),$ and Google has its own distributed object storage. I am more familiar with AWS, so I will stick with using S$3$ as an example.

Object storage vs databases

Why do we store the video in object storage$($like S$3),$ why not in databases$($MySQL$,$ MongoDB, etc...$)?$ It is commonly known that storing an object file, such as videos or images, is not the best practice for many reasons$$slower queries, not scalable, etc.. $($for more detail, check out this resource$)$

Speeding up the upload to object storage$($S$3)$

AWS provides multipart upload and transfer acceleration service that helps developers increase object upload performance. Using multipart upload, upload speed increases by splitting a large video into smaller chunks and sending it to the object storage separately. Transfer Acceleration helps us provide a consistent network experience and lower latency. Youtube is part of Google, so in reality, they have their own ways to achieve this performance increase, but I believe this concept applies similarly. If you are interested in learning more about this. There is an amazing article for more information.

Now that Puppy.mp$4$ is stored in the object storage, the video is sent to a transcoding server. The primary purpose of the transcoding server is to convert a video format to other formats. This makes it easier for Youtube to provide a better video$-$watching service for users.

In brief sentences, some of the reasons why video transcoding is done:

reducing a large$-$size video to a smaller size

encode a video format$($e$.$g$.$ mp$4)$ into many other formats

multiple resolution types

$($In this blog, we are going to focus on something other than what video transcoding is and how it works since we want to focus on the breadth of architecture.$)$

After the video transcoding, the transcoded video data is sent to another object storage$($e$.$g$.$ S$3)$ and saved.

As soon as the video transcoding is done, the transcoding server sends a request to a server$$completion server that handles the metadata of transcoded video and every post$-$processing required after the transcoding. This completion server then saves the metadata to a database.

Here, we will choose MySQL database to store the metadata. In reality, Youtube uses a database called Vitess. It is created to solve its own unique scalability challenges.

NoSQL database has its own advantages as well. Especially, when we deal with a large number of videos, the efficient way to access video details, such as title and metadata, is by providing centralized access; In Video table, for example, each video detail is associated with its unique key. write summery