Is faster than Amazon CloudFront Why

TYPO3 with: Amazon S3 and CloudFront CDN for static content

Everything was better in the past, one would think. With regard to the loading times of a website, this saying does not apply at all, because meanwhile a website not only includes a clean output of the code and compatibility with different devices, no, loading times are also becoming more and more important.

In the past (and unfortunately still today with many pages) you had to wait five, ten or even twenty seconds for the content of your desire to show up, nowadays nobody wants to wait that long for a website. The operators of the major search providers have also recognized this fact and are penalizing websites that take too long to load.

To go into the optimization of a website and to show what is possible with TYPO3, I already have an article on the topic TYPO3 and Google Page Speed written. This article is now about the extension of the optimization from back then. Especially for those website operators whose visitors come from abroad, the content delivery network topic becomes interesting very quickly. Because if a site is built up quickly in Germany, for example, that does not mean that it is also overseas. And this is exactly where the strengths of a content delivery network come into play, because the data is usually not only stored on a server that is located somewhere, but on many servers around the world. Each visitor then receives the data from their nearest server. It should be clear to everyone that an image ends up on an American PC faster if it is also sent from an American instead of a German server.

Another advantage of using a content delivery network is the reduction in the number of requests that are sent to the server. When a domain is called, a request must be made for each file that is to be received. Unfortunately, we don't have an infinite number of requests available in the browser, but only very few. However, since parallel requests distributed over several domains are not a problem, it pays off to move files such as CSS, JS and images to another domain. While the actual website is called up at www.example.com, images, CSS and JS files are loaded from cdn.example.com. The browser no longer has to carry out all requests on a domain, but can scan several domains in parallel until all data has been loaded. In addition, the CDNs are very fast and offer a high bandwidth for downloading. Unfortunately, good content delivery networks are not available for free.

Here is an overview of which points I would like to work through in the article

  • Small concept and goal
  • Preparatory actions
  • Setup of the AWS S3 Bucket
  • Set up the CloudFront CDN
  • Setup of the S3 / CDN domains
  • Extension of the CDN with a certificate (free of charge from AWS)
  • Creating the access credentials
  • Setup and configuration of the TYPO3
    • Procedure in the existing project, restrictions and strange things
    • Outsourcing of CSS, JS, images etc.
  • Extension of the Grunt deployment for CSS / JS etc.
  • CacheControl and Expires for data in the S3 bucket
  • Clear the CloudFront cache
  • Reaching the PageSpeed ​​100/100 again with CDN
  • Related Links

 

Small concept and goal

In my article I will save all files, i.e. the CSS, the JavaScript, the fonts, the icons, images of the layout and also images of the content in an Amazon S3 bucket. I use this storage space as Origin (source) for CloudFront to make the data available in the worldwide CDN. TYPO3 is configured in such a way that the editor can save data directly in the Amazon S3 bucket and then select it again in content elements. In theory, there would be no need for a fileadmin / folder with this constellation, as everything is stored in the Amazon S3 bucket (including thumbnails generated by TYPO3). Once the configuration has been completed, the server on which TYPO3 is running will only execute TYPO3 and deliver the HTML for the front end. All other static data is fed via the content delivery network.

 

Preparatory actions

In any case, you will need the following things in order to implement the planned concept:

  • A website with TYPO3, preferably already in the current version
  • the TYPO3 extension for AWS S3 access from TYPO3 (available in TER, AWS S3 FAL Driver)
  • Access to your own domain administration in order to be able to create CNAMEs for your own domain
  • an Amazon AWS account (available from Amazon AWS)
  • if you want to deploy into the CDN like me, SHH access your console (must be installed: npm, grunt, grunt-contrib-concat, grunt-contrib-uglify, grunt-contrib-cssmin, grunt-aws-s3)
  • Coffee (beer and harder things are also allowed)

 

Setup of the AWS S3 Bucket

First you should create a new bucket in the AWS S3 area. You can find the S3 service in the AWS Management Console (https://eu-central-1.console.aws.amazon.com/console/home?nc2=h_m_mc®ion=eu-central-1) in the "Storage & content delivery"with the entry"S3". You can give the bucket a name, but the name will be unique in the entire S3 space. The domain name that you want to use later has proven itself. In addition, you should select the region where the data should be saved. On best in your area, as this can be decisive for the speed in the TYPO3 backend later:

 

Create the bucket and open the properties, select the "Permissions" section. For this bucket you should set the permissions to "Everyone" with the property "List" to activated:

 

With the button "Edit bucket policy"You have to save another policy at this point:

{
"Version": "2012-10-17",
"Id": "Policy234234251123",
"Statement": [
        {
"Sid": "Stmt12313423123",
"Effect": "Allow",
"Principal": "*",
"Action": [
"s3: GetObjectVersionAcl",
"s3: GetObjectVersionTorrent",
"s3: GetObject",
"s3: GetObjectAcl",
"s3: GetObjectTorrent",
"s3: GetObjectVersion"
            ],
"Resource": "arn: aws: s3 ::: static.example.com/*"
        }
    ]
}

 

With this policy, all files in this bucket are made public. This is important so that new data uploaded via TYPO3 can be read from the CDN later.

Important NOTE: If you plan to operate your site with the HTTPS protocol, it would of course not be bad to connect the CDN later via HTTPS in order to avoid warnings or blocked content that only comes via HTTP. In this case you should use the button "Edit CORS Configuration"store the following policy:

 

 

 

 

As you can see, I am already setting the Access-Control-Allow-Origin for my domain in the AWS S3 Bucket. Later, other HTTP headers are also set directly here in the AWS S3 bucket and simply passed on to the CDN. This has the advantage that I can, for example, set the cache headers per file in S3 without having to do anything in the CDN. The CDN will later take over all set HTTP headers. But more on that later.

You should now select the area "Static website hosting"Open in the properties. You have to select the option"Enable website hosting"activate, and when"Index Document"for example index.html and at"Error Document"For example, enter error.html. The other areas in the properties of this bucket are of no interest at the moment.

 

Set up the CloudFront CDN

An empty bucket is now available to you. Now you can create a content delivery network that uses your S3 bucket. You can find this in your AWS Management Console under the entry "CloudFront" under "Storage & content delivery"in the AWS Services. Create a new distribution here with the button"Create Distribution". You will be asked what you want to create a distribution. That would then be" Web ":

 

 

In the next step, you will be faced with some setting options. In the end it should look something like this, an explanation of the important points can be found under the picture:

  • Origin Domain Name: Here you can enter the origin you have created, i.e. the AWS S3 bucket. It is important here that the full endpoint is specified (there is also a selection where you can select buckets, provided that everything has been done correctly above)
  • Viewer Protocol Policy: Important if you plan to use SSL. I have this setting on "HTTPS Only". In principle, only HTTPS requests can be made to this CDN
  • Forward headers: Very important! Change that to "Whitelist" and add the three existing entries in the right-hand column below. With this setting, all HTTP headers that should belong to a file are sent to the browser through the CDN. So if we later get an expires date for a file the CDN delivers this setting from the S3 bucket in exactly the same way, which is very important for PageSpeed ​​in the further process.
  • Compress Objects Automatically: Set this setting to "Yes". This option is also very important for Google PageSpeed, but otherwise very useful. The CDN recognizes certain file types (css, js, xml etc) and automatically compresses them with gzip, provided that the request from the browser includes "Accept-Encoding: gzip". If this is not the case, the file will not be compressed by the CDN either.
  • Price Class: If you have worldwide visitors and want to have a good performance everywhere, you use all "Edges", otherwise an option of your choice.
  • SSL Certificate: Even if you plan to use SSL, leave it on default for the time being. I'll come back to that later.

You have now made all settings accordingly and can create the distribution. You will then find them in the overview of the CloudFront distributions.

 

Setting up the S3 / CDN domains

You now have a bucket in which you can store the data and you have created a CloudFront distribution. You can reach your bucket at static.example.com.s3.amazonaws.com and in the list of your CloudFront distributions you can see the domain name of your content delivery network: gsreshd272hsdad.cloudfront.net. Your files can be accessed later via these two domains. Not very tingly, is it?

For this reason, we will now create a CNAME entry for the bucket and for the CDN. This then results in the following:

That can be seen. I hosted my domains with df.eu. To do this, I have to go to the domain settings, there in the name server settings and create a CNAME entry for my domain. This is different with every provider, with some there is a period at the end, with some not:

 

Test the bucket and CDN

After you have created the CNAME for the S3 bucket and the CDN, you should soon see at least one error (AccessError or NotFound) with both subdomains. You can also upload a file (example.png) using the AWS S3 console. You should then be able to access this file via the following URL's:

 

You can also check the whole thing with curl on the console, then you will immediately see the headers:

 

 

 

The same for the CDN:

 

 

 

The entry X-Cache when calling Curl for the first time would be interesting here: "Miss from cloudfront", which means that the file was not yet in the CDN and is then stored with this first request. With the second request it looks like this:

 

 

 

 

Setting up the CDN with a certificate (free of charge from AWS)

So far so good. If you do not plan to use SSL and the CDN via SSL (which I would recommend), you can skip this point.

If your website runs over SSL, it of course makes sense to also run the remaining files that are then fetched via the CDN when the page is loaded via SSL. There are in your AWS Management Console under the point "Security & Identity" the entry "Certificate Manager". This can be used to create and manage certificates. In the Certificate Manager, click on" Request a certificate, and you will see the following input option:

 

Enter your future CDN domain here and click on "Review and request". In the following you click on "Confirm and request". Amazon will send you several approval emails in which you identify yourself as the domain owner and confirm the certificate. The emails go to the technical contact person who is stored in your domain. Additionally to hostmaster @, postmaster @, administrator @, admin @ and webmaster @. So make sure that at least one email address can be received.

After the email has been received and you have clicked the "I approve" button, the status in the Certificate Manager should change to "Issued". In the AWS Console, you can go back to the overview of the CloudFront distributions, click on your created distribution and under the tab "general"the button"Edit"Click to edit the CDN. In the"SSL Certificate"you set up"Custom SSL Certificate"um. Here you can select the created and confirmed certificate. You also get a new option,"Custom SSL client support":

 

As already said, the certificate is free of charge. And because it doesn't cost anything, it can't be as much as a paid one. You don't get your own IP address for your CDN. As a result, the client must support SNI for the certificate to work. You can find out which browsers support SNI in the list on the Server Name Indication page at wikipedia. In short, the right browser and operating system can do it. Combinations such as Windows XP or IE <7, however, do not work.

If you save the distribution, it will take a few minutes for it to be active. You should then be able to load the file via https://cdn.example.com/beispiel.png.

 

 

Creating the access credentials

So that you can write data from TYPO3 and possibly also with Grunt into your AWS S3 bucket, you need an access key and an access secret key. In the AWS Management Console you will find your name at the top right, if you click on it there is the entry "in the drop-down"Security credentials". On the new page, in the segment"Access Keys (Access Key ID and Secret Access Key)"Create access keys, but no more than two. You should write down the ID and the key right away, as the secret access key can no longer be viewed later.

 

Setup and configuration of the TYPO3

Finally - it's TYPO3's turn. Install the Amazon S3 FAL driver (CDN) Extension of different and very. This is currently approved for TYPO3 6.2 LTS and TYPO3 7 LTS. The extension has no further dependencies and integrates a FAL driver into TYPO3, with which data can be managed in an AWS S3 bucket. The extension already includes the complete AWS PHP SDK and provides (almost) all functions. Unfortunately, there are a few minor problems and limitations that I will go into later.

After the extension is installed, you create a new data record of the type File Storage on the top level. In the tab "Generally"Enter a name (for example Amazon S3) and choose under the tab"configuration"the type"Amazon S3". You can configure the storage as follows (explanation below the picture):

 

These entries and selections are important in the Driver Configuration area:

  • Amazon S3: Bucket: This is where the name of your bucket comes in (not the endpoint!)
  • Amazon S3: Region: Here you select the region for which you have created the bucket.
  • Amazon S3: Key: Your Access Key ID
  • Amazon S3: Secret Key: Your Secret Access Key
  • Amazon S3: Public base URL: Here you enter the CDN subdomain of your CloudFront. This is very important because this URL will also appear in the source text of your TYPO3 afterwards, if you store images and files. This ensures that the data is always transferred from the CDN.
  • Cache header: max age: This entry is also important. I have saved a period of one month here. This is the length of time a file is stored in the CDN before it is retrieved from the AWS S3 bucket. More on that later.

 

When you have set everything so far, you can save this configuration. If you now go to the file list module in your TYPO3, you will also see the Amazon S3 storage in addition to the fileadmin / directory:

They can use this storage just like the fileadmin. Create folders, upload and delete files. Above all, you can later select these files from the storage in all content elements in order to use them. The speed is really good, I haven't noticed any major delays in contrast to the local fileadmin.

 

Procedure in the existing project, restrictions and strange things

New project: If you start a fresh project, everything is fine, you just have to always use the Amazon S3 storage so that all data is in the CDN. Of course, you can also use the fileadmin storage in parallel to provide data from the local server.

Existing project: The whole thing gets a bit trickier.With the integration of FAL in TYPO3, each file is internally assigned a UID and the UID of the storage. I am not aware of a simple change from fileadmin to Amazon S3. This means that you first have to copy all the images that are maintained into the Amazon S3 storage and then reassign them. If you have not always uploaded the data cleanly in Fileadmin first, but have selected direct upload in the element, this can sometimes be stressful, as all files are then in the / user_upload / folder.

Copy restrictions: As of now it was not possible for me to copy complete folders from Fileadmin to the Amazon S3 Storage via the filelist. The backend pummels around because of the lack of "Write Permission". Probably a bug in the storage driver.

Move restrictions: see a point above

Curiosities: Multiple Upload: As soon as I upload several images in the TYPO3 backend at the same time, I get an exception "_proccessed_ / x / x /" is not a resource "when I update the file list. There also seems to be a bug when converting the thumbnails in the backend The temporary folders are not available, but the named folder will be created the next time the file list is updated.

Odd: Thumbs are missing in the backend: If the thumbnails are not displayed in the file list, it could be because the path to the converted image is incorrect. I had the problem that after uploading files there were always too many slashes in the path (/_proccessed_//a//1/beispiel.png) I couldn't find a solution, so I fixed the FAL driver myself:

In the file

typo3conf / ext / aus_driver_amazon_s3 / Classes / Driver / AmazonS3Driver.php

I changed the return in the getPublicUrl () method at line 145:

This one

 

 

becomes this:

 

 

 

Outsourcing of CSS, JS, images etc.

Now that your editors are distributing images and data in the CDN, you should also move your CSS files, your JavaScript, the images for the template, fonts and all the other stuff requested by the browser to the CDN. You can either create a new bucket and a new CDN with new subdomains to further distribute the requests and reduce round trips, or you can use the existing one and save all data there. You can create a directory structure in the AWS S3 Bucket like you use it in your TYPO3 theme. It is best to divide the directories as follows:

  • / files / css
  • / files / fonts
  • / files / images
  • / files / icons
  • / files / js

 

Copy all your static data there and integrate them (css and js resources) accordingly via TypoScript as an external resource with the full path. You have to make sure that nothing is concatenated or compressed locally, since the files are then integrated locally again. In any case, the source code should say something like this:

<link rel="stylesheet" type="text/css" href="https://cdn.example.com/files/css/main.min.css" media="all">

<script src="https://cdn.example.com/files/js/main.min.js" type="text/javascript"></script>

If images that are used in the CSS are correctly referenced there, they are also loaded from the CDN.

Images that you need for the template and use in TypoScript or in FLUITEMPLATE should of course also be referenced externally via the CDN.

If your CSS or JS data change, the upload process of the files has to be repeated every time. If you want to reduce your requests and automatically upload the data to the S3, I have in the article "Pagespeed Insights 100/100 for TYPO3"showed how data can be properly concatenated and compressed with grunt. All JS and CSS includes are reset

 

 

and the data generated with grunt are integrated in the footer as a complete HTML tag:

 

 

In the next point I extend grunt with the function of having this data automatically placed in the CDN.

 

Extension of the Grunt deployment for CSS / JS etc.

In the article "Pagespeed Insights 100/100 for TYPO3"I have already shown a complete configuration of how to cleanly concatenate and compress JS and CSS files (important for PageSpeed ​​100/100). If you are already running this or a similar configuration, you can also use grunt wonderfully to convert the Store template images, CSS files, JS files and other resources in your AWS S3 bucket.

Building on my previous article, I am now expanding the Gruntfile.js by uploading the data.

First I install the grunt-aws-s3 module:

manfred $: npm install grunt-aws-s3
manfred $:

Then I create a JSON file that stores my AWS Key ID and my AWS Secret Key:

manfred $: touch ~ / aws-s3.json
manfred $:

It is best to put this file in the home directory. Not in the DocRoot or in a directory managed by GIT that may go public.

The access credentials are placed in this file:

 

 

 

 

You can now adapt the Gruntfile.js accordingly, I'll show you my complete configuration again. Explanation after the code:

 

 

 

  • grunt.initConfig.concat: Concatenate all files, CSS as well as JS. Important to minimize requests!
  • grunt.initConfig.uglify: Compress JS file
  • grunt.initConfig.cssmin: Minimize CSS file
  • grunt.initConfig.aws: The file with the keys for AWS S3 is read here
  • grunt.initConfig.aws_s3: The upload to the bucket is set here. It is important that gzip is deactivated, as the CDN will automatically take care of the compression. As you can see, when you run grunt, everything that belongs to the template is automatically uploaded to the bucket. The point params.CacheControl and params.Expires in the respective files is very important: {} Nodes. You will find out why in the next point.

 

CacheControl and Expires for data in the bucket

How the CDN with the S3 bucket is structured should now be clear to you. To sum it up, AWS S3 serves you as file storage, CloudFront as CDN, whereby CloudFront in this constellation fetches the data from the S3 bucket. So if you do not set an Expire or a CacheControl, the resource is loaded from the S3 bucket every time a request is made. The S3 bucket is billed according to requests. Well, is the doorbell ringing? :-) So make sure that the static content can always be cached sufficiently by the CDN, so you can reduce the number of requests to S3 and keep the costs there low. As a new AWS user, you will receive 50GB of data transfer and 2,000,000 requests every month free of charge for the first year. You can find an overview of the pricing as a link at the end of the article.

 

Clear the CloudFront cache

After the data is stored in the CDN with the corresponding header when it is accessed via the CDN domain, it remains there. So it may well be possible that you have to remove the data from the CloudFront in order to get the current version from the AWS S3 bucket. You can do this quite easily by switching to the list of your CloudFront distributions in the AWS Management Console. Click on the ID of your CloudFront distribution and open the "Invalidations" tab. There is a button there "Create Invalidation". A popup opens with an input field. Here you can specifically specify what should be removed from the CDN. Some examples are already displayed. If you want to remove everything, it is sufficient to enter a single asterisk *. By clicking on"Invalidate"The CDN is emptied and the data is requested again from the S3 bucket on the next hit. The process can take a few minutes.

 

Reaching the PageSpeed ​​100/100 again with the CDN

Well, I had imagined this topic to be more exciting. If you have followed everything so far (Compress activated in CDN, CacheControl and Expires set in FAL and in grunt), the PageSpeed ​​should still be 100/100 :-)

 

Related Links

Here again a list of many useful and required links