When the big boys run websites (basically any brand name site you can think off) they use Content Delivery Networks (CDN’s) to host most of their content, especially images, stylesheets, files to download and other static content. The reason they do this is that the less they have to host themselves the less load they have on their servers, and the more content they can host closer to the end user the quicker the user can download it. The most famous CDN is probably Akamai, that almost run their own parallell internet. Akamai and other CDN providers cost big bucks though, so it is nothing for us mere mortals.
But thanks to Google anyone can now run their own CDN for free on Googles servers. It is really easy to set up and storing files for downloads, stylesheets etc on Google instead of on your own site takes the load of your servers (and consumes much less bandwidth of your hosting account) and speeds things up for the end users. It’s a win win situation, and it is also really really cool! If you are interested in more information about how to get the maximum performance from your web site then I recommend you read the excellent post Performance on a Shoe String on 24ways.
What is Google App Engine?
Using Google App Engine you can run web applications on Google’s servers. That means that you can benefit from Google’s huge world-wide server farms, it means that it is really easy to scale and to integrate with othe Google applications (for example using Google authentication in your applications). At the moment you have to write your applications in Python (don’t worry: no coding at all needed to use Google App Engine as a CDN, just keep reading), but hopefully they will expand it to other languages soon (personally I want to run PHP and CodeIgniter on Google App Engine!). The App Engine is Googles response to Amazons very successfull web services S3 (for storage) and EC2 (for computing). Amazons services are very powerful, but they do require a deeper level of technical knowledge to use than Google App Engine.
Currently Google App Engine is in a Preview Release (= beta), but it is free for anyone to join, all you need is a Google Account and a cell phone (more about this later). What you get is 500MB of free storage and around 5 million free pageviews a month, if you use more than that there is a small cost (see the Google App Engine blog for more details). The cost for these extra resources are almost the same as for Amazons Web Services, and with the freebies and ease of use thrown in Google App Engine is a bargin.
How to set up your own CDN
To use Google App Engine as your own personal CDN you need to install some things on your computer and edit a few configuration files. All this work is a one time thing though, after that all you need to do is run a simple program to upload new files to Google. Sorry to say that the scripts you download is for Windows only, if you are on a Mac or using Linux then you need to make your on script to do what deploy_digitalistic_cdn.bat does (if you do so please add this to the comments of this post for any one else to use).
- Since Google App Engine only works with the programming language Python you need to download and install Python on your computer. If you have a Mac or run Linux you most probably already have Python installed, so you can skip this step. Download the correct installation file for your OS from Python 2.5.2 from http://www.python.org/download/ and install it. Use the default settings, except install it under “Program Files” instead of directly on the C: drive (or install it wherever you want, but in then you need to modify the scripts below.
Download the Google App Engine SDK from http://code.google.com/appengine/downloads.html and install it. During the SDK installation it will check if you have Python or not, so if you have a Python installation problem you will know it already here. The Google App Engine SDK is needed to be able to write and upload applications to Google. Just use the default settings when installing the SDK.- Sign up for Google App Engine at appengine.google.com. For this you need a Google account (your GMail address for example, if you dont have one it is free to create one).
Once you are signed up you need to create an application, so just click on the button “Create an Application” and give your application a name (called “application identifier”). This name needs to be unique among all users applications, so it might take a while to find a unique one. In my case I used “digitalisticcdn”. Save your new application. After you have created your first application you need to specify your cell phone number. Google will then send you a SMS with a code that you enter into their site. This confirms that you are the owner of this Google App Engine account (so don’t use it for spamming ;).- Download the file http://digitalisticcdn.appspot.com/files/digitalisticcdn.zip (hosted on my private CDN!) and unzip it to your harddrive. If you want you can rename the unzipped directory from “digitalisticcdn” to whatevery you want, for example the name of your own application. It doesnt really matter, it just makes easier for you to keep track of things in the future.
- Use a text editor to edit the app.yaml file in the digitalisticcdn directory. Change “application: digitalisticcdn” to “application: <your application identifier” and save the file. This will tell Google App Engine what application to upload your files to.
- Now it is time to add all the images, stylesheets, files, videos etc you want to upload to Google to the folders in the digitalisticcdn directory. Put all images into the /images folder etc. You can create any number of subfolders inside the images, files, stylesheets etc folders (for example /images/webhostninja.com/ninja.gif). You can always add more files at a later time, so if you just want to set things up to work you can skip this step for now. There is already an image in the /images folder for you to test that all is working as it should be.
- Download http://digitalisticcdn.appspot.com/files/deploy_digitalistic_cdn.bat and edit it in a text editor. This file needs to point out your Python installation, your Google App Engine installation and your digitalisticcdn directory. If you installed the Google App Engine SDK in the default directory and Python in C:/Program Files/ then you don’t have to worry about those settings. Just change the last part of the file to point to your digitalisticcdn directory. Keep in mind that all paths with spaces in needs to be surrounded by quotes.
- Double click on the newly changed deploy_digitalistic_cdn.bat file to upload all the files in the digitalisticcdn directory to Google. The first time you do this you need to specify your Google username and password.
You now have your own private CDN! Go to <your-application-identifier>.appspot.com/images/ninja.gif (in my case digitalisticcdn.appspot.com/images/ninja.gif) to see that it works.
How to use your private CDN
To use the files you upload to your Google App Engine CDN you just need to use the URL to the file on your site. If you want to show an image of a cool ninja from WebHostNinja.com you would just use digitalisticcdn.appspot.com/images/ninja.gif as your image source in your HTML. The same goes for stylesheets, files to download or whatever else you want to share on your CDN.
At any time you can add new files to your digitalisticcdn subdirectories (/images, /stylesheets etc) and run the deploy_digitalistic_cdn.bat file to upload them to Google. If you remove files from your digitalisticcdn directory and then run the bat file they will be deleted from your Google application.
You can check the statistics of your Google Application at appengine.google.com. For example you can see how much bandwidth and disk space you are using. It will take quite a lot of files and usage for you to use up the resources you get for free, but if you have a super popular site then it is worth taking a look here every now and then.
If you have read so far and found all this usefull then please Digg it. Thanks!
php i all ära men codeigniter? 🙂
I alla fall, prova på lite python! Det är riktigt trevligt språk, många likheter med php och django är ett grymt ramverk för att göra webb.
Please change the url behind appengine.google.com (dellete http:).
Andreas: Håller på med ett större projekt med Python och Django på Google App Engine just nu, så jag provar Python för fullt. Föredrar fortfarande PHP och CodeIgniter, men det är ju en smaksak.
Niek: thanks for catching my typo, it’s fixed
Andreas,
This is a great article for those wanting to use the Google App Engine as a quasi CDN. I would not recommend using App Engine as a CDN as it is not an edge based solution providing downloads from servers closer to the user. The analytical reporting on objects stored in App Engine is also very poor and leaves much to be desired.
Anything free always has its drawbacks but for content that is not crucial to you business it will work fine. I would recommend many of the month-to-month CDN’s that are out on the market for content that is critical to your business.
Daniella Newmark
castmetrix.com
This solution is definitely gaining momentum, it will be very interesting to see if global companies start to use storage clouds for faster distribution. The performance of enterprise content systems like Documentum is very poor, they need to cache content all over the web. Google, Amazon, Nervanix are all close to what people are looking for.
http://www.ContentWorkspace.com/solutions/google.htm
I think its a good way to get banned by google from every using any of their services. Did you read the eula? your probably not allowed to do this. Also could you change your font. I can baely read anything on your pages.
Joel: What in Googles EULA states that I can not store non-pornographic content that I own the copyrights to and do not use for spamming on Google App Engine? I have read the TOS, Privacy Notice and Program Policy and as far as I can tell this should be OK. I would think that right now Google should be happy about anybody using and promoting Google App Engine, they are in for a hard fight with Amazon, Microsoft etc about this space.
I don’t think it’s a good candidate for CDN, not so much because of the EULA, but because it’s really meant for applications, and not for storage, (like S3 *is*, I’m betting google has something under their hat that *will* do that) and as such has a hard limit of 1000 separate files of maximum 1MB size. (The 1000 files is actually already a problem when you want to make use of exisiting python libraries.)
Eric & ContentWorkSpace: Agreed, Google App Engine is not the _perfect_ CDN. Amazon S3 is a much more flexible and powerful solution when it comes to storage, and dedicated CDNs are of course much more powerful. But Google App Engine is free, easy to use even for people that are not programmers and a quick fix.
I wouldn’t suggest using Google App Engine as a CDN for a big enterprise project, but for any webmaster/blogger sharing big files for downloads etc I think Google App Engine is a good solution. Using it as a CDN is also an easy way to get started with Google App Engine.
Nice idea.
But I would suggest, a “deploy_digitalistic_cdn.py” instead of the “deploy_digitalistic_cdn.bat” on the article, just for cross-platformness’ sake.
Newbie question maybe, but how do I remove files I no longer need?
Hello,
Nice blog ! This is a great article on AppEngine. I wanna see more articles like this in the future.
Janos: to remove files form your CDN just remove the files in question from your local digitalisticcdn directory (or whatever you named it) and do a new upload. That deletes the files from your Google App Engine account as well.
Edmar: Thanks 🙂 Hoping to soon follow this post with a real example of how to use Google App Engine. Keep your eyes open, especially if you are a Ning fan 🙂
Oh. That was kind of lame. Thanks. Both for the excellent article and the clarification.
A quick question. I am using the cdn app to store my css bat bash-script etc on app engine. Now could you tell me how i could retrieve the all the files that i uploaded to my app and store them onto a new pc.
Sort of syncing data via google app engine.
Regards
Hareem.
PS: If this sounds totally stupid. Please do not get angry. I am just a newbie.
Hareem I feel very very angry with you.
Fantastic walk through! Looks like I have something to keep me occupied this rainy evening. 🙂
I came up with this post from a suggestion given in http://vagabundia.blogspot.com about Google App, and found the explanations clear and easy to understand, even for people that do not speak English as a mother language. Really, thank you very much. I will visit this site often from now on. 🙂
Same as Jam, I came from Vagabundia’s post and agree is a good explanation and a very good idea, I just started to use it.
Many thanks Andreas 😉
PD: how can i change (only a little) the start page of my new “appspot” site? (of course thinkout remove your credits)
opps XD
.. think is the ‘index.html’ file included on your digitalisticcdn.zip file, didn’t it?
Yes Solrac your right, the startpage is all in the index.html file (and the CSS file it is importing). Just change what you need and reupload it to your Google App Engine site and your should be all done.
I setup the app engine on google and see the url in Manage Start Page but if I click it or enter it I get a page cannot be displayed. My Bat file seems to work pretty fast so I’m not sure I’m transfering anything either.
@Len If you try running the command in the bat file from a command promt you can see if it uploads something or not, and if there is a problem you will (hopefully) also see a helpfull error message.
Excellent walkthrough Andreas. I am a newbie and have a related problem. I have built a website on Google Sites and am hosting it currently at Dreamhost.com. I would love to host it on the Google App engine so that I can eliminate hosting fees. Is there anyway I can do this? I am already done with steps 1-4 of your walkthrough.
Thanks
Rajesh
@Rajesh Without having looked into the details of this I suspect that you will have problems just moving a Google Site over to Google App Engine. If you can export the Google Site as HTML then you can just upload those HTML files to GAE, otherwise I think you might be stuck where you are.
Hey people. Check out my blog it now has CDN by google app engine. Thanks google and thanks Andreas !
Hi,
I don’t have the app_engine folder under c:\program files\google\ but there are other Google apps there.
Is there a new place to download this from?
Ah.. N00b!
There’s a .MSI AND a .zip that you need.
*skulks away with a red face.
Well, Mac has a GUI for that 😛
I mentioned your article in my blog post about YSlow optimization.
Hey there,
Now that Java is also running on Google App Engine, would that be possible to see such an implementation of a CDN, in Java, on GAE?
@Zackatoustra It shouldn’t be a problem to do the CDN on GAE in Java, I will put it on my list of todos. I have also thoughts on some other pratical uses for GAE that I might post about soon 🙂
This is fantastic, thank you!
Thanks guys that was extremely helpful!
sandeep verma
(http://sandeepverma.wordpress.com)
Have you tried using drydrop for this?
@John nope, havent tried Drydrop, but it sounds really interesting. I will definetly try it next time I need to upload stuff to GAE. Thanks for the tip!
Nice Article.
How to make the app engine cookie free and catchable?
Thanks.
Just what I was looking for. Your instructions worked perfectly. I’m seeing a speed up of a minimum of 100ms over my current host on images. It’s also nice to see that google takes care of cache settings, etc.
Thanks!
It works!
Two small questions though:
1) How do I set expiration for the content to be somewhere well in the future (it’s all static content and want to maximize the cache time)
2) How do I modify the app.yaml file to add a single crossdomain.xml file to the root dir. (where index.html) is located. I currently modified my app.yaml file like this:
– url: /.*
static_files: crossdomain.xml
upload: crossdomain.xml
because it was the only way I could add the file to the root directory but would like to include the index.html file also as an error page. How can I do this?
Kind regards,
Wim
@Win or anyone lookiking for similar thing for content expiration and HTTP 304 status. please see my post on this.
http://vangel.3ezy.com/archives/67-Google-CDN-for-my-website,-304-status-and-improving-performance,-and-a-DDOS-attack-of-HALOOP.html
Thanks Andreas, this was a very helpful post. You saved my lazy ass . heh.