programming

Google App Engine limitations, and how to get around them

Standard

Using Google App Engine has great advantages, but there are also serious limitations in the platform that it’s important to be aware of. Not all applications can be implemented and executed with Google App Engine. Google is working hard on removing a lot of these limitations, and they will probably do so eventually, but in the meantime that isn’t really any help if you are writing a web app today.

This post was originally published in swedish on Mashup.se, my blog about swedish mashups and APIs.

Only Python
If you don’t know Python you don’t have choice if you want to use Google App Engine, you just have to learn. It’s not a difficult programming language to learn if you already know how to program. Django, the Python framework that really speeds up writing applications for Google App Engine, is also quite easy to pick up (tip: Use the Google App Engine helper for Django). Before I started with Google App Engine I hadn’t written a single line pf Python and after a few intense days of concentration and coffee consumption I knew Python, Django and Google App Engine quite well.

Most 3rd party python libraries work perfectly on Google App Engine (Django is just one example), but there are limitations. Only libraries that are 100% Python can be used, so if the library has any code in C it can’t be used on Google App Engine. If a librarry has any code that makes a HTTP request or similar it can’t be used on Google App Engine either. All HTTP requests have to be done via Google App Engines URL Fetch API.

Differences between local dev environment and the production environment
One of the advantages with Google App Engine is that there is a local development server that simulates how Google App Engine works when an application is in production. This makes it easy to develop an application locally and then deploy it to the live servers on Google App Engine. As a developer it is important to pay attention to the differences between executing an application on the local server compared to the production environment. It’s no fun to spend time writing code that just don’t work once deployed.

The biggest difference betweent the two environments are that the requests made to 3rd parties work differently. If you have an application that uses the Delicous API it will work fine locally, but once deployed it wont work at all. The reason is that Delicous is blocking all requests from Google App Engine IP-addresses. Same thing is true for the Twitter API due to some HTTP headers that Google App Engine sets (Twitter claims to have fixed this now, haven’t had a chance to test yet). To avoid these problems you need to test your application often in the live environment, especially when you use APIs to call other services.

Datastore
The Google App Engine datastore has some limitations due to being a distributed database. The most obvious limitation is that there isn’t an “OR” operator in GQL (the Datastore version of SQL), but that is easily handled when coding. A more annoying limitations is that it is not possible to create a new entity (data object) in the Datastore via the Google App Engine dashboard unless there already exists an entity of this type, and that there is a real noticable delay between what one can see in the Datastorer in the Dashboard compared to what really is stored in the Datastore. This makes it very hard to really check what data that is stored in the Datastore in any given moment, which makes debugging more difficult.

No scheduled processes
One of the most restrictive limitations with Google App Engine is that there is no way to start a process, other than via a HTTP request. There is no type of recurring scheduled process (like a cron job) and no triggers or hooks to use to start a process when a special event occures. Almost all web applications need som kind of scheduled process to function correctly – to clean up old data, send emails, consolidate statistics or fetch data from an RSS feed once an hour.

The easiest way to get around this limitation is to create a cron job on another server that calls an URL in the Google App Engine application. If you have a lot of visitors you can also perform the background process as part of a user request. For example you could import data from an RSS feed when a user logs in to your application. This will of course make the user experience slow and there is no guarantee that users will perform the action to trigger the background process att the right times.

No matter which solution is implemented one will quickly bump into the next limitation of Google App Engine, that only short processes are permitted.

Only short processes
Something I have learned the hard way is that Google App Engine only is built to handle applications where a users makes a request and quickly gets an answer back. There is no support for a process that  executes for a longer time, and for the time being this limitations seems to be approximatly 9 secondes/process. After that an exception is thrown and the process killed. It does not end there, even if you are nowhere close to use the assigned CPU resources you can quickly use too many resources with long running processes since they have their own unique resource pool. If you use too many resources you application is shut down for 24 hours, and right now there is no way to buy extra resources.

This is a really serious limitation in Google App Engine that it is really difficult to get around. If you need heavy processes it is recommended that you use Amazon EC2 or something similar. To handle (not get around) the limitation you have to handle exceptions in a nice way and use transactions. More about this in William Vambenepe’s very informative post Emulating a long-running process (and a scheduler) in Google App Engine. He has some tips on how to get around this limitation, even if it is not recommended since you risk that your application is shut down.

More limitations
There are more limitations, read  Google App Engine: The good, the bad, and the ugly? for a longer list. Just keep in mind that some of the limitations mentioned in that post already have been addressed by Google.

To summerize you need to really know what limitations there are in Google App Engine before you spend time and energy on developing an application. If the limitations are not a problem then there is a lot to gain by using Google App Engine.

Why I fell for CodeIgniter

Standard

Since leaving Kapow a week and a half ago I have been coding more than I have for the last 3 years combined, and I have done it all in PHP using the framework CodeIgniter. I looked at quite a few frameworks, CakePHP and Ruby on Rails for example, and quite a few applications/blogging platforms/Content Management Systems that can be hacked and adapted, wordpress and Drupal for example. In the end CodeIgniter won the day, and so far I am extremenly happy with my choice. There are a handfull reasons why I prefer CodeIgniter:

  • It is PHP, this might be a dealbreaker for some, but for me that is a huge plus. Primarily the advantages of this is that I can easily host my creations on basically any cheap web host, that there are plenty of libraries and resources out there to make my life easier and also that I know PHP. For the last reason I could have gone with Ruby on Rails or Java as well, but it put all Python frameworks out of the competition.
  • CodeIgniter is very easy to install and as easy to deploy. All you need for things to play nicely is an Apache server, a MySQL database and a copy of the CodeIgniter files. Deploying and setting up things are the most boring thing when it comes to writing your own apps, so it is a must for me that it is a breeze, I just dont have the patience to deal with deployment problems.
  • Great documentation, the CodeIgniter user guide is excellent. This is a huge difference from many other frameworks and platforms, especially the ones developed by an open source community (CodeIgniter is developed by Ellis Labs, the guys behind the Expression Engine blogging platform). The developer community is also very active and knowledgable, so what isn’t in the user guide is in the CodeIgniter forums.
  • Finally, a framework that improves my productivity. Most frameworks tries to do to much and are so huge and rigid that there is a huge learning curve if you want to doing anything but a “Hello World” app. CodeIgniter helps me with the stuff I need help with and doesn’t meddle in the rest. There is no need to hack 10 plugins of different qualities together to get what I need (like in Drupal), and that just makes developing fun as it should be. It is also the first MVC (Model-View-Controller) framework that helps me organise my code in a good way, something that I usually suck at otherwise.

In short I recommend that anybody that knows PHP and want a light weight, good framework checks out CodeIgniter. A good place to start are the CodeIgniter video tutorials, and if you get a bit deeper into things Elliot Haughin has a great blog that often covers CodeIgniter and he also have some great libraries that are well worth looking at (CodeIgniter libs for Twitter, Flickr and Akismet for example). Another great resource is the blog of Derek Allard, Technology Architect at Ellis Labs.

The Evolution of Mashup Development – From Hacking to Assembling

Standard

The current trends are more that business users gets more abilities to solve their own problems and for programmers to give the business users the tools to do so. There are some tools like this already that are very popular (see Explaining Enterprise Mashups), but we have only seen the start of this development. These trends will change the roles for both programmers and business users in fundamental ways and it is interesting to see how that could play out. This is one likely scenario of what will happen during the next couple of years.

Yesterday
Programmers were building applications and integrations by programming. There were very little finished modules to reuse and combine, so a lot of logic had to be written from scratch (most talk about code reuse is nothing but talk). This was not only Mainframes and Cobol, but also a lot of Java, C++ on Windows and Solaris etc.

Now
Programmers building applications using ready made modules. These modules might be full applications, libraries or, in some cases, widgets. This means that less logic needs to be written from scratch (Apache Jakarta projects have changed the lifes of most Java programmers for example).

In 1-3 years (early adopter geeks today)
Programmers and early adopter business users are assembling solutions using ready made components. These solutions are called mashups today, but are probably not going to have that name in a few years, instead the “mashup style” of solution development are going to have become mainstream and be a natural part of applications (just see how mashups are built with Facebook Applications today). The components are widgets that have been developed by programmers to function as building blocks in bigger solutions.

In 3+ years
Business users are assembling solutions, programmers are building widgets.

From Hacking to Assembling
Of course there will always be programmers that are programming low level solutions like kernels and compilers (at least those are low level to me) and business users that just want their problems solved without them having to deal with any technology. But overall the focus for programmers will go from building the solution to building the building blocks (aka widgets). For business users the focus will go from waiting for a program that solves their problem to be developed to being able to assemble a solution (aka mashup) that will address their problem. Overall the focus will move from hacking to assembling.

Great Tools for Business Users Needed
This means that business users need to be more aware of what building blocks are available. Currently there are a few early attempts to develop systems to handle this – IBM’s Mashup Hub is a good example. Furthermore there is a need for easy to use systems for business users to combine all those widgets. This is probably the hottest area in mashups right now, with everyone wanting to be in on this – BEA Pages, IBM QEDWiki, PageFlakes and iGoogle to just mention a few. But so far it has been with very limited success and just a little more than proofs of concepts or cool AJAX playgrounds. Let’s see what will happen in 2008, I for one if looking forward to seeing new cool innovative solutions to these problems.

What I learned at JavaOne 2007

Standard

I am just back from a week in San Francisco at JavaOne where I was one of 12.000 geeks that got together to get to know more about the latest and greatest in Java development. Since it was almost 2 years ago I worked full time with Java development this almost felt like a trip back in time for me, back to the time when Java was equal to my professional life, and what I have worked with the last few years (Web 2.0, mashups, web scraping etc) is still considered new and exotic in the Java world.

It was also very interesting for me to compare JavaOne with the Web 2.0 Expo that I attended a few weeks ago. JavaOne is almost strictly for developers that are using a mature technology that has not really taken any real leaps lately. The Web 2.0 Expo was for both techies and business people and it looking forward at new technologies and business oppertunities.

Keynotes and General Sessions
Most of the Keynotes and the General Sessions were also about running Java on all kinds of devices (cell phones, ATMs etc) and generally about how great Java is, so nothing new there. But there were a few things that caught my eye:

  • JavaFX Script – Sun has decided to take up the fight with Microsoft’s Silverlight and Adobe’s Flash, unfourtunatly they decided to give it a name that most people will confuse with “JavaScript” and Adobe’s “Flex” (just try saying JavaFX Script 10 times quickly). This is a very interesting move since it suddenly makes it easy for the whole Java community to develop Rich Internet Applications. This and Silverlight has the potential to make the web a much more interesting place.
  • Netbeans 6 – There was a quite impressive demo of Netbeans 6 and how it can be used for programming Ruby on Rails, much better than RadRails I am using now. Of course it was a demo and I havent had time to test the Netbeans 6 preview yet, but as soon as I have to dig down into Rails again Netbeans is my choice.
  • Blu-ray – There was some semi-desperat plees (=competitions) to get Java developers involved in the Blu-ray vs HD-DVD fight. Sun is squarly behind Blu-ray (they mentioned that it ran Java about 100 times). The Blu-ray demos were cool, but personally I just want one format to win quickly so I know what player to buy.

Java and Web 2.0
Most of the presentations were of course about hardcore Java stuff, and I skipped those. Instead I went to all presentations about Mashups, RSS, Atom and REST (acctually I held a presentation about Mashups myself, more about that in a later post). It is pretty clear that all the Web 2.0 technologies are viewed as some distant hype by most of the Java community.

The only really cool thing I saw in regards to Java and Mashups was a couple of demos of jMaki. It is a project developed by Sun and it is basically a framework where java developers can easily program Mashups. The great thing was what is called the “Glue” which is an event bus that enables widgets from different providers like Yahoo and Dojo. jMaki has a great future if it ever moves into the Enterprise world and it could be a real step forward for both Java and Web 2.0.

Another interesting Sun research project is Project Caroline that enables java developers to control all resouces from the code, ie create new server instances on the fly, set up new file systems etc etc. If this ever moves beyond just being a half implemented research project it could open up to a lot of competition for Amazons S3 and EC2.