It’s a well-known fact that most users choose weak passwords. Choosing a strong password is actually much harder than it might seem at first. Huge wordlists constructed from previously leaked password databases are readily availably online and we consistently pick passwords which are at or near the top of those lists.

In a worst-case (but increasingly common) scenario, where the actual database of hashed passwords is stolen, a half-decent cracking program will be able to spit out these weak passwords almost instantly, leading to large numbers of accounts being compromised. This is especially true for simple unsalted password hashes, which companies have been found to be using in some of the recent scandals, despite it being acknowledged as very poor practice for many years. Cloud Foundry stores passwords using bcrypt, which is deliberately designed to hinder password-cracking, but even so, weak passwords are still vulnerable.

So what can be done to help people choose stronger passwords which are not easily cracked? Some sites have complicated and annoying password policies which force you to include a combination of numbers, symbols and different-case letters in your password, despite the fact that this usually just makes your password hard to remember without actually making it stronger (as depicted in the famous XKCD “correct horse battery staple” cartoon).

Most users are familiar with the use of password-strength meters. In Cloud Foundry we plan to use a meter not just as a guideline for choosing a good password, but also to define the system policy. So rather than saying “you must satisfy these X annoying rules”, we simply say “you must get this score or higher on the meter”.

Obviously, the question then arises as to what makes a good “score”. The code we use is based on the calculations from Dan Wheeler’s zxcvbn project (https://github.com/lowe/zxcvbn), which uses a combination of common password dictionaries and checking for patterns such as dates, keyboard sequences (hence the project name) and other typical “bad” password choices. The dropbox tech blog about the project gives an excellent overview of the algorithms used and the rationale for choosing this approach. It’s not perfect – finding a weak password with a reasonable score isn’t too difficult, particularly if you consider languages other than English, but it’s definitely a good start and something we hope to improve on in future releases.

The main difference is that while the original project is a browser-based Javascript library, we will be using a server-side implementation both to provide feedback and to actually enforce a particular policy. The port of the zxcvbn library, as well as a sample application (both in Scala), are available on github. The sample application is also running on Cloud Foundry at https://szxcvbn.cloudfoundry.com/password.html. You can either use the ajax-enabled UI or directly submit requests to it and get a JSON response. Here’s an example using curl

$ curl -H "Accept: application/json" -d password=correcthorsebatterystaple https://szxcvbn.cloudfoundry.com/

which returns

{"score":8,"crack_time_s":2.0372004064749978E9,"match_sequence":[{"start":0,"end":6,"token":"correct","dictName":"english","matchedWord":"correct","rank":1525},{"start":7,"end":11,"token":"horse","dictName":"passwords","matchedWord":"horse","rank":494},{"start":12,"end":18,"token":"battery","dictName":"english","matchedWord":"battery","rank":3845},{"start":19,"end":24,"token":"staple","dictName":"english","matchedWord":"staple","rank":14066}],"crack_time":"63.38 years","entropy":45.21165314373938}

In our production implementation, the interface will only return the score plus the server’s required score (to allow the UI to tell the user how close they are to providing a strong enough password). The sample application is more verbose, providing information on how it has analyzed the supplied password string.

This article explains how to use Cloud Foundry APIs from a user application using the built in identity management solution in the User Account and Authentication Service (UAA). The UAA acts (amongst other things) as an OAuth 2.0 Authorization Server, granting access tokens to Client applications for them to use when accessing Resource Servers in the platform, such as the Cloud Controller. This article describes the responsibilities of a Client application and the mechanics of setting one up. It uses a simple example Client application (available on github), and recasts it into various forms to help developers with different language and tool preferences to get to grips with the topic (Ruby, Java, Grails).

Quick Introduction to OAuth2

In a typical OAuth2 scenario (as developed in the sample application) OAuth2 is a protocol with 4 main participants:

1. Client application, often a web application
2. Resource Owner, usually a User
3. Resource Server (another web application or web service)
4. Authorization Server

The User tells an Authorization Server that he trusts the Client to access the Resource Server on his behalf. In OAuth2 terms we are going to see a sample Client application in which the Authorization Server grants a bearer token to the Client using an Authorization Code flow. This scenario is composed of 4 steps:

1. Client redirects User to Authorization Server /oauth/authorize to authorize a token grant
2. Authorization Server authenticates the User and obtains his approval, redirecting back to the Client with a one-time Authorization Code
3. Client contacts the Authorization Server directly at /oauth/token, exchanging the Authorization Code for an Access Token
4. Client presents the Access Token in a request to Resource Server

Responsibilities of a Client Application

A Client application has these responsibilities:

1. Register with the UAA and get a client id and secret
2. Do not collect User credentials – delegate to the Login Server
3. Send a unique state parameter with each request to the /oauth/authorize endpoint, and check it later in a callback
4. Keep the Access Token and any associated Refresh Token a secret
5. Watch out for access denied responses from the Resource Server and take appropriate action
6. Use the Login server if there is one
7. Don’t log the User out of Cloud Foundry unless they explicitly ask for it

Register With the UAA

A Client application must have a pre-existing relationship with the UAA. This is a standard OAuth2 feature, and is necessary to ensure that the platform can control access and give Users peace of mind that there is some control over the access to their data and applications.

When a Client registers with the UAA the owner or developer of the Client provides a redirect URI (or list of them) where it will accept callbacks from the UAA, and in return will get an id and a secret, and a list of scopes that it is allowed to access on behalf of a user. The sample application is registered in the local UAA by default with these properties:

{
  "client_id": "app",
  "client_secret": "appclientsecret",
  "scope": ["cloud_controller.read", 
    "cloud_controller.write",
    "openid", ...],
  "redirect_uri": ["http://localhost"]
}

The scope values allow the client to operate the Cloud Controller (read and write) and to get the user’s profile data (“openid”). These are similar to the scopes granted to vmc in production. Note that if you were to register a client with the UAA on cloudfoundry.com you would expect the scopes would be limited by default (e.g. we would not normally expect Client applications to be able to change user’s passwords).

The redirect URI in the example above is actually not the value you will see in a local UAA by default: there it would be an empty list. An empty redirect URI in the client registration allows a client to request a redirect to any address, so this would not be allowed in production. In production a redirect is required to prevent an attack by a bad guy who might want to steal authorization codes or access tokens (see the earlier CSRF article for details of the attacks).

Client registrations on cloudfoundry.com are available on a case-by-case basis only at the moment, so please contact the Support Team (support@cloudfoundry.com) for help if you would like to try this in the real system. The Identity Team are also working on some features to automate client registration and open it up to more users, so keep an eye out on the cloudfoundry.org and cloudfoundry.com blog sites for news.

Authentication

When a Client registers with the UAA it will get an id and a secret which must be used any time a Client needs to authenticate itself with the platform. Principally this will be at the OAuth2 token endpoint /oauth/token, when exchanging an authorization code for an access token. The token endpoint requires a Client to send the id and secret in a standard HTTP Basic header, e.g.

POST /oauth/token HTTP/1.1
Authorization: Basic YXBwOmFwcGNsaWVudHNlY3JldA==

{
  access_token: FUYGKRWFG.jhdfgair7fylzshjg.o98q47tgh.fljgh,
  expires_in: 43200,
  client_id: app,
  scope: openid,cloud_controller.read
}    

State Parameter

As protection against Cross Site Request Forgery (CSRF), the OAuth2 spec allows for a state parameter to be sent in the authorization request, and if it is there the Authorization Server sends it back in the callback that contains the authorization code. It is the Client’s responsibility to generate a value for the state and to assert that it issued that particular value when it comes back. The value of the state parameter should be unique (e.g. random), and not easily generated by an attacker. The Client can implement the state in one of two ways, either by storing the value in between the two requests as a key in a user session, or by encoding some information in it that can be verified when it sees it again (e.g. a unique id in a special format encrypted with a secret that only the Client knows).

When the callback comes to the Client application with an authorization code in the query parameters, the Client extracts the state parameter and verifies that it generated the value. If it cannot verify this then it should not proceed: an unrecognised state value can be a sign of a CSRF attack and it should be rejected.

In most cases Client applications will use a library to manage the authorization and token grant, and the library will take care of the state for you. It is the browser that actually sends both requests, the first to the Authorization Server, and the second to the Client application, so in principle a user session is available and that is normally the easiest way to implement the state check. Many of the libraries (e.g. omniauth-oauth2 as used in the Ruby sample, and Spring Security OAuth in the Java sample) do this for you. Some do not (e.g. Scribe as used in the Grails sample), in which case you should take care to add that feature to your application; i.e. it is not recommended to use the Grails sample as it is.

Using the Access Token

Once you have a token, your Client application can send it in a request header to protected resources on the platform. E.g. getting the current user’s profile information from the UAA:

GET /userinfo HTTP/1.1
Authorization: Bearer eYFDLKJHLKJ.DYOGFKHDLJDF.uIOFDUF

HTTP/1.1 200 OK
{
  "user_id":"ec5797e2-24dd-44c9-9e92-f03d0ef9d11e",
  "user_name":"marissa",
  "given_name":"Marissa",
  "family_name":"Bloggs",
  "name":"Marissa Bloggs",
  "email":"marissa@test.org"
}

Note that Cloud Foundry, unlike some older OAuth2 providers (e.g. Facebook), in general does not accept an access token as a request or form parameter. You must use the Authorization header, as recommended in the OAuth2 specification.

Responding to Unauthenticated or Access Denied Errors

If your access token is expired or invalid, you should see an HTTP 401 UNAUTHORIZED response from a protected resource. For example:

GET /userinfo HTTP/1.1
Authorization: Bearer GARBAGE

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="UAA/oauth", error="invalid_token", error_description="Invalid access token: GARBAGE"

{
  "error":"invalid_token",
  "error_description":"Invalid access token: "
}

The error response will tell the Client the difference between an expired or invalid token. In either case you can ask for a new one, and if the problem was an expiry then you might also have a refresh token that you can exchange for a new access token without needing to get the user’s approval again (the refresh token would have been granted at the same time as the original access token and it is the Client’s responsibility to hold on to it).

Or if the token has insufficient scope to access the resource you asked for, you should see an HTTP 403 FORBIDDEN. For example:

GET /Users HTTP/1.1
Authorization: Bearer eYFDLKJHLKJ.DYOGFKHDLJDF.uIOFDUF

HTTP/1.1 403 FORBIDDEN
WWW-Authenticate: Bearer error="insufficient_scope", error_description="Insufficient scope for this resource", scope="scim.read"
{
  "error":"insufficient_scope",
  "error_description":"Insufficient scope for this resource","scope":"scim.read"
}

In both cases there was a JSON body, and a WWW-Authenticate header to tell you what went wrong, and a little bit about how you can fix it. The fact that a Bearer token is required is indicated, and in the case of the scope issue, also the required scope that you are missing. Your client or user may not have permission to acquire a token with that scope, in which case there is nothing you can do, but in other cases the information is useful and you may be able to get another token that does what you need.

The example responses above are from UAA resources (i.e. APIs provided by the UAA itself). At the time of writing the 401 and 403 responses from the Cloud Controller may not resemble precisely the examples above, but they should be similar.

Requesting Scope for an Access Token

In the sample applications the tokens are all acquired without specifying a scope explicitly, so they have the default scopes allowed to the Client and User. This is always an option, but in general it is good practice for clients to only ask for tokens with a specific scope that allows them to do as much as they need but no more – e.g. do not ask for cloud_controller.write if you only need cloud_controller.read. This allows the UAA to alert the user specifically to what your application wants to do, and in turn they can make a better decision about whether or not to allow it. A user is more likely to grant permission for a third party application to read data than write it, for instance.

The Login Server

The Login server is a UI for the UAA in cloudfoundry.com whose features, design and implementation will be covered in a separate article. For the purposes of this article it suffices to say that a Client application should us the Login server if it is available for the OAuth2 /oauth/authorize endpoint. This ensures that applications authenticating with Cloud Foundry have a uniform user experience including seamless single sign on (SSO).

Logout

A Client application can have user sessions of its own (the sample implementations all do) and so it makes sense for it to have a logout feature. The Login Server (or UAA if there is none) has a logout endpoint as well (/logout.do), and if a Client application wants to it can provide a link to that endpoint, and specify a redirect to itself so that the User lands back in a familiar place after being logged out (see the sample application code for examples, e.g. /logout.do?redirect=http://myapp.cloudfoundry.com/loggedout). If the User clicks that link then he will clear his session with all Cloud Foundry applications, which might not be something he wanted to do. A Client application can and should present Users with the option to logout of Cloud Foundry, once they have logged out of the local session, but the User should be made aware of what they are doing if they follow that link.

The Sample Client Applications

The sample applications accompanying this article are designed to be as simple as possible – the minimum needed to get an interaction between a user and the Cloud Controller, mediated by a UAA bearer token. More complicated functionality should be easy to add, and there are libraries to help do that which we haven’t used in the samples just to keep them simple.

The roles played in the system are

• Authorization Server = UAA
• Resource Server = Cloud Controller (or a locally deployed fake)
• Client = the sample application

All the implementations of the sample have the same features:

• an authentication filter that only allows valid Cloud Foundry users to use the rest of the application
• a home page that lists the properties of the currently authenticated user
• an /apps page that lists the currently deployed applications of the current user in Cloud Foundry
• a /logout link that clears the local session in the Client itself and then gives the user the option to also logout from Cloud Foundry

The authentication filter is responsible for acquiring an access token from the UAA and storing it in a local session, so it can be re-used without having to ask the UAA on every request (optional, but recommended). The filter also uses the token to build a representation of the User (so we can say “Hello” on the home page). It does this simply by sending it to the /userinfo endpoint on the UAA (a similar feature is available from virtually all OAuth2 providers, e.g. the /me endpoint at Facebook). In a real Client application the user data could be omitted if all you care about is the access token, or it could be used more extensively, e.g. to correlate with a local user database and assign additional, application-specific roles and permissions.

The /apps endpoint also uses the access token. It sends it to the /apps endpoint on the Cloud Controller, gets back a list of the user’s deployed applications, and then renders it. Obviously you could do more than that with the token to build out a richer application, and depending on the runtime or framework you are using there are tools available from Cloud Foundry to help you do that. So, for example, if your application is running on the JVM you can use the Cloud Foundry Java Client to interact with the Cloud Controller in a more convenient way than directly through the JSON endpoints. Similarly, if your application is in Ruby, then there is a Cloud Foundry Ruby Gem that you can use to do similar things.

In all cases you should inject the access token in the appropriate place in the library API and not have to collect user credentials, even though the libraries still support that (for a limited time).

Deploying the Sample

The samples can all be deployed locally for testing purposes, and the UAA instance defaults to one at http://localhost:8080/uaa. They also all default to using a fake Cloud Controller which is the API sample from the UAA repo. The fake Cloud Controller and the UAA can both be launched from the command line using Maven (see instructions in the UAA README):

$ git clone git@github.com:cloudfoundry/uaa.git
$ cd uaa
$ mvn install
$ cd samples/api
$ mvn tomcat:run -P integration

The samples also all work with the real UAA and the real Cloud Controller, but you need a client registration to make that work. If you have a real Cloud Foundry client you should use login.cloudfoundry.com for authentication and user approvals (this is the holder of the single-sign-on state for the platform) and uaa.cloudfoundry.com for the machine endpoints (principally /userinfo in this use case).

Appendix: Sample Implementation Details

The sample applications are in Github under an umbrella project. You can clone them using git:

$ git clone https://github.com/cfid/uaa-samples.git

or you can download the source code from the download link on the github website.

Ruby

The Ruby sample is written in sinatra which has a nice DSL for declaring web endpoints and filters. It uses omniauth-oauth2 to handle the OAuth2 Authorization Code flow and rest-client to access the Resource Servers. There is only one file app.rb and you can launch it from a command line like this (assuming you have Ruby 1.9.2 and bundle on your PATH):

$ bundle # do this once
$ ruby app.rb
...
>> Listening on 0.0.0.0:4567, CTRL+C to stop

The application is now running on http://localhost:4567.

Omniauth works by providing strategy implementations for specific token providers, and there isn’t a UAA strategy available officially yet, so the application adapts an existing abstract “oauth2″ strategy. This strategy provides an endpoint at /auth/oauth2 which is where the redirect and callback to the UAA is handled for you. The main thing the application has to do is tell this endpoint the client credentials and also how to send them to the token endpoint on the UAA. The default behaviour of omniauth-oauth2 strategy follows an older specification than the one implemented by the UAA, so we have to do a bit of work to get the client_options and token_params in the form required. (Since the existing abstract strategy doesn’t match the UAA very well we think it might be better to implement an UAA strategy using the [UAA gem][UAAGEM]. We are exploring this possibility on the Cloud Foundry Identity Team, and we need to decide on the implementation before we publish an omniauth strategy for general use.)

The application responds to environment variables for picking up settings of the remote URLs etc:

• UAA_LOGIN_SERVER = location of the Cloud Foundry Login Server where the Authorization endpoint lives. Defaults to the value of UAA_TOKEN_SERVER, but should be set to https://login.cloudfoundry.com in production.
• UAA_TOKEN_SERVER = location of the UAA. Defaults to http://localhost:8080/uaa. In production use https://uaa.cloudfoundry.com.
• CLOUD_CONTROLLER_SERVER = the Cloud Controller root URL. Defaults to http://localhost:8080/api. In production use https://api.cloudfoundry.com.
• CLIENT_ID = the id in the Client registration. Defaults to app.
• CLIENT_SECRET = the secret in the Client registration. Defaults to appclientsecret.

Grails

Grails is an opinionated web development and developer productivity tool that uses Groovy, Spring and a set of its own DSLs and conventions. Because of it’s heavy use of convention over configuration it is possible to build quite rich applications with relatively little code.

All the application logic is in HomeController.groovy and there is some configuration in Config.groovy and some properties files with names like application-*.;properties. To launch the application you can import it into a Grails aware IDE (like Spring Tool Suite), or run it on the command line (with Grails 2.1.1):

$ grails -Dgrails.server.port.http=9080 RunApp

The application shows up on [http://localhost:9080/grapps]. Here we show the application being launched on port 9080 so as not to clash with a UAA that you might be running locally. You could also run the UAA, API and Client applications all in the same container on port 8080 (e.g. by building war files and deploying them, or in the IDE with drag and drop).

The sample uses the Grails Scribe plugin to handle the Authorization Code flow. Scribe is quite rich, but too old to support well the most up to date versions of the OAuth2 specification, so we had to do quite a lot of work to make it work with the UAA. Once it is configured it works though and the rest of the Grails community seems to like the Scribe plugin for other OAuth providers like Facebook and Google.

The application responds to environment variables for picking up settings of the remote URLs etc:

• UAA_PROFILE = a short name for the profile, which is converted into a properties file name to pick up the URL information needed. Default value is default, and properties files are provided for cloud and vcap profiles.
• CLIENT_ID = the id in the Client registration. Defaults to app.
• CLIENT_SECRET = the secret in the Client registration. Defaults to appclientsecret.

Java and Spring

The Spring sample uses Spring Security OAuth on the client side (as opposed to the server which is what the UAA is). The filter for authentication is a combination of a standard filter from Spring Security OAuth, and one from the UAA common JAR (a library used to build the UAA, but also useful for building Clients and Resource Servers).

The application logic is in TreeController.java, which uses a RestOperations to access the Cloud Controller API resources. The rest of the features are provided by Spring Security and are configured in the main Spring configuration file at WEB-INF/spring-servlet.xml. The main aspect of that which is important is the filter that handles authentication with the UAA:

<http ...>
    <intercept-url pattern="/**" access="hasRole(&#39;uaa.user&#39;)" />
    ...
    <custom-filter ref="oauth2ClientFilter" after="EXCEPTION_TRANSLATION_FILTER" />
    <custom-filter ref="socialClientFilter" before="FILTER_SECURITY_INTERCEPTOR" />
</http>

With those filters in place the servlet request is automatically populated with a Principal object containing the user’s profile. Spring injects it into the HomeController so that the information can be displayed to the user in home.jsp. The oauth2ClientFilter is responsible for setting up the access token in a context that can be used by the RestOperations (which is an OAuth2RestTemplate) in HomeController.

To launch the application drop it into the same container as the UAA and API running locally (e.g. drag and drop in the IDE). Or run it on the command line separately using

$ mvn tomcat7:run -Dmaven.tomcat.port=9080

The application responds to environment variables for picking up settings of the remote URLs etc:

• UAA_PROFILE = a short name for the profile, which is converted into a properties file name to pick up the URL information needed. Default value is empty, and properties files are provided for cloud and vcap profiles.
• CLIENT_ID = the id in the Client registration. Defaults to app.
• CLIENT_SECRET = the secret in the Client registration. Defaults to appclientsecret.

This week VMware released the Scripted code editor on GitHub: https://github.com/scripted-editor/scripted.

Scripted, a JavaScript editor from VMware, is a general purpose code editor intended to be very lightweight with an initial focus on giving a great JavaScript editing experience– particularly around content assist and awareness of module systems. It is a browser-based editor that runs locally on a developer’s machine with a Node.js instance serving the editor code and performing the editor operations. The only pre-req for running Scripted is that you have a recent version of Node.js installed. Scripted is implemented in 100% JavaScript, HTML and CSS. If you are interested in more background on Scripted, you can read more about it on the SpringSource.org blog.

Features

• Fast startup, lightweight
• Syntax highlighting for JavaScript, HTML and CSS
• Errors and warnings:
  • JSLint is integrated to provide error/warning markers on JavaScript code
  • AMD and CommonJS module resolution: There is basic resolution where unresolved references will be marked as errors
• Content assist:
  • Basic content assist for HTML, CSS
  • For JavaScript, content assist is driven by a type inferencing engine which is aware of AMD/CommonJS module dependencies and also uses JSDoc comments to help it understand the code
• Hovers: Hovering over a JavaScript identifier will bring up the inferred type signature
• Navigation: Press F8 on an identifier (that the inferencer has recognized) and the editor will navigate to the declaration. This also works on module identifiers (e.g., in define() clauses)
• Formatting: JSbeautify is integrated
• Sidepanel: Alongside the main editor, a sidepanel can be opened. Currently this can be used to host a second editor
• Key binding to external command: Key bindings in the editor can invoke external commands (less, mvn, etc.)

There is much more detail on these features in the wiki documentation.

For Node.js Development

As listed in the features above, Scripted understands the CommonJS module system, as employed by Node.js apps. Understanding modules means two key things:

• References to non-existent modules can be reported at editing time.
• Knowing the module, we can look inside and from the contents offer appropriate content assist where the module is being used.

The following screenshot shows Scripted being used on a Node.js module and in this case an invalid module reference is being reported:

In that same piece of code, here we can see that because we recognized the module, content assist is correctly proposing the two methods from the users module, called getUser(id) and getUserCount():

The keen eyed amongst you will notice that in the content assist proposals the return value of getUser(id) was shown as a Person. This was inferred by using JSDoc that was attached to the definition of getUser(id) in the users module:

Knowing the return value then enables smart content assist at the location where the return value of the function is referenced.

For an integrated experience with Cloud Foundry, you can use the key binding configurations for Scripted to cause a vmc operation to execute from the editor. Configuration of Scripted is done through a .scripted file at the root of the project, a bit like a .virc file for vi. This file is a JSON format document, and the supported configuration options are covered in the documentation (see the section on configuration here). Using the exec-keys config option, it is possible to connect a key binding to invocation of a command. Here is some vmc related configuration:

{
    "exec": {
        "onKeys": {
            "ctrl+shift+alt+p": {
                "name": "vmc push scriptedapp",
                "cmd": "vmc push -n scriptedapp --runtime node08 --path .",
                "timeout":60000
            },
            "ctrl+shift+alt+u": {
                "name": "vmc update scriptedapp",
                "cmd": "vmc update -n scriptedapp",
                "timeout":60000
            }
        }
    }
}

In this next screenshot you can see:

• On the right the help panel is open and the new key bindings are listed for our vmc operations.
• The command output currently goes to the JS console (this will be improved!). The console here is showing the command has been invoked and the push was successful.

Want To Try It Out?

If you wish to try it out for yourself, arm yourself with a copy of Node.js then jump onto the Scripted GitHub page for instructions on how to get started.

Future Plans

Over the next few months we are going to focus on a few things:

• Even smarter content assist and improved navigation options.
• More side panel contents. This is something I haven’t focused on in this article but it is shown in the video available on the project homepage. The side panel is intended to host information relevant to the task you are trying to achieve in the main editor. This might be code, documentation, search results or a preview. Watch the video to see the side panel in action.
• A plugin system for extending Scripted. Plugins, like Scripted itself, will be 100% JavaScript, HTML and CSS.
• Debugging integration. Exploring integration with tools like Chrome Dev Tools and Node.js inspector.

We open-sourced Scripted to accelerate adoption and collect feedback. If you want to help us shape the editor, please join in the discussion. There is a scripted-dev Google Group for discussing it and a jira issuetracker for logging bugs, enhancement requests and voting on existing issues to ensure they are prioritized appropriately. If you want to start hacking on the codebase yourself we are definitely open to submissions–see the GitHub page for more information. The codebase is very new at the moment so there isn’t really a steep learning curve.

Please try it out! https://github.com/scripted-editor/scripted.

–Andy Clement

Andy Clement is a staff engineer in the SpringSource division of VMware, based in the languages and tools lab in Vancouver. He has more than ten years experience in Enterprise Application Development and now spends his time building tools for languages like AspectJ, Groovy and JavaScript and frameworks like Grails. He currently oversees the Groovy Grails Tool Suite deliverable, a variant of the Spring Tool Suite with a focus on Groovy and Grails.

As an active committer on Spring Security OAuth and the Cloud Foundry UAA, one of the questions I get asked the most is: “When and why would I use OAuth2?” The answer, as often with such questions, is “it depends.” However, I must admit, there are some features of OAuth2 that make it compelling in a wide variety of situations, especially in systems composed of many lightweight web services. This article guides you through updating a system to be secured with OAuth2 and the decision points for choosing to build such a system.

There is a strong trend at the moment towards distributed systems with lightweight architectures based on plain text web services (usually JSON). In this article we concentrate on these services and the systems they are part of, and look at some options for their basic security needs. An example of such a system is the open platform as a service, Cloud Foundry, in which the UAA acts as an OAuth2 provider. Using Cloud Foundry as an example also indicates that the trend in lightweight services is driven by a related trend towards cloud-based platforms for application deployment, both in the Internet at large and in the enterprise.

I recently published a blog post that provides an overview of the UAA: Introducing the UAA and Security for Cloud Foundry. The remainder of this article is about securing REST services in with OAuth generically and not necessarily specifically to the Cloud Foundry UAA, but I will use the UAA as an example to illustrate how it works.

What is a Lightweight Service?

The kind of service that is building the new architectures typically has some or all of these features:

• HTTP transport. Cheap and easy to deploy, and supported by all cloud platforms.
• Text-based message content, usually JSON. The emphasis is on readability and sometimes providing easy support to front-end browser and mobile developers.
• Small, compact messages, and quick responses. Although there is a strong parallel trend of services with streaming responses for “push” data, those are generally not implemented the same way, and HTTP is not the best transport in those cases.
• REST-ful, or at least inspired by the REST architectural style.
• Some degree of statelessness, either ‘share-nothing,’ or very careful use of server-side state, to enable horizontal scalability.
• Interoperability. The consumers of the service might be a dynamic population, completely unknown or unknowable at design time, using a variety of platforms and languages.

No two people or organizations will agree on what to call these services. So, although we need a working definition, it isn’t important what the precise details are. Roy Fielding’s academic work on REST is very influential, but religious wars are fought (and never won) on what it should mean in practice. To avoid being drawn into that, I will be deliberately inclusive and just refer to the services as lightweight services over HTTP, or just “lightweight services.”

What Are the Security Requirements for a Lightweight Service?

It is possible, but unlikely, that a service might be so lightweight that it doesn’t require any security, for example if it is purely informational. If security is required, it will be for the usual reasons, i.e., identity and permission; knowing who is asking for a resource, and calculating if they are permitted to use it. Identity and permission are often known as “authentication” and “authorization” (especially in Spring Security), but using those terms would cause confusion later because “authorization” is part of the OAuth2 domain language and means something different there. “Authorization” is also the name of a standard HTTP header, which is also a bit different as it is more about transporting of data than calculating permission.

So the basic requirements are identity and permissions, and the choices all come down to these nuts and bolts questions:

• how are identity and permission information conveyed to a service?
• how is this information decoded and interpreted?
• what data is needed to make the access decision (user accounts, roles, ACLs, etc.)?
• how is the data managed, who is responsible for storing and retrieving it?

The rest of this article deals with those questions using a couple of example approaches, including OAuth2, but does not attempt to be a complete description of available features and implementations even in that case.

HTTP Basic Authentication

For a simple system based on shared secrets (e.g., username and password for user accounts), HTTP Basic is something of a lowest common denominator. It is supported on practically all servers natively out of the box and there is ubiquitous support on the client side in all languages. Most, if not all clients, even interpret the authority section of a URL as data to be used for Basic authentication.

As a client, the only thing you need to do for Basic authentication is to include an Authorization header in an HTTP request, composed of the username and password, separated by a colon and then Base64 encoded. E.g., in Ruby (1.9) using RestClient:

require 'restclient'
require 'base64'
auth = "Basic " + Base64::strict_encode64("#{username}:#{password}") 
response = RestClient.get("https://myhost/resource",:authorization=>auth)

On a command line with curl you could do this:

$ curl "https://$username:$password@myhost/resource"

This looks, and is, extremely simple to implement. The only difficulty is in where you get the credentials from (the username and password). If your client is a web application, which is very common for these lightweight services, you might collect the credentials from a user in a simple HTML form.

As long as you ensure that all requests are protected by a secure socket layer, Basic authentication is fine for systems where all the participants can easily share secrets securely. If there is only one resource server (a lightweight service, or a user facing application) this isn’t difficult because the data only needs to be accessed by one component and can even be managed in config files if they are static. For example, the UAA uses Basic authentication for endpoints that are intended to be accessed only by other components in the platform kernel (i.e., machine clients). Some of those endpoints (e.g. /varz) have a password stored in a config file, and some (e.g. /oauth/token) use a database backend to store credentials. For explanations of what those endpoints do in the UAA see the introductory blog or the UAA docs.

User or Client Roles and Permissions

For very simple use cases, authentication is the only form of permission needed. If you can authenticate you can access the resource, otherwise not. But often it is not enough and the service has to be able to make a decision based on finer-grained information about the authenticated party. A simple example of such a decision is role-based access, which is very common, and is also supported out of the box by many server platforms. For instance, the service has a resource that it wants to be available only to users in the ADMIN role. When an authenticated request arrives, the service has to map the authentication to some account data and extract the role assignments. It then checks if the ADMIN role is present and if so grants access, otherwise not.

For role-based access to make sense you need to be able to categorize user accounts. In other words. there have to be at least two classes of users (e.g., USER and ADMIN, which implies at least two users), and someone has to manage the assignments of accounts to roles. This is a good deal more complicated than simple shared password data, but as long as the account data is static it can still be managed in a config file, or in a database otherwise. Crucially, this is still easy to do, as long as the system stays simple and can manage its own account data, or get access to an external system where the data is managed for it.

More complicated access decisions involving different and more detailed fine-grained business data are also possible (even common), but those use cases are beyond the scope fof this article. It’s time to start looking at other things and get down to the OAuth2 discussion.

OAuth2 and Centralized Identity Management

As a system grows and the number of components that need access to authentication services and to user account data increases, or the user account data is changing frequently or growing rapidly, the simple approaches to security will start to become a maintenance problem. What is needed is a centralized approach to identity management, and a service for the whole system that can provide and manage that data in a secure way. This is where standards like OAuth2 come into play. They introduce a new layer in the architecture, and some extra complexity that can be hard to get to grips with at first, but the benefits will be worth it in many cases.

Quick Introduction to OAuth2

OAuth2 is a protocol enabling a Client application, often a web application, to act on behalf of a User, but with the User’s permission. The actions a Client is allowed to perform are carried out by a Resource Server (another web application or web service), and the User approves the actions by telling an Authorization Server that he trusts the Client to do what it is asking. Clients can also act as themselves (not on behalf of a User) if they are permitted to do so by the Authorization Server.

The most common way for a Client to present itself to a Resource Server is using a bearer token, as covered in the core OAuth2 specification. The token is obtained from the Authorization Server, with the User’s approval if necessary, and stored by the Client. Then when it needs to access a Resource Server the Client sends a special HTTP header in the form:

Authorization: Bearer <TOKEN_VALUE>

The token value is opaque to a client, but can be decoded by a Resource Server so it can check that the Client and User have permission to access the requested resource.

Common examples of Authorization Servers on the Internet are Facebook, Google, and Cloud Foundry all of which also provide Resource Servers (the Graph API in the case of Facebook, the Google APIs in the case of Google, and the Cloud Controller in the case of Cloud Foundry).

OAuth2 and the Lightweight Service

OAuth2 is, at its heart, an authentication protocol for lightweight services, which are Resource Servers in the domain language of the specification. A Client application that wants to access a protected resource sends an authorization header, a bit like in the Basic authentication case. E.g., in Ruby:

auth = "Bearer #{token}"
response = RestClient.get("https://myhost/resource",:authorization=>auth)

or on the command line:

$ curl -H "Authorization: Bearer $TOKEN" https://myhost/resource

As with the Basic authentication, the mechanics are extremely simple, and that is one thing that makes OAuth2 bearer tokens attractive for clients of lightweight services.

The token is opaque to the Client, but the Resource Server can decode it into some finer grained information about the Client and the level of access that the token represents. It then uses this information to make an access decision. This is not the only feature of OAuth2, but on its own makes OAuth2 a lot more powerful than simple Basic authentication: The token itself carries more information than a Basic header.

A typical minimal set of information in a token would be the Client ID, a target Resource ID and a set of approved scopes (more on those later). Normally the Resource Server has an ID, and it should check that it matches the one in the token in order to prevent token misuse. Beyond that, it is entirely up to the Resource Server what to do with the decoded token data. If the Resource Server is able to coordinate with the Authorization Server, it can arrange for specific fine-grained data to be encoded in the token, relating to the access decision it wants to make. For example the Authorization Server might include role assignments that are recognized by the Resource Server. Tokens could, and do, also contain other information, like a unique identifier for the current user (if there is one).

The UAA stores group assignments in its native user data, and those show up in the access tokens as scopes. The scope values themselves in the UAA are period-separated, but it is up to the Resource Server to interpret them any way it needs. For example, a Resource Server calling itself ‘dashboard’ might change its access decision based on whether it finds a scope ‘dashboard.user’ or ‘dashboard.admin’, and those are the names of groups in the user accounts. These semantics are not part of any standard specification, but OAuth2 leaves them up to implementations to interpret according to their need, so the UAA takes advantage of that.

The Role of the Client Application

For a Client, obtaining a token makes things slightly complicated, and in this case it is usually not enough just to know a username and password. Also, one of the key reasons for OAuth2 to exist is so that Client applications do not need to collect user credentials (as they did with Basic authentication). Here is where the learning curve for OAuth2 gets steeper.

The Client has to obtain an access token, and to do that it has to be pre-registered with the Authorization Server, and it has to authenticate itself at the token endpoint. The UAA uses Basic authentication for this endpoint, as suggested by the OAuth2 specification. If a Client is not acting on behalf of a User, the Authorization Server might allow it to obtain tokens in its own right, directly from the token endpoint. If a client with id “myclient” has been registered with the relevant authorized grant type you could obtain a token like this:

$ curl "https://myclient:mysecret@uaa.cloudfoundry.com" -d grant_type=client_credentials -d client_id=myclient

The result would be JSON containing an access token and some meta data, for example:

{ 
  access_token: FUYGKRWFG.jhdfgair7fylzshjg.o98q47tgh.fljgh,
  expires_in: 43200,
  client_id: myclient,
  scope: uaa.admin 
}

(This is not a valid token, nor is it a valid client id for the UAA on cloudfoundry.com, just dummy data for illustration purposes.)

The example so far has been for a Client authenticating and obtaining an access token in its own right, which the specification calls the Client Credentials Grant Type. Another grant type is Authorization Code. This is the most common and one that allows the Client to delegate the authentication of Users. Application providers then need to educate Users that they should never enter their credentials in any other application than the Authorization Server (or its trusted delegates), so that a malicious Client application has less chance of tricking a User into revealing his credentials. This use case is one of the main reasons to use OAuth2 and a big driver of the OAuth2 specification itself.

The Authorization Code token grant proceeds in four steps:

Sequence diagram, auth-code-flow, with an already authenticated user (steps 2-4):

1. Authorization Server authenticates the User any way necessary. This step is required but is specific to the Authorization Server implementation and not part of the OAuth2 specification (e.g., it is different for Google and Cloud Foundry).
2. Client starts the authorization flow and obtains approval from the Authorization Server to act on the User’s behalf. The approval is required, but the details are not specified in the OAuth2 specification.
3. At this point, if successful, the Authorization Server issues an authorization code (opaque one-time token). This step, and the next one, are described in detail in the OAuth2 specification.
4. Client exchanges the authorization code for an access token.

Because there are multiple steps here and normally the first two would require a short conversation with the User, the Authorization Server has to direct the flow through the use of HTTP redirects (status code 302 and a Location header). The Client specifically asks for a redirect to itself in order to obtain the authorization code at step 3. One of the security threats in OAuth2 is a malicious Client stealing tokens by asking for an arbitrary redirect, so Authorization Servers protect against this by requiring Clients to register one or more redirect URIs.

The Authorization Server is so called because it provides an interface for users to confirm that they authorize the Client to act on their behalf (step 2 above). The specification wisely leaves the details entirely up to the implementation, so how the authorization (a.k.a approval) is obtained is unspecified. The UAA, and the Spring Security Oauth2 project that it builds on, provide a simple form-based interface in the general case, but also allow auto-approval of certain clients (e.g., if they are deemed by the Authorization Server owners to be part of the platform).

Client Registration and Scopes

The OAuth2 specification mentions Client registration explicitly and it is an important part of the protection against various security threats. It is also the first point of contact for most application developers with OAuth2, e.g., it’s a pre-condition if you want to look at Google calendar data, or manage the applications deployed on Cloud Foundry. There are some core elements of a Client registration required by the specification (an identifier and a secret if the Client is trusted), and several that are recommended (legal scope values, and registered redirect URIs). Authorization Servers often require additional information, describing the application and the owner of the registration.

The scope values are the hardest to understand for a newbie because they are arbitrary strings and don’t necessarily mean anything to the Client at first. Fortunately for the User, the Authorization Server can usually find a default set of scopes from somewhere, and usually knows how to render the values in a way that a User will understand (e.g., from Facebook, “do you allow this application (<URL>) to access your personal data, including email address and photos”). But for a hapless Client owner it can be quite hard to discover the valid scope values and get them registered. In the end, the Client might have to send a speculative request and look at the response from the Resource Server: The specification recommends that valid scope values are included in the HTTP 403 response as part of the WWW-Authenticate header.

You can find out (some of) the valid Google API scopes in the OAuth2 playground–they are mostly URLs starting with “https://www.googleapis.com” (but some are not, maybe the older ones). The API docs themselves (i.e., the Resource Servers) are very thin on scope documentation.

In the case of the UAA there is a UAA Security Model document that lists the known scope values as used by the UAA itself and the Resource Servers that it knows about in the Cloud Foundry platform (e.g., the Cloud Controller). In theory, though, and this is quite a powerful feature of OAuth2, there is nothing to stop a Client registering any value as a scope, and a Resource Server that is completely unknown to the Authorization Server can accept that value. To protect against malicious clients, the UAA extracts a resource ID dynamically from a scope value (scope up to the last period is a resource ID) and includes it in the access token as the “aud” (audience) field. We hope that eventually this will provide an opportunity for the platform to be used as a general purpose OAuth2 provider for all applications, not just those that belong to or consume the platform.

The UAA on cloudfoundry.com at present is only accepting Client registration by a manual process through the Product Management group in Cloud Foundry, but if you would like to deploy a Client using, or a Resource Server accepting, cloudfoundry.com bearer tokens, feel free to contact us.

Authentication and the Authorization Server

OAuth2 on its own does not provide an authentication protocol for the Users of the Authorization Server. It strongly suggests that Client applications should use Basic authentication for accessing the token endpoint, but it says nothing about the authentication of Users when their approval is needed for a token grant (only that they must be authenticated). This is a good thing because it makes authentication completely orthogonal to the approval process, and Authorization Servers are free to implement the authentication any way they choose.

Out of the box, the UAA supports form-based authentication backed by a database of user accounts. But because authentication is not part of the OAuth2 specification, it is easy to modify to support other authentication mechanisms or data sources. For example, with a few lines of configuration you could switch from form-based authentication to Open ID authentication with Google or Yahoo, or to an enterprise directory service (e.g., Active Directory or OpenLDAP) for the data store.

The authentication logic in the UAA can also be extracted out into a completely separate server component, so it can be independently developed and styled. There is a sample login server which allows authentication either with Open ID (Google, Yahoo, etc.) or using an HTML form and a password in the UAA database. Modifying it to support a SAML IDP such as VMWare Horizon or an AD back end for the user account data is trivial because both are well supported by the underlying Spring Security stack.

Alternatives

We have seen that there is a trade-off between the complexity of OAuth2 and a simpler system based purely on shared secrets and a shared user account database. There are other alternatives that are equally or more complex than OAuth2, the main example being SAML. SAML and OAuth2 share a lot of aims, but the implementation is different, and the touch points for developers are very different. SAML is extremely rich and has many features and offshoots that users find indispensable, and it is by default very secure. It is also hard to set up and configure, and often leads to very large amounts of XML being sent with each HTTP message (often so much that it would dwarf the average JSON payload). This last point is probably the thing that puts people off SAML for the sort of lightweight services discussed here, but there is no reason, in principle, why it could not be used.

In environments where SAML is already well established there are also ways to combine it with OAuth2. For example, the specifications allow for a SAML grant in which a client exchanges a SAML assertion for an OAuth2 token, which can then be used as if it came from any other source.

Another alternative to OAuth2 is to write your own system with the same or a subset of features. Many systems have been forced to adopt this approach just because the specifications have been rather long in gestation. There’s nothing intrinsically wrong with doing it yourself, especially if your needs are simple, but many people feel that with security especially there is value in using a standard protocol. Security is very high profile these days and any mistake you make with a non-standard implementation is going to reflect badly if (or rather when) things go wrong.

In Conclusion

So what’s the punchline? Why would you use OAuth2 to secure a lightweight service? It certainly isn’t the simplest approach you could possibly adopt, so you have to be aware of the benefits that would lead you to accept the extra complexity. OAuth2 is well suited to lightweight services in medium to large distributed systems, where User and Client account data need to be centralized, and where you need some control over the expiry and validity of access tokens. The fine-grained information that can be packed into an access token (and which is derived from the centrally managed account data) provides significant benefits over a simple protocol like HTTP Basic authentication. Fortunately, the complexity can also be hidden by client libraries (e.g., Spring Security OAuth2 for Java, or the UAA Gem or Signet for Ruby, or JSO for JQuery). In addition, since OAuth2 is a commonly used standard these days, he number and quality of client libraries is increasing steadily. To put things round the other way, I wouldn’t bother with my own OAuth2 provider if my system only had one or two Resource Servers, or if Client applications never had to act on behalf of a User. I would consider using something simple like Basic authentication, and when I needed more I might switch to someone else’s OAuth2 provider (e.g., Cloud Foundry) if it supported the features I needed.

Naturally, this short article is not going to answer all the questions that arise about OAuth2 and identity management. If you want more information, join the community: check out the source code of the UAA and hang out on the Spring Security OAuth Forum or the Cloud Foundry Developers Group.

We have been working on making it easier to deploy Clojure apps to Cloud Foundry [1]. Clojure is a functional language that runs on the JVM that allows rapid prototyping of applications while offering great support for integration with Java libraries. Many interesting projects–for example, Twitter’s own realtime processing framework Storm–are written in Clojure.

To help to run Clojure apps in Cloud Foundry, we have written a library called claude that you can use to easily access the different services offered by Cloud Foundry, like MongoDB or RabbitMQ.

Now let’s see how you can start deploying your Clojure apps to Cloud Foundry.

Deploying Your Clojure Apps to Cloud Foundry

We assume that you are using leiningen for your Clojure projects.

First you will add the latest version of claude into your project dependencies:

[claude "0.2.1-SNAPSHOT"] Then run lein deps to obtain the project dependencies.

Once you have your project set up then you need to declare your project main function:

:main clojure-demo.server

See this project project.clj file for more details.

Then on your main namespace you need to add a :gen-class like in this example:

(ns clojure-demo.server
  (:gen-class)
  (:require [noir.server :as server]
            [claude.core :as cf]))

If you are creating a web app then another tweak you need to perform is to inform your app of the port in which it has to run. You can obtain the port from the VMC_APP_PORT environment variable. Here’s an example:

(defn -main [& m]
  (let [mode (if (cf/cloudfoundry?) :prod :dev)
        port (Integer. (get (System/getenv) "VMC_APP_PORT" "8080"))]
    (server/start port {:mode mode
                        :ns 'clojure-demo})))

Also, as you can see in that code snippet, we are using the claude.core/cloudfoundry? helper to see if we are running in the cloud. In that case, we can set the web app environment to :prod.

So far we have a basic Clojure project that can be run from a .jar file. Let’s create that jar file and deploy it to Cloud Foundry. We asume that you are already logged into Cloud Foundry using vmc. Run the following commands inside your Clojure project folder:

mkdir cf
lein uberjar
cp target/clojure-demo-0.1.0-SNAPSHOT-standalone.jar cf/
cd cf

There we created a new folder to hold our app .jar file. Now we are ready to deploy it. In this case, I will bind one instance of each service provided by Cloud Foundry. Therefore, you will see a lot of output in the following transcript. Let’s start with vmc push:

vmc push
Would you like to deploy from the current directory? [Yn]: Y
Application Name: clojure-demo
Detected a Standalone Application, is this correct? [Yn]: Y
1: java
2: java7
3: node
4: node06
5: node08
6: ruby18
7: ruby19
Select Runtime


: java7
Selected java7
Start Command: java $JAVA_OPTS -cp clojure-demo-0.1.0-SNAPSHOT-standalone.jar clojure_demo.server

Up to here we told vmc that we are deploying a standalone application, that the runtime is java7 and we provided our startup command. Make sure to change the application name on your own deployment.

Let’s continue our deployment:

Application Deployed URL [None]: ${name}.${target-base}
Memory reservation (128M, 256M, 512M, 1G, 2G) [512M]: 512
How many instances? [1]: 1

If we are deploying a web application, then we need to provide the URL for our app. In our case we let vmc build the URL based on our application name. In this case it will be under http://clojure-demo.cloudfoundry.com. Keep in mind that you only need a URL if you are deploying a web app.

Now let’s create and bind one instance of each service type to our app:

Bind existing services to 'clojure-demo'? [yN]: N
Create services to bind to 'clojure-demo'? [yN]: y
1: blob
2: mongodb
3: mysql
4: postgresql
5: rabbitmq
6: redis
What kind of service?: 1
Specify the name of the service [blob-9dcdc]:
Create another? [yN]: y
1: blob
2: mongodb
3: mysql
4: postgresql
5: rabbitmq
6: redis
What kind of service?: 2
Specify the name of the service [mongodb-cb862]:
Create another? [yN]: y
1: blob
2: mongodb
3: mysql
4: postgresql
5: rabbitmq
6: redis
What kind of service?: 3
Specify the name of the service [mysql-2358b]:
Create another? [yN]: y
1: blob
2: mongodb
3: mysql
4: postgresql
5: rabbitmq
6: redis
What kind of service?: 4
Specify the name of the service [postgresql-de020]:
Create another? [yN]: y
1: blob
2: mongodb
3: mysql
4: postgresql
5: rabbitmq
6: redis
What kind of service?: 5
Specify the name of the service [rabbitmq-3422d]:
Create another? [yN]: y
1: blob
2: mongodb
3: mysql
4: postgresql
5: rabbitmq
6: redis
What kind of service?: 6
Specify the name of the service [redis-29cde]:
Create another? [yN]: N

Note that we did this for our example’s sake. Your app might not need as many services or you might choose to not use any of them. It’s all up to you.

Now we probably want to save this configuration inside a manyfest.yml file so we don’t need to type all that again the next time we deploy our app from scratch.

Would you like to save this configuration? [yN]: y
Manifest written to manifest.yml.

Now vmc will stage your app and it will start it automatically:

Creating Application: OK
Creating Service [mongodb-cb862]: OK
Binding Service [mongodb-cb862]: OK
Creating Service [mysql-2358b]: OK
Binding Service [mysql-2358b]: OK
Creating Service [postgresql-de020]: OK
Binding Service [postgresql-de020]: OK
Creating Service [blob-9dcdc]: OK
Binding Service [blob-9dcdc]: OK
Creating Service [rabbitmq-3422d]: OK
Binding Service [rabbitmq-3422d]: OK
Creating Service [redis-29cde]: OK
Binding Service [redis-29cde]: OK
Uploading Application:
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (5M): OK
Push Status: OK
Staging Application 'clojure-demo': OK
Starting Application 'clojure-demo': OK

You can access a demo app here.

While this seems to be a little long, in fact we detailed every single step to get you up and running. To recap this is what we need did:

• Add the claude dependency to your project.
• Add a (:gen-class) to your main namespace. This is a Clojure/Java requirement
• Generate your uberjar.
• Deploy your app providing a start up command

And that’s it! Now create a free account at Cloud Foundry and let’s see your awesome Clojure apps deployed to Cloud Foundry!

1 *NOTE: At the time of this writing Clojure is unofficially supported at Cloud Foundry. *

The value proposition of using the cloud is that “things will be easy this way.” Building and operating large scale clouds, however, is rarely easy. As Mark Lucovsky said in his recent interview, “Who said large scale distributed systems are simple?” While solutions like Cloud Foundry make building core cloud technology less daunting, it’s still not easy.

In this guest blog, Lucas Carlson, Founder and CEO of AppFog, illustrates the story of how AppFog was built using Cloud Foundry open PaaS technology and what they learned along the way.

Core Cloud Technology is Hard

Running a simple Cloud Foundry-based service is not very hard. Check out Micro Cloud Foundry^TM.

Building one that is suitable for production and enterprise workloads is fairly hard.

Building a Cloud Foundry-based service with one API end-point to run in 6+ different availability zones on 4+ independent infrastructure vendors and that is suitable for production and enterprise workloads, at scale and in general availability? Now that is really hard.

Hitting the Ground Running with Cloud Foundry

As you can imagine, ever since AppFog announced GA, people have been asking how we did it. How did we figure out how to scale up a public cloud offering for Cloud Foundry that delivers one interoperable interface across multiple clouds, multiple availability zones and multiple vendors?

First: The Cloud Foundry codebase is really very good. In choosing to build using Cloud Foundry we eliminated a number of the potential serious challenges and major time sinks likely with this sort of project.

Second: The leverage that we were able to gain from the additional tools, code and services from within the Cloud Foundry ecosystem dramatically accelerated our development.

Third: Having a non-Cloud Foundry-based public PaaS already in GA and already at scale was a massive boost for us. We had the operational experience in-house needed to pull off a great PaaS.

AppFog’s ops experience was invaluable for us. Although PaaS is closely tied to the NoOps movement, NoOps doesn’t mean no ops, it only means no ops for the developer. As anyone who has run Cloud Foundry knows, a lot of ops is necessary and essential to running PaaS.

And most importantly, we had a vibrant community of tens of thousands of developers (the users of PHP Fog) that we could consult to help us define, test, and evolve AppFog through private beta to Public Beta and now to GA.

We Listened and Delivered

It was this community that helped us figure out what AppFog at GA would need to be, how it would need to work, and what our pricing would be. This was what devs asked us for:

• The ability to easily scale apps across dozens or hundreds of load-balanced instances, even on the free plan
• Must run apps on the fastest servers available in the infrastructure (yes, we mean, for example, M2.4XL)
• Must be able to run apps across multiple infrastructures (Rackspace, HP, AWS, Azure–and with no price difference)
• Must simplify pricing. No more weird packages and bundles and names. Make it simple and affordable and most of all make it fair.
• True interoperability in the cloud, including single-click, zero-code migrations between cloud vendors, eliminating all traces of vendor lock-in
• Target of 30-second deployments to AWS, Rackspace, HP, Azure, etc.
• Support for any and all languages in the Cloud Foundry codebase
• Support for any and all relational and NoSQL data stores**

This is a daunting list. But it’s what devs wanted.

So let’s get into the nitty gritty… how did we actually do this?

How We Built a PaaS Around Cloud Foundry

This is a complex story. Fortunately, a few months ago, Jeremy Voorhis gave a talk at QCon in London describing how AppFog built a commercial system by using the Cloud Foundry OSS bits alongside our own set of custom extensions and enhancements. This is far and away the best place to start for understanding the Cloud Foundry-to-AppFog evolution. The slides for the talk can be found here and the video here.

We won’t go over everything in Jeremy’s hour-long talk here, but we’d like to cover some of the main points.

First, Jeremy clears up some of the confusion surrounding the signifier “Cloud Foundry.” On one hand, CloudFoundry.com is a hosted PaaS offering from VMware that runs on the vSphere virtualization platform based on the open source project found on CloudFoundry.org. VMware also has open sourced the tooling it uses to manage CloudFoundry.com–Cloud Foundry BOSH–and exposes the exact BOSH releases used live in production on their hosted service.

We do not run on the hosted VMware service but instead we are built from the same open source Cloud Foundry project. This open source distributed system enabling teams to manage and deploy apps and services over their lifecycle in the cloud is the core of the AppFog service.

AppFog is unique within the current Cloud Foundry ecosystem because we have built an infrastructure-agnostic Cloud Foundry service.

The beauty of Cloud Foundry is that it can, in principle, be implemented in conjunction with infrastructure offerings ranging from an OpenStack-based IaaS like Rackspace or HP to AWS, Joyent, Citrix, and beyond. This inherent multi-cloud portability is why the Cloud Foundry project was so inspiring to us.

For us, the ability to deploy and redeploy across infrastructures within the lifecycle of an application is the key to building a next-generation PaaS. It also, we think, fulfills the underlying mission of PaaS to: (a) sharpen product team focus in development; (b) produce a shorter feedback loop within app lifecycles, and; (c) provide the possibility of near-instant horizontal scalability.

Cloud Foundry’s Role in the Evolution of Cloud Computing

Even for those who are familiar with the space of cloud computing in general, it’s important to review historical context. For us at AppFog, Cloud Foundry has provided some of the crucial building blocks to assembling the first next-generation PaaS. But what do we mean by that?

First, it’s important to get some background. PaaS in its initial forms was enabled by a number of historical breakthroughs such as:

• The rise of data centers via companies like Rackspace in the 1990s
• The emergence of server virtualization via VMware in the early ‘00s
• The introduction of APIs to virtualized resources via AWS in the mid ‘00s
• The more recent rise of the DevOps paradigm with their associated and derived tools like Puppet and Chef.

But this trajectory really began coming to fruition in just the past two years. Tools like Cloud Foundry and OpenStack have emerged that enable full application lifecycle management. What this means is that the “platform” in “platform as a service” can now encompass all aspects of the application lifecycle outside of development. Jeremy refers to this as “NoOps” in the talk. What it means is emphatically not that no one does Ops work anymore. Instead, it means that responsibility for Ops work is passed on to the platform layer and remains there for most if not all of the app’s lifecycle.

Without a comprehensive tool like Cloud Foundry, we never would have been able to overcome the gap between (a) first-generation PaaS platforms, which we were essentially little more than gateways for (unwieldy) infrastructure management, and (b) next-generation PaaS like AppFog, which provide a far more comprehensive set of user benefits.

AppFog’s Modifications of Cloud Foundry

AppFog began as an effort to bring a great user interface and multi-cloud experience to the hosted Cloud Foundry ecosystem. We saw a number of stumbling blocks standing between the Cloud Foundry codebase and a full-fledged PaaS offering. The current Cloudfoundry.org project has CLI and various IDE integrations, but our vision included the full user experience, including fast sign-up, a web console, pre-built example applications that became what we call Jumpstarts, add-ons, and more.

While some Cloud Foundry users are indeed quite comfortable operating the system from the command line, Eclipse or the Spring Tool Suite (just as many AppFog users do), we also wanted to provide users with an elegant and intuitive console from which they could manage all of their apps. An example of the AppFog console is below: