Cloud Foundry Blog

Cloud Foundry Integration for Eclipse Now Supports Tunneling to Services

Today we announce a new release of Cloud Foundry Integration for Eclipse which features the ability to open a tunnel to any Cloud Foundry data service. Now Eclipse users can use familiar client applications to directly analyze, manipulate, or port the data contained in their Cloud Foundry applications.

The Cloud Foundry Data Tunneling service can be created from the Services table in the Cloud Foundry Eclipse server editor. The new integration (version 1.1.0)  allows users to create data tunneling (also known as ‘Caldecott‘) from within Eclipse Indigo JEE or SpringSource Tool Suite (STS) version 2.9.0 or higher.

Follow the documented instructions to install the Cloud Foundry Integration for Eclipse. If you had previously installed the Cloud Foundry plugin in Eclipse or STS, the update will be automatically detected or you can manually check for updates in the IDE.

Creating a Tunnel

You can create a tunnel to connect to any running service in your Cloud Foundry account. Go to the Applications tab in the Cloud Foundry Server view. There is a listing of existing services under the Services section. Creating a tunnel is performed by simply right-clicking on a service in the table and selecting “Open Caldecott Tunnel” menu action:

This will automatically publish a tunneling application to your account, if one isn’t already published, and start or restart the application as necessary. This tunneling application will appear in the list of published applications as Caldecott. Cloud Foundry will also automatically bind the selected service to the Caldecott application prior to opening a tunnel.

The process of creating a tunnel requires services to be bound to the tunneling application. Cloud Foundry will automatically stop a running Caldecott application, bind the service, and restart the application to create the tunnel. Therefore, existing tunnels may be closed when a new tunnel is created, if the desired service was not previously bound to the tunneling application. Multiple tunnels can be opened to different services in your account at the same time, as long as all the related services are already bound to the Caldecott application prior to creating the data tunnels.

In general, any operation that requires the tunneling application to be stopped, restarted or removed will result in all existing  tunnels being disconnected. Manually unbinding a service from the Caldecott application through the Services table will also result in the associated tunnel being disconnected.

Once a tunnel is successfully created, a dialogue will open displaying the data service tunnel information necessary to use a tool like the Eclipse Data Source Explorer to connect to a data service in a Cloud Foundry server. The information displayed includes a username, password, and local port number:

As URLs are needed for database connections in tools like the Eclipse Data Source Explorer, they are automatically constructed for certain data services like MySQL and PostgreSQL. That way, users do not need to manually construct them when creating a database connection.

Right-clicking on the tunnel entry in the tunnel dialogue will display several context menu actions to copy field values to the clipboard. The menu action to copy the URL appears if the service is of MySQL or PostgreSQL type:

Managing Tunnels

Once a tunnel is opened, it can be managed from the Services table in the Cloud Foundry server editor. A sortable “Tunnel” column indicates which services have an open tunnel:

Right-clicking on a service allows users to either disconnect the tunnel or show the tunnel information.

Data tunnels can also be managed in the Servers view by right-clicking on a server instance and options will be presented to manage individual tunnels via wizard flow or disconnecting all existing tunnels:

These Servers view context menu actions are only displayed if the server has at least one active tunnel.

Creating Database Connections

Using the service tunnel information, database connections can then be created with the Eclipse Data Source Explorer, which is available by default in STS and Eclipse Indigo JEE installations:

Once a database connection is established, the database can be accessed through the Data Source Explorer:

To disconnect the tunnel, all data service connections must be disconnected. Therefore, active data service connections created by tools, such as the Data Source Explorer, must be terminated prior to disconnecting the tunnel from the Services table.

Cloud Foundry Integration for Eclipse 1.1.0 can be installed or updated from the official update site:

http://dist.springsource.com/release/TOOLS/cloudfoundry/

- The Cloud Foundry Team

Don’t have a Cloud Foundry account yet? Signup for free today

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

High Level Features of the UAA

The User Account and Authentication Service (UAA) in Cloud Foundry is responsible for securing the platform services and providing a single sign on for web applications. A previous article introduced the UAA and placed it in the context of the platform, and here we go into a bit more detail and describe the features of the UAA individually:

Centralized Identity Management

Applications that want to act on behalf of a User, for instance to view or push apps to the users Cloud Foundry account, need to authenticate the User against the platform. Since Cloud Foundry traditionally has been secured by username (email) and password, one way to do that would be to ask the User for his credentials and forward them to the Cloud Controller. This is a bad thing for two reasons:

  1. It opens up a security threat. If Users can routinely be tempted to type their passwords into any old application that says it provides a platform-related service, there is nothing to stop such applications from stealing the credentials and using them for their own purposes.
  2. It restricts the platform to always and only supporting username-password authentication. This is architecturally brittle, and also probably not what a lot of Cloud Foundry users want, especially for enterprise installations.

Centralized identity management is the way forward. There should be one place where a User knows he can authenticate safely and not risk giving away secrets to a rogue application. The authentication mechanism should be decided centrally with no dependencies on the applications that consume it. This is the role of the UAA.

Single Sign On

Because it is centralized, the UAA can provide a Single Sign On (SSO) service for applications in the Cloud Foundry platform. The cloudfoundry.com platform has several UI components and where they need to be secure they delegate to the UAA for authentication. Examples are the support site and the Micro Cloud Foundry site. (There are no other components in the core Cloud Foundry open source platform that have a UI, so it’s not really necessary to have an SSO feature until you grow the platform beyond the core use cases.)

The UI components that need SSO use the /userinfo endpoint in the UAA, which is just a regular OAuth2 protected resource (an example of the UAA acting as a Resource Server). Here is an example output from the devuaa instance, where TOKEN has been set already to a valid OAuth2 access token:

$ curl -H "Authorization: Bearer $TOKEN" http://devuaa.cloudfoundry.com/userinfo { "user_id":"fb180054-9cbb-441d-96e7-5c3d65861e9c", "user_name":"marissa", "given_name":"Marissa", "family_name":"Bloggs", "name":"Marissa Bloggs", "email":"marissa@test.org" }

The caller would only get an error response if he tried the /userinfo endpoint without a valid access token, so the caller already believes that the user is authenticated but knows nothing about his identity. What the endpoint is doing is giving the caller (e.g., a web application) enough information about the current user to identify him and to be able to customize responses. For example, when you sign into the support site you will see a link to “My unsolved tickets.”

Applications that are part of the platform can be (and have been in cloudfoundry.com) configured in the UAA to be auto-approved (i.e., not require that the user explicitly approve the access token grant). This makes the user experience a lot nicer, and gives the impression of a single logical platform, concealing the implementation as a set of separate components.

The is no reason that applications deployed on Cloud Foundry, or indeed elsewhere, could not also use the UAA as an SSO service. Another blog article is in the pipeline to show you how to take advantage of that in your own applications.

Delegating Access to Services

Single Sign On is a huge benefit to client applications, but the primary role of the UAA is as an OAuth2 Authorization Server. (As a side note, the SSO feature is implemented as a delegated access to the userinfo service, so that was just a specific case of the more general feature.)

The basic delegation model is covered in the OAuth2 specification. In a nutshell, the Authorization Server grants access tokens to Client applications, but only if the Resource Owner (User) approves the access. An example Resource Server in Cloud Foundry is the Cloud Controller, which manages users’ applications and services. The UAA supports all the core OAuth2 token grant types, but the ones that are most relevant for delegating access are authorization_code, which we visited briefly above, and implicit, which is similar, but less secure and aimed at script clients in the browser rather than webapp clients.

The access tokens are supposed to be opaque to Client applications, but they carry information that is relevant to Resource Servers. In the case of the UAA, the token carries some basic information about the user and some other details that can be used by a Resource Server to limit access. By default, the UAA issues base64 encoded JSON tokens, based on the JSON Web Token (JWT) specification. Here’s an example of a decoded token:

{ "exp": 1341079110, "user_id": "65dddc5d-d566-42ee-88f7-9f0098ee7f45", "user_name": "vcap_tester@vmware.com", "email": "vcap_tester@vmware.com", "scope": [ "uaa.user", "cloud_controller.read", "cloud_controller.write", "password.write", "openid" ], "aud": ["openid","cloud_controller","password"], "client_id": "vmc" }

Hint: Because of the way the token is constructed, if you want to eyeball the contents of a token, a quick and dirty way to do it is to simply base64 decode the whole thing. The result is not parseable, but you can see the contents. A Resource Server that is also a registered Client can also decode a token at the /check_token endpoint (using HTTP Basic authentication). It would do this if it prefers not to use the JWT, or wants more up to data information (an access token can be long lived and the user account data might change).

The token contents are as follows:

  • ‘exp’: expires at (seconds since start of epoch)
  • ‘user_id’: an immutable unique identifier for the user
  • ‘user_name’: the user name that the user authenticated with
  • ‘email’: the primary email address of the user (for legacy Cloud Foundry accounts the email and the user name are the same)
  • ‘scope’: the level of access granted by the user to this token.
  • ‘aud’: the audience of the token. This token was intended for the Cloud Controller and two other resources managed by the UAA itself: “password” and “openid”.
  • ‘client_id’: a unique identifier for the client to whom the token was granted.

The security features of the token rely on a Resource Server denying access to a resource if any of the following conditions applies:

  • the token has expired
  • the audience does not include the Resource Server
  • the scope is insufficient

Valid scope values are determined by the Resource Servers and they are free to interpret them in any way they need. The UAA has some conventions about the scope values it uses that help Resource Servers to pull out useful information from the scopes. The Client is allowed to ask for specific scopes when it acquires the token, but the UAA limits the values that are available to individual clients. For example, the “vmc” client that was granted the token above is not allowed to acquire a token that has scope outside the values shown (e.g., it can’t create new user accounts).

Scope values in OAuth2 are arbitrary strings, but in the UAA they normally have a resource id (as in the “aud” field) and some level of access (e.g., “read”, “write” and “admin”) separated by a period. Exceptions are the scope values defined in standard specifications like the “openid” value above which is defined in the OpenID Connect specification and gives the Client application access to the user’s profile data (read only).

The scopes in a token are a combination of values from various sources; what the client asked for, what the client is allowed to ask for, and additional group assignment information appended from the User account (“uaa.user” in the example above, signifying that the user is known to the UAA but not given any special status).

User Account Management

One of the main functions of the UAA is to store user account data, and provide endpoints for provisioning and inspecting user accounts. For these purposes the UAA implements the standard Simple Cloud Identity Management (SCIM) API, exposing an endpoint to authorized clients at /Users, and authenticating them using OAuth2 access tokens. For example:

$ curl -H "Authorization: Bearer $TOKEN" https://devuaa.cloudfoundry.com/Users { [ { "id": "4df2ddcb-6314-495b-b077-e1693f54f6a4"}, { "id": "1ce62b22-ea56-4d62-b300-c7a6c1fd9106"}, { "id": "de3ca343-fe9e-4b61-90b0-0eb97810028a"}, { "id": "a41afc1c-3006-41c9-9d3b-5ebb88f37153"}, { "id": "639d9baa-aaf9-488c-a693-d45c11c39594"}, ... ] } $ curl -H "Authorization: Bearer $TOKEN" https://devuaa.cloudfoundry.com/User/4df2ddcb-6314-495b-b077-e1693f54f6a4 { { "id": "4df2ddcb-6314-495b-b077-e1693f54f6a4", "userName": "marissa", "givenName": "Marissa", "familyName": "Bloggs", "emails": [ { "value": "marissa@test.org"} ], "groups": [ { "externalId": "uaa.user"} ] } }

A Client application with sufficient permission can manage User accounts in its own right. The Cloud Controller does this, as do the new services which are in the pipeline to support future improvements to Cloud Foundry.

Client Application Registration

The UAA keeps a registry of Client applications, and what they are allowed to do, so that it can manage the process of issuing tokens (and in particular deny a token grant if the client is invalid, or if it is asking for a scope that it is not allowed). It does this by exposing service endpoints at /oauth/clients/**, and protects them using OAuth2 token authentication. For example:

$ curl -v -H "Authorization: Bearer $TOKEN" 'https://devuaa.cloudfoundry.com/oauth/clients' { "admin": { "scope": ["uaa.none"], "client_id": "admin", "authorized_grant_types": [ "client_credentials" ], "authorities": ["clients.secret", "clients.read", "clients.write"] }, "app": { "scope": ["openid","cloud_controller.read","cloud_controller.write"], "client_id": "app", "authorized_grant_types": ["authorization_code","password","refresh_token"], "redirect_uri": ["http://localhost"], "authorities": ["uaa.none"] }, ... }

The fields in a client registration are:

  • ‘client_id’: a unique identifier.
  • ‘client_secret’: only provided when the registration is first created, or the secret is being changed. Also if the authorized grant types includes “implicit” a secret is not allowed (implicit clients are untrusted).
  • ‘scope’: the valid scope values that the UAA will allow in a token issued to this client acting on behalf of a User (so not with Client Credentials grant).
  • ‘authorized_grant_types’: the grant types that this client is allowed, as listed in the OAuth2 specification.
  • ‘authorities’: the scope values that can be requested by the client if it is asking for a “client_credentials” token grant. The “app” client in the example above is not authorized to request a “client_credentials” grant, so it has a default value “uaa.none” in this field.
  • ‘redirect_uri’: a list of callback URIs that this client is allowed to ask for. Any client that needs implicit or authorization code grants must provide at least one callback URI. The UAA allows it to ask for a callback that start with the value provided in the registration, so it can redirect to multiple places in the same application but only register one callback.
  • Additional fields (primitives or JSON objects) are also stored. The UAA uses a field ‘client_name’ to provide a user-friendly name for the client on the token grant approval page.

As things stand in cloudfoundry.com, if you have an application that you would like to register as a Client application, you should contact the Product Management team via the support site. If you want to provide a Resource Server that accepts OAuth2 tokens granted by the UAA, you should also register the server as a Client application. We are working on providing some user-facing features allowing you to use some of the UAA services including client registration, and those will be announced on the cloudfoundry.com blog site when they are available.

Other UAA Resources

There are some more services provided by the UAA in addition to those described already. Here is a brief list of all the UAA endpoints (with valid scopes if it is an OAuth2 resource):

  • OAuth2 Authorization Server: oauth/authorize and /oauth/token endpoints.
  • User info endpoint (for SSO): /userinfo, scope openid.
  • Token decoding endpoint: /check_token
  • Login info endpoint (open to anyone): /login
  • SCIM user account management: /Users endpoint, scopes [scim.read, scim.write].
  • Password changes: /Users/{id}/password endpoint, scope password.write
  • Token management, e.g. cancelling an approval: /oauth/users/{id}/tokens and /oauth/clients/{id}/tokens, scopes [tokens.read, tokens.write]
  • Client registration: /oauth/clients, scopes [clients.read, clients.write, clients.secret]
  • Password strength meter: /password
  • Management endpoints, used by the Cloud Foundry platform internally: /health and /varz

More detail is available on these and the other services provided by the UAA in the API documentation.

Note: the description of UAA scope values and the precise details of the client registrations are features of the (in progress) 1.1.0 release. You can find the code for this release on the “scopes” branch in GitHub (or on master once 1.1.0 is released). At the time of writing, the devuaa.cloudfoundry.com server is up to date with these features, but uaa.cloudfoundry.com is not.

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Introducing the UAA and Security for Cloud Foundry

Cloud Foundry is a distributed system with many components front and back end. If you are familiar with the Cloud Foundry architecture you have probably noticed that the Cloud Controller exposes its functionality via lightweight HTTP APIs. The internal components also use the same approach to communicate with each other. Up until recently this was done using a custom authentication mechanism which had some drawbacks. This blog post will walk you through the changes that we are making in this area.

We created a new component to handle all external user-facing security concerns named the User Account and Authentication Service or UAA for short. It has been live in cloudfoundry.com since March 2012, and is now starting to play a big role in the interactions between Cloud Foundry users and the platform. This article explains the basic features of the UAA and the role that it plays for users and administrators of Cloud Foundry.

Design Goals of the UAA

  • Web applications need to be able to authenticate users and access platform resources on their behalf without collecting user credentials. (Several examples of applications collecting user credentials have been brought to our attention but until the UAA was available, there was no way to offer an alternative.)
  • The system needs to be able to grow, with more components being added by the platform and by users (for example providing management UIs for existing applications or for deploying new applications or services).
  • It should provide Single Sign On (SSO) for all user-facing applications in the platform.
  • It should support cross-platform and polyglot programming for users of the platform APIs.
  • It should be easy to strategize the authentication mechanism for social (e.g., sign on with GitHub) and enterprise (e.g., corporate AD) scenarios.

Security is not something to skimp on during implementation. One way to be sure that you have the bases covered and to reassure users that their data is safe is to use standard protocols and interfaces because they have been vetted by many security engineers and reviewed by a board of approvers. Open standards tend to get a large community behind the implementation and documentation of client libraries and utilities, so they make it much easier to support a wide range of users. For that reason, we chose to use some of the latest and greatest open standards in the UAA, the most important of which is OAuth2. Also worth mentioning are SCIM and OpenID Connect (the latter spec is still unstable so we haven’t fully implemented it yet).

Quick Introduction to OAuth2

OAuth2 is a protocol enabling a Client application, often a web application, to act on behalf of a User, but with the User’s permission. The actions a Client is allowed to perform are carried out on a Resource Server (another web application or web service), and the User approves the actions by telling an Authorization Server that he trusts the Client to do what it is asking. Clients can also act as themselves (not on behalf of a User) if they are permitted to do so by the Authorization Server.

The Cloud Foundry UAA is an Authorization Server and also a Resource Server (e.g. for resources to do with user accounts). Another Resource Server in Cloud Foundry is the Cloud Controller, which is the main interface for managing user’s applications and services. There are some more detailed descriptions of authentication and authorization flows, including sequence diagrams, later in this article.

UAA Quick Start

The quickest way to see the UAA in action is to point your browser at http://uaa.cloudfoundry.com. If you have a Cloud Foundry account you can login and see a basic home page just confirming that you are authenticated. Log out by clicking on the link at the top right.

On a command line you can also interrogate the UAA a bit to see what it is doing:

$ curl -H "Accept: application/json" uaa.cloudfoundry.com/login { "timestamp":"2012-06-21T10:52:35-0700", "commit_id":"b19eb89", "prompts":{ "username":["text","Username"], "password":["password","Password"] } }

The response tells you when the UAA was built, the git commit id from GitHub and the information it needs at its login endpoint to authenticate a user.

The code is also deployed with its default configuration as a Cloud Foundry app called devuaa: http://devuaa.cloudfoundry.com. You can use this deployment for testing if you don’t want to deploy it yourself. Out of the box, the UAA runs with an in-memory back end, and one user account (username: marissa, password: koala). To build the server yourself and see it working locally you can clone the project and follow the instructions in the README on GitHub.

The normal flow for a web application wishing to act on behalf of a user is the Authorization Code Flow. Here is a sequence diagram of the interactions when the user is already authenticated:

Sequence diagram, auth-code-flow, with an already authenticated user (steps 2-4)

Out of the box, the UAA also has an OAuth2 client called app, so you can see part of the OAuth2 Authorization Code Flow by visiting this link in your browser. Log in as marissa and you might see the user approval page where marissa confirms that she allows the Client known as ‘app’ to access her profile data (scope=openid).

You might not see that page because someone else may already have been there and approved the access and obtained an access token, so the server will not need to prompt again. The access token is a memo for the approval, so if no access token was granted, then you will see the approval page. Approve the grant by pressing the “Authorize” button and you should see the browser bounce back to http://localhost/nolink (intentionally non-existent) with a query parameter containing an authorization code.

Getting an access token is another step (which the Client application would normally take), exchanging the authorization code for the access token on the Token Endpoint at https://devuaa.cloudfoundry.com/oauth/token. There is no UI for the Token Endpoint, so nothing to see in the browser, but if you visit that location you should be prompted for authentication details because the Token Endpoint uses HTTP Basic to authenticate Client applications.

The command line client that most people use with Cloud Foundry is vmc. For historic and practical reasons, vmc collects user credentials directly on the command line, so it’s not a browser flow. The UAA advertises what credentials it requires on its /login endpoint, and vmc collects the data and sends it, getting a token in return. OAuth2 has a Resource Owner Password grant type that could have been used for the legacy Cloud Foundry, but is too restrictive for future purposes because it is limited to username-password authentication. Instead of using that, we used the Implicit Flow with a couple enhancements. This is how it works:

Sequence diagram, vmc-implicit-flow:

For a detailed description of all the features of the UAA in version 1.0 see the UAA Reference Blog Post.

Login Server

The UAA can and does handle authentication of users with Cloud Foundry accounts. It only knows how to do username-password authentication, and only for registered users, which is fine for basic use cases, e.g., for vmc users who are accustomed to logging in with a password. To go beyond that, for instance to allow users to authenticate with social network accounts, or with an external identity provider like VMware Horizon, or an enterprise directory, you have two choices. Either 1) fork the UAA and modify the authentication filters and login UI, or 2) deploy another component that can handle authentication and delegate everything else to the UAA. We call 2) a Login Server, and that is the design we will be adopting soon in cloudfoundry.com. There are two sample apps (one in Java and one in Ruby) that show you how to implement a Login Server in the samples directory of the UAA.

Conclusion

The UAA handles all the user-facing security concerns in Cloud Foundry, and also provides some back-end services for other components in the platform, as well as some that can be used by a user’s own applications. Its primary role is as an OAuth2 Authorization Server, but many other services are provided in connection with user accounts and client registrations. We will soon be providing more detailed examples of how to use the UAA in the form of some more articles on the cloudfoundry.org and cloudfoundry.com blog sites. The range of services supported by the UAA and its registered clients is also expanding so watch out for blogs on new features.

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Cloud Foundry and Open PaaS at OSCON

The Cloud Foundry team is at the premier Open Source conference, O’Reilly OSCON this week. What a difference a year makes! This time last year, the Cloud Foundry Open Source project and the idea of an open Platform as a Service (and PaaS generally) was a new concept to many developers. This year, there has been a clear buzz around clouds – particularly open cloud platforms –and sessions on these topics have been very busy.

Cloud Foundry had a number of sessions: Josh Long has been as popular as ever, talking about Spring and Cloud Foundry topics; Raja Rao DV delivered a masterclass in node.js on Cloud Foundry; and I hosted a birds-of-a-feather group to discuss the idea of messaging and integration in the cloud using technologies like RabbitMQ. We also had some of the developers talking about and demonstrating BOSH, the deployment tool that was released to the community a couple of months ago. The exciting part here was that Vadim Spivak and Oleg Shaldybin showed BOSH deploying a full Cloud Foundry environment onto PistonCloud’s OpenStack-based cloud!

Deploying Cloud Foundry to PistonCloud

The SpringSource and Cloud Foundry expo hall booth has been busy, too. Everyone picked up a Micro Cloud Foundry USB stick in their conference schwag bags. Lots of people stopped by to show us their apps running on CloudFoundry.com and had the opportunity to earn a great hoodie.

The Cloud Foundry and SpringSource booth

For me as a Developer Advocate, it has also been fantastic to meet up with many of our partners in the Cloud Foundry ecosystem to see where they are taking their own offerings. I had the pleasure of spending time with Adron Hall talking about Tier 3Iron Foundry, and all kinds of other topics around the evolution of the Cloud Foundry community. If you are not familiar with Iron Foundry, it’s an exciting project that extends Cloud Foundry to the .NET environment.

Our partners over at AppFog brought some “No Vendor Lock-in” placards to the show, and handed out free ice cream outside the front of the Oregon Convention Center. Now, the draw of ice cream is always a good one, but for me it was a great opportunity to chat with the team and talk about what they are up to. AppFog are based locally in Portland, so it was nice to get to see them on “home turf.” I’m a big fan of their nice web console and the fact that they offer EC2, OpenStack and other options for deployment. AppFog have also been running a contest enabling developers to wish the OpenStack project a Happy Birthday, which looks like a lot of fun.

The ActiveState team are here too talking about Stackato, their private enterprise PaaS based on Cloud Foundry. A few months ago we posted that FeedHenry can deploy mobile apps to CloudFoundry.com, but some organisations might prefer to deploy those apps behind an enterprise firewall. Diane Mueller showed me a great example of Stackato and FeedHenry working together, delivering secure mobile webapps (see the sneak peek in the screenshot here).

Apart from all of these Cloud Foundry-based ecosystem partners, it has also been great to chat with our friends on the team at MongoDB, and get feedback from developers at the sessions and on the show floor about what they thought of Cloud Foundry and how I can help them move forward to a PaaS environment.

What a great event. I hope to be back here next year!

Watch our Facebook and Google+ pages for peeks at images from OSCON 2012.

Andy Piper, Cloud Foundry Team

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Using Cloud Foundry to Create the #ContributingCode Competition App

A situational application generally has a short lifespan and sidesteps formal requirements so that it can be developed quickly with a narrow focus. Cloud-based application development and deployment platforms lend themselves very well for such applications. This blog post will demonstrate a situational application that I developed and guide you to use cloud platform services to aid in your web development.

This app was built to register participants for a VMware Service Foundation pilot project called #ContributingCode, a program that involves an intense coding competition for students that is set to begin July 13, 2012. While there are several existing web applications available to do basic registration functionality, my custom application had additional requirements to provide:

  • Seamless user experience without having to navigate across sites
  • Flexibility in the registration set up
  • A highly customizable user interface

I developed this app to be very generic so it can be used for other events by just changing the content and CSS as required.

Requirements

A web application for handling registrations where users can create teams or join existing ones. Specifically, the owner of the team has the ability to:

  • Add or delete members on the team
  • Provide the ability for members to leave a team
  • Send email notifications for registration, add requests and any modification to the team
  • Send team announcements and view team summary using an admin panel

Application Architecture

Application Design

My application architecture called for an array of data services to be accessible by a core and standalone worker app. I considered using an IaaS like Amazon EC2, but realized I would have to take a raw Ubuntu Amazon Machine Image (AMI) and install Ruby, Rails, MySQL, Redis, MongoDB and then configure all of them to communicate with both the core and standalone worker application. Noting the amount of time, effort and possibility of errors involved in that approach, I chose to use Cloud Foundry PaaS. All the above steps could either be avoided or performed within seconds. Having such a platform with an array of data services and frameworks, supporting auto reconfiguration, and an NGNIX web server made my job far faster and easier.

The application is a Rails 3.2.6 app using MySQL for storing data, Mongo GridFS to store images, SendGrid for email delivery and Redis for Resque operations. There is a main app and a standalone worker app to perform the Resque operations in the background. The applications run on CloudFoundry.com and are bound to MySQL and Redis service instances provided by CloudFoundry.com. Using the services provided by CloudFoundry.com means that we can leverage Cloud Foundry’s ability to auto-reconfigure service bindings automatically.The reasoning behind this and the broader system environment is explained below.

Choice of Application Framework

The Rails framework includes everything I needed to create a database-backed web application using the
Model-View-Controller (MVC) pattern. Also the ActiveRecord, the base for the models in a Rails application provides database independence, basic CRUD functionality, advanced query capabilities, and the ability to relate models to one another. ActionMailer is a component of Rails which provides a framework for email services. I also leverage a number of existing Ruby gems for this web application to perform various tasks.

Background Processing

A standalone app executes background tasks using Redis and Resque to email the users. This was primarily done to improve the user experience by sending the emails lazily in the background and responding to post requests as soon as the database entries have been made. For example, while registering as a new user, we need to send in a confirmation email which delays the response that the server has to send to the client as it takes a while to send the email. So when the job of sending an email is queued to Redis and later processed by the worker app the response is instantaneous. To learn how to run Resque workers on Cloud Foundry refer to this blog and there is more on Resque and Redis here.

Communications

We use the SendGrid email delivery system to deliver the email. There is clear documentation on how to use SendGrid with Rails here. In order to make the emails richer for our customization needs, we used HTML layouts. The notifier.rb file shows how we can include the layout in the email.

def register_email(name, email)
    @name = name
    mail( :to             => email, 
          :subject       =>  "Thank you for registering for contributingcode",
          :template_path  => 'mailer',
          :template_name  => 'register' )
  end

Data Storage

Relational persistance made the most sense for most of the data on this application. Each logical model, and operations on the model, are well defined. The models for this app are quite straightforward; User, Team, Member (which associates the users to the teams they belong to), JoinRequest, and AddRequest. The MySQL database was picked for its popularity. Additionally, we thought that a team avatar option would make the app more user friendly, so we extended the storage services to include MongoDB to use GridFS. The carrierwave gem for uploading the images and storing them with MongoLab and GridFS using the mongoid gem. Setting this up requires reading environment variables on application startup. This can be accomplished by creating a file in the initializer folder as seen here.

Single Sign On via GitHub

Github can be used for Single Sign On into apps using OAuth2 and REST APIs. OAuth2 is a protocol that lets third party clients or apps request authorization to protected resources, such as a user’s profile data, in a particular service without exchanging sensitive usernames and passwords. Instead, it uses interactions via the browser to obtain a code which is then exchanged for an access token and can be revoked by the service at any point in time. Think of the access token as a valet key which gives access to a subset of features. This is preferred over Basic Authentication, because tokens can be limited to specific types of data and can be revoked by users at any time. Specifically, we used omniauth-github gem for authentication through Github. This reduces the burden of having a custom login system. More on OAuth2 can be found here. Github was used for authentication as the participants of our code competition are required to have an account to submit the code to Github. More importantly, the authentication is done by Github and the app does not store any passwords. This is where we may encounter a drawback to OAuth2, which is the unavailability of the service you use for authentication, e.g., Github in our case, will in turn affect the users’ ability to log into your app.

User Experience

The use of modal box and having a single page app improves the user experience as the user is never more than two clicks away from the target section.

Twitter Bootstrap was used to enhance the look and feel by utilizing the type-ahead feature for autocomplete functionality and carousel was used to display the teams and contributors. The front-end form validator, native jquery plugin, validationhelper is written  as a wrapper around validate.js  to validate forms in the frontend with the help of Twitter Bootstrap. This was made as a separate open source code and can be used irrespective of the backend framework. Using front end validation reduces unwanted server requests. For example, the email entered while registering can be validated before the user registration form is submitted to the server.

Deploying the App on CloudFoundry.com

Deployment was the easiest step of all and took just seconds to push the app to production. Here are the steps you can follow to push this app to Cloud Foundry using your credentials. Fork the project.

git clone git@github.com:/contributingcode.git contributingcode
cd contributingcode
bundle install;bundle package

Before precompiling the assets make sure mongo is running in the local and set the following environmental variables.

export mysql_pwd='your mysql password'
export github_client_id= 'github client id'
export github_client_secret= 'github client secret'
rake assets:precompile

Start the Core app.

$ vmc push --runtime ruby19 --nostart
Would you like to deploy from the current directory? [Yn]: y
Application Name: contributingcode
Detected a Rails Application, is this correct? [Yn]: y
Application Deployed URL [contributingcod.cloudfoundry.com]: y
Memory reservation (128M, 256M, 512M, 1G, 2G) [256M]: 256M
How many instances? [1]: 1
Create services to bind to 'contributingcode'? [yN]: y
1: mongodb
2: mysql
3: postgresql
4: rabbitmq
5: redis
What kind of service?: 2
Specify the name of the service [mysql-3e25d]: mydb
Create another? [yN]: y
1: mongodb
2: mysql
3: postgresql
4: rabbitmq
5: redis
What kind of service?: 5
Specify the name of the service [redis-2a4c2]: myque
Create another? [yN]: n
Would you like to save this configuration? [yN]: y
Manifest written to manifest.yml.
Creating Application: OK
Creating Service [mydb]: OK
Binding Service [mydb]: OK
Creating Service [myque]: OK
Binding Service [myque]: OK
Uploading Application:
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (558K): OK   
Push Status: OK

Start the worker app. Rename the manifest file if you have one.

vmc push ccworker  --nostart
Would you like to deploy from the current directory? [Yn]: y
Detected a Rails Application, is this correct? [Yn]: n
1: Rails
2: Spring
3: Grails
4: Lift
5: JavaWeb
6: Standalone
7: Sinatra
8: Node
9: Rack
10: Play
Select Application Type: 6
Selected Standalone Application
1: java
2: node
3: node06
4: ruby18
5: ruby19
Select Runtime [ruby18]: 5
Selected ruby19
Start Command: bundle exec rake VERBOSE=true QUEUE="*" resque:work    
Application Deployed URL [None]: 
Memory reservation (128M, 256M, 512M, 1G, 2G) [128M]: 128M
How many instances? [1]: 1
Bind existing services to 'ccworker'? [yN]: y
1: mydb
2: myque
Which one?: 1
Bind another? [yN]: y
1: mydb
2: myque
Which one?: 2
Create services to bind to 'ccworker'? [yN]: n
Would you like to save this configuration? [yN]: y
Manifest written to manifest.yml.
Creating Application: OK
Binding Service [mydb]: OK
Binding Service [myque]: OK
Uploading Application:
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (39K): OK   
Push Status: OK

Set the configuration. Set the Github and Mongolab credentials for core app.

vmc env-add contributingcode github_client_id= 'github client id'
vmc env-add contributingcode github_client_secret= 'github client secret'
vmc env-add contributingcode mongodb_host= 'host'
vmc env-add contributingcode mongodb_port= 'port'
vmc env-add contributingcode mongodb_username= 'username'
vmc env-add contributingcode mongodb_password= 'password'
vmc env-add contributingcode mongodb_db= 'db name'

Similarly, set the environmental variables to the worker app too.

The standalone worker app requires the same environment as the core app and thus all the environment variables should be set for the worker app as well. In addition, set the SendGrid credentials.

vmc env-add ccworker github_client_id= 'github client id'
vmc env-add ccworker github_client_secret= 'github client secret'
vmc env-add ccworker mongodb_host= 'host'
vmc env-add ccworker mongodb_port= 'port'
vmc env-add ccworker mongodb_username= 'username'
vmc env-add ccworker mongodb_password= 'password'
vmc env-add ccworker mongodb_db= 'db name'
vmc env-add ccworker sendgrid_username= 'sendgrid username'
vmc env-add ccworker sendgrid_password= 'sendgrid_password'

While running on localhost set mysql_pwd as follows:

export mysql\_pwd='mysql password'

Finally to start your app on CloudFoundry.com.

vmc app start contributingcode
vmc app start ccworker

Visit contributingcode.cloudfoundry.com to view my  web application. Your application should look similar, but as the core app and worker app names are subject to availability, your app name will be different from ‘contributingcode’.

To get the app running, fork the app from here.

The Right Tools For the Job

The bootstrapping of the app was done in one day with the basic registration system in place. Relying on some expert website mockups, I had all the functionality and HTML in place in just a couple of weeks. With some further testing and fine tuning, the app was easily deployed to CloudFoundry.com and open for registrations. While, the website needed several updates with new content and sections (e.g. FAQ’s) it was all done with barely any downtime. Using a single Cloud Foundry command “vmc update” reduced the overhead in updating the website in multiple iterations to one step. This confirmed that we had made the right choice in using a PaaS rather than standing up our own infrastructure. This registration app has registered over 100 students, helping them to connect with each other and learn about the theme of the competition. I hope this blog helps you jumpstart your own web application development as well.

– Magizharasu Thirunavukkarasu, Developer Advocate Intern

Don’t have a Cloud Foundry account yet?  Sign up for free today

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email