Cloud Foundry Blog

Continuous Integration to Cloud Foundry.com Using Jenkins in the Cloud

Posted on February 28, 2013 by Mark Prichard

Continuous integration using Jenkins is increasingly seen as an effective tool for reducing the cycle time from product backlog to receiving actual user feedback. This can result in real increases in developer and team productivity when combined with an Open PaaS such as Cloud Foundry.

In this guest blog, Mark Prichard, Senior Director of Product Management for CloudBees®, explains how to create a fully automated environment for application build, test and deployment to Cloud Foundry.com using Jenkins in the Cloud via the CloudBees DEV@cloud™ build service.

Spring Framework is the most widely adopted development model for Java-based enterprise applications, used by millions of developers for its powerful abstractions and declarative support for crucial application infrastructure concepts. Jenkins is the world’s number one open-source continuous integration (CI) platform, with deep roots in the Java community and 60,000 installations worldwide. These two technologies are part of the fabric that has made Java a highly productive, agile programming environment for business applications. With the rise of cloud computing and Platform as a Service (PaaS), developers want CI services to build, test and deploy their Spring applications entirely in the cloud, using elastic, on-demand resources. You can now use Jenkins via CloudBees DEV@cloud to continuously deploy to Cloud Foundry.com.

Usage of Continuous Integration and Delivery for Release Management

In our most recent annual Jenkins CI user survey, a couple of facts stood out: first, 74% of users are building Java applications with Jenkins (although we also see significant development in C/C++, Javascript, Python, C#, PHP, Ruby, Scala and Groovy); and second, we are seeing a really big up-tick in usage by large organizations, with 60% growth among the very largest group. Overall, the number of installations is up 66% in the last year, with over 83% of those surveyed using Jenkins for what they consider mission-critical development. Looking at those results, it seems very clear that many of those large, mission-critical applications are using Java with Spring Framework – with CI provided by Jenkins.

Another key trend that is growing in importance daily is Continuous Delivery. More and more organizations are looking to embrace an agile model in which stringent, automated testing allows enhancements or “micro-releases” to go live without the traditional waterfall release cycles. We are seeing a major shift in enterprise software development to cloud-based, continuous delivery, with fully automated quality, coverage, functional and performance tests gating live deployments. This is the new best practice, and it is now available for Cloud Foundry development.

In this blog post, I’m going to show you how easy it is to set up a CI job using Jenkins via CloudBees DEV@cloud service to automatically build, test and deploy a rich Spring Framework application to Cloud Foundry.com.

Using Jenkins in the Cloud to Continuously Deploy Spring Apps to Cloud Foundry

Here’s an overview of the process:

Link your Cloud Foundry and CloudBees accounts using OAuth 2.0 for secure and automatic deployment
Clone a Git repo and set up a Jenkins job to build automatically when changes are pushed (we’ve provided ClickStarts™ that automatically do these tasks for you)
Set up and configure Jenkins Cloud Foundry deployment plugin to push your application to Cloud Foundry if your build succeeds

That’s it: your application and its services are now running live on Cloud Foundry!

Let’s take a quick look at all this cool stuff. First, we need a way to allow secure access from your CloudBees DEV@cloud account to the Cloud Foundry deployment services. CloudBees and Cloud Foundry both support the industry-standard OAuth 2.0 protocol, allowing you to establish a trust relationship between your accounts without the need for either party to store account details like passwords from the other service. Go to cloudfoundry.cloudbees.com, which will redirect you to the Cloud Foundry log in. Log in and authorize your CloudBees account to deploy to your Cloud Foundry account.

Next we want to set up a build job. That’s really easy, even if you’ve never used Jenkins before: we’ve set up ClickStarts that will clone private Git repositories with a couple of fully-featured Cloud Foundry/SpringSource examples from GitHub (springmvc-hibernate and petclinic-grails) and then create Jenkins jobs for those builds. From the Cloud Foundry ClickStarts launch page all you need to do is click on the Spring icon, enter the name you want to use for your build and in a few seconds you’ll be taken to the Jenkins build job, which will start automatically as soon as the repository is available.

At this point, you have everything set up: a private Git repo to use for development, a Jenkins CI build job and automatic deployment to Cloud Foundry. The build job has been configured to use your CloudBees account name as part of the Deployment Hostname (e.g. springmvc-hibernate-< youraccount>.cloudfoundry.com), but of course you can change that if you like. You can now clone a local copy of the source code repository (click on the Repositories tab on the toolbar for full details) and you have a fully automated, cloud-based develop-build-test-deploy Continuous Delivery environment for your Spring/Grails applications. Every time you push an upstream commit, Jenkins will run a complete build/test and deploy to Cloud Foundry.

As you can see by browsing the console output for the build job, this is a substantial application that uses many aspects of Spring Framework, including Hibernate for the mapping between the Java application and two data services, Redis and PostgreSQL, that also need to be provisioned on Cloud Foundry.com and bound to the application. That is all done as part of the deployment; you can see the details in the Services section of the host service configuration.

Once the build and deployment are complete, you can go into the console output for the job and see all the details of the deployment. If you have vmc set up on your workstation, you can use vmc apps and vmc services to verify those deployments, like this:

All you have to do now is browse to those URLs and you’re up and running with Spring/Grails and Jenkins in the cloud – enjoy!

Summary

Cloud Foundry together with CloudBees Jenkins in the Cloud service give you a complete Continuous Deployment solution for your enterprise Spring Framework and Grails projects:

You have access to all the capabilities of a fully managed Jenkins as a service – you don’t have to set up Jenkins or the build systems.
You can use Jenkins to deploy your Java Spring applications seamlessly to Cloud Foundry.
You always have enough build capacity — CloudBees dynamically adds more build machines as you need them.
It’s free to get started.

Watch the video: Continuous Integration to Cloud Foundry.com Using Jenkins in the Cloud
Docs: https://developer.cloudbees.com/bin/view/DEV/DeployAppsToCloudFoundry
Give it a try: cloudfoundry.cloudbees.com

About the Author: Mark Prichard, Senior Director of Product Management for CloudBees, speaks and blogs regularly as an evangelist for Platform as a Service. He came to CloudBees after 13 years at BEA Systems and Oracle, where he was product manager for the WebLogic Platform. A graduate of St John’s College, Cambridge and the Cambridge University Computer Laboratory, Mark works in Los Altos, CA.

Open Standards in Cloud Foundry Identity Services

Posted on February 19, 2013 by Dale Olds

Cloud Foundry includes a component called the User Account and Authentication (UAA) service which was introduced about a year ago. It provides user account management and authentication for developers that push applications to cloudfoundry.com, including single signon with support.cloudfoundry.com, micro.cloudfoundry.com, and more on the way. It also provides delegated authorization capabilities to partner sites so that they can interact with Cloud Foundry on a user’s behalf without access to user credentials.

An introduction and details of the UAA from a developer perspective can be found in earlier blog posts. In this post I will focus on the UAA’s integration of three emerging identity standards and what that implies for some specific Cloud Foundry deployment patterns.

Goals

The goal of the UAA is to make it easier, simpler, more secure and more convenient for users to get to the features of Cloud Foundry. The value is in the open PaaS and the applications it enables. So the identity services provided by the UAA should not only be useful for cloudfoundry.com and applications on it, but for pure open source deployments, managed private clouds, and even the micro cloud running on a thumb drive.

To achieve this goal, we have implemented and integrated new standards in a way that supports various deployment scenarios, such as multiple identity providers and external authentication mechanisms.

Fusing Identity Standards for a Stronger Platform

We have woven identity services into the platform using three standard protocols: OAuth2 for delegated authorization, OpenID Connect for session and authentication information, SCIM for user and group management. Of course we provide support for other standards as well, such as JSON Web tokens (JWT), SAML2, and OpenID 2.0. Even support for the legacy identity stalwart LDAP is in process. But what is particularly interesting is how we have combined the three primary protocols.

OAuth2 — this is the foundational protocol and provides delegated authorization. It provides a way for web applications to ask users to allow access to cloudfoundry on the user’s behalf. It provides authentication and SSO as a side effect. The app gets one or more secure tokens — some that the app can use to identify the user, and some that it can present to the cloudfoundry API to act on behalf of the user. But OAuth2 does not specify the token format. We chose another standard, JWT, for that.
OpenID Connect — provides identity information for the authenticated user. OAuth2 does not specify how the user’s information can be obtained by an app, just an opaque token. OpenID connect adds the ability for an app to efficiently and safely get authenticated user attributes like user name or email address.
SCIM — User and group provisioning. User account management is out of scope for OAuth2, but SCIM specifies simple user provisioning as well as groups, and it’s extensible to new types. Since OAuth2 requires that web applications preregister with the authorization server (UAA), but does not specify how these registrations are done, we use a SCIM extension for that.

Imagine two web applications: – one manages users and groups for access to cloudfoundry.com – one displays results of monitors a user’s running applications as a service — like our partner Appsecute provides.

You could diagram the interactions of the apps and three protocols like this:

The three protocols complement each other well so far, but there is a piece still missing.

Connecting the Missing Link

For delegated authorization to work, the user needs a unit of delegation — some handle so they can say: I want to allow application X to read my application list on cloudfoundry.com, but not actually modify anything. This notion of specifying names for roles or permissions is called the token ‘scope’ in OAuth2. We needed some way to support management of these scopes, their existence, what users have access to them, what web applications can ask for them — and do it all in a way that naturally supports their use in authorization decisions.

Therefore, we mapped SCIM groups to OAuth2 scopes. In other words, you can create a group in the UAA and register a web app with that group name as a scope. When a user logs into the web app, it’s redirected to the UAA. If the user is in the group and they choose to authorize the web app with that scope, the web app gets a signed token that contains that scope.

We use this in our own cloudfoundry dashboard application. The system reliability engineers access the dashboard app, which is a UAA client app and always requests a token with dashboard.user scope. The UAA denies the scope unless the user is in the dashboard.user group.

What this gets us is that the dashboard app does not have any user account code, and it never gets the user’s credentials. It merely redirects to the UAA if there is no access token in the authorization header. If there is an access token it validates the token, and then checks if it has dashboard.user in the scope. For the system monitors, this means no need for shared accounts or passwords and better control of which users have access.

This is particularly useful when there are multiple such applications. Users get SSO among the applications, the applications are each more secure, access can be easily controlled by groups in the UAA.

Extensible Authentication Services

A somewhat recent addition is that we have separated the authentication service from the rest of the UAA. In OAuth2 terms this is separating the authorize and token endpoints. We can have different implementations of the authentication service for different needs. So far implementations are available for username and password (on cloudfoundry.com), for various flavors of openid2, and for SAML2 identity providers.

Why this is Cool

This is all quite useful, but how it can be used in a cloudfoundry deployment is best illustrated by some examples:

Suppose cloudfoundry is running in a managed instance for an enterprise. The enterprise has a suite of apps that can use the full advantage of the PaaS — easy deployment, scale up, scale down, with a myriad of frameworks and services. All the apps can use the UAA at the core for SSO and role (group) based authorization. Add a user to a group via SCIM, and the corresponding scope will be available at next login to all apps.

Cloudfoundry.com includes various web apps to help developers. Some of them only need to know who the user is for SSO. The openid scope is perfect for them. Suppose a new site wants to also provide personalized content based on the user’s running apps. We can create a registration for that app that restricts its scope to listing the user’s apps. If that app is compromised, nothing in the data of the user’s apps can be changed.

Suppose you are a developer running a private cloud in an enterprise. With Cloud Foundry you get these features:

You can create and deploy applications exactly the same as apps on cloudfoundry.com with access to the multitude of frameworks and services, exactly the same as apps on cloudfoundry.com.
Your apps can register with the UAA in your cloudfoundry instance, which means they don’t have to manage users or authentication. All apps in your instance can have SSO and common groups for authorization.
Further, you can configure the UAA in your instance to include a SAML Login server, which gets user authentication from the corporate SAML IdP in front of an Active Directory instance. All apps now have SSO with the corporate IdP.
Eventually you’d like a user’s AD group membership for be available to your app as an OAuth2 scope. Put the group membership in the SAML assertion and it can be mapped to a scope by the SAML Login Server — this part isn’t actually implemented yet, but is here to show future possibilities.

Cloud Foundry identity services are built on open standards and open source. They increase security and reduce friction so users can take advantage of the open PaaS.

NTT Communications, World’s Leading Telecom, Joins Cloud Foundry Core

Posted on February 13, 2013 by Hideki Kurihara

As the world’s leading telecom, NTT Communications has experience delivering global cloud services to enterprise customers. Enterprises in Japan are seeking the openness, choice, agility and extensibility that an Open PaaS can provide.

In this guest blog, Hideki Kurihara, Product Lead for NTT Communications’ Global Cloud Services, explains why NTT Communications selected Cloud Foundry as the basis for its CloudⁿPaaS and why it chose to join Cloud Foundry Core.

Test NTT’s CloudⁿPaaS for Cloud
Foundry Core Compatibility here.

We are excited to join the Cloud Foundry Core community as proof of our commitment to making cloud portability a reality for Japanese developers. As the world’s leading telecom, everyday we see customers interested in using PaaS to be more agile. But we also hear concerns about vendor lock-in and ability to meet the needs of a complex enterprise environment. We chose to build CloudⁿPaaS on top of Cloud Foundry because of its multi-cloud nature, ability to integrate with existing assets, and solid API foundation for adding management and monitoring features. Using Cloud Foundry as the base, we are extending CloudⁿPaaS for developers and enterprise customers in Japan. Together with other Cloud Foundry Core partners, we are delivering cloud portability to Japanese users as well as global users of Cloud Foundry.

The Drivers for Open PaaS in Japan

The customer community in Japan wants more agility, choice, extensibility and manageability when it comes to cloud. Customers across sectors are looking to innovate faster in response to a very dynamic environment. For example, cloud has been used extensively for emergency infrastructure services in response to recent natural disasters. There has been a sharp increase in demand for cloud services to support smartphone and tablet application development. More global businesses have started considering cloud services as part of their global IT standardization so applications can be ported on demand from one place to another.

Customers here also want choice – choice to add new frameworks and services, and even to port their applications back on-premise or to another provider in Japan and other countries. Enterprises tend to prefer customization to augment their business. But they also value industry or open standards in the services they consume so they can deploy their application assets in the most appropriate place. When it comes to PaaS, application portability needs to be taken more seriously. Open PaaS, that is PaaS based on an open or industry standard, is what customers are talking about here.

Choosing and Building a PaaS Around Cloud Foundry

Meeting the needs of developers and operations for a PaaS that is open, extensible, and easily managed in the enterprise environment is difficult. We chose to build CloudⁿPaaS on top of Cloud Foundry specifically because of its:

1) Openness and portability

2) Flexibility – in configuration and ability to integrate with existing enterprise assets and services

3) Extensibility – the ability to add frameworks and services, but even beyond that, to build UI based tools that make it easy to manage and add resources to applications and navigate very large installations

We’ve relied heavily on the open source nature of Cloud Foundry. We have added memcached, our own filesystem service, and support for Java web application servers (Resin/Resin Pro) so our customers’ existing applications can work seamlessly on CloudⁿPaaS. But Cloud Foundry Core provides our customers’ developers the choice to use a common floor of runtimes and services when programming their applications. This gives them assurance they can port their applications to a Cloud Foundry Core provider in another geography not served by NTT communications or to a Cloud Foundry instance in their data centers.

Once in their own data centers, customers typically have a heterogeneous mix of infrastructure on which they wish to deploy Cloud Foundry, depending on the SLA and tenancy models. We initially deployed Cloud Foundry on top of our Enterprise Cloud services platform using vSphere and vCloud Director. We use this instance as our own Cloud Foundry development environment and plan to use a similar environment for future enterprise/private PaaS. On the other hand, the public multi-tenant instance of CloudⁿPaaS has been deployed on a different cloud platform with attributes and services more tailored for developer centric markets.

Cloud Foundry is designed with sufficient flexibility to satisfy a huge variety of needs for PaaS environments. The Cloud Foundry system consists of several components, and users can set up their environment by choosing the necessary combination and number of components for their needs Fig. 1

Cloud Foundry supports a wide range of environments—from a private PaaS on a single server to a huge public PaaS on a cluster of over a thousand servers—and can be set up to suit the scale as well as its reliability and support features. Moreover, users can add components on demand, so it is possible to start with the minimum configuration and increase the scale as the load grows.

Enterprise integration is one key area where we see area for adding value to our customers on top of Cloud Foundry. The Cloud Foundry services gateway component addresses the ability to add enterprise data services that our customers request for HA, recovery, and backup. The service gateway itself is nothing more than a REST gateway for binding any service, but the real magic is in the service node implementation that allows multi-node deployments for HA and clustering. We are working to create our own multi-node deployments of services using this component such as the aforementioned filesystem and memcached services.

NTT Communications’ Customizations of Cloud Foundry for Enterprises in Japan

Although Cloud Foundry is a promising PaaS software solution that has many good features, as described above, it is still not perfect. For example, integration with other NTT Communications’ services was naturally not provided. Therefore, we have been making efforts to extend Cloud Foundry to meet the specific needs of our enterprise customers and developers in Japan.

Contributions and Customizations to Cloud Foundry

(1) Reliability of Cloud Foundry components

Because Cloud Foundry was a very young project when we started our development, NTT Communications and NTT R&D (Software Innovation Center) have been examining the performance and scalability of Cloud Foundry by conducting various tests and fixing any problems revealed by them. For example, we solved a problem involving an important component that was a single point of failure and a problem where some components could not be restored after failures by fixing the source code and adding additional external systems.

(2) Convenience of Cloud Foundry

Since convenience is important for commercial services, we created an installer for the virtual machine container (VMC), which is the console for PaaS users, and we are now developing a function for linking the VMC and version control systems such as Git.

(3) Control Panel for user application management

Although Cloud Foundry has a flexible component system as described above, it is essential to provide a more user-friendly environment to fully leverage its features. Therefore, we have developed a control panel that allows users to manage their applications on CloudⁿPaaS. The control panel displays a list of the user’s applications, information about each application, status of resources for each application instance, and each application’s logs that have been stored by means of CloudⁿPaaS services. Using the control panel, operators can flexibly scale-out or scale-up their applications instances. Fig. 2

Building a Cloud Foundry Community in Japan

The presence of active communities is indispensable in the long-term evolution of open source software. We launched the Japan Cloud Foundry Group and are fostering a developers’ and users’ community of Cloud Foundry in Japan. In addition, we will open the source code created by group members by committing them to the official Cloud Foundry repository, and we also provide the know-how in workshops.

About the Author: Hideki Kurihara is the Product Lead for Global Cloud Services, including PaaS, for NTT Communications. Prior to his current position, he was engaged in NTT Communications’ North American hosting and cloud business for 10 years, serving in managerial positions such as General Manager of the Enterprise Hosting business unit and Vice President of Global Product Strategy at its subsidiaries. Hideki holds a Master of Science degree from University of Tokyo in Japan and a Master of Business Administration degree from INSEAD in France.