Cloud Foundry Blog

Want to Contribute to Cloud Foundry? Come on in!

Posted on May 16, 2013 by Andy Piper

Cloud Foundry is an Open Platform-as-a-Service, and an Open Source project. It has attracted phenomenal interest from the community - including partners, companies using the code internally, and those individual developers with a passion for getting involved. You can find the source code on Github. Community contributions are what help to make the platform so extensible. We are always happy when we receive a Github pull request to offer new functionality or fixes! We also appreciate bug reports submitted through Github Issues.

Looking at the Cloud Foundry project as a whole though… where should you start?

As you might imagine, there are a lot of moving parts in a PaaS. At Pivotal, we have teams working on the backend (the main runtime environment for Cloud Foundry, also known as ‘vcap’); the frontend (the tools and APIs for deploying apps, such as the ‘cf’ gem); services (database, message queues, and related resources); and Cloud Foundry BOSH, the installer and deployment tool, which has different plugins targeting infrastructures like AWS, OpenStack and vSphere. In the future, we would love to see even more Cloud Provider Interfaces (CPIs) for BOSH to allow Cloud Foundry to be deployed onto additional clouds.

One of the best ways to get involved with the community and the codebase is to start with the Github repositories and the mailing lists. If you are not sure where particular functionality resides within the project as a whole, and you’ve had a good look at Github already, then we have a number of mailing lists where you can ask for help. vcap-dev is for those working on the PaaS itself; bosh-users is for those who want to understand how to use BOSH to deploy Cloud Foundry; and bosh-dev is for the elite hackers (like Dr Nic Williams!) who want to help us make BOSH even more awesome. If you participate in the mailing lists you will also be able to keep up with what others are working on, and learn who some of the leading developers on the project are. If you want to get an idea of the general direction of development, you’ll find that in our roadmap.

Once you are up-and-running with Cloud Foundry, or if you know to which part of the project you want to contribute, then consider the size of an initial contribution. If you are new to the project you might want to start with something small (some Open Source projects call these kind of small changes to smooth out any rough edges “paper cuts”). If you have something larger, we are always keen to work with you to understand how we can integrate your code. Before sending a pull request, take a look at our OSS contribution information in the project readme, which includes details on the Contributor License Agreement for individuals and corporations.

We thought it would be good to highlight a recent pull request which was a superb example of a major addition to the project. Recently we received a contribution via a detailed Github pull request, adding Oracle support to some of the major components inside Cloud Foundry: the User Account and Authentication service (UAA) and the Cloud Controller. Originally, these components were built to use Postgres as a backend, and more recently MySQL support was added. Since many enterprises already use Oracle as their main relational data store, it is great to give them this additional storage option.

Taking a look at the specific request for the Cloud Controller, we were really struck by a few things. The first one is that the pull request itself is very detailed, and walks through the various changes required in order to replace the database with Oracle.

This pull request is a lot to digest. If you want you can review the single feature PR that lead to this PR for some more concise context into some of the decisions I made in creating this PR. Here is a summary of the more notable changes I had to make and some context for why I made those changes:

This level of detail is not always required with every patch, of course, but for something this extensive it was very welcome. Another aspect of this patch is that was was made of up several individual changes (things like tweaks to the database schema, differences between Oracle datatypes and those used by other databases, etc), and they were broken down into separate commits which made it easier to consume. It also helped that this set of changes was extensively discussed on the vcap-dev mailing list before this larger pull request was submitted, so the engineering team were aware of some of the other impacts it would have.

Although the resulting pull request was detailed and clear, it has taken a few days to make all of the changes, as the core code was still being worked on. This was actually one of the rarer cases where we had to do more work to get the patches integrated – and that shows how willing we are to work with contributions. Smaller changes are extremely welcome too, and often much quicker to digest. Don’t be put off by this larger example – come and start hacking with us!

We hope this post highlights some of the things that help to make it smoother to contribute to the project. To recap:

review the contribution guidelines and complete the CLA;
if you want to submit a major code change, discuss it on the relevant mailing list beforehand;
think about how the pull request is structured, and submit changes in separate consumable commits (rather than as one monolithic change) where appropriate;
the patch should be well-tested, should include tests if possible, and should pass our Travis CI process;
the changes should be clearly documented in the request, especially when the updates are more extensive than correcting a few typos.

The past few months have seen the Cloud Foundry Open Source community grow hugely, and mailing list activity continues to be at an all-time high – so we’re really looking forward to continuing to work with other developers to make the platform even more awesome! Thanks for being on this journey with us.

Andy Piper @andypiper

Developer Advocate, Cloud Foundry

UK Charity Raises Record Donations Powered by Cloud Foundry

Posted on April 30, 2013 by Colin Humphreys

For Comic Relief – a UK based charity, fifty minutes of downtime during peak fundraising hours could result in half their annual online donations income being lost. This blog describes how Cloud Foundry was used to provide an Enterprise grade donations platform that helped Comic Relief raise a record £75million (about US $115 million) in funds for their humanitarian projects.

Guest Blog by Colin Humphreys, CloudCredo, Inc.

Comic Relief is a well known charity based in the UK which strives to create a just world free from poverty. They raise millions of pounds through two big fundraising campaigns – Red Nose Day and Sport Relief. The money raised is spent in the best possible way to tackle the root causes of poverty and social injustice. All the money raised by the public is spent by Comic Relief to help poor and disadvantaged people in the UK and the world’s poorest countries.

Seven Hours to Maximise Funding For The Entire Year

A year of planning for Red Nose Day 2013 resulted in a 6 week media campaign culminating in 7 hours of prime time television coverage on the 15th March. Their donations platform is required to process in the region of 600,000 transactions in this 7 hour period, handle in excess of 10,000 concurrent call centre operators, and handle peaks of 500 donations completing per second from web, mobile and call centers.

These requirements pose an operational nightmare. A fifty minute outage at peak traffic, allowed within an annual “four nines”(99.99%) uptime SLA, could result in 50% of the charity’s annual income being lost. Comic Relief require a platform delivered with no outages; there won’t be a second chance for another year. Feedback on the performance of the platform under high load from real users is only received on an annual basis. To demonstrate the difficulty of this challenge, JustGiving, one of the largest charity-orientated service providers, had a huge outage beginning early on Red Nose Day 2013.

For the previous eight years the donations applications have been delivered via a monolithic Java application and RDBMS backend, deployed and scaled with the help of twelve partners. This setup suffered from single points of failure, high recovery times from potential failures, and a lack of consistency between environments in the deployment pipeline. Comic Relief were also looking to remove their reliance on any single third-party supplier so as to not be limited in their technology choices.

Working Prototype Delivered in a Single Day on CloudFoundry.com

Moving from traditional development methodologies to a lean, agile approach delivered by a small team required a leap of faith from Comic Relief. Their fears were allayed by delivering a working prototype via CloudFoundry.com on the first day of application development. This was critical to receiving constant feedback from Comic Relief, and gave a starting point from which to iterate on the applications and the platform.

The software that drives the ability to take donations is a series of small micro-applications decoupled via queuing technologies. The service-layer application generates events that are used to mutate an eventually consistent view of the donations domain, enabling failure handling and fault tolerance across the distributed architecture. The applications were all developed to the platform contract represented by Cloud Foundry.

Vendor Neutrality and High Availability are Primary Architectural Goals

Developing the applications to the Cloud Foundry contract brought vendor neutrality to Comic Relief’s donations deployment. Wherever Cloud Foundry ran the donations suite could run too. This commoditised the infrastructure vendors and freed Comic Relief from vendor-related constraints, allowing infrastructure choices to be based on price, performance and efficiency – rather than forced by vendor lock-in.

Developing on the operational capabilities of Cloud Foundry played a major role in meeting Comic Relief’s requirements. The data services were enhanced with redundancy and replication, intelligent connection load-balancing was added to each instance, and a distributed global load-balancing/fault tolerance service added to recover quickly from single or multiple instance outages. The service was distributed across multiple AWS EC2 regions and zones, and also London-based VMware vCloud/vSphere IaaS provider, Carrenza (See Figure).

Service distributed across multiple regions and zones

Enhancing Cloud Foundry’s capabilities involved the same Continuous Delivery-based patterns employed to develop the donations applications. The platform pipeline was based on Jenkins orchestrating Cloud Foundry BOSH, with automated testing using Cucumber and RSpec, automated load testing using Grinder, and automated security testing with Zed Attack Proxy in tandem with a custom suite. The platform pipeline for Cloud Foundry then formed part of a larger integration pipeline for continuous delivery of the donations service as a whole to production-like environments. Frequent and regular deployments following extensive automated testing generated the confidence required to make any necessary changes to such a mission critical system.

Record Earnings With No Outage

So how did it actually perform on the night? Shortly after 01:30 GMT, comedian Russell Brand announced the money raised stood at £75,107,851 – passing the previous high of £74.3m. The new record was delivered by a platform that suffered absolutely no outage in receiving donations – the first time this has ever been achieved by a donations platform for Red Nose Day.

I’ve been delivering platforms for nearly fifteen years; I can’t recall ever being more excited about a new technology than I was on the day Cloud Foundry was released. I’m exceptionally proud to have taken the promise of Cloud Foundry and delivered real value in a mission-critical scenario.

About the Author: Colin Humphreys is CEO of CloudCredo, a team of people with extensive knowledge in running and customizing Cloud Foundry. He led the installation of the first Cloud Foundry to deliver SLA-driven production services, delivers tooling to the Cloud Foundry community, and is a regular conference speaker on PaaS-related topics. He also organizes the London PaaS User Group.

NTT Contributes Nise BOSH, a Tool to Speed Up BOSH Development

Posted on April 15, 2013 by Matthew Kocher, Vikram Rana

NTT, the world’s leading telecom, has been active in fostering the Cloud Foundry developer and user community in Japan. We’re excited about NTT Lab’s latest open source community contribution to Cloud Foundry – Nise BOSH, a lightweight BOSH emulator for locally compiling, releasing and testing services. BOSH is an open source tool chain for release engineering, deployment and lifecycle management of large-scale distributed applications and services, such as Cloud Foundry. Nise BOSH allows developers and cloud operators who use BOSH to speed up the development feedback cycle while saving effort and money. In this post, we’ll explain how all this works and why it’s useful.

Normal BOSH Deploy Cycle

When you do a bosh deploy of a large scale distributed service such as Cloud Foundry to a cluster, BOSH does a number of things on your behalf:

Prepares deployment
Compiles packages – Calculates all packages and their dependencies that need to be compiled. It then begins compiling the packages and storing their output in the blobstore. The number of workers specified in the deployment configuration determines how many VMs can be created at once for compiling.
Prepares DNS
Creates bound missing VMs – Creates new VMs, deletes extra VMs.
Binds instance VMs
Prepares configuration
Updates/deletes jobs – Deletes unneeded instances, creates needed instances, updates existing instances if they are not already updated. This is the step where things get pushed live.

In step #2 BOSH spins up a number of worker VMs to compile your code packages, then installs your compiled code in one or more VMs called stemcells. Normally all of these VMs are running somewhere on an IaaS provider such as VMware vSphere or AWS EC2.

Using Nise BOSH to Speed Up the Feedback Cycle

This is great in production, but when iterating on a BOSH Package, spinning up multiple VMs for compilation and deployment makes for a slow feedback cycle and a lot of ssh’ing into servers. It can also be costly, as you may have to pay for development resources on an IaaS, e.g. on AWS. Nise BOSH is a great help when developing a BOSH package. A Package is a collection of source code along with a script that contains instruction how to compile it to binary format and install it, with optional dependencies on other pre-requisite packages. Without Internet access or an IaaS, Nise BOSH will compile the packages necessary to run the job and install them on the box that you’re on, saving you the time, expense and complexity of using remote resources on AWS or another IaaS. Let’s see how this works.

First, start on a machine that looks like your target Stemcell. A Stemcell is a VM template with an embedded BOSH Agent. The Stemcell used for Cloud Foundry is a standard Ubuntu distribution. For this you can use Vagrant, a handy tool to create easily reproducible environments on your local laptop. Install Vagrant and then in a terminal:

$ vagrant init lucid64 http://files.vagrantup.com/lucid64.box 
$ vagrant up
$ vagrant ssh

You should now be in an Ubuntu lucid64 environment on your local machine. Now install Nise BOSH –

$git clone git@github.com:nttlabs/nise_bosh.git
$cd nise_bosh
$bundle install

Next you need to get a BOSH Cloud Foundry release – a BOSH release is a collection of source code, configuration files and startup scripts used to run services, along with a version number that uniquely identifies the components.

$git clone git@github.com:cloudfoundry/cf-release.git 
$cd cf-release 
$git submodule sync 
$git submodule update --init --recursive

You can now modify the Cloud Foundry open source code in cf-release/src as your project needs require. Once you are ready to test your changes, you need to create the release, but first install the BOSH CLI -

$gem install bosh_cli 
$bosh create release

Before you can deploy your release you’ll need a BOSH manifest file we’ll call deploy.conf. An example of the contents for a Nise BOSH manifest can be found in the Nise BOSH doc.

Now run Nise BOSH from the nise_bosh directory to launch dea_next. Without Internet access or an IaaS, Nise BOSH will compile the packages necessary to run the job and install them on the box that you’re on. Then just run run-job start to start the jobs in /var/vcap/bosh/etc/monitrc locally on your box.

$sudo PATH=$PATH bundle exec ./bin/nise-bosh ~/cf-release ~/deploy.conf dea_next
$./bin/run-job start

Nise BOSH is all set up to help you play with dea_next all by itself, but eventually it might enable to you develop and run Cloud Foundry right on your local machine. We see the many community contributions like NTT Nise BOSH as proof of the power of an Open PaaS ecosystem to advance the project. Next time you’re working on a BOSH release, give it a try.

–Matthew Kocher and Vikram Rana, Cloud Foundry Team

Use of Hybrid PaaS Now and In the Future

Posted on April 9, 2013 by Jonas Partner

From its beginning, Cloud Foundry has been committed to providing developers and enterprises choice of deployment options spanning public and private clouds. In this post, Jonas Partner, CEO of CloudCredo and OpenCredo, shares how their enterprise customers are using PaaS to deploy applications that span public and private clouds to increase availability, add extra capacity and prevent lock-in.

As one of the co-founders of OpenCredo I have been actively working with Cloud Foundry since its early days. We were one of the first companies to launch a production application on Cloud Foundry with the help of our friends at Carrenza. More recently we have established CloudCredo to help both enterprise customers wishing to host Cloud Foundry on their own infrastructure and cloud service providers wishing to offer Platform as a Service. We helped Ospero to bring a Cloud Foundry-based PaaS running on vCloud Director to market to better serve their customers.

With the growing availability of public Cloud Foundry instances, we are seeing an increase in interest from enterprise customers in hybrid cloud as a way to extract maximum value from their existing infrastructure while enabling capacity on demand through expansion into public cloud. In this blog we will discuss why hybrid PaaS with Cloud Foundry is gaining interest, some of the ways people are using it today and how we hope to be using it in the future.

Why Cloud Foundry for Hybrid PaaS

The availability of a PaaS that seamlessly spans in-house data centers and various public cloud offerings makes hybrid cloud a compelling commercial proposition. For example, it serves to drive competition among public cloud providers. Extra capacity can be bought from the most cost competitive provider, or spread across a number of providers, as determined by the best commercial fit for the need. To date the majority of public PaaS offerings have been closed source and can’t be run within the enterprise data center. Combine this form of lock-in with some well-publicized outages and most large enterprises simply aren’t willing to consider betting everything on a single instance cloud provider. We see the open, extensible nature of Cloud Foundry and the increasing number of companies offering hosted Cloud Foundry instances as key to making enterprises comfortable with PaaS.

For us the number of companies offering or intending to offer a Cloud Foundry instance is reaching a critical mass, giving enterprises a competitive set of choices of where to run their applications. The Cloud Foundry Core initiative also means that customers can rely on public Cloud Foundry Core compatible instances to meet certain minimum requirements. Taken alongside improved support for running Cloud Foundry on public IaaS such as EC2, enterprises now have three real options for deploying cloud applications: 1) on a Cloud Foundry instance running in house on existing infrastructure, 2) on a Cloud Foundry instance running on large scale public IaaS such as EC2 or 3) on one of the increasing number of cloud providers offering Cloud Foundry as a service.

Being able to pick a company offering a PaaS hosted within national boundaries is already broadening the applicability of PaaS to regulated industries that have requirements on data sovereignty and how data is managed. There are still many other considerations around technology stacks and non-functional requirements that determine whether Cloud Foundry is suitable for a particular workload.

Simple Hybrid Cloud Use Cases

We are seeing significant interest in hybrid PaaS as a way of adding capacity in peak periods and increasing availability. Currently this only works out of the box for simpler cases where 1) there is no need to massively scale the persistence technology and 2) the application’s database does not need to be shared between services on either side of the public private cloud divide. Where enterprises employ SOA it can also be challenging to find a subset of the services that can be moved out to the public cloud. While services can span public and private cloud, this adds complexity, potential performance challenges and cost where data stored in the public cloud is separately billable.

The factors that determine the simplicity of an application for the purposes of a hybrid cloud scenario are primarily how coupled it is to the rest of the enterprise. This coupling can take the form of persisted data, messaging and application services. The simplest case is an application that has none of these forms of dependencies – with the increasing adoption of SOA it is increasingly rare to find these types of applications. When considering the potential use of hybrid cloud, the cost benefit versus the complexity due to connectedness to the rest of the enterprise will be one of the key trade-offs.

Unless for some reason you have a large number of island-like applications then batch processing applications can often provide a fertile ground in the early days of adopting a hybrid cloud deployment strategy. Batch applications often combine minimum data connectivity to the rest of the application, often running on a data snapshot, with providing cost benefit from scaling up just for the batch run.

Scalable Database Services

The open source Cloud Foundry project provides a number of open data technologies that can be configured to “scale up” as single instance services out of the box. We see incorporation of scale-out, multi-instance database services as key to widening the use cases that can be addressed by Cloud Foundry and are currently working to address this opportunity for our customers.

At CloudCredo we recently open sourced a simple Cassandra service integration for Cloud Foundry on GitHub. The initial implementation was targeted mainly at development and test usage since it was single instance and therefore didn’t provide any way to make the database scale out or highly available. We are now extending that out to a multi-node cluster and we intend that other services such as MongoDB will also follow.

Scalable Databases as a Service can be plugged in to Cloud Foundry very easily by leveraging the Service Broker. This pattern allows the cluster to be managed from outside of Cloud Foundry but for the ‘external’ cluster service to be exposed to clients as a running Cloud Foundry service. The benefits of this model include Cloud Foundry features such as central configuration management and automatic service binding from applications.

While horizontal scaling and clustering of open database services opens up new potential, what really excites us is the potential to use the multi-data center capabilities of persistence technologies in conjunction with hybrid PaaS. Cassandra and other open data technologies such as Riak make it possible to have a logical database which spans multiple data centers and therefore potentially both public and private cloud.

A PaaS which integrates multi-data center databases that are easy to deploy and provide data synchronization would make solving problems such as disaster recovery and transparent scaling of both database and application code much easier. Disaster recovery currently relies on expensive low latency RPO systems that require manual, offsite back-ups be taken to mirror production. A multi-node, multi-data center data service would have the potential to automatically mirror data across data centers as part of the online transaction processing of the application. Having effectively the same database available within both public and private PaaS would also simplify the process of scaling out.

Conclusion

The growing popularity and increasing number of ways to consume Cloud Foundry is making hybrid PaaS an appealing reality for many enterprises. Cloud Foundry provides the Service Broker to facilitate integration of new, clustered open data technologies and legacy clustered databases, easing enterprise adoption in cases where the database services that come as part of the platform are not sufficient. The combination of reliable open source technologies that can deal with databases and the increasing availability of Cloud Foundry is an exciting option for those trying to use PaaS within the enterprise. Being able to solve these sorts of traditionally expensive problems with a PaaS will make PaaS even more compelling for large enterprises.

About the Author: Jonas Partner is CEO of OpenCredo, a software consultancy and delivery firm with offices in the United Kingdom and North America. Recently, Jonas co-founded CloudCredo, a company focused on helping businesses get Cloud Foundry up and running on the infrastructure of their choice. Before establishing OpenCredo in 2009, Jonas was a consultant with Spring Source where he contributed to the Spring Integration project. Jonas maintains interests in Machine Learning and the scalability of highly concurrent enterprise systems.

Anchora Brings Cloud Foundry Core Instance to China

Posted on April 3, 2013 by Dr. Wei-Min Lu

A fast growing mobile apps market alongside the increasing need for a unified enterprise private cloud platform requires a new generation of open PaaS solutions in China.

In this guest blog, Wei-Min Lu, founder and CEO of Anchora, explains why Anchora selected Cloud Foundry as the basis for its MoPaaS platform and why it chose to join Cloud Foundry Core.

Test MoPaaS for Cloud
Foundry Core Compatibility here.

Today we are excited to join Cloud Foundry Core as proof of our commitment to cloud application portability across a choice of public and private clouds. Here in China, there is growing demand for an open PaaS that enables developers to build mobile applications and enterprises to build private or dedicated clouds. We built MoPaaS, a cloud application engine based on the Cloud Foundry service platform. Cloud Foundry gives us the flexibility to offer both a public cloud platform with extensions for mobile and data applications and a private/dedicated PaaS solution that integrates with enterprises’ existing infrastructure. We also provide services within Cloud Foundry applications allowing enterprises to take advantage of big data infrastructure for use cases like cloud storage and synchronization, analytics and search.

The Need for Open PaaS in China

It is projected that PaaS will grow at 30% annually over the next few years. Higher growth is expected to occur in China due to 1) a rapidly expanding mobile application market requiring an agile back-end services platform and 2) the need for cloud application portability and the rise of applications taking advantage of big data infrastructure in the enterprise.

China is now the world’s biggest smart phone market with 390 million units projected to be sold this year – more than 1/3 of smart phones to be sold globally in 2013. Businesses of all sizes are looking to take advantage of this trend, increasing the number of mobile apps by 35% annually. There is a growing need for an agile platform with additional services required for mobile applications, such as push notification, geospatial data storage and short message services. Due to the limited resources on the client device, mobile applications require a platform that is able to process large quantities of data in the cloud. A new generation of Open PaaS is needed to handle additional mobile-specific requirements.

With more than 70% of companies implementing PaaS over the next two years, we see emerging needs for PaaS solutions in the China enterprise IT market to consolidate existing applications and infrastructure while building an ecosystem of new enterprise applications in a consistent and unified manner. Enterprise application portability is increasingly becoming a critical need – businesses expect to be able to move their applications for optimal service, regulatory compliance, and data privacy requirements. As enterprises are realizing the impact and starting to build apps to take advantage of big data, there is an urgent need for more sophisticated platform services with big data infrastructure. This includes scalable services for log analysis, cloud storage and synchronization, information clustering, analytics, and customized search.

Building a Public PaaS Using Cloud Foundry

We built MoPaaS to serve developers’ needs to create applications and services without worrying about backend IT tasks. It helps them simplify and automate application lifecycle management, including development, deployment, and operation, significantly reducing IT infrastructure expenditures and devops costs and time. MoPaaS is built on an intelligent cloud service platform infrastructure that consists of two integrated parts:

Service platform: extends the Cloud Foundry service platform to provide better support for mobile applications, platform monitoring and management, and service extensibility.
Data platform: based on innovative information chain management (ICM) technology. It offers a simple interface for developers to support data driven applications.

Fig 1. MoPaaS Cloud Application Engine Architecture

MoPaaS provides necessary devops automation, masks complexities from developers and enhances usability by offering additional functions:

UI: providing ease of use for developers of various levels and a CLI providing more flexibility and control for experienced developers. Developers can use the MoPaaS UI for devops tasks like application deployment and performing resource allocation.
MDSS (mass data storage service): a high performance distributed storage system that provides cloud storage for user data and application source code management.
Application monitoring and management: a streamlined visual tool and web console for monitoring and managing applications, services, and environment variables.
MoPaaS Services: Mobile specific services, such as notifications, short messages, and locations etc. Data services, such as scalable services for log analysis, cloud storage and synchronization, information clustering, and customized search.

Building an Enterprise PaaS Solution Using Cloud Foundry

MoPaaS also provides an intelligent IaaS-PaaS solution for enterprises and service providers to build dedicated clouds to boost service agility, reduce IT expenditures and devops costs while increasing the portability, scalability and reliability of enterprise applications. This solution offers necessary granular controls and customer configuration flexibility provided by IaaS and the devops automation and ease of app creation provided by PaaS. The MoPaas architecture strictly separates PaaS from IaaS services.

Fig 2. MoPaaS Intelligent Cloud Platform

In addition to the services offered by Cloud Foundry, MoPaaS offers a comprehensive array of data services as native Cloud Foundry services and integrates various legacy services.

Fig 3. MoPaaS Services

MoPaaS effectively addresses a wide range of the enterprise cloud platform requirements, including:

Application consolidation: there is a need for building and running an ecosystem of enterprise applications on a cloud-based platform in a consistent and unified manner. However, many of the applications in enterprises have been around for quite some time and were not designed for the cloud. Among the key requirements, legacy services used by those applications need to be integrated into the cloud platform or upgraded to new services of the cloud platform. MoPaaS addresses the application consolidation and migration issues effectively, as the MoPaaS platform not only supports basic services, but also adds new data services to meet enterprise big data requirements using Cloud Foundry service gateway and seamlessly integrates legacy services used by existing applications, such as Oracle Database, Microsoft SQL Server, and IBM DB2 using Cloud Foundry service broker (Figure 3).
Data platform: Big data has increasingly become top of mind across organizations, from the data center to the boardroom. Companies are seeking systems and tools capable of tackling the challenges of massive data growth. Integrated with Hadoop-based Information Chain Management (ICM) data services, MoPaaS enables organizations to deploy a central platform to solve end-to-end data problems that involve any combination of information ingestion, processing, storage, exploration, analytics, and retrieval. With MoPaaS enterprise developers can easily integrate sophisticated big data services with their applications as native Cloud Foundry services. These include log analysis, cloud storage and synchronization, information clustering, and customized search. In addition, with ICM data services, MoPaaS provides new capability for monitoring and managing applications on the cloud platform.
Application portability: MoPaas architecture strictly separates PaaS from IaaS services. MoPaaS furnishes applications with an infrastructure-agnostic execution method, standardizes the method of exposing services such as databases, messaging and queuing, runtime, and management, and standardizes service credentials across runtimes. Thus MoPaaS keeps the applications portable in case they need to move the PaaS layer and application services to another infrastructure provider. Additionally, MoPaaS ICM mechanism enables automatic application data transferring along with the associated applications via its data synchronization service. Developers don’t have to worry about infrastructure-specific requirements for their applications, because the MoPaaS platform takes care of these concerns for them.

Summary

The initial success of Cloud Foundry-based PaaS providers demonstrates the strength of the Cloud Foundry open PaaS solution and the power of its open source ecosystem and community. This is a win for everyone invested in the success of Cloud Foundry.

We are excited to become the first company in China to join the Cloud Foundry Core community. By being a member of the Core ecosystem, Anchora is committed to the open PaaS concept and delivering cloud portability to Chinese application developers and service providers.

About the Author: Wei-Min Lu is the founder and CEO of Anchora. Anchora provides both MoPaaS, a public open cloud platform with extensions for mobile and data applications, and a dedicated PaaS solution that facilitates application consolidation and cloud platform portability for enterprises. Dr. Lu has over 15 years experience in product development and marketing. Before he founded Anchora in 2008, he served in key engineering and management positions at IBM and NASA-JPL. His background is in cloud computing, cybernetics, machine learning, information retrieval and storage technologies. Dr. Lu holds a Ph.D. in Electrical Engineering/Math from CalTech and a B.Sc. from Tsinghua University, China.

Cloud Foundry Integration for Eclipse Can Now Launch External Command-line Applications Using Service Tunnels

Posted on March 14, 2013 by Nieraj Singh

In the course of developing, testing and deploying cloud applications, developers sometimes need to directly access the application data. The Cloud Foundry Integration for Eclipse advances this capability and now features the ability to launch external command-line applications for Cloud Foundry services like MySQL and PostgreSQL. The plugin offers Spring and Java developers using Eclipse or STS a way to manipulate, or port the data contained in their Cloud Foundry applications more easily.

An earlier release of the Cloud Foundry plugin (1.1.0) introduced tunneling support for data services, and the command-line application feature in 1.4.0 now extends the support by automatically resolving tunnel values when launching a command-line application for a Cloud Foundry service. In addition, a set of predefined commands for executing commonly used command-line applications are available by default, and ready to use if the external command-line application is installed and set in the operating system’s PATH environment variable.

These commands are launched from within Eclipse or STS, and executed in an external command-line terminal. New commands can be defined either through a new Eclipse preference page for Cloud Foundry, or by opening a Service Tunnel Commands wizard from the Services table in the Cloud Foundry Server editor. The commands are stored as Eclipse preferences and therefore persisted across Eclipse runtime sessions.

In this blog, we will describe the steps to define and launch a new mysqldump command that automatically generates a dump file for a Cloud Foundry MySQL service. This blog assumes the reader has basic understanding of how to create a Cloud Foundry server in Eclipse, as well as create services through the Cloud Foundry Server editor’s Applications tab.

The command-line application feature in 1.4.0 allows users to launch an external application by simply selecting a service in the Services table in the Cloud Foundry Server editor and right-clicking to execute a particular command from the Services table context menu (see Figure below). The launch process then automatically creates a tunnel if one was not already created and executes the command in the operating system’s default terminal application. Terminal applications can also be changed per command definition. For example, in Linux the default terminal used by the feature is xterm, yet a user can change a command to use a different terminal like gnome-terminal.

Predefined Commands

The Cloud Foundry Integration for Eclipse version 1.4.0 provides a list of predefined commands that are ready to use for MySQL, PostgreSQL, MongoDB, and Redis services, provided the corresponding applications are installed. The commands are accessible by right-clicking on a service in the Services table in the Cloud Foundry Server editor:

The list of commands are:

MySQL:

mysql
mysqldump

MongoDB:

mongo
mongodump
mongorestore

PostgreSQL:

psql

Redis:

redis-cli

If these applications are already set in the operating system’s PATH environment variable, these commands can be executed right away without changes, and are launched in an external terminal like Terminal.app for Mac OS X, cmd.exe for Windows, and xterm for Linux.

For example, launching mysql for a MySQL service in the Cloud Foundry editor will automatically connect the mysql application to the MySQL service in the Cloud Foundry server through a tunnel, provided mysql is installed and set in the OS PATH environment variable. The end result is an external terminal window opening outside of Eclipse, displaying the mysql application prompt:

Since the tunnel credentials are resolved automatically for the external application by the Cloud Foundry plugin, the mysql application automatically connects to the data service and opens the mysql prompt without asking the user for credentials.

If new commands need to be added, or existing ones edited, they can be done so by either selecting Command Definitions in the Services table context menu, or by opening Eclipse Preferences -> Cloud Foundry -> Service Tunnel External Application Commands

Creating a New mysqldump Command

In our example, we will be adding a new mysqldump command that creates a particular output dump file called myoutputfile.txt. Although a mysqldump command already exists by default as part of the set of predefined commands, we will create a separate one where a dump file is explicitly set.

Commands are associated per service vendor, so the new mysqldump command will apply to any MySQL service regardless of the Cloud Foundry server.

To add a new command, choose Command Definitions from the Services table context menu to open the Service Tunnel Commands wizard, select the desired service, and click “Add”.

This will open the Command Definition wizard:

Two required properties are needed to define a command:

A unique display name. Two or more commands with the same name cannot exist for a given service.
An executable location, or just the name of the application if already set in the OS PATH environment variable.

Optionally, a user can also define command options that are passed to the application when it is being executed, and environment variables that are set by the Cloud Foundry plugin prior to launching the command. Option variables using the pattern ${variablename} can be used in the options definition.

These variables are grouped into two types:

Tunnel-specific variables that are reserved for tunnel values like username, password, port, and database name. These variables are automatically resolved by the Cloud Foundry plugin when the command is launched.
Additional variables that will be prompted to the user in a dialogue when the command is executed.

Some applications like psql require that passwords be set as environment variables. As with command options, variables using the pattern ${variablename} can be used in place of environment variable values.

In addition, the terminal definition can also be adjusted for that particular command, and optionally applied to all current and future commands by editing the terminal launch command value in the External Command Line Terminal section.

To facilitate creating custom commands, the list of predefined commands are available via a drop-down control, and can be used as templates for new definitions:

Let’s select mysqldump from the list. It will now populate the wizard with the default definition for myslqdump, including the necessary options required to run the application. Since we already have a default mysqldump command, an error will appear asking the user to set a different display name:

Now, we want to edit the display name and give it a unique name, and also set a specific name for the dump file: myoutputfile.txt. In addition, we are changing the location of the mysql executable as we do not have it set in the OS PATH environment variable.

Clicking “Finish” in the Command Definition wizard will now add the command to the list of commands for the MySQL service:

Clicking “Finish” once more in the Service Tunnel Commands wizard will persist the changes. At this point, clicking “Cancel” will alternately cancel any changes made to any services.

Now let’s right-click on the MySQL service in the Services table, and we’ll see the newly added mysqldump with outputfile command in the context menu. Selecting this command will execute it, and if necessary, automatically create a tunnel connection if one wasn’t already present.

After the command executes, the following dump file is created in the user’s home directory:

If instead of executing the custom mysqldump with outputfile that was created above, a user runs the default mysqldump command where the dump file is defined using an ${output file} variable, a dialogue will open prompting the user for the name of the output file prior to executing the command. The file name can include an absolute path, or if just specifying a name, it will be created relative to the user’s home directory.

Variations in Terminal Launch

The command-line feature for 1.4.0 is supported in Mac OS X, Linux and Windows. However, there are differences in how a command is launched in an external terminal. For Mac OS X, the default terminal is Terminal.app, and the Cloud Foundry plugin executes the command and sets any environment variables in a temporary .sh script file that it creates and then passes as an argument to Terminal.app. This temporary file is deleted after the process running the command is launched.

For Windows and Linux, no temporary script file is created, and instead the command and its options are passed directly to the external process as process arguments.

Installing Cloud Foundry Integration for Eclipse 1.4.0

To install version 1.4.0, follow the documented instructions to install Cloud Foundry Integration for Eclipse. Earlier versions of the integration, starting from 1.0.0 can be updated from within Eclipse or STS. Note that updates for older Cloud Foundry plugins prior to 1.0.0 are not supported, and must first be uninstalled before installing 1.4.0.

Only Eclipse JEE Indigo and Juno, as well as STS 2.9.0 and GGTS 3.0.0 and higher are supported.

Cloud Foundry is Open and Pivotal

Posted on March 7, 2013 by James Watters

For those who have not heard, Cloud Foundry, the open platform-as-a-service project, is now part of the recently announced Pivotal Initiative, an independent entity funded by VMware and EMC. (For a preview of our Paul Maritz-led Pivotal Initiative, check out this Wired article).

Our independent status is quite meaningful and gives us the ability to engage with our large ecosystem in new ways. Adding full-time external committers has always been a goal of the team, and we are engaged with several organizations about putting dedicated resources on the extended engineering team –we believe this to be a very important step forward. The scale of these external investments is significant and a major milestone in our growth. The heart of Cloud Foundry, however, really comes from individual community contributions and users, so of course, we invite you to join us. All you need to do is send a pull-request.

As more and more organizations contribute to Cloud Foundry a transparent roadmap to the community is paramount. In January we began publishing a quarterly plan as part of our community documentation and since then our mailing list activity has hit an all-time high. We will be prolific in communicating the updated roadmap and updates on new work including logging APIs, .NET support, Cloud Foundry BOSH enhancements and more.

Cloud independence and multi-cloud support are important beliefs for us and the broader Pivotal Initiative. Expect open interfaces and support for a variety of clouds, with continued development on AWS, Openstack, vCloud and vSphere environments.

Our own financial and human investment in the project continues to grow. Cloud Foundry is staffed with over forty full time internal engineers, augmented by an additional team of twenty five spread across PM, developer advocacy and community management.

Last week we began actively selling software update and support subscriptions to large customers, another important milestone. Enterprise interest in building large scale internal agile platforms with Cloud Foundry is tremendous and we expect an interesting new backlog of feature requests from these engagements. The only proprietary code in these subscriptions is our web console, so any new enhancements will immediately benefit the larger community.

Finally we have again weighed the best governance model for the project and believe the current corporate sponsored, Apache 2 licensed, pull request driven approach, remains the right one. Adding external committers will also enhance the diversity and strength of the team. That said, the massive growth of the community and ecosystem requires mediating a diverse set of needs and we will always be open to other governance models for the project in the future.

Our mission is to become the most popular platform for new applications and the services used to build them across public and private clouds. If your organization would like to join us in building something great please contact us here.

James Watters, Head of Product

The Cloud Foundry Team

Continuous Integration to Cloud Foundry.com Using Jenkins in the Cloud

Posted on February 28, 2013 by Mark Prichard

Continuous integration using Jenkins is increasingly seen as an effective tool for reducing the cycle time from product backlog to receiving actual user feedback. This can result in real increases in developer and team productivity when combined with an Open PaaS such as Cloud Foundry.

In this guest blog, Mark Prichard, Senior Director of Product Management for CloudBees®, explains how to create a fully automated environment for application build, test and deployment to Cloud Foundry.com using Jenkins in the Cloud via the CloudBees DEV@cloud™ build service.

Spring Framework is the most widely adopted development model for Java-based enterprise applications, used by millions of developers for its powerful abstractions and declarative support for crucial application infrastructure concepts. Jenkins is the world’s number one open-source continuous integration (CI) platform, with deep roots in the Java community and 60,000 installations worldwide. These two technologies are part of the fabric that has made Java a highly productive, agile programming environment for business applications. With the rise of cloud computing and Platform as a Service (PaaS), developers want CI services to build, test and deploy their Spring applications entirely in the cloud, using elastic, on-demand resources. You can now use Jenkins via CloudBees DEV@cloud to continuously deploy to Cloud Foundry.com.

Usage of Continuous Integration and Delivery for Release Management

In our most recent annual Jenkins CI user survey, a couple of facts stood out: first, 74% of users are building Java applications with Jenkins (although we also see significant development in C/C++, Javascript, Python, C#, PHP, Ruby, Scala and Groovy); and second, we are seeing a really big up-tick in usage by large organizations, with 60% growth among the very largest group. Overall, the number of installations is up 66% in the last year, with over 83% of those surveyed using Jenkins for what they consider mission-critical development. Looking at those results, it seems very clear that many of those large, mission-critical applications are using Java with Spring Framework – with CI provided by Jenkins.

Another key trend that is growing in importance daily is Continuous Delivery. More and more organizations are looking to embrace an agile model in which stringent, automated testing allows enhancements or “micro-releases” to go live without the traditional waterfall release cycles. We are seeing a major shift in enterprise software development to cloud-based, continuous delivery, with fully automated quality, coverage, functional and performance tests gating live deployments. This is the new best practice, and it is now available for Cloud Foundry development.

In this blog post, I’m going to show you how easy it is to set up a CI job using Jenkins via CloudBees DEV@cloud service to automatically build, test and deploy a rich Spring Framework application to Cloud Foundry.com.

Using Jenkins in the Cloud to Continuously Deploy Spring Apps to Cloud Foundry

Here’s an overview of the process:

Link your Cloud Foundry and CloudBees accounts using OAuth 2.0 for secure and automatic deployment
Clone a Git repo and set up a Jenkins job to build automatically when changes are pushed (we’ve provided ClickStarts™ that automatically do these tasks for you)
Set up and configure Jenkins Cloud Foundry deployment plugin to push your application to Cloud Foundry if your build succeeds

That’s it: your application and its services are now running live on Cloud Foundry!

Let’s take a quick look at all this cool stuff. First, we need a way to allow secure access from your CloudBees DEV@cloud account to the Cloud Foundry deployment services. CloudBees and Cloud Foundry both support the industry-standard OAuth 2.0 protocol, allowing you to establish a trust relationship between your accounts without the need for either party to store account details like passwords from the other service. Go to cloudfoundry.cloudbees.com, which will redirect you to the Cloud Foundry log in. Log in and authorize your CloudBees account to deploy to your Cloud Foundry account.

Next we want to set up a build job. That’s really easy, even if you’ve never used Jenkins before: we’ve set up ClickStarts that will clone private Git repositories with a couple of fully-featured Cloud Foundry/SpringSource examples from GitHub (springmvc-hibernate and petclinic-grails) and then create Jenkins jobs for those builds. From the Cloud Foundry ClickStarts launch page all you need to do is click on the Spring icon, enter the name you want to use for your build and in a few seconds you’ll be taken to the Jenkins build job, which will start automatically as soon as the repository is available.

At this point, you have everything set up: a private Git repo to use for development, a Jenkins CI build job and automatic deployment to Cloud Foundry. The build job has been configured to use your CloudBees account name as part of the Deployment Hostname (e.g. springmvc-hibernate-< youraccount>.cloudfoundry.com), but of course you can change that if you like. You can now clone a local copy of the source code repository (click on the Repositories tab on the toolbar for full details) and you have a fully automated, cloud-based develop-build-test-deploy Continuous Delivery environment for your Spring/Grails applications. Every time you push an upstream commit, Jenkins will run a complete build/test and deploy to Cloud Foundry.

As you can see by browsing the console output for the build job, this is a substantial application that uses many aspects of Spring Framework, including Hibernate for the mapping between the Java application and two data services, Redis and PostgreSQL, that also need to be provisioned on Cloud Foundry.com and bound to the application. That is all done as part of the deployment; you can see the details in the Services section of the host service configuration.

Once the build and deployment are complete, you can go into the console output for the job and see all the details of the deployment. If you have vmc set up on your workstation, you can use vmc apps and vmc services to verify those deployments, like this:

All you have to do now is browse to those URLs and you’re up and running with Spring/Grails and Jenkins in the cloud – enjoy!

Summary

Cloud Foundry together with CloudBees Jenkins in the Cloud service give you a complete Continuous Deployment solution for your enterprise Spring Framework and Grails projects:

You have access to all the capabilities of a fully managed Jenkins as a service – you don’t have to set up Jenkins or the build systems.
You can use Jenkins to deploy your Java Spring applications seamlessly to Cloud Foundry.
You always have enough build capacity — CloudBees dynamically adds more build machines as you need them.
It’s free to get started.

Watch the video: Continuous Integration to Cloud Foundry.com Using Jenkins in the Cloud
Docs: https://developer.cloudbees.com/bin/view/DEV/DeployAppsToCloudFoundry
Give it a try: cloudfoundry.cloudbees.com

About the Author: Mark Prichard, Senior Director of Product Management for CloudBees, speaks and blogs regularly as an evangelist for Platform as a Service. He came to CloudBees after 13 years at BEA Systems and Oracle, where he was product manager for the WebLogic Platform. A graduate of St John’s College, Cambridge and the Cambridge University Computer Laboratory, Mark works in Los Altos, CA.

Open Standards in Cloud Foundry Identity Services

Posted on February 19, 2013 by Dale Olds

Cloud Foundry includes a component called the User Account and Authentication (UAA) service which was introduced about a year ago. It provides user account management and authentication for developers that push applications to cloudfoundry.com, including single signon with support.cloudfoundry.com, micro.cloudfoundry.com, and more on the way. It also provides delegated authorization capabilities to partner sites so that they can interact with Cloud Foundry on a user’s behalf without access to user credentials.

An introduction and details of the UAA from a developer perspective can be found in earlier blog posts. In this post I will focus on the UAA’s integration of three emerging identity standards and what that implies for some specific Cloud Foundry deployment patterns.

Goals

The goal of the UAA is to make it easier, simpler, more secure and more convenient for users to get to the features of Cloud Foundry. The value is in the open PaaS and the applications it enables. So the identity services provided by the UAA should not only be useful for cloudfoundry.com and applications on it, but for pure open source deployments, managed private clouds, and even the micro cloud running on a thumb drive.

To achieve this goal, we have implemented and integrated new standards in a way that supports various deployment scenarios, such as multiple identity providers and external authentication mechanisms.

Fusing Identity Standards for a Stronger Platform

We have woven identity services into the platform using three standard protocols: OAuth2 for delegated authorization, OpenID Connect for session and authentication information, SCIM for user and group management. Of course we provide support for other standards as well, such as JSON Web tokens (JWT), SAML2, and OpenID 2.0. Even support for the legacy identity stalwart LDAP is in process. But what is particularly interesting is how we have combined the three primary protocols.

OAuth2 — this is the foundational protocol and provides delegated authorization. It provides a way for web applications to ask users to allow access to cloudfoundry on the user’s behalf. It provides authentication and SSO as a side effect. The app gets one or more secure tokens — some that the app can use to identify the user, and some that it can present to the cloudfoundry API to act on behalf of the user. But OAuth2 does not specify the token format. We chose another standard, JWT, for that.
OpenID Connect — provides identity information for the authenticated user. OAuth2 does not specify how the user’s information can be obtained by an app, just an opaque token. OpenID connect adds the ability for an app to efficiently and safely get authenticated user attributes like user name or email address.
SCIM — User and group provisioning. User account management is out of scope for OAuth2, but SCIM specifies simple user provisioning as well as groups, and it’s extensible to new types. Since OAuth2 requires that web applications preregister with the authorization server (UAA), but does not specify how these registrations are done, we use a SCIM extension for that.

Imagine two web applications: – one manages users and groups for access to cloudfoundry.com – one displays results of monitors a user’s running applications as a service — like our partner Appsecute provides.

You could diagram the interactions of the apps and three protocols like this:

The three protocols complement each other well so far, but there is a piece still missing.

Connecting the Missing Link

For delegated authorization to work, the user needs a unit of delegation — some handle so they can say: I want to allow application X to read my application list on cloudfoundry.com, but not actually modify anything. This notion of specifying names for roles or permissions is called the token ‘scope’ in OAuth2. We needed some way to support management of these scopes, their existence, what users have access to them, what web applications can ask for them — and do it all in a way that naturally supports their use in authorization decisions.

Therefore, we mapped SCIM groups to OAuth2 scopes. In other words, you can create a group in the UAA and register a web app with that group name as a scope. When a user logs into the web app, it’s redirected to the UAA. If the user is in the group and they choose to authorize the web app with that scope, the web app gets a signed token that contains that scope.

We use this in our own cloudfoundry dashboard application. The system reliability engineers access the dashboard app, which is a UAA client app and always requests a token with dashboard.user scope. The UAA denies the scope unless the user is in the dashboard.user group.

What this gets us is that the dashboard app does not have any user account code, and it never gets the user’s credentials. It merely redirects to the UAA if there is no access token in the authorization header. If there is an access token it validates the token, and then checks if it has dashboard.user in the scope. For the system monitors, this means no need for shared accounts or passwords and better control of which users have access.

This is particularly useful when there are multiple such applications. Users get SSO among the applications, the applications are each more secure, access can be easily controlled by groups in the UAA.

Extensible Authentication Services

A somewhat recent addition is that we have separated the authentication service from the rest of the UAA. In OAuth2 terms this is separating the authorize and token endpoints. We can have different implementations of the authentication service for different needs. So far implementations are available for username and password (on cloudfoundry.com), for various flavors of openid2, and for SAML2 identity providers.

Why this is Cool

This is all quite useful, but how it can be used in a cloudfoundry deployment is best illustrated by some examples:

Suppose cloudfoundry is running in a managed instance for an enterprise. The enterprise has a suite of apps that can use the full advantage of the PaaS — easy deployment, scale up, scale down, with a myriad of frameworks and services. All the apps can use the UAA at the core for SSO and role (group) based authorization. Add a user to a group via SCIM, and the corresponding scope will be available at next login to all apps.

Cloudfoundry.com includes various web apps to help developers. Some of them only need to know who the user is for SSO. The openid scope is perfect for them. Suppose a new site wants to also provide personalized content based on the user’s running apps. We can create a registration for that app that restricts its scope to listing the user’s apps. If that app is compromised, nothing in the data of the user’s apps can be changed.

Suppose you are a developer running a private cloud in an enterprise. With Cloud Foundry you get these features:

You can create and deploy applications exactly the same as apps on cloudfoundry.com with access to the multitude of frameworks and services, exactly the same as apps on cloudfoundry.com.
Your apps can register with the UAA in your cloudfoundry instance, which means they don’t have to manage users or authentication. All apps in your instance can have SSO and common groups for authorization.
Further, you can configure the UAA in your instance to include a SAML Login server, which gets user authentication from the corporate SAML IdP in front of an Active Directory instance. All apps now have SSO with the corporate IdP.
Eventually you’d like a user’s AD group membership for be available to your app as an OAuth2 scope. Put the group membership in the SAML assertion and it can be mapped to a scope by the SAML Login Server — this part isn’t actually implemented yet, but is here to show future possibilities.

Cloud Foundry identity services are built on open standards and open source. They increase security and reduce friction so users can take advantage of the open PaaS.

NTT Communications, World’s Leading Telecom, Joins Cloud Foundry Core

Posted on February 13, 2013 by Hideki Kurihara

As the world’s leading telecom, NTT Communications has experience delivering global cloud services to enterprise customers. Enterprises in Japan are seeking the openness, choice, agility and extensibility that an Open PaaS can provide.

In this guest blog, Hideki Kurihara, Product Lead for NTT Communications’ Global Cloud Services, explains why NTT Communications selected Cloud Foundry as the basis for its CloudⁿPaaS and why it chose to join Cloud Foundry Core.

Test NTT’s CloudⁿPaaS for Cloud
Foundry Core Compatibility here.

We are excited to join the Cloud Foundry Core community as proof of our commitment to making cloud portability a reality for Japanese developers. As the world’s leading telecom, everyday we see customers interested in using PaaS to be more agile. But we also hear concerns about vendor lock-in and ability to meet the needs of a complex enterprise environment. We chose to build CloudⁿPaaS on top of Cloud Foundry because of its multi-cloud nature, ability to integrate with existing assets, and solid API foundation for adding management and monitoring features. Using Cloud Foundry as the base, we are extending CloudⁿPaaS for developers and enterprise customers in Japan. Together with other Cloud Foundry Core partners, we are delivering cloud portability to Japanese users as well as global users of Cloud Foundry.

The Drivers for Open PaaS in Japan

The customer community in Japan wants more agility, choice, extensibility and manageability when it comes to cloud. Customers across sectors are looking to innovate faster in response to a very dynamic environment. For example, cloud has been used extensively for emergency infrastructure services in response to recent natural disasters. There has been a sharp increase in demand for cloud services to support smartphone and tablet application development. More global businesses have started considering cloud services as part of their global IT standardization so applications can be ported on demand from one place to another.

Customers here also want choice – choice to add new frameworks and services, and even to port their applications back on-premise or to another provider in Japan and other countries. Enterprises tend to prefer customization to augment their business. But they also value industry or open standards in the services they consume so they can deploy their application assets in the most appropriate place. When it comes to PaaS, application portability needs to be taken more seriously. Open PaaS, that is PaaS based on an open or industry standard, is what customers are talking about here.

Choosing and Building a PaaS Around Cloud Foundry

Meeting the needs of developers and operations for a PaaS that is open, extensible, and easily managed in the enterprise environment is difficult. We chose to build CloudⁿPaaS on top of Cloud Foundry specifically because of its:

1) Openness and portability

2) Flexibility – in configuration and ability to integrate with existing enterprise assets and services

3) Extensibility – the ability to add frameworks and services, but even beyond that, to build UI based tools that make it easy to manage and add resources to applications and navigate very large installations

We’ve relied heavily on the open source nature of Cloud Foundry. We have added memcached, our own filesystem service, and support for Java web application servers (Resin/Resin Pro) so our customers’ existing applications can work seamlessly on CloudⁿPaaS. But Cloud Foundry Core provides our customers’ developers the choice to use a common floor of runtimes and services when programming their applications. This gives them assurance they can port their applications to a Cloud Foundry Core provider in another geography not served by NTT communications or to a Cloud Foundry instance in their data centers.

Once in their own data centers, customers typically have a heterogeneous mix of infrastructure on which they wish to deploy Cloud Foundry, depending on the SLA and tenancy models. We initially deployed Cloud Foundry on top of our Enterprise Cloud services platform using vSphere and vCloud Director. We use this instance as our own Cloud Foundry development environment and plan to use a similar environment for future enterprise/private PaaS. On the other hand, the public multi-tenant instance of CloudⁿPaaS has been deployed on a different cloud platform with attributes and services more tailored for developer centric markets.

Cloud Foundry is designed with sufficient flexibility to satisfy a huge variety of needs for PaaS environments. The Cloud Foundry system consists of several components, and users can set up their environment by choosing the necessary combination and number of components for their needs Fig. 1

Cloud Foundry supports a wide range of environments—from a private PaaS on a single server to a huge public PaaS on a cluster of over a thousand servers—and can be set up to suit the scale as well as its reliability and support features. Moreover, users can add components on demand, so it is possible to start with the minimum configuration and increase the scale as the load grows.

Enterprise integration is one key area where we see area for adding value to our customers on top of Cloud Foundry. The Cloud Foundry services gateway component addresses the ability to add enterprise data services that our customers request for HA, recovery, and backup. The service gateway itself is nothing more than a REST gateway for binding any service, but the real magic is in the service node implementation that allows multi-node deployments for HA and clustering. We are working to create our own multi-node deployments of services using this component such as the aforementioned filesystem and memcached services.

NTT Communications’ Customizations of Cloud Foundry for Enterprises in Japan

Although Cloud Foundry is a promising PaaS software solution that has many good features, as described above, it is still not perfect. For example, integration with other NTT Communications’ services was naturally not provided. Therefore, we have been making efforts to extend Cloud Foundry to meet the specific needs of our enterprise customers and developers in Japan.

Contributions and Customizations to Cloud Foundry

(1) Reliability of Cloud Foundry components

Because Cloud Foundry was a very young project when we started our development, NTT Communications and NTT R&D (Software Innovation Center) have been examining the performance and scalability of Cloud Foundry by conducting various tests and fixing any problems revealed by them. For example, we solved a problem involving an important component that was a single point of failure and a problem where some components could not be restored after failures by fixing the source code and adding additional external systems.

(2) Convenience of Cloud Foundry

Since convenience is important for commercial services, we created an installer for the virtual machine container (VMC), which is the console for PaaS users, and we are now developing a function for linking the VMC and version control systems such as Git.

(3) Control Panel for user application management

Although Cloud Foundry has a flexible component system as described above, it is essential to provide a more user-friendly environment to fully leverage its features. Therefore, we have developed a control panel that allows users to manage their applications on CloudⁿPaaS. The control panel displays a list of the user’s applications, information about each application, status of resources for each application instance, and each application’s logs that have been stored by means of CloudⁿPaaS services. Using the control panel, operators can flexibly scale-out or scale-up their applications instances. Fig. 2

Building a Cloud Foundry Community in Japan

The presence of active communities is indispensable in the long-term evolution of open source software. We launched the Japan Cloud Foundry Group and are fostering a developers’ and users’ community of Cloud Foundry in Japan. In addition, we will open the source code created by group members by committing them to the official Cloud Foundry repository, and we also provide the know-how in workshops.

About the Author: Hideki Kurihara is the Product Lead for Global Cloud Services, including PaaS, for NTT Communications. Prior to his current position, he was engaged in NTT Communications’ North American hosting and cloud business for 10 years, serving in managerial positions such as General Manager of the Enterprise Hosting business unit and Vice President of Global Product Strategy at its subsidiaries. Hideki holds a Master of Science degree from University of Tokyo in Japan and a Master of Business Administration degree from INSEAD in France.