Cloud Foundry Blog

Static.com Adds Hadoop Support for Cloud Foundry

In this guest post, Jake Farrell, CTO for Static.com, explains how the major shift in the hosting industry towards platforms for high developer productivity and data-centric applications led them to build their Platform as a Service using Cloud Foundry with Hadoop support.

Test Static.com for Cloud
Foundry Core Compatibility here.

The hosting industry is undergoing a dramatic transformation due to increasing demand for self-service cloud based capabilities. To greet this shift, Static.com, a subsidiary of Catalog.com, is leveraging 19 years of hosting experience, and adopting Cloud Foundry as the foundation for its Platform as a Service (PaaS). This move demonstrates the company’s commitment to providing modern, easily-installed and managed cloud-based solutions to hosting companies and enterprises.

To address the demand for self-service cloud hosting, Static has designed the Static Cloud Manager, a robust yet easy-to-use interface. Static’s Cloud Manager leverages Cloud Foundry to allow end users and enterprises to configure, deploy and scale applications and services in seconds. Additionally, Static has expanded on Cloud Foundry’s core platform and developed a growing catalog of OSS applications, frameworks and services for end users to choose from.

With this growing array of options comes a greater need for insight into application metrics and usage patterns. Static has integrated Apache Hadoop with Cloud Foundry as one of the first steps in what it sees as the cloud’s next major offering: Analytics as a Service (AaaS). Thanks to the power and flexibility of Cloud Foundry, these robust new offerings are rapidly evolving the cloud.

Static adds Apache Hadoop to Cloud Foundry

1KGT2fx0PxpqAHVcJmD8G7gxQw9kyMGOrNRYpnw

As PaaS and Infrastructure as a Service (IaaS) become the new hosting standard, having the ability to gain insight into usage and the performance of sites and cloud-based applications is essential. Static believes that the AaaS offering will allow users to make educated decisions about their sites and applications based on usage patterns. The first step to achieving this is to be able to store and crunch large amounts of big data into usable metrics, and Apache Hadoop fits this need perfectly.

Adding services like Apache Hadoop to Cloud Foundry allows Static’s users to easily add cost-effective big data solutions to address custom application needs. Adding Apache Hadoop as a pluggable service inside Cloud Foundry is the first step to automating AaaS to any site or application within the PaaS, enabling customizable metrics and immediate insight into usage patterns.

The Shift in Hosting

Traditional hosting offers minimal choices to the user, forcing them into shared pools where sites and applications are not guaranteed any dedicated resources. Shared hosting has worked for many users and hosting companies, but Static has seen an increase in demand from users who are switching to Virtual Private Servers (VPS) solely for the ability to have a dedicated set of resources and the freedom to choose what they run. This use case places the burden on the user to install, setup and administer the entire process, from start to finish.

Static’s goal was to rethink this approach and to allow for users to have the ability to choose:

• An expanding applications list ready to be installed with one click
• All the latest languages and version to choose from
• Latest frameworks ready to be use
• Full control over memory limits
• Ability to scale instances easily
• Choose from array of services
• Sync code between test and production environments
• Backup and rollback applications and services with ease

1gG0q_vq_MT7px9tHTWBvEikx0zGSHB1PHC0T

Static has leveraged Cloud Foundry to help make this a reality with the Static Cloud Manager, an easy-to-use but robust interface which interacts with Cloud Foundry. Users are able to 1-click create a Platform as a Service application with a wide array of frameworks and services while taking advantage of integrated hosting features such as domain registration, DNS, billing, support, Virtual Private Servers (VPS), and email.

The Cloud Network

Having choices does not stop at applications and services. Static is working to create a unified cloud network of datacenters and hosting companies that will:

• Run your application in multiple data centers for high availability
• Migrate applications seamlessly from one datacenter to another while maintaining uptime
• Choose your application and services based on data centers capable of meeting your needs

11cHte9j7Z3fTOwqDaOSPyCfkgoRzEiZs8msL

Static’s Commitment to Open Source Development

Static believes that open source software development drives innovation and leads to higher quality software. We are deeply committed to the open source ecosystem and being a leader in the Cloud Foundry community. Static is proud to contribute back to the Apache Hadoop project, as well as Cloud Foundry, through services such as snapshot functionality to the file system service, chef cookbooks, bug fixes and enhancements. These contributions can also be found on our GitHub account at http://github.com/staticcorp.

About the Author: Jake Farrell is the Chief Technical Officer for Static Corporation. He specializes in scalable distributed back end infrastructure systems. He is actively involved in numerous projects with the Apache Software Foundation and is the Project Management Committee Chair for the Apache Thrift project.

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Want to Contribute to Cloud Foundry? Come on in!

Cloud Foundry is an Open Platform-as-a-Service, and an Open Source project. It has attracted phenomenal interest from the community - including partners, companies using the code internally, and those individual developers with a passion for getting involved. You can find the source code on Github. Community contributions are what help to make the platform so extensible. We are always happy when we receive a Github pull request to offer new functionality or fixes! We also appreciate bug reports submitted through Github Issues.

Looking at the Cloud Foundry project as a whole though… where should you start?

As you might imagine, there are a lot of moving parts in a PaaS. At Pivotal, we have teams working on the backend (the main runtime environment for Cloud Foundry, also known as ‘vcap’); the frontend (the tools and APIs for deploying apps, such as the ‘cf’ gem); services (database, message queues, and related resources); and Cloud Foundry BOSH, the installer and deployment tool, which has different plugins targeting infrastructures like AWS, OpenStack and vSphere. In the future, we would love to see even more Cloud Provider Interfaces (CPIs) for BOSH to allow Cloud Foundry to be deployed onto additional clouds.

One of the best ways to get involved with the community and the codebase is to start with the Github repositories and the mailing lists. If you are not sure where particular functionality resides within the project as a whole, and you’ve had a good look at Github already, then we have a number of mailing lists where you can ask for help. vcap-dev is for those working on the PaaS itself; bosh-users is for those who want to understand how to use BOSH to deploy Cloud Foundry; and bosh-dev is for the elite hackers (like Dr Nic Williams!) who want to help us make BOSH even more awesome. If you participate in the mailing lists you will also be able to keep up with what others are working on, and learn who some of the leading developers on the project are. If you want to get an idea of the general direction of development, you’ll find that in our roadmap.

Once you are up-and-running with Cloud Foundry, or if you know to which part of the project  you want to contribute, then consider the size of an initial contribution. If you are new to the project you might want to start with something small (some Open Source projects call these kind of small changes to smooth out any rough edges “paper cuts”). If you have something larger, we are always keen to work with you to understand how we can integrate your code. Before sending a pull request, take a look at our OSS contribution information in the project readme, which includes details on the Contributor License Agreement for individuals and corporations.

We thought it would be good to highlight a recent pull request which was a superb example of a major addition to the project. Recently we received a contribution via a detailed Github pull request, adding Oracle support to some of the major components inside Cloud Foundry: the User Account and Authentication service (UAA) and the Cloud Controller. Originally, these components were built to use Postgres as a backend, and more recently MySQL support was added. Since many enterprises already use Oracle as their main relational data store, it is great to give them this additional storage option.

Taking a look at the specific request for the Cloud Controller, we were really struck by a few things. The first one is that the pull request itself is very detailed, and walks through the various changes required in order to replace the database with Oracle.

This pull request is a lot to digest. If you want you can review the single feature PR that lead to this PR for some more concise context into some of the decisions I made in creating this PR. Here is a summary of the more notable changes I had to make and some context for why I made those changes:

This level of detail is not always required with every patch, of course, but for something this extensive it was very welcome. Another aspect of this patch is that was was made of up several individual changes (things like tweaks to the database schema, differences between Oracle datatypes and those used by other databases, etc), and they were broken down into separate commits which made it easier to consume. It also helped that this set of changes was extensively discussed on the vcap-dev mailing list before this larger pull request was submitted, so the engineering team were aware of some of the other impacts it would have.

Although the resulting pull request was detailed and clear, it has taken a few days to make all of the changes, as the core code was still being worked on. This was actually one of the rarer cases where we had to do more work to get the patches integrated – and that shows how willing we are to work with contributions. Smaller changes are extremely welcome too, and often much quicker to digest. Don’t be put off by this larger example – come and start hacking with us!

We hope this post highlights some of the things that help to make it smoother to contribute to the project. To recap:

  1. review the contribution guidelines and complete the CLA;
  2. if you want to submit a major code change, discuss it on the relevant mailing list beforehand;
  3. think about how the pull request is structured, and submit changes in separate consumable commits (rather than as one monolithic change) where appropriate;
  4. the patch should be well-tested, should include tests if possible, and should pass our Travis CI process;
  5. the changes should be clearly documented in the request, especially when the updates are more extensive than correcting a few typos.

The past few months have seen the Cloud Foundry Open Source community grow hugely, and mailing list activity continues to be at an all-time high – so we’re really looking forward to continuing to work with other developers to make the platform even more awesome! Thanks for being on this journey with us.

Andy Piper @andypiper

Developer Advocate, Cloud Foundry

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

NTT Contributes Nise BOSH, a Tool to Speed Up BOSH Development

NTT, the world’s leading telecom, has been active in fostering the Cloud Foundry developer and user community in Japan. We’re excited about NTT Lab’s latest open source community contribution to Cloud Foundry – Nise BOSH, a lightweight BOSH emulator for locally compiling, releasing and testing services. BOSH is an open source tool chain for release engineering, deployment and lifecycle management of large-scale distributed applications and services, such as Cloud Foundry. Nise BOSH allows developers and cloud operators who use BOSH to speed up the development feedback cycle while saving effort and money. In this post, we’ll explain how all this works and why it’s useful.

Normal BOSH Deploy Cycle

When you do a bosh deploy of a large scale distributed service such as Cloud Foundry to a cluster, BOSH does a number of things on your behalf:

  1. Prepares deployment
  2. Compiles packages – Calculates all packages and their dependencies that need to be compiled. It then begins compiling the packages and storing their output in the blobstore. The number of workers specified in the deployment configuration determines how many VMs can be created at once for compiling.
  3. Prepares DNS
  4. Creates bound missing VMs – Creates new VMs, deletes extra VMs.
  5. Binds instance VMs
  6. Prepares configuration
  7. Updates/deletes jobs – Deletes unneeded instances, creates needed instances, updates existing instances if they are not already updated. This is the step where things get pushed live.

In step #2 BOSH spins up a number of worker VMs to compile your code packages, then installs your compiled code in one or more VMs called stemcells. Normally all of these VMs are running somewhere on an IaaS provider such as VMware vSphere or AWS EC2.

fig1

Using Nise BOSH to Speed Up the Feedback Cycle

This is great in production, but when iterating on a BOSH Package, spinning up multiple VMs for compilation and deployment makes for a slow feedback cycle and a lot of ssh’ing into servers. It can also be costly, as you may have to pay for development resources on an IaaS, e.g. on AWS. Nise BOSH is a great help when developing a BOSH package. A Package is a collection of source code along with a script that contains instruction how to compile it to binary format and install it, with optional dependencies on other pre-requisite packages. Without Internet access or an IaaS, Nise BOSH will compile the packages necessary to run the job and install them on the box that you’re on, saving you the time, expense and complexity of using remote resources on AWS or another IaaS. Let’s see how this works.

First, start on a machine that looks like your target Stemcell. A Stemcell is a VM template with an embedded BOSH Agent. The Stemcell used for Cloud Foundry is a standard Ubuntu distribution. For this you can use Vagrant, a handy tool to create easily reproducible environments on your local laptop. Install Vagrant and then in a terminal:

$ vagrant init lucid64 http://files.vagrantup.com/lucid64.box 
$ vagrant up
$ vagrant ssh

You should now be in an Ubuntu lucid64 environment on your local machine. Now install Nise BOSH –

$git clone git@github.com:nttlabs/nise_bosh.git
$cd nise_bosh
$bundle install

Next you need to get a BOSH Cloud Foundry release – a BOSH release is a collection of source code, configuration files and startup scripts used to run services, along with a version number that uniquely identifies the components.

$git clone git@github.com:cloudfoundry/cf-release.git 
$cd cf-release 
$git submodule sync 
$git submodule update --init --recursive

You can now modify the Cloud Foundry open source code in cf-release/src as your project needs require. Once you are ready to test your changes, you need to create the release, but first install the BOSH CLI -

$gem install bosh_cli 
$bosh create release

Before you can deploy your release you’ll need a BOSH manifest file we’ll call deploy.conf. An example of the contents for a Nise BOSH manifest can be found in the Nise BOSH doc.

Now run Nise BOSH from the nise_bosh directory to launch dea_next. Without Internet access or an IaaS, Nise BOSH will compile the packages necessary to run the job and install them on the box that you’re on. Then just run run-job start to start the jobs in /var/vcap/bosh/etc/monitrc locally on your box.

$sudo PATH=$PATH bundle exec ./bin/nise-bosh ~/cf-release ~/deploy.conf dea_next
$./bin/run-job start

Nise BOSH is all set up to help you play with dea_next all by itself, but eventually it might enable to you develop and run Cloud Foundry right on your local machine. We see the many community contributions like NTT Nise BOSH as proof of the power of an Open PaaS ecosystem to advance the project. Next time you’re working on a BOSH release, give it a try.

–Matthew Kocher and Vikram Rana, Cloud Foundry Team

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Use of Hybrid PaaS Now and In the Future

From its beginning, Cloud Foundry has been committed to providing developers and enterprises choice of deployment options spanning public and private clouds. In this post, Jonas Partner, CEO of CloudCredo and OpenCredo, shares how their enterprise customers are using PaaS to deploy applications that span public and private clouds to increase availability, add extra capacity and prevent lock-in.

As one of the co-founders of OpenCredo I have been actively working with Cloud Foundry since its early days. We were one of the first companies to launch a production application on Cloud Foundry with the help of our friends at Carrenza. More recently we have established CloudCredo to help both enterprise customers wishing to host Cloud Foundry on their own infrastructure and cloud service providers wishing to offer Platform as a Service. We helped Ospero to bring a Cloud Foundry-based PaaS running on vCloud Director to market to better serve their customers.

With the growing availability of public Cloud Foundry instances, we are seeing an increase in interest from enterprise customers in hybrid cloud as a way to extract maximum value from their existing infrastructure while enabling capacity on demand through expansion into public cloud. In this blog we will discuss why hybrid PaaS with Cloud Foundry is gaining interest, some of the ways people are using it today and how we hope to be using it in the future.

Why Cloud Foundry for Hybrid PaaS

The availability of a PaaS that seamlessly spans in-house data centers and various public cloud offerings makes hybrid cloud a compelling commercial proposition. For example, it serves to drive competition among public cloud providers. Extra capacity can be bought from the most cost competitive provider, or spread across a number of providers, as determined by the best commercial fit for the need. To date the majority of public PaaS offerings have been closed source and can’t be run within the enterprise data center. Combine this form of lock-in with some well-publicized outages and most large enterprises simply aren’t willing to consider betting everything on a single instance cloud provider. We see the open, extensible nature of Cloud Foundry and the increasing number of companies offering hosted Cloud Foundry instances as key to making enterprises comfortable with PaaS.

For us the number of companies offering or intending to offer a Cloud Foundry instance is reaching a critical mass, giving enterprises a competitive set of choices of where to run their applications. The Cloud Foundry Core initiative also means that customers can rely on public Cloud Foundry Core compatible instances to meet certain minimum requirements. Taken alongside improved support for running Cloud Foundry on public IaaS such as EC2, enterprises now have three real options for deploying cloud applications: 1) on a Cloud Foundry instance running in house on existing infrastructure, 2) on a Cloud Foundry instance running on large scale public IaaS such as EC2 or 3) on one of the increasing number of cloud providers offering Cloud Foundry as a service.

Being able to pick a company offering a PaaS hosted within national boundaries is already broadening the applicability of PaaS to regulated industries that have requirements on data sovereignty and how data is managed. There are still many other considerations around technology stacks and non-functional requirements that determine whether Cloud Foundry is suitable for a particular workload.

Simple Hybrid Cloud Use Cases

We are seeing significant interest in hybrid PaaS as a way of adding capacity in peak periods and increasing availability. Currently this only works out of the box for simpler cases where 1) there is no need to massively scale the persistence technology and 2) the application’s database does not need to be shared between services on either side of the public private cloud divide. Where enterprises employ SOA it can also be challenging to find a subset of the services that can be moved out to the public cloud. While services can span public and private cloud, this adds complexity, potential performance challenges and cost where data stored in the public cloud is separately billable.

The factors that determine the simplicity of an application for the purposes of a hybrid cloud scenario are primarily how coupled it is to the rest of the enterprise. This coupling can take the form of persisted data, messaging and application services. The simplest case is an application that has none of these forms of dependencies – with the increasing adoption of SOA it is increasingly rare to find these types of applications. When considering the potential use of hybrid cloud, the cost benefit versus the complexity due to connectedness to the rest of the enterprise will be one of the key trade-offs.

cloud credo fig1

Unless for some reason you have a large number of island-like applications then batch processing applications can often provide a fertile ground in the early days of adopting a hybrid cloud deployment strategy. Batch applications often combine minimum data connectivity to the rest of the application, often running on a data snapshot, with providing cost benefit from scaling up just for the batch run.

Scalable Database Services

The open source Cloud Foundry project provides a number of open data technologies that can be configured to “scale up” as single instance services out of the box. We see incorporation of scale-out, multi-instance database services as key to widening the use cases that can be addressed by Cloud Foundry and are currently working to address this opportunity for our customers.

At CloudCredo we recently open sourced a simple Cassandra service integration for Cloud Foundry on GitHub. The initial implementation was targeted mainly at development and test usage since it was single instance and therefore didn’t provide any way to make the database scale out or highly available. We are now extending that out to a multi-node cluster and we intend that other services such as MongoDB will also follow.

Scalable Databases as a Service can be plugged in to Cloud Foundry very easily by leveraging the Service Broker. This pattern allows the cluster to be managed from outside of Cloud Foundry but for the ‘external’ cluster service to be exposed to clients as a running Cloud Foundry service. The benefits of this model include Cloud Foundry features such as central configuration management and automatic service binding from applications.

cloud credo fig2

While horizontal scaling and clustering of open database services opens up new potential, what really excites us is the potential to use the multi-data center capabilities of persistence technologies in conjunction with hybrid PaaS. Cassandra and other open data technologies such as Riak make it possible to have a logical database which spans multiple data centers and therefore potentially both public and private cloud.

cloud credo fig3

A PaaS which integrates multi-data center databases that are easy to deploy and provide data synchronization would make solving problems such as disaster recovery and transparent scaling of both database and application code much easier. Disaster recovery currently relies on expensive low latency RPO systems that require manual, offsite back-ups be taken to mirror production. A multi-node, multi-data center data service would have the potential to automatically mirror data across data centers as part of the online transaction processing of the application. Having effectively the same database available within both public and private PaaS would also simplify the process of scaling out.

Conclusion

The growing popularity and increasing number of ways to consume Cloud Foundry is making hybrid PaaS an appealing reality for many enterprises. Cloud Foundry provides the Service Broker to facilitate integration of new, clustered open data technologies and legacy clustered databases, easing enterprise adoption in cases where the database services that come as part of the platform are not sufficient. The combination of reliable open source technologies that can deal with databases and the increasing availability of Cloud Foundry is an exciting option for those trying to use PaaS within the enterprise. Being able to solve these sorts of traditionally expensive problems with a PaaS will make PaaS even more compelling for large enterprises.

About the Author: Jonas Partner is CEO of OpenCredo, a software consultancy and delivery firm with offices in the United Kingdom and North America. Recently, Jonas co-founded CloudCredo, a company focused on helping businesses get Cloud Foundry up and running on the infrastructure of their choice. Before establishing OpenCredo in 2009, Jonas was a consultant with Spring Source where he contributed to the Spring Integration project. Jonas maintains interests in Machine Learning and the scalability of highly concurrent enterprise systems.

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Anchora Brings Cloud Foundry Core Instance to China

A fast growing mobile apps market alongside the increasing need for a unified enterprise private cloud platform requires a new generation of open PaaS solutions in China.

In this guest blog, Wei-Min Lu, founder and CEO of Anchora, explains why Anchora selected Cloud Foundry as the basis for its MoPaaS platform and why it chose to join Cloud Foundry Core.

Test MoPaaS for Cloud
Foundry Core Compatibility here.

Today we are excited to join Cloud Foundry Core as proof of our commitment to cloud application portability across a choice of public and private clouds. Here in China, there is growing demand for an open PaaS that enables developers to build mobile applications and enterprises to build private or dedicated clouds. We built MoPaaS, a cloud application engine based on the Cloud Foundry service platform. Cloud Foundry gives us the flexibility to offer both a public cloud platform with extensions for mobile and data applications and a private/dedicated PaaS solution that integrates with enterprises’ existing infrastructure. We also provide services within Cloud Foundry applications allowing enterprises to take advantage of big data infrastructure for use cases like cloud storage and synchronization, analytics and search.

The Need for Open PaaS in China

It is projected that PaaS will grow at 30% annually over the next few years. Higher growth is expected to occur in China due to 1) a rapidly expanding mobile application market requiring an agile back-end services platform and 2) the need for cloud application portability and the rise of applications taking advantage of big data infrastructure in the enterprise.

China is now the world’s biggest smart phone market with 390 million units projected to be sold this year – more than 1/3 of smart phones to be sold globally in 2013. Businesses of all sizes are looking to take advantage of this trend, increasing the number of mobile apps by 35% annually. There is a growing need for an agile platform with additional services required for mobile applications, such as push notification, geospatial data storage and short message services. Due to the limited resources on the client device, mobile applications require a platform that is able to process large quantities of data in the cloud. A new generation of Open PaaS is needed to handle additional mobile-specific requirements.

With more than 70% of companies implementing PaaS over the next two years, we see emerging needs for PaaS solutions in the China enterprise IT market to consolidate existing applications and infrastructure while building an ecosystem of new enterprise applications in a consistent and unified manner. Enterprise application portability is increasingly becoming a critical need – businesses expect to be able to move their applications for optimal service, regulatory compliance, and data privacy requirements. As enterprises are realizing the impact and starting to build apps to take advantage of big data, there is an urgent need for more sophisticated platform services with big data infrastructure. This includes scalable services for log analysis, cloud storage and synchronization, information clustering, analytics, and customized search.

Building a Public PaaS Using Cloud Foundry

We built MoPaaS to serve developers’ needs to create applications and services without worrying about backend IT tasks. It helps them simplify and automate application lifecycle management, including development, deployment, and operation, significantly reducing IT infrastructure expenditures and devops costs and time. MoPaaS is built on an intelligent cloud service platform infrastructure that consists of two integrated parts:

  • Service platform: extends the Cloud Foundry service platform to provide better support for mobile applications, platform monitoring and management, and service extensibility.
  • Data platform: based on innovative information chain management (ICM) technology. It offers a simple interface for developers to support data driven applications.

CF_Core_Fig01

Fig 1. MoPaaS Cloud Application Engine Architecture

MoPaaS provides necessary devops automation, masks complexities from developers and enhances usability by offering additional functions:

  • UI: providing ease of use for developers of various levels and a CLI providing more flexibility and control for experienced developers. Developers can use the MoPaaS UI for devops tasks like application deployment and performing resource allocation.
  • MDSS (mass data storage service): a high performance distributed storage system that provides cloud storage for user data and application source code management.
  • Application monitoring and management: a streamlined visual tool and web console for monitoring and managing applications, services, and environment variables.
  • MoPaaS Services: Mobile specific services, such as notifications, short messages, and locations etc. Data services, such as scalable services for log analysis, cloud storage and synchronization, information clustering, and customized search.

Building an Enterprise PaaS Solution Using Cloud Foundry

MoPaaS also provides an intelligent IaaS-PaaS solution for enterprises and service providers to build dedicated clouds to boost service agility, reduce IT expenditures and devops costs while increasing the portability, scalability and reliability of enterprise applications. This solution offers necessary granular controls and customer configuration flexibility provided by IaaS and the devops automation and ease of app creation provided by PaaS. The MoPaas architecture strictly separates PaaS from IaaS services.

CF_Core_Fig02

Fig 2. MoPaaS Intelligent Cloud Platform

In addition to the services offered by Cloud Foundry, MoPaaS offers a comprehensive array of data services as native Cloud Foundry services and integrates various legacy services.

CF_Core_Fig03

Fig 3. MoPaaS Services

MoPaaS effectively addresses a wide range of the enterprise cloud platform requirements, including:

  • Application consolidation: there is a need for building and running an ecosystem of enterprise applications on a cloud-based platform in a consistent and unified manner. However, many of the applications in enterprises have been around for quite some time and were not designed for the cloud. Among the key requirements, legacy services used by those applications need to be integrated into the cloud platform or upgraded to new services of the cloud platform. MoPaaS addresses the application consolidation and migration issues effectively, as the MoPaaS platform not only supports basic services, but also adds new data services to meet enterprise big data requirements using Cloud Foundry service gateway and seamlessly integrates legacy services used by existing applications, such as Oracle Database, Microsoft SQL Server, and IBM DB2 using Cloud Foundry service broker (Figure 3).

  • Data platform: Big data has increasingly become top of mind across organizations, from the data center to the boardroom. Companies are seeking systems and tools capable of tackling the challenges of massive data growth. Integrated with Hadoop-based Information Chain Management (ICM) data services, MoPaaS enables organizations to deploy a central platform to solve end-to-end data problems that involve any combination of information ingestion, processing, storage, exploration, analytics, and retrieval. With MoPaaS enterprise developers can easily integrate sophisticated big data services with their applications as native Cloud Foundry services. These include log analysis, cloud storage and synchronization, information clustering, and customized search. In addition, with ICM data services, MoPaaS provides new capability for monitoring and managing applications on the cloud platform.

  • Application portability: MoPaas architecture strictly separates PaaS from IaaS services. MoPaaS furnishes applications with an infrastructure-agnostic execution method, standardizes the method of exposing services such as databases, messaging and queuing, runtime, and management, and standardizes service credentials across runtimes. Thus MoPaaS keeps the applications portable in case they need to move the PaaS layer and application services to another infrastructure provider. Additionally, MoPaaS ICM mechanism enables automatic application data transferring along with the associated applications via its data synchronization service. Developers don’t have to worry about infrastructure-specific requirements for their applications, because the MoPaaS platform takes care of these concerns for them.

Summary

The initial success of Cloud Foundry-based PaaS providers demonstrates the strength of the Cloud Foundry open PaaS solution and the power of its open source ecosystem and community. This is a win for everyone invested in the success of Cloud Foundry.

We are excited to become the first company in China to join the Cloud Foundry Core community. By being a member of the Core ecosystem, Anchora is committed to the open PaaS concept and delivering cloud portability to Chinese application developers and service providers.

About the Author: Wei-Min Lu is the founder and CEO of Anchora. Anchora provides both MoPaaS, a public open cloud platform with extensions for mobile and data applications, and a dedicated PaaS solution that facilitates application consolidation and cloud platform portability for enterprises. Dr. Lu has over 15 years experience in product development and marketing. Before he founded Anchora in 2008, he served in key engineering and management positions at IBM and NASA-JPL. His background is in cloud computing, cybernetics, machine learning, information retrieval and storage technologies. Dr. Lu holds a Ph.D. in Electrical Engineering/Math from CalTech and a B.Sc. from Tsinghua University, China.

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Cloud Foundry is Open and Pivotal

For those who have not heard, Cloud Foundry, the open platform-as-a-service project, is now part of the recently announced Pivotal Initiative, an independent entity funded by VMware and EMC. (For a preview of our Paul Maritz-led Pivotal Initiative, check out this Wired article).

Our independent status is quite meaningful and gives us the ability to engage with our large ecosystem in new ways. Adding full-time external committers has always been a goal of the team, and we are engaged with several organizations about putting dedicated resources on the extended engineering team –we believe this to be a very important step forward. The scale of these external investments is significant and a major milestone in our growth. The heart of Cloud Foundry, however, really comes from individual community contributions and users, so of course, we invite you to join us. All you need to do is send a pull-request.

As more and more organizations contribute to Cloud Foundry a transparent roadmap to the community is paramount. In January we began publishing a quarterly plan as part of our community documentation and since then our mailing list activity has hit an all-time high. We will be prolific in communicating the updated roadmap and updates on new work including logging APIs, .NET support, Cloud Foundry BOSH enhancements and more.

Cloud independence and multi-cloud support are important beliefs for us and the broader Pivotal Initiative. Expect open interfaces and support for a variety of clouds, with continued development on AWS, Openstack, vCloud and vSphere environments.

Our own financial and human investment in the project continues to grow. Cloud Foundry is staffed with over forty full time internal engineers, augmented by an additional team of twenty five spread across PM, developer advocacy and community management.

Last week we began actively selling software update and support subscriptions to large customers, another important milestone. Enterprise interest in building large scale internal agile platforms with Cloud Foundry is tremendous and we expect an interesting new backlog of feature requests from these engagements. The only proprietary code in these subscriptions is our web console, so any new enhancements will immediately benefit the larger community.

Finally we have again weighed the best governance model for the project and believe the current corporate sponsored, Apache 2 licensed, pull request driven approach, remains the right one. Adding external committers will also enhance the diversity and strength of the team. That said, the massive growth of the community and ecosystem requires mediating a diverse set of needs and we will always be open to other governance models for the project in the future.

Our mission is to become the most popular platform for new applications and the services used to build them across public and private clouds. If your organization would like to join us in building something great please contact us here.

James Watters, Head of Product

The Cloud Foundry Team

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

NTT Communications, World’s Leading Telecom, Joins Cloud Foundry Core

As the world’s leading telecom, NTT Communications has experience delivering global cloud services to enterprise customers. Enterprises in Japan are seeking the openness, choice, agility and extensibility that an Open PaaS can provide.

In this guest blog, Hideki Kurihara, Product Lead for NTT Communications’ Global Cloud Services, explains why NTT Communications selected Cloud Foundry as the basis for its Cloudn PaaS and why it chose to join Cloud Foundry Core.

Test NTT’s Cloudn PaaS for Cloud
Foundry Core Compatibility here.

We are excited to join the Cloud Foundry Core community as proof of our commitment to making cloud portability a reality for Japanese developers. As the world’s leading telecom, everyday we see customers interested in using PaaS to be more agile. But we also hear concerns about vendor lock-in and ability to meet the needs of a complex enterprise environment. We chose to build Cloudn PaaS on top of Cloud Foundry because of its multi-cloud nature, ability to integrate with existing assets, and solid API foundation for adding management and monitoring features. Using Cloud Foundry as the base, we are extending Cloudn PaaS for developers and enterprise customers in Japan. Together with other Cloud Foundry Core partners, we are delivering cloud portability to Japanese users as well as global users of Cloud Foundry.

The Drivers for Open PaaS in Japan

The customer community in Japan wants more agility, choice, extensibility and manageability when it comes to cloud. Customers across sectors are looking to innovate faster in response to a very dynamic environment. For example, cloud has been used extensively for emergency infrastructure services in response to recent natural disasters. There has been a sharp increase in demand for cloud services to support smartphone and tablet application development. More global businesses have started considering cloud services as part of their global IT standardization so applications can be ported on demand from one place to another.

Customers here also want choice – choice to add new frameworks and services, and even to port their applications back on-premise or to another provider in Japan and other countries. Enterprises tend to prefer customization to augment their business. But they also value industry or open standards in the services they consume so they can deploy their application assets in the most appropriate place. When it comes to PaaS, application portability needs to be taken more seriously. Open PaaS, that is PaaS based on an open or industry standard, is what customers are talking about here.

Choosing and Building a PaaS Around Cloud Foundry

Meeting the needs of developers and operations for a PaaS that is open, extensible, and easily managed in the enterprise environment is difficult. We chose to build Cloudn PaaS on top of Cloud Foundry specifically because of its:

1) Openness and portability

2) Flexibility – in configuration and ability to integrate with existing enterprise assets and services

3) Extensibility – the ability to add frameworks and services, but even beyond that, to build UI based tools that make it easy to manage and add resources to applications and navigate very large installations

We’ve relied heavily on the open source nature of Cloud Foundry. We have added memcached, our own filesystem service, and support for Java web application servers (Resin/Resin Pro) so our customers’ existing applications can work seamlessly on Cloudn PaaS. But Cloud Foundry Core provides our customers’ developers the choice to use a common floor of runtimes and services when programming their applications. This gives them assurance they can port their applications to a Cloud Foundry Core provider in another geography not served by NTT communications or to a Cloud Foundry instance in their data centers.

Once in their own data centers, customers typically have a heterogeneous mix of infrastructure on which they wish to deploy Cloud Foundry, depending on the SLA and tenancy models. We initially deployed Cloud Foundry on top of our Enterprise Cloud services platform using vSphere and vCloud Director. We use this instance as our own Cloud Foundry development environment and plan to use a similar environment for future enterprise/private PaaS. On the other hand, the public multi-tenant instance of Cloudn PaaS has been deployed on a different cloud platform with attributes and services more tailored for developer centric markets.

Cloud Foundry is designed with sufficient flexibility to satisfy a huge variety of needs for PaaS environments. The Cloud Foundry system consists of several components, and users can set up their environment by choosing the necessary combination and number of components for their needs Fig. 1

fa2_fig02

Cloud Foundry supports a wide range of environments—from a private PaaS on a single server to a huge public PaaS on a cluster of over a thousand servers—and can be set up to suit the scale as well as its reliability and support features. Moreover, users can add components on demand, so it is possible to start with the minimum configuration and increase the scale as the load grows.

Enterprise integration is one key area where we see area for adding value to our customers on top of Cloud Foundry. The Cloud Foundry services gateway component addresses the ability to add enterprise data services that our customers request for HA, recovery, and backup. The service gateway itself is nothing more than a REST gateway for binding any service, but the real magic is in the service node implementation that allows multi-node deployments for HA and clustering. We are working to create our own multi-node deployments of services using this component such as the aforementioned filesystem and memcached services.

NTT Communications’ Customizations of Cloud Foundry for Enterprises in Japan

Although Cloud Foundry is a promising PaaS software solution that has many good features, as described above, it is still not perfect. For example, integration with other NTT Communications’ services was naturally not provided. Therefore, we have been making efforts to extend Cloud Foundry to meet the specific needs of our enterprise customers and developers in Japan.

Contributions and Customizations to Cloud Foundry

(1) Reliability of Cloud Foundry components

Because Cloud Foundry was a very young project when we started our development, NTT Communications and NTT R&D (Software Innovation Center) have been examining the performance and scalability of Cloud Foundry by conducting various tests and fixing any problems revealed by them. For example, we solved a problem involving an important component that was a single point of failure and a problem where some components could not be restored after failures by fixing the source code and adding additional external systems.

(2) Convenience of Cloud Foundry

Since convenience is important for commercial services, we created an installer for the virtual machine container (VMC), which is the console for PaaS users, and we are now developing a function for linking the VMC and version control systems such as Git.

(3) Control Panel for user application management

Although Cloud Foundry has a flexible component system as described above, it is essential to provide a more user-friendly environment to fully leverage its features. Therefore, we have developed a control panel that allows users to manage their applications on Cloudn PaaS. The control panel displays a list of the user’s applications, information about each application, status of resources for each application instance, and each application’s logs that have been stored by means of Cloudn PaaS services. Using the control panel, operators can flexibly scale-out or scale-up their applications instances. Fig. 2ntt

Building a Cloud Foundry Community in Japan

The presence of active communities is indispensable in the long-term evolution of open source software. We launched the Japan Cloud Foundry Group and are fostering a developers’ and users’ community of Cloud Foundry in Japan. In addition, we will open the source code created by group members by committing them to the official Cloud Foundry repository, and we also provide the know-how in workshops.

About the Author: Hideki Kurihara is the Product Lead for Global Cloud Services, including PaaS, for NTT Communications. Prior to his current position, he was engaged in NTT Communications’ North American hosting and cloud business for 10 years, serving in managerial positions such as General Manager of the Enterprise Hosting business unit and Vice President of Global Product Strategy at its subsidiaries. Hideki holds a Master of Science degree from University of Tokyo in Japan and a Master of Business Administration degree from INSEAD in France.

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Recent Changes in Node.js Modules Support

Since Cloud Foundry introduced npm support, we added several improvements that make deploying Node.js applications with dependencies easier, faster and more transparent to developers.

Easy development

When we introduced npm support in Node.js, in addition to generating “npm-shrinkwrap.json” file with locked down dependencies you were required to remove the “node_modules” folder. If this was too disruptive, you could create an “cloudfoundry.json” file with an “ignoreNodeModules” property set to ‘true’.

This wasn’t an obvious solution. So we decided to change it. Detection of native modules is now done automatically by analyzing module contents. To lock down module versions, you can simply use the “npm shrinkwrap” command and your application is ready to be pushed with or without bundled modules. With this change, we have deprecated ignoreNodeModules property.

If you want to store your application in a git repository without node modules, the next time someone clones it, he/she can just push it to CloudFoundry.com without any modifications.

Fast deployment

In addition to  npm, we also improved our cache system. There are two levels of caching now — fetched and compiled modules. All modules that have been fetched from the npm registry are cached locally and pulled from that cache from that point on. If a module contains native code, it gets compiled on Cloud Foundry and the compiled version gets cached as well.

Based on the information from npm-shrinkwrap.json, Cloud Foundry checks if the module has been provided with the application itself. If it is there, and contains native code, it rebuilds it as needed. If the module hasn’t been provided with the app, Cloud Foundry checks the local cache for the given module, and if it needs compilation, provides the cached version.

As module contents can be modified by users, we are using hash of module files as a cache key.

Local modules

There may be situations when you need to customize an existing node module or move a part of your application as a reusable module. In this case, you would want your local module to be used without any modifications and be rebuilt, should it contain native code.

With automatic detection of native code – such modules will be rebuilt on Cloud Foundry and cached, based on their contents for your next push.

More informative logging

In addition to these node improvements, you can now see a detailed log of what’s going on in log files. You can get log messages by running:

vmc logs application-name

We can see if npm support is enabled (npm-shrinkwrap.json was provided) and which node version we are using:

Installing dependencies. Node version 0.6.8

For each node module we can see its installation status:

Installing express@2.5.11 from local path

In case of a failure we can see an error message and/or npm error output.:

Failed installing bcrypt@0.5.0: Node version requirement >= 0.6.0 is not compatible with the current node version 0.4.12

Node-gyp

Modules using node-gyp for the installation process are now supported on CloudFoundry.com for node08 runtime. For node06 and node04 runtimes, Cloud Foundry falls back to node-waf if the module provides the wscript file.

Git URLs

Another recent improvement is that we also support git URLs in npm-shrinkwrap.json. They can be specified in “from” or “version” fields according to npm requirements and both methods are supported. We have a separate cache for git modules and they also get compiled on Cloud Foundry if they have native extensions.

Example

In the previous blog post on auto-reconfiguration we used a Node app, “calipso”,  and performed additional steps for npm support. Let’s see how pushing calipso works now.

First, we clone calipso source from github:

$ git clone git://github.com/cliftonc/calipso.git

Then, we install node modules and lock them down.

$ npm install
$ npm shrinkwrap

Now, we can push our application.

$ vmc push calipso-app --runtime=node08

As we push we add mongodb service and set memory to 256M.

Would you like to deploy from the current directory? [Yn]:
Detected a Node.js Application, is this correct? [Yn]:
Application Deployed URL [calipso-app.cloudfoundry.com]:
Memory reservation (128M, 256M, 512M, 1G, 2G) [64M]: 256M
How many instances? [1]:
Bind existing services to 'calipso-app'? [yN]:
Create services to bind to 'calipso-app'? [yN]: y
  1: blob
  2: mongodb
  3: mysql
  4: postgresql
  5: rabbitmq
  6: redis
What kind of service?: 2
Specify the name of the service [mongodb-62f4d]:
Create another? [yN]:
Would you like to save this configuration? [yN]:
Creating Application: OK
Creating Service [mongodb-62f4d]: OK
Binding Service [mongodb-62f4d]: OK
Uploading Application:
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (82K): OK
Push Status: OK
Staging Application 'calipso-app': OK
Starting Application 'calipso-app': OK

That’s it! We didn’t need to configure our application at all. In addition,  notice that staging process was faster and the preparation was effortless. When we look at the logs we can see how everything went. Try it and see how easy it is to manage your dependencies in Node.js applications now.

- Maria Shaldibina
The Cloud Foundry Team

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Preserving Cloud Application Portability – Introducing Cloud Foundry Core

The Cloud Foundry team is happy to announce Cloud Foundry Core  -  a program designed to preserve cloud application portability.

In the cloud computing world, preserving a choice of clouds is critical. The risks of being locked into a single cloud are substantial. Pricing, reliability, geographic location and compliance can all vary between clouds. Business requirements will evolve over time, necessitating the ability to move between clouds, whether public to private, private to public or between public cloud providers.

Cloud Foundry Core provides a baseline of common capabilities and an open mechanism to instantly validate application portability.

Cloud Foundry Core defines specific versions for application services and runtimes and introduces a system of current, next, and deprecated to provide access to feature innovation, enhanced performance and greater stability as applications continue to evolve.

AppFogTier 3Uhuru SoftwareMicro Cloud Foundry™ and CloudFoundry.com are now Core compatible as part of their commitment to preserving cloud portability.

-Dekel Tankel
The Cloud Foundry Team

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

New Release of Micro Cloud Foundry

We are excited to announce a new version of Micro Cloud Foundry™ with a new set of features. Among those new features is a new process to streamline frequent updates so that we can maintain compatibility between Micro Cloud Foundry and any Cloud Foundry-based clouds, including CloudFoundry.com.

If you don’t have a Cloud Foundry
account yet, sign up here.

Micro Cloud Foundry is a complete version of the Cloud Foundry open PaaS, but it runs in a single virtual machine on a developer’s computer. Micro Cloud Foundry exemplifies how a multi-cloud approach to PaaS can help developers easily develop and test their applications locally and deploy to any Cloud Foundry-based clouds with no code or configuration changes.

Download the new Micro Cloud Foundry here.

In this blog, we will review notable new features and focus on how we will be releasing Micro Cloud Foundry going forward.

What’s New

We have a new Micro Cloud Foundry for you to download. Since the release of the last version (1.2), there have been many new features that have been implemented for Cloud Foundry. Now these features have been made available on Micro Cloud Foundry. Let’s review some of them:

  • Standalone apps support: This feature enables a new class of apps including background apps such as Resque workers, apps with bring-your-own-container, and Spring background tasks.
  • Enhanced Ruby support: While Cloud Foundry always offered auto-reconfiguration for relational databases for Ruby apps, we’ve since extended it to all service types, enabling many more applications to deploy to Cloud Foundry without changing a single line of code. We now also offer improved support for Rails 3.1 and 3.2 apps, Rack as a supported framework, the ability to run JRuby apps, and apps that specify git URLs in their Gemfiles along with numerous small improvements to make a much wider range of applications work well in Cloud Foundry.
  • Enhanced Java support: Java 7 support, which enables applications such as vert.x and containers that need NIO such as Netty.
  • Enhanced Scala support: We now offer explicit support for Play 2.0. You can also deploy apps using a Scala framework such as Unfiltered that requires Jetty and Blue Eyes that additionally requires the NIO support in Java 7.
  • Enhanced Node support: We took a comprehensive look at our Node.js support and made a series of improvements to make Node.js apps shine. We now offer NPM support, bringing it up to par with the Gemfile support for Ruby apps including support for packages specified as git URLs. We also offer auto-reconfiguration of all services for node.js apps.

Download the new Micro Cloud Foundry from the download site. While there, you may notice a few changes, especially the availability of multiple versions and curiously new version names. Let us explain what they mean.

Going Forward

One of the ways developers use Micro Cloud Foundry is to deploy applications during development, where Micro Cloud Foundry is used as proving ground. Rather than installing a web server (Tomcat, etc.), runtimes (Java, Ruby, etc.), and services (Postgres, MongoDB, etc.), you can do a single download of Micro Cloud Foundry, boot it up, and deploy your applications using ‘vmc push’. You can run Java and Node.js apps in debug mode, take advantage of JVM hotswap to obviate the need to restart apps after changes, use a shell to access applications and services in the Micro Cloud Foundry VM, and so on. And you could do all this without even being connected to the network thanks to the offline capability of Micro Cloud Foundry (allowing you to develop apps for the cloud while flying through the clouds!).  Once you have your app working, you can push it to CloudFoundry.com, your own in-house Cloud Foundry cloud, or one of Cloud Foundry’s multi-cloud partners.

For all of this flow to really work, it is imperative that Micro Cloud Foundry has the same functionality as CloudFoundry.com. This means we must continually update Micro Cloud Foundry to keep up with all the new improvements in the core code. Our manual process of building and testing the Micro Cloud Foundry VM simply wouldn’t be up to the task. So we took a hard look at how to make this a better process, starting with this release.

This is how we will proceed:

  • Frequent releases: We will now release Micro Cloud Foundry whenever a change is pushed to CloudFoundry.com (subject to passing automated testing). If you track the cf-release repository, you will see it tagged with a release version such as v116, v114, and so on. Micro Cloud Foundry versions will match the same tags (along with a timestamp) so you can tell if you need to update Micro Cloud Foundry in order to catch up with changes in CloudFoundry.com. Since the launch over a year ago, we have been updating CloudFoundry.com about twice a week so you can expect a new Micro Cloud Foundry release at the same frequency. Later, we will consider releasing even more frequently for those who want to stay on the bleeding edge.
  • Automated building and testing: In the earlier releases, while we had scripts that would build Micro Cloud Foundry, there was some manual process involved. We also had a QA team testing Micro Cloud Foundry. This was in part necessitated by the requirement to run on multiple platforms (Fusion or Workstation for Mac, Windows, and Linux). Given our desire to update Micro Cloud Foundry bits at the same frequency as CloudFoundry.com, automated building and testing becomes an obvious requirement. We have already automated building Micro Cloud Foundry and doing a basic verification. Over the coming months, we will automate most, if not all, aspects of building, testing, and distributing Micro Cloud Foundry. Of course, Micro Cloud Foundry like the rest of Cloud Foundry, is an open source project, so you can definitely pitch in.
  • Improved user interface: Automated testing requires that the tests be able to configure Micro Cloud Foundry. When we decided to automate building and testing, we realized that while there is a way to do this through the command interface (by directing terminal input and output), it is difficult to make this work in a robust manner. Independent of this, we have been pondering a web interface that will provide an experience similar to a home network router. In a few weeks, we will provide a REST interface to Micro Cloud Foundry along with a web interface replacing the terminal interaction.

We hope you will like what Micro Cloud Foundry has in store. You can follow our frequent releases on the Micro Cloud Foundry site.

-Ramnivas Laddad and Matt Boedicker
The Cloud Foundry Team

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email