CloudFoundry.com Blog

In this guest post, Jake Farrell, CTO for Static.com, explains how the major shift in the hosting industry towards platforms for high developer productivity and data-centric applications led them to build their Platform as a Service using Cloud Foundry with Hadoop support.

Test Static.com for Cloud
Foundry Core Compatibility here.

The hosting industry is undergoing a dramatic transformation due to increasing demand for self-service cloud based capabilities. To greet this shift, Static.com, a subsidiary of Catalog.com, is leveraging 19 years of hosting experience, and adopting Cloud Foundry as the foundation for its Platform as a Service (PaaS). This move demonstrates the company’s commitment to providing modern, easily-installed and managed cloud-based solutions to hosting companies and enterprises.

To address the demand for self-service cloud hosting, Static has designed the Static Cloud Manager, a robust yet easy-to-use interface. Static’s Cloud Manager leverages Cloud Foundry to allow end users and enterprises to configure, deploy and scale applications and services in seconds. Additionally, Static has expanded on Cloud Foundry’s core platform and developed a growing catalog of OSS applications, frameworks and services for end users to choose from.

With this growing array of options comes a greater need for insight into application metrics and usage patterns. Static has integrated Apache Hadoop with Cloud Foundry as one of the first steps in what it sees as the cloud’s next major offering: Analytics as a Service (AaaS). Thanks to the power and flexibility of Cloud Foundry, these robust new offerings are rapidly evolving the cloud.

Static adds Apache Hadoop to Cloud Foundry

As PaaS and Infrastructure as a Service (IaaS) become the new hosting standard, having the ability to gain insight into usage and the performance of sites and cloud-based applications is essential. Static believes that the AaaS offering will allow users to make educated decisions about their sites and applications based on usage patterns. The first step to achieving this is to be able to store and crunch large amounts of big data into usable metrics, and Apache Hadoop fits this need perfectly.

Adding services like Apache Hadoop to Cloud Foundry allows Static’s users to easily add cost-effective big data solutions to address custom application needs. Adding Apache Hadoop as a pluggable service inside Cloud Foundry is the first step to automating AaaS to any site or application within the PaaS, enabling customizable metrics and immediate insight into usage patterns.

The Shift in Hosting

Traditional hosting offers minimal choices to the user, forcing them into shared pools where sites and applications are not guaranteed any dedicated resources. Shared hosting has worked for many users and hosting companies, but Static has seen an increase in demand from users who are switching to Virtual Private Servers (VPS) solely for the ability to have a dedicated set of resources and the freedom to choose what they run. This use case places the burden on the user to install, setup and administer the entire process, from start to finish.

Static’s goal was to rethink this approach and to allow for users to have the ability to choose:

• An expanding applications list ready to be installed with one click
• All the latest languages and version to choose from
• Latest frameworks ready to be use
• Full control over memory limits
• Ability to scale instances easily
• Choose from array of services
• Sync code between test and production environments
• Backup and rollback applications and services with ease

Static has leveraged Cloud Foundry to help make this a reality with the Static Cloud Manager, an easy-to-use but robust interface which interacts with Cloud Foundry. Users are able to 1-click create a Platform as a Service application with a wide array of frameworks and services while taking advantage of integrated hosting features such as domain registration, DNS, billing, support, Virtual Private Servers (VPS), and email.

The Cloud Network

Having choices does not stop at applications and services. Static is working to create a unified cloud network of datacenters and hosting companies that will:

• Run your application in multiple data centers for high availability
• Migrate applications seamlessly from one datacenter to another while maintaining uptime
• Choose your application and services based on data centers capable of meeting your needs

Static’s Commitment to Open Source Development

Static believes that open source software development drives innovation and leads to higher quality software. We are deeply committed to the open source ecosystem and being a leader in the Cloud Foundry community. Static is proud to contribute back to the Apache Hadoop project, as well as Cloud Foundry, through services such as snapshot functionality to the file system service, chef cookbooks, bug fixes and enhancements. These contributions can also be found on our GitHub account at http://github.com/staticcorp.

About the Author: Jake Farrell is the Chief Technical Officer for Static Corporation. He specializes in scalable distributed back end infrastructure systems. He is actively involved in numerous projects with the Apache Software Foundation and is the Project Management Committee Chair for the Apache Thrift project.

Cloud Foundry is an Open Platform-as-a-Service, and an Open Source project. It has attracted phenomenal interest from the community - including partners, companies using the code internally, and those individual developers with a passion for getting involved. You can find the source code on Github. Community contributions are what help to make the platform so extensible. We are always happy when we receive a Github pull request to offer new functionality or fixes! We also appreciate bug reports submitted through Github Issues.

Looking at the Cloud Foundry project as a whole though… where should you start?

As you might imagine, there are a lot of moving parts in a PaaS. At Pivotal, we have teams working on the backend (the main runtime environment for Cloud Foundry, also known as ‘vcap’); the frontend (the tools and APIs for deploying apps, such as the ‘cf’ gem); services (database, message queues, and related resources); and Cloud Foundry BOSH, the installer and deployment tool, which has different plugins targeting infrastructures like AWS, OpenStack and vSphere. In the future, we would love to see even more Cloud Provider Interfaces (CPIs) for BOSH to allow Cloud Foundry to be deployed onto additional clouds.

One of the best ways to get involved with the community and the codebase is to start with the Github repositories and the mailing lists. If you are not sure where particular functionality resides within the project as a whole, and you’ve had a good look at Github already, then we have a number of mailing lists where you can ask for help. vcap-dev is for those working on the PaaS itself; bosh-users is for those who want to understand how to use BOSH to deploy Cloud Foundry; and bosh-dev is for the elite hackers (like Dr Nic Williams!) who want to help us make BOSH even more awesome. If you participate in the mailing lists you will also be able to keep up with what others are working on, and learn who some of the leading developers on the project are. If you want to get an idea of the general direction of development, you’ll find that in our roadmap.

Once you are up-and-running with Cloud Foundry, or if you know to which part of the project you want to contribute, then consider the size of an initial contribution. If you are new to the project you might want to start with something small (some Open Source projects call these kind of small changes to smooth out any rough edges “paper cuts”). If you have something larger, we are always keen to work with you to understand how we can integrate your code. Before sending a pull request, take a look at our OSS contribution information in the project readme, which includes details on the Contributor License Agreement for individuals and corporations.

We thought it would be good to highlight a recent pull request which was a superb example of a major addition to the project. Recently we received a contribution via a detailed Github pull request, adding Oracle support to some of the major components inside Cloud Foundry: the User Account and Authentication service (UAA) and the Cloud Controller. Originally, these components were built to use Postgres as a backend, and more recently MySQL support was added. Since many enterprises already use Oracle as their main relational data store, it is great to give them this additional storage option.

Taking a look at the specific request for the Cloud Controller, we were really struck by a few things. The first one is that the pull request itself is very detailed, and walks through the various changes required in order to replace the database with Oracle.

This pull request is a lot to digest. If you want you can review the single feature PR that lead to this PR for some more concise context into some of the decisions I made in creating this PR. Here is a summary of the more notable changes I had to make and some context for why I made those changes:

This level of detail is not always required with every patch, of course, but for something this extensive it was very welcome. Another aspect of this patch is that was was made of up several individual changes (things like tweaks to the database schema, differences between Oracle datatypes and those used by other databases, etc), and they were broken down into separate commits which made it easier to consume. It also helped that this set of changes was extensively discussed on the vcap-dev mailing list before this larger pull request was submitted, so the engineering team were aware of some of the other impacts it would have.

Although the resulting pull request was detailed and clear, it has taken a few days to make all of the changes, as the core code was still being worked on. This was actually one of the rarer cases where we had to do more work to get the patches integrated – and that shows how willing we are to work with contributions. Smaller changes are extremely welcome too, and often much quicker to digest. Don’t be put off by this larger example – come and start hacking with us!

We hope this post highlights some of the things that help to make it smoother to contribute to the project. To recap:

review the contribution guidelines and complete the CLA;
if you want to submit a major code change, discuss it on the relevant mailing list beforehand;
think about how the pull request is structured, and submit changes in separate consumable commits (rather than as one monolithic change) where appropriate;
the patch should be well-tested, should include tests if possible, and should pass our Travis CI process;
the changes should be clearly documented in the request, especially when the updates are more extensive than correcting a few typos.

The past few months have seen the Cloud Foundry Open Source community grow hugely, and mailing list activity continues to be at an all-time high – so we’re really looking forward to continuing to work with other developers to make the platform even more awesome! Thanks for being on this journey with us.

Andy Piper @andypiper

Developer Advocate, Cloud Foundry

Cloud Foundry Blog

Tag Archives: open source