Cloud Foundry Blog

About James Bayer


Remote Dependencies, Convenience, Risk and Other Considerations for Operating Distributed Systems

IMG_0329 One deeply held principle by experienced distributed system operators that I have worked with is that you should have no external dependencies to your software other than the ties to minimum requirements of the OS such as common system libraries, utilities, and the kernel of the base OS. This approach should enable recreating a distributed system deployment without any dependencies on the outside world. When something goes wrong, you should have control over your own destiny. Reliance on any external dependency that is managed or hosted by someone else introduces risk that something outside your system can affect your ability to restore and recreate the system any time you need to.

To use a simple metaphor, imagine your system is represented by Jenga blocks and it falls over as Jenga towers inevitably do. However, instead of being able to rebuild your tower you find out that a mandatory required component at the base of your tower is missing or unavailable. No matter what you try, you cannot rebuild the tower exactly how it was before. Your new tower is going to behave differently in unexpected ways and you might topple over because you do not understand all the behaviors when using different building blocks combined in a different way.

Some of the original designers of the software deployment project for Cloud Foundry named BOSH (Mark Lucovsky, Vadim Spiwak, Derek Collison) embraced this principle and tried to create a prescriptive framework that encouraged this approach. They had experience managing large scale distributed systems at Google (the web services APIs). Kent Skaar also did similar for SaaS provider Zendesk. Given a software release that references specific versions of multiple software packages (known as a BOSH release), an instantiation of that release (a BOSH deployment) can be reconstructed at any time with the deployment configuration (a BOSH deployment manifest), the base OS images (the BOSH stemcells) and the software release (the BOSH packages and job templates for applying configuration); at any point in time, properly implemented BOSH releases of large scale distributed systems can be recreated without external dependencies. That means this holds true even when the internet is unavailable.

BOSH does give you the framework hooks to break out of this prescriptive principle and use external dependencies or at least external dependency formats if you choose to for convenience or other reasons. Dr Nic Williams recently implemented tooling to use apt packages instead of compiling from source. another example: some of the Pivotal big data software intentionally targets CentOS/RHEL only and therefore only ships rpm packages rather than compiling Hadoop. A guiding principle is that you should be mindful of the tradeoffs you are making of convenience vs risk and tying your release to only one OS distributor.

Examples of the tradeoffs:

  • relying on an externally hosted package manager like apt-get could affect the availability or correctness of that dependency when you need it most
  • relying on debian packages could prevent someone from using your release unmodified with a CentOS image

A recent real-world example demonstrated the risk of an external dependency changing unexpectedly. The coreos/etcd project that Cloud Foundry is using for storing stateful configuration data for the new Cloud Foundry Health Manager codebase had one of the dependencies (goraft/raft) force push to master of their git repository that overwrote some git history required by git to work properly. This situation has limited the flexibility of some users to make code modifications on several previous releases of Cloud Foundry without some tedious intervention.

A common reaction when learning about Cloud Foundry BOSH is to question the prescriptive guidance to compile from source when commonly used distributed package management systems exist in the Linux distributions. My recommendation is to understand the tradeoffs involved and make the best choice for your situation. You should explicitly call out external dependencies if you have them in your system. When your tower inevitably falls over, know how to rebuild it.

Thanks to Jose Hernandez for the Jenga image

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Deploying Tomcat 7 Using the Standalone Framework

The new standalone framework support greatly increases the number of different types of non-Web applications that can run on Cloud Foundry, including application servers. This tutorial will walk you through the steps to deploy a “hello world” application in a Tomcat 7 container on Cloud Foundry. Currently, Cloud Foundry leverages Tomcat 6 to host Java web applications. While the team is working to support Tomcat 7 as a first-class container, it is very straightforward to use the standalone application support to run Tomcat 7 in the meantime, which is particularly useful for applications that leverage Servlet 3.0. The basic outline involves installing your application into a local Tomcat 7 instance, making minor modifications to the configuration and pushing the entire contents of Tomcat 7 and your application to Cloud Foundry as a standalone application.

Step 1 – Download Apache Tomcat

Download Apache Tomcat 7 to the location where you will use the vmc command line tool. If you do not have vmc, then follow these instructions to install it. I downloaded Apache 7.0.27, which is currently the latest version 7.0 release and the file name is apache-tomcat-7.0.27.zip. All of the commands throughout this tutorial will assume that the present working directory is the Tomcat 7 base directory.

Step 2 – Extract Tomcat and Update Permissions

Extract the tomcat zip file to a local directory. I changed the permissions of the bin/*.sh scripts to have executable permissions.

unzip apache-tomcat-7.0.27.zip
cd apache-tomcat-7.0.27
chmod +x bin/*.sh

Linux and OSX users should be able to test the scripts on their local systems before pushing them to Cloud Foundry’s Ubuntu-based server environment. Unfortunately, Windows users cannot test the bin/startup.sh script changes locally first, but the modifications are really quite simple.

Step 3 – Edit Startup Scripts

bin/startup.sh

Tomcat is typically started with the bin/startup.sh script. In order for Tomcat to use the same shell that invokes startup.sh instead of spawning a new shell, change the execution argument in the last line of startup.sh from “start” to “run“:

exec "$PRGDIR"/"$EXECUTABLE" start "$@"

to:

exec "$PRGDIR"/"$EXECUTABLE" run "$@"

bin/catalina.sh

Instead of using a pre-defined static port, we would like Tomcat 7 to use the port assigned by Cloud Foundry, which will be stored in the VCAP_APP_PORT environment variable when deployed. Place the following bash code near the top of catalina.sh after the initial comments. Just so that we can run this locally as well without modifying the code, this code will assign a static port number of 8080 if the dynamic port is not available as an environment variable.

# USE VCAP PORT IF IT EXISTS, OTHERWISE DEFAULT TO 8080

if [ -z ${VCAP_APP_PORT} ]; then

export VCAP_APP_PORT=8080

fi

export JAVA_OPTS="-Dport.http.nonssl=$VCAP_APP_PORT $JAVA_OPTS"

Step 4 – Edit Tomcat Configuration

conf/server.xml

Set the port attribute of the Server element to -1, which disables the Tomcat shutdown port. Cloud Foundry does not use the shutdown port because it issues a “kill -9 PID” command to stop any standalone app instance. We want to avoid any potential port conflicts with other applications that are running on the same Droplet Execution Agent (DEA), so only using a single http port is the current recommendation for standalone applications running on Cloud Foundry.

Server port=”-1” command=”SHUTDOWN”

Since Cloud Foundry handles the load balancing for you without using the AJP connector, you should disable the AJP connector to ensure we do not get a port conflict by commenting out the section shown below.

  Define an AJP 1.3 Connector on port 8009  

The Connector element should use the port provided in the JAVA_OPTS environment variable, which we have set previously in the catalina.sh script.

Connector port="${port.http.nonssl}" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443"

At this point, I recommend trying the edits on your local server to see if Tomcat 7 starts up as expected. In order to test whether the VCAP_APP_PORT is being used, I recommend using a command shell to assign a sample port such as 8082.

jbayer$ export VCAP_APP_PORT=8082
jbayer$ bin/startup.sh

The console should not return (it should block while Tomcat is running) and one of the last lines in the console output should be:

INFO: Starting ProtocolHandler ["http-bio-8082"]

If that is the case, you should be able to visit http://localhost:8082 to see if the welcome page is there.

Tomcat 7 Running Locally

Tomcat 7 Running Locally

At this point you may want to back up (zip) the entire Tomcat 7 directory with the customizations you have done thus far so you can reuse it on other applications later.

Step 5 – Install your application

In order to show a Tomcat 7 feature, I used Servlet 3.0 which now has support for Servlet annotations as shown below in the simple Servlet.

package tomcat7;

import java.io.IOException; import javax.servlet.ServletException; import javax.servlet.annotation.WebServlet; import javax.servlet.http.*;

@WebServlet("/Servlet3") public class Servlet3 extends HttpServlet {

    public Servlet3() {
        super();
    }
    
    protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        response.getWriter().println("Hello from Servlet 3.0!");
    }
    

}

Cloud Foundry uses the ROOT web application with Tomcat 6, so let’s replicate that same behavior for Tomcat 7.

First delete the existing ROOT application.

jbayer$ rm –r webapps/ROOT

Now take your web application and explode it into the webapps/ROOT location. If you have a .war file then the command looks like this:

jbayer$ unzip –d webapps/ROOT ~/dev/mytomcat7.war

Check your application locally to see if it functions properly.

Tomcat 7 With Servlet 3.0 Running Locally

Step 6 – Push the application to Cloud Foundry

Execute the vmc command from the Tomcat 7 base directory, selecting many of the default selections. Notice that vmc auto-detects that it is a “Standalone Application,” and you simply need to provide the startup script path and map an unique URL. Note that you will get an error if the URL is not unique.

jbayer$ vmc push mytomcat7 
Would you like to deploy from the current directory? [Yn]: 
Detected a Standalone Application, is this correct? [Yn]: 
1: java 
2: node 
3: node06 
4: ruby18 
5: ruby19 
Select Runtime : 1 
Selected java 
Start Command: bin/startup.sh 
Application Deployed URL [None]: mytomcat7.cloudfoundry.com 
Memory reservation (128M, 256M, 512M, 1G, 2G) [512M]: 256M 
How many instances? [1]: 
Bind existing services to 'mytomcat7'? [yN]: 
Create services to bind to 'mytomcat7'? [yN]: 
Would you like to save this configuration? [yN]: y 
Manifest written to manifest.yml. 
Creating Application: OK 
Uploading Application: 
  Checking for available resources: OK 
  Processing resources: OK 
  Packing application: OK 
  Uploading (23K): OK 
Push Status: OK 
Staging Application 'mytomcat7': OK 
Starting Application 'mytomcat7': OK

Tomcat 7 With Servlet 3.0 on CloudFoundry.com

At the end of the push process, you will be offered the option to write the configuration to a manifest file. Here is the resulting manifest.mf file that got written to the Tomcat 7 base directory from our deployment. Note that it contains the startup command, bin/startup.sh. By having this file present in the root directory vmc will read from this file and skip the interactive questions the next time you push this application.

---
applications:
  .:
    url: mytomcat7.cloudfoundry.com
    command: bin/startup.sh
    runtime: java
    framework:
      info:
        exec:
        description: Standalone Application
        mem: 64M
      name: standalone
    name: mytomcat7
    instances: 1
    mem: 256M

Conclusion

Other containers such as Jetty would follow a similar pattern as described above. Most applications should be able to use the existing frameworks Cloud Foundry makes available. Should the need arise to customize or bring your own container, Cloud Foundry standalone application support is a great option.

- James Bayer

The Cloud Foundry Team

Try Cloud Foundry on CloudFoundry.com for free

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Cloud Foundry Multi-Cloud Options Keep Multiplying: BOSH CPI Support for OpenStack

Cloud Foundry provides a consistent model for deploying and running applications across multiple clouds. This multi-cloud approach preserves developer choice and flexibility, both today and in the future.

Following on the heels of the BOSH hackathon at the recent OpenStack conference, Piston Cloud today announced plans to distribute and support Cloud Foundry on OpenStack, joining existing BOSH support for vSphere and Amazon Web Services. Piston Cloud will offer this new integrated capability in a future release of Piston Enterprise OS and the new project will be submitted to the OpenStack satellite ecosystem for future consideration as an OpenStack incubation project. Details on the announcement are available in the official press release.

The OpenStack BOSH CPI, like both OpenStack and Cloud Foundry itself, is an open source project under the Apache 2 license. User and developer community contributions are welcomed, and patches can be submitted at https://github.com/piston/openstack-bosh-cpi.

This effort is still in the early stages, but with core Nova team members @0×44 and @(jk0) leading the effort, and @vadimspivak and the Cloud Foundry team on call, rapid progress is expected.

The Piston Cloud project joins other community contributions around Cloud Foundry BOSH including the work being done by Nic Williams of Engine Yard. See his recent blog on his experiences with BOSH, as well as his public talk this Wednesday (May 2, 2012).  Join in on the updates and conversations in the Cloud Foundry BOSH user group.

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email