Cloud Foundry Blog

About Nima Badiey

Nima Badiey heads ecosystem partnerships and business development for Cloud Foundry at Pivotal. His goal is to make Cloud Foundry better by partnering with leading cloud technology providers. He is a 17 year veteran in Silicon Valley with various management positions at Sun Microsystem, Deloitte, Flickr, Six Apart, Joyent and VMware.
    Find more about me on:
  • twitter

CollabNet’s CloudForge Available on run.pivotal.io Marketplace

The following is a guest post by Kelly Lanspa from the CloudForge product team.

CloudForge from CollabNet is a collaborative software development and application lifecycle management platform. It includes source code management, Git/Subversion hosting and bug tracking on one platform, with backup services, additional storage and secure role-based user access to manage distributed teams. CloudForge is integrated with Pivotal’s hosted Cloud Foundry service, enabling users to easily build-test-deploy and scale apps.

cloudfoundry floudforge diagram

From the marketplace console (or the cf command line utility), select CloudForge then choose one of the packages available. When you select a package by clicking on “buy this plan”, CloudForge will require you to select a unique name for your instance which will be used to create a new organization name in CloudForge. Each “space” in Cloud Foundry will map to a unique CloudForge organization, and users can create different CloudForge accounts for different Cloud Foundry Applications or Spaces, but this isn’t strictly required. You can always deploy your code from a shared CloudForge repository to different Spaces. As CloudForge is a shared service, it does not need to bind to a specific Cloud Foundry application.

pivotal marketplace

Create a new instance called “PivotalDemo” in your default Space (typically “development”) but do not bind it to a specific application. Note: the account name in CloudForge needs to be unique.

Add the CloudForge service in your Space. When you have successfully created a new CloudForge instance it will be listed in your Space Services. Click on “Manage” to finish configuring your CloudForge account.

Space

The instance name you chose for CloudForge service will be used for your new CloudForge account. By default, any user clicking on “Manage” within Cloud Foundry will be automatically logged into CloudForge via OAuth as the Cloud Foundry Space user. However, you will need to choose a username and a password to access your SVN and Git repositories. These SCM services do not support oAuth and are accessed directly from your SCM client, IDE or your command line. Changing your first and last name are optional.

create account

Regardless of how many developers you have it is always useful to maintain version control, even if you are the only developer. Using a Cloud-based SCM provider like CloudForge means that you always have a full copy of your source code backed up and ready for retrieval from any computer. It also makes it very easy to add collaborators on your projects. In order to add collaborators simply click on Admin->Manage Users.

manage users

You can invite as many collaborators as you like (depending upon your plan) to join your CloudForge account regardless of whether they are users on Cloud Foundry (the admin CloudForge account is created when the CloudForge service is selected from the run.pivotal.io marketplace, but subsequent team members do not have to be Cloud Foundry users). Each user will create a new username on CloudForge so you can track their commits, issue updates, and comments separately as you develop your application.

invite users

Now that you have invited others to collaborate on your projects it’s time to create your first project. Click on Projects->New Project.

new project

Create a new CloudForge Project to manage your development. After you name your project you can add a SVN or Git repository (or both). If you want to include Bug/Issue Tracking, Agile Planning, Wiki, or Discussions you can also choose to add TeamForge to your project. All of these services will be provisioned within your project and the direct links for each provided on your project landing page.

project links

Once you have coded your application and merged all of your branches into a stable trunk you can easily deploy the app to Cloud Foundry using the CF command utility or directly from the Spring Tool Suite. You are able to deploy to any CloudFoundry target, Space, or Org; your CloudForge project is not bound to a specific instance of Cloud Foundry. Feel free to create as many Projects in CloudForge as you need. Your account can be used to manage multiple applications and projects across several Spaces within Cloud Foundry. If you using the Spring Tool Suite IDE, we recommend installing the CollabNet Desktop as well. This allows you to manage your CloudForge account, including all repositories and all tasks, from within the same environment that manages your code.

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Monitoring Java Apps with AppDynamics

The following is a guest blog post by Dustin Whittle, Developer Evangelist at AppDynamics.

AppDynamics is an Application Performance Management company that offers solutions to monitor a variety of applications running on public clouds or in private data centers. App Dynamics is excited to support Pivotal’s Cloud Platform by making it easy to monitor Java apps running on Cloud Foundry and Pivotal’s Web Services.

Monitor Apps on Pivotal Web Services

The AppDynamic Java agents are included in the default Java buildpack for Cloud Foundry, so if you have AppDynamics monitoring running, the Cloud Foundry DEA will auto-detect the service and enable the agent in the buildpack. If you start the AppDynamics monitoring for an app already running, just restart the app and the DEA will autodetect the new service.

1) Sign-up for a Pivotal Web Services account and AppDynamics Pro SaaS account

Screen Shot 2014-02-03 at 5.25.54 PM

In the future, Pivotal Web Services will include the AppDynamics SaaS APM services, so you’ll only need to signup for Pivotal Web Services and it will automatically create an AppDynamics account. But for now, don’t forget to signup signup for AppDynamics because you need both accounts!

2) Download the Cloud Foundry CLI (Command Line Interface)

Pivotal Web Services has both a web based GUI as well as a full featured linux style command line interface (CLI). Once you have a PWS account, you can download a Mac, Windows or Unix CLI from the “Toosl” tab in the PWS dashboard or directly for OSX, Linux, and Windows.

Pivotal Web Services CLI

3) Sign-in With Your Pivotal Web Services Credentials

Using the CLI, log into your Pivotal Web Services account. Remember to preface all commands given to Cloud Foundry with cf. Individual Cloud Foundry PaaS clouds are identified by their domain API endpoint. For PWS, the endpoint is api.run.pivotal.io. Cloud Foundry will automatically target your default org (you can change this later) and ask you to select a space (a space is similar to a project or folder where you can keep a collection of apps).

$ cf login

Cloud Foundry CLIÂ

Monitoring Apps on Pivotal Web Services

1) Clone the Spring Trader demo application

The sample Spring Trader app is provided by Pivotal as a demonstration. This can be used to show how monitoring works. First git clone the app from the Github repository.

$ git clone https://github.com/cloudfoundry-samples/rabbitmq-cloudfoundry-samples

2) Create a user provided service to auto-discover the AppDynamics agent

$ cf create-user-provided-service demo-app-dynamics-agent -p "host-name,port,ssl-enabled,account-name,account-access-key"

Cloud Foundry CLI

Tip: Find out more about deploying on PWS in the Java build pack docs.

3) Use the Pivotal Web Services add-on marketplace to add a cloud based AMQP + PostgreSQL database instance

$ cf create-service elephantsql turtle demo-db

$ cf create-service cloudamqp lemur demo-amqp

Cloud Foundry CLI

4) Bind PostgreSQL, AMQP, and AppDynamics services to the app

$ git clone https://github.com/cloudfoundry-samples/rabbitmq-cloudfoundry-samples

$ cd rabbitmq-cloudfoundry-samples/spring

$ mvn package

$ cf bind-service demo-app demo-app-dynamics-agent

$ cf bind-service demo-app demo-amqp

$ cf bind-service demo-app demo-db

Cloud Foundry CLI

5) Push the app to production with the CLI Upload your app with the cf push command, or (as in the example below) you can specify additional parameters including memory, number of app instances, name and the exact app path.

$ cf push demo-app -i 1 -m 512M -n demo-app -p target/rabbitmq-spring-1.0-SNAPSHOT.war

Cloud Foundry CLI

Spring AMQP Stocks Demo App

Spring Trader

Pivotal Web Services Console

Pivotal PaaS CloudFoundry

Production Monitoring with AppDynamics Pro

To get more out of your app monitoring, signup for an AppDynamic Pro account with enhanced features and code level visibility into application performance problems.

AppD Dashboard

AppD

Watch the Full Installation Video

If you missed any part of the instructions, you can watch Dustin walk through each step in this screen cast video, or reach him on Twitter @Dustin Whittle.

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Migrating a Cloud Foundry PaaS to Run on OpenStack

The following is a guest blog post by Julian Fischer (hello@anynines.com, @railshoster) founder and CEO or AnyNines, a Cloud Foundry and Rails hosting service operated by Avarteq GmbH in Saarbrücken, Germany.

Cloud Foundry is well known for simplifying application portability from one CF-based PaaS to another, but how simple is it to move an entire, live, Cloud Foundry installation from one underlying IaaS to another? We asked the team at Pivotal, who recounted their experience moving the Cloud Foundry instance at run.pivotal.io from one Amazon AWS availability zone to another in 40 minutes. If Pivotal could do it in less than an hour, could we?

After running the AnyNines Cloud Foundry PaaS on vCloud, we decided to move our underlying IaaS installation over to OpenStack (see the official AnyNines OpenStack migration announcement for a more details, and also check out our AnyNines blog). This decision was motivated in part because we wanted to build a competence in the emerging synergy between Cloud Foundry and OpenStack, and to gain experience in this domain for our growing cloud hosting and consulting business.

We already had experience running OpenStack, and more recently took an active contributory role as well with the release of OpenStack Swift Cloud Foundry service giving simple access to OpenStack’s Amazon S3-like object store. Operating a Cloud Foundry PaaS layer on top of OpenStack was simply the logical choice.

Before Getting Started

Before jumping into the fray, we experimented with several migration scenarios, ultimately deciding to start with a virgin Cloud Foundry deployment on top of OpenStack (as opposed to a moving the currently deployed CF stack). Customer uptime was our primary concern and we didn’t want to affect the existing production environment, just in case we needed to revert to the old stack. Deploying Cloud Foundry is relatively straightforward and the incremental cost of running two CF deployments for a short period mitigated the risk of breaking anything in production. In addition, a second CF platform allowed us to test the migration before actually performing it on production system.

Deploying Cloud Foundry using BOSH meant we could use the same manifest as our production deployment, with only minor adjustments for the cloud plugins, network configurations and resource allocations (see the Gist below to compare the details of each manifest). With only a few lines of difference, it was a relatively straightforward exercise to deploy a new CF stack.

network:                                     network:
  cloud_properties:                           cloud_properties:
    name: "anynines"                            name: "anynines"
  ip: 5.22.x.x                              |   net_id: 12345678-9101-1121-1236-142355d67ca5
  netmask: 255.255.255.0                    |   type: manual
  gateway: 5.22.x.x                         |   label: private
  dns:                                      |   ip: 10.10.10.10
  - 109.234.x.x                             <
  - 109.234.x.x                             <

resources:                                    resources:
  persistent_disk: 100000                       persistent_disk: 100000
  cloud_properties:                             cloud_properties:
    ram: 1024                               |     instance_type: any-infr-small
    disk: 8192                                    disk: 8192
    cpu: 2                                  <

cloud:                                        cloud:
  plugin: vcloud                            |   plugin: openstack
  properties:                                   properties:
    vcds:                                   |     openstack:
      - url: https://cloud.example.com      |       auth_url: https://auth.example.com:5000/v2.0
        user: anynines                      |       username: anynines
        password: secret                    |       api_key: secret
        entities:                           |       tenant: anynines-tenant
          organization: anynines            |       default_key_name: secret-key
          virtual_datacenter: anynines-vdc  |       default_security_groups: ["anynines"]
          vapp_catalog: Bosh                |       private_key: /root/.ssh/secret-key.pem
          media_catalog: Bosh               <
          vm_metadata_key: cf-agent-env     <
          description: Bosh                 <

Preparing the New OpenStack Environment

Preparing the new OpenStack environment involved the following steps:

  1. Deploy Bosh in the new OpenStack Cloud Foundry environment.
  2. Change all configuration variables in the deployment manifest to suit the new environment. We switched from static IPs to DNS names for nats, nfs and db access. This required adjusting a few settings in the new environment.
  3. Setup a new SSL termination gateway and configure it to use the new gorouter.
  4. Running a mirrored Cloud Foundry deployment will result in two different domain names. This will require a post-deployment step to change the domain names back to the original (post migration). To avoid this additional name change, adjust the DNS system included in CF BOSH to respond to both domains (For AnyNines we used a9s.eu and a9sapp.eu). Insert both domains in the CF BOSH powerdns database as shown in this Gist. The DNS is queried from all instances deployed using CF BOSH and the new environment will be able to connect to an SSL gateway without hitting the live endpoint.
  5. Deploy a clone of the existing Cloud Foundry installation to the new environment.

Migrating Apps, Databases, Configurations, …

Once the OpenStack infrastructure is ready, focus attention on migrating key system state parameters, as follows:

  1. Transfer all persistent disks from the vCloud installation to the new OpenStack environment.
  2. Store the gorouter routing table for later comparison in the new environment.
  3. Shutdown all instances with persistent disks. This may include cc_db, uaa_db, nfs_server and all service nodes. Additionally, stop the health manager to avoid a situation whereby the Cloud Controller may try to restart all applications.
  4. Sync persistent disks (again).
  5. Start cc_db, uaa_db, nfs_server for the first step of the migration.

TIP: OpenStack requires minor adaptations to the Cloud Controller database. For faster processing of encrypted entries, use a script (AnyNines used a small Ruby script) to update all services with the new host IP. In addition, ensure all application environment variables are correctly updated.

  1. Start cloud controller, uaa and service nodes. At this point , the new environment should be ready to re-start all existing applications.
  2. Start health manager to enable app health monitoring and logic re-start all previously running applications.
  3. To validate the startup, compare the routing table of the new environment with the old parameters.
  4. Start any remaining instances.
  5. Adjust the old gateway to point to the new environment.

At this point, the new environment should be up and running. Be sure to clear up (or archive) the old environment. Don’t forget to revert any DNS settings and to remove the CF BOSH DNS hack.

Post Migration – Facts and Lessons Learned

With solid prep work and some practice (to adjust our recipe and process flows), the whole migration took less than one hour:

  1. Customer downtime was limited to only 30 minutes, all performed in a maintenance window. The downtime was required as we had to freeze the cc_db state and prevent any changes to customer apps/data during the migration.
  2. Start to finish, the migration took about 45 minutes.
  3. Several hundred live customer apps and services, with their data, were migrated within this timeframe.
  4. The startup on the new system took only 10 minutes – that’s the time it took to get all customer applications up and running.

As with any migration, the most time consuming part was the prep work involved and the investment in quality time experimenting beforehand to ensure we had a sound and repeatable recipe. We took care to perform the work in a planned maintenance window, so as not to affect customers. We also created several backup and risk mitigation plans – “just in case” scenarios – to restore any changes back to their original state in case the migration didn’t go as planned.

Ultimately we proved our original premise that Cloud Foundry was a platform robust enough to support full stack cloud migrations. This was a critical requirement for us as a business, and equally important to customers who want to protect their cloud investment with migration flexibility. Prep work minimized the risks, practice gave us confidence, and a recipe ensured a repeatable process from start to finish.

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Sending Email from Cloud Foundry Java Applications with SendGrid

The following is a guest blog post by Scott Motte (@scottmotte), Developer Evangelist at SendGrid, a cloud based SMTP email delivery and management service.

A few weeks ago SendGrid released their sendgrid-java helper library. The library goes a step further in sticking with Cloud Foundry’s “it just works” experience and making your life as a developer easier. Previously, you had to write a large amount of boilerplate code using JavaMail. Now, you can send email with just a few lines of code.

How Does It Work?

At your Cloud Foundry Command Line Interface, add SendGrid as a service and bind it your app.

cf create-service sendgrid  
cf bind-service

Next, Install the vcapenv library, and install the sendgrid-java library.

Then add the following code to your project:

import com.github.scottmotte.Vcapenv;
import com.github.sendgrid.SendGrid;

Vcapenv vcapenv = new Vcapenv();
String sendgrid_username = vcapenv.SENDGRID_USERNAME();
String sendgrid_password = vcapenv.SENDGRID_PASSWORD();

SendGrid sendgrid = new SendGrid(sendgrid_username, sendgrid_password);

sendgrid.addTo("example@example.com");
sendgrid.setFrom("other@example.com");
sendgrid.setSubject("Hello World");
sendgrid.setText("My first email through SendGrid");

sendgrid.send();

That’s it. Now you can start sending emails from your Java application! Cloud Foundry developers can send 25,000 emails every month for free. Check out the Cloud Foundry documentation page to get started.

Video Demo

For a working demo of Cloud Foundry + SendGrid, just follow the easy steps in this example video. It uses the sample spring-attack app.

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Continuous Deployment and Application Delivery with CloudMunch

The following is a guest post from Rosmi Chandy, Software Developer at CloudMunch, a continuous deployment and application delivery platform.

CloudMunch recently announced the availability of our integration with Cloud Foundry. CloudMunch is a cloud-based solution for continuous integration, testing and release management. It manages the typical DevOps activities for a project, enabling developers to deploy applications to Cloud Foundry powered clouds, such as Pivotal’s http://run.pivotal.io PaaS. CloudMunch (1) tracks user repos (e.g., Github) for any updates or changes which would (2) trigger the predefined CloudMunch pipeline. (3) The pipeline validates the code, compiles/builds the app, and if all tests conditions are met (4) automatically deploys app updates to Cloud Foundry.

Before Getting Started

Before configuring a CloudMunch account for continuous deployment:

  • Setup a Github account and keep your credentials handy
  • Ensure a copy of your app is available on Github. Start with a simple Java/Spring app to try things out.
  • Setup an account on a Cloud Foundry powered PaaS Cloud. Start with http://run.pivotal.io/register and register for a new account. Use the code “cloudmunch” for a limited availability 30 day free trial.

Step 1: Configure Build Deploy Pipeline

Log into CloudMunch using your Github account credentials. You should see a list of projects in your Github account. Select and import the correct project into CloudMunch. In this example, we import dummy app called “Sample” which is a simple Java app for demo only. Successful import will show “This is a CloudMunch App” message in the browser. This example has a manually created “build.xml” script to package the code as a *.war file.

img1

CloudMunch automatically identifies that this is a Java app and adds basic validation steps into the pipeline as below.

img2

Add an “ANT builder app” at the end of the pipeline build using the build.xml. Once configured the build pipeline is set up to continuously perform builds on updates. The updated build pipeline looks like this:

img3

Step 2: Register a Cloud Foundry Account

CloudMunch requires secure access to your Cloud Foundry account. This is implemented using Cloud Foundry’s OAuth 2.0 protocol-based solution as basic username/password mechanism are not as secure. The registration process will direct the browser to the Cloud Foundry login screen (e.g., http://console.run.pivotal.io) to request access tokens for CloudMunch.

img4

After registering the Cloud Foundry account, either import the running applications in Cloud Foundry to CloudMunch or create new applications in Cloud Foundry. In this example, we will import an app into CloudMunch.

img5

At this point, CloudMunch has the details required to deploy the “Sample” project to your Cloud Foundry application instance.

Step 3: Edit Build Step for Continuous Deployment During the Build

Edit the build pipeline again to add a deployment app as a last step in the pipeline to continuously deploy into your environment.

img6

Add “Deploy to Cloud Foundry” app as a last step of the continuous build. Configure the app to use the provider you registered, as shown below.

img7

The new pipeline looks like this:

img8

You should now be all set for continuous deployment!

Step 4: Check-in a Code Change. Watch Build/Deploy Pipeline in Action

Try it out. Make a change in your code and check in the update to your Github account.

img9

Once the updated code is checked-in, the build triggers automatically. All the validation build steps are executed and the project gets deployed to Cloud Foundry.

img10

Here is the sample app deployed in Cloud Foundry. img11

Done! Your project now runs in a cloud-based, continuous delivery system with code changes in Github directly hitting the Cloud Foundry platform.

Summary

CloudMunch lets developers choose the development and operations tools that are best suited to the application and business needs by providing plug-and-play cloud application lifecycle management platform:

  1. Automation of development and operations processes – Build, Test, Deployment, Monitoring.
  2. Collaboration – Seamless flow of real time data and updates to enable Dev, Test and Ops to be on the same page, every time, all the time.
  3. Orchestration and Release Management – Automated, customizable workflows and environments between Dev-Test-Staging-Production.
  4. Cloud Delivery – Auto deployment and auto scaling of applications.

img12

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Monitoring Cloud Foundry Applications with New Relic

A couple of weeks ago, Ben Hale blogged about the new Cloud Foundry Java Buildpack, highlighting some great new features and our design principles – the “it just works” experience.

The new buildpack provides the opportunity for a new level of configuration and setup for popular add-on services, such as application performance and monitoring tools. The Java buildpack now includes automated configuration for the New Relic application monitoring agent. If you create a New Relic service and bind it to an application, the buildpack will set up the New Relic agent automatically when the application is staged. After the app has been staged with the New Relic configuration and traffic is flowing to the app, it generally takes several minutes for application traffic to show up in your New Relic dashboard.

nr_overview_page

If you bind a New Relic service to an existing application, you must restart the app so the buildpack can check for new services and auto-configure the environment accordingly. Without a restart, the buildpack is not aware that new auto-configurable services have been added.

On your Pivotal CF hosted web console, the “Services” section lists the active services in each space, and identifies how many (if any) apps have been bound to a specific service instance. To access the New Relic dashboard, just click on the “Manage” button below the New Relic service instance, and make sure you have at least one app bound to the New Relic service instance. The “Manage” button will open a new tab in your browser window, and automatically log you into your New Relic dashboard.

newrelic-service

How It Works

One of the extension points provided by the Java buildpack is the framework component, which allows the buildpack to customize the application deployment. The Java buildpack includes a New Relic framework component that adds New Relic-specific configuration. The New Relic framework component checks to see if a New Relic service has been bound to the application being deployed. If a New Relic service is bound, the buildpack adds the New Relic Java agent to the JVM startup command and passes the licenseKey to the agent.

The Java buildpack maintains a current copy of the latest New Relic agent, which is defined in the New Relic YAML file. To ensure fast deployment of apps to DEAs, a copy of the New Relic .jar file is cached in the local blobstore, which is hosted on Amazon’s S3, for run.pivotal.io. The cache ensures the buildpack does not have to request a new jar file from New Relic each time an app is pushed to run.pivotal.io.

Currently, only the Cloud Foundry Java buildpack supports auto-configuration of the New Relic agent and monitoring services. Depending on feedback from users, we may consider adding auto-configuration for additional runtimes in the future.

For a quick demo of the automated provisioning for New Relic (and to try it for yourself), just follow the easy steps in this video demo below.

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email