On the night of March 29, 2012, we upgraded our Tomcat from version 6 to version 7 as part of our normal production upgrade schedule.

Early in the morning of March 30th, we discovered that Java apps that were created or re-pushed were receiving an error.

At 9am PT that morning we identified the specific issue causing the error.  The script catalina.sh was modified in Tomcat 7 to start Tomcat using eval exec instead of exec.  We were setting the CATALINA_OPTS environment variable to include values for -Dhttp.nonProxyHosts that contained a pipe character, which was interpreted by eval as Unix pipe. This affected the application start command, leading to a failure to start the application.  This change did not affect our QA environment, as there was no proxy configured (or required), and thus the problem escaped our testing.  Once we understood the issue, our first priority was to minimize the impact on our users.  The best course of action was deemed to be reverting back to the last known good version (Tomcat 6).

The version rollback and validations completed at 1:20pm PT.

This problem impacted about 60 people who were performing pushes or creating apps during the time Tomcat 7 was in place.

We are learning from our mistake.  We will start by adding tests based on this incident, and will continue investigating other mechanisms to prevent similar issues from occurring in the future.

We would like to apologize to all of you who suffered from this problem.

- Ramnivas Laddad
Cloud Foundry Frameworks Engineering

Application logging is essential to monitoring applications and diagnosing problems, but in a cloud universe where file systems may be ephemeral and application instances are provisioned dynamically keeping track of your application logs can be a challenge. Fortunately with Cloud Foundry you can get fast, multi-instance, cloud-proof application logging in just a few steps with RabbitMQ

In this blog post I’ll show you how to configure a Spring application so that it logs messages via AMQP. But first, what are the benefits? The primary motivations are that it’s fast and it doesn’t rely on the local file system of your application (which can disappear in a cloud environment). It’s also very easy to set up multiple applications to log to the same message broker, allowing you to aggregate log messages. Finally, you have the power to route and process those messages, for example allowing you to index them for search and store them in a blob store.

So how do you go about logging messages via the broker? If you’re happy to use Spring, you’re already ahead of the game. One of the best kept secrets of the Spring AMQP project is an AMQP Log4j appender that allows you log messages to any AMQP broker. All you have to do is configure it.

Application changes

As an example, I’m going to use the Spring travel application. The same principles apply to any Java, servlet-based web application.

In order to send log messages to a non-standard location, you have to attach the appropriate appender to a logger. Typically, you attach it to the root logger. The easiest way to do this is to add the appender to your Log4j configuration file, be it a properties file or XML. The Spring travel application uses XML configuration, so adding the AMQP appender is a simple case of inserting the appender definition into log4j.xml:

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE log4j:configuration PUBLIC "-//APACHE//DTD LOG4J 1.2//EN" "log4j.dtd">
    <log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
        ...
        <appender name="amqp" class="org.springframework.amqp.rabbit.log4j.AmqpAppender">
            <param name="ExchangeName" value="amq.topic" />
            <param name="ExchangeType" value="topic" />
            <param name="RoutingKeyPattern" value="logs.spring-travel" />
            <param name="ApplicationId" value="spring-travel" />
        </appender>
        ...
    </log4j:configuration>

and then adding the appender to whichever loggers you want. In the case of the root logger:

    <root>
        <priority value="info" />
        <appender-ref ref="console" />
        <appender-ref ref="amqp"/>  <!--   <-- Add this line -->
    </root>

These simple changes will result in the Spring Travel application logging messages to a RabbitMQ broker – as long as it’s running on the default port and with the standard ‘guest’ account. You can configure the connection settings for the broker in log4j.xml if you know the broker host and port information in advance, but Cloud Foundry assigns these values dynamically. Fortunately it is easy to configure your Cloud Foundry application to use those dynamically assigned values.

Since the connection settings for the various Cloud Foundry services are injected into an application at runtime (via an environment variable), we have to programmatically configure Log4j at runtime as well. This is a simple case of adding a ServletContextListener that checks whether the application is running inside Cloud Foundry and if so, updates the connection settings of the “amqp” appender.

The full code is in the application sample, but the basic process involves initializing the logging from log4j.xml, getting the appender by name, closing it (so that any existing AMQP connections are dropped) and then updating the connection settings.

Of course, we need to make sure the listener actually runs! So, drop the listener definition into the applications web descriptor:

    <listener>
        <listener-class>org.springframework.samples.travel.web.Log4jConfigListener</listener-class>
    </listener>

The last piece of the puzzle is to make sure that AmqpAppender is on the application’s classpath, which means you need to add the spring-rabbit library as a dependency:

    <dependency>
        <groupId>org.springframework.amqp</groupId>
        <artifactId>spring-rabbit</artifactId>
        <version>1.0.0.RELEASE</version>
    </dependency>

And because we’re using the CloudEnvironment class to extract the RabbitMQ connection settings from Cloud Foundry, we also need the cloudfoundry-runtime JAR, although that’s already part of the Spring Travel sample project:

    <dependency>
        <groupId>org.cloudfoundry</groupId>
        <artifactId>cloudfoundry-runtime</artifactId>
        <version>0.8.1</version>
    </dependency>

Everything is now in place, so we’re ready to deploy the application to Cloud Foundry. All that’s unique to this deployment is that you have to bind a RabbitMQ service called “rabbitmq-logs” to the application. This is because that’s the name of the service the code in Log4jConfigListener looks for. If you want to try this app yourself, just clone the repository and use the latest vmc to deploy it:

$ git clone https://github.com/pledbrook/spring-travel.git
$ cd spring-travel/server
$ mvn package
$ vmc push --path target
Would you like to deploy from the current directory? [Yn]: 
Application Name: spring-travel
Detected a Java SpringSource Spring Application, is this correct? [Yn]: 
Application Deployed URL [spring-travel.cloudfoundry.com]: koala-travel
Memory reservation (128M, 256M, 512M, 1G, 2G) [512M]:    
How many instances? [1]: 
Bind existing services to 'spring-travel'? [yN]: 
Create services to bind to 'spring-travel'? [yN]: y
1: mongodb
2: mysql
3: postgresql
4: rabbitmq
5: redis
What kind of service?: 2
Specify the name of the service [mysql-6117e]: travel-rdbms
Create another? [yN]: y
1: mongodb
2: mysql
3: postgresql
4: rabbitmq
5: redis
What kind of service?: 4
Specify the name of the service [rabbitmq-abd55]: rabbitmq-logs
Create another? [yN]: 
Would you like to save this configuration? [yN]: y
Manifest written to manifest.yml.
Creating Application: OK
Creating Service [travel-rdbms]: OK
Binding Service [travel-rdbms]: OK
Binding Service [rabbitmq-logs]: OK
Uploading Application:
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (41K): OK
Push Status: OK
Staging Application 'spring-travel': OK
Starting Application 'spring-travel': OK

Don’t forget to change the application URL if you’re deploying to cloudfoundry.com, otherwise you may be told that it’s already in use and the deployment will fail.

Once you have successfully deployed the application and it starts up, those log messages will be flying! Hmmm…but where will they be flying to?

Who ate all the messages?

RabbitMQ will shift those log messages at a tremendous rate and retain them up to a point, but those messages need to be routed somewhere if you want to view them easily or search the logs in a week’s time. We obviously need to pick up those messages and put them into a data store for easy retrieval.

For this blog post, I’m going to create a simple Grails application that consumes the messages and stores them in a MongoDB instance. I could use a relational database, but ACID and referential integrity aren’t important in this scenario.

There are two parts to this application: a message consumer that stores the log messages in the database and a UI for viewing the messages. For the first part, I just declare dependencies on the RabbitMQ and MongoDB plugins (removing the Hibernate plugin) and create a handler service that consumes the messages and stores them into MongoDB:

    package org.example

    import org.springframework.amqp.core.Message

    class MessageStoreService {
        static transactional = false

        static rabbitSubscribe = [
                name: "amq.topic",
                routingKey: "logs.#",
                messageConverterBean: ""]

        void handleMessage(Message msg) {
            def props = msg.messageProperties
            def logMessage = new LogMessage(
                    message: new String(msg.body, props.contentEncoding ?: "UTF-8"),
                    appName: props.receivedRoutingKey - "logs.",
                    category: props.headers.categoryName,
                    level: props.headers.level).save(flush: true)
        }
    }

The handleMessage() method is invoked each time a message is received from the amq.topic exchange whose routing key matches “logs.#”. It just extracts the relevant information from the AMQP message and stores that in a LogMessage instance, which maps to a MongoDB record:

    package org.example

    class LogMessage {
        String appName
        String message
        String category
        String level

        static constraints = {
            category nullable: true
            level nullable: true
        }
    }

Note that the application name is pulled from the routing key of the message. Finally, the application has a scaffolded controller for LogMessage so that users can view the log messages:

    package org.example

    class LogMessageController {
        static scaffold = true
    }

The above are the only extra classes you need on top of a fresh Grails application. You will also need to add some RabbitMQ configuration to grails-app/conf/Config.groovy:

    rabbitmq {
        connectionfactory {
            username = 'guest'
            password = 'guest'
            hostname = 'localhost'
            consumers = 5
        }

        queues = {
            exchange name: "amq.topic", type: topic, durable: true, autoDelete: false
        }
    }

The key entry is the declaration of the ‘amq.topic’ exchange.

The full source is of this application is available on GitHub and can be deployed directly to any Cloud Foundry account. All you need to do is create and bind a MongoDB service to the application and bind “rabbitmq-logs” (or whatever RabbitMQ service you’re using for logging). You will also need to download and install Grails 2 to try this application.

$ git clone https://github.com/pledbrook/cf-log-app.git
$ cd cf-log-app
$ grails compile
$ grails
grails> cf-create-service rabbitmq rabbitmq-logs
Service 'rabbitmq-logs' provisioned.
grails> cf-create-service mongodb mongo-logs
Service 'mongo-logs' provisioned.
grails> prod cf-push --appname=cf-log-apps --services=rabbitmq-logs,mongo-logs
Building war file
| Done creating WAR target/cf-temp-1332242548101.war
> 
Application Deployed URL: 'cf-log-apps.cloudfoundry.com'? 

Creating application cf-log-apps at cf-log-apps.cloudfoundry.com with 512MB and services [rabbitmq-logs, mongo-logs]:
2012-03-20 11:22:40,498 [main] WARN  client.RestTemplate  - GET request for "http://api.cloudfoundry.com/apps/cf-log-apps" resulted in 404 (Not Found); invoking error handler
 OK
Uploading Application:
  Checking for available resources:
 OK
  Processing resources:
 OK
  Packing application:
 OK
  Uploading (6K):
 OK

Trying to start Application: 'cf-log-apps'......
Application 'cf-log-apps' started at http://cf-log-apps.cloudfoundry.com

As before, don’t forget to change the application name if you follow the above instructions. You can also see the application live on cloudfoundry.com with some log messages from the grailstwitter sample application.

Power logging

This is of course a very basic application but it does demonstrate how easy it is to aggregrate the logs from multiple applications into a single location. It would also be fairly easy to add a separate application that archives logs to a blob service, all the while ensuring low latency inside the applications that are performing the logging. This is the power of logging via a message broker.

Why stop at Java? In a polyglot environment, logging via AMQP will allow you to aggregate logs from all sorts of frameworks, such as Node.js and Ruby on Rails. As long as the consumer can handle any differences between message formats, you’re good to go.

- Peter Ledbrook
Cloud Foundry Developer Relations

The services offered in Cloud Foundry are necessary for writing any serious application. Our aim is to make it easy to configure and consume these services. In addition to the auto-reconfiguration described in the Using Cloud Foundry Services with Ruby: Part 1 – Auto-reconfiguration blog post we also have support for manual service property lookup as well as library calls to obtain a pre-configured connection object.

Library call to obtain client object

For each supported service type there are corresponding library calls to obtain a pre-configured client object. This makes it easy to use in your code since you don’t have to lookup connection properties, instead you can rely on the library to do the work for you.

Here are some examples for the supported service types:

• Relational database (PostgreSQL)

  require 'cfruntime/postgres'
  client = CFRuntime::PGClient.create_from_svc('postgres-test')
  ...
  

• Relational database (MySQL)

  require 'cfruntime/mysql'
  client = CFRuntime::Mysql2Client.create_from_svc('mysql-test')
  ...
  

• Document database (MongoDB)

  require 'cfruntime/mongodb'
  connection = CFRuntime::MongoClient.create_from_svc('mongo-test')
  db = connection.db
  ...
  

• Key-Value store (Redis)

  require 'cfruntime/redis'
  client = CFRuntime::RedisClient.create_from_svc('redis-test')
  ...
  

• Messaging (RabbitMQ)
  • AMQP Client

    require 'cfruntime/amqp'
    client = CFRuntime::AMQPClient.create_from_svc('rabbit-test')
    ...
    

  • Carrot

    require 'cfruntime/carrot'
    client = CFRuntime::CarrotClient.create_from_svc('rabbit-test')
    ...
    

These library calls all use the “create_from_svc” method where you need to specify the name of the service you are connecting to. If you only have a single service of a specific type bound to the app, then you can omit the service name and use the “create” method instead.

You can also provide connection parameters that can be used for local testing. This can be done using a rescue clause since the cloud connection method throws an exception if a cloud service can’t be located. The following is an example of using the “create” method to connect to a single MongoDB service bound to the app with an added rescue clause to provide a localhost db connection for running locally.

require 'cfruntime/mongodb'
...
db = (CFRuntime::MongoClient.create.db rescue Mongo::Connection.new("localhost", 27017).db("db"))
...

The following table shows the available methods and parameters for each service type:

* The proxy returned for the MongoDB Connection has a no-argument ‘db’ method to get access to the DB object for the database created for the CloudFoundry service.

Service configuration properties lookup

For applications where you want more control, we provide a service configuration properties look-up library. To use this, you can programmatically check whether you are running in a cloud environment, and then use the library to look up a service specific properties needed for manual connection configuration.

What services are supported and what properties are exposed for these services?

All services have the following properties provided in a hash:

• :label
• :version
• :name
• :username
• :password
• :host
• :port

Each supported service also provides the following properties:

• Relational database (PostgreSQL, MySQL)
  • :database – the name of the database
• Document database (MongoDB)
  • :db – the name of the database
• Key-Value store (Redis)
  • no additional properties
• Messaging (RabbitMQ)
  • :url – the connection URL

Here is a brief MongoDB example:

require 'cfruntime/properties'

if CFRuntime::CloudApp.running_in_cloud?
  @service_props = CFRuntime::CloudApp.service_props('myservice')
else
  @service_props = {}
  @service_props[:host] = 'localhost'
  @service_props[:port] = 27017
  @service_props[:db] = 'testdb'
end
db = Mongo::Connection.new(@service_props[:host], @service_props[:port]).db(@service_props[:db])
if CFRuntime::CloudApp.running_in_cloud?
  db.authenticate(@service_props[:username], @service_props[:password])
end

Conclusion

In the previous blog post, you learned about Ruby auto-reconfiguration, and in this post we have covered manual configuration. We are pleased to offer these two new approaches to make it even easier to connect to Cloud Foundry services from your Ruby applications. Please feel free to send us any feedback via the Cloud Foundry Support Forums.

- Thomas Risberg
The Cloud Foundry Team

Don’t have a Cloud Foundry account yet?  Sign up for free today

Right from the launch, Cloud Foundry supported auto-reconfiguration of Spring and Rails apps that use a relational database service.  This allowed deploying such an app without changing a single line of code.  Recently, we extended this support for Spring apps to cover all services (Redis, Mongo, and Rabbit).  We are now extending this support for all services for Rails and making this available for Sinatra apps as well.  In this blog, we will explore how auto-reconfiguration works with Rails and Sinatra applications.

Auto-reconfiguration in action

To demonstrate auto-reconfiguration, we will grab an application from github and deploy it to Cloud Foundry without modification.  Let’s use lamernews, a Sinatra app that uses Redis.

mycomp:lamernews$ vmc push lamernews 
Would you like to deploy from the current directory? [Yn]: 
Application Deployed URL ["lamernews.cloudfoundry.com"]: 
Detected a Sinatra Application, is this correct? [Yn]: 
Memory Reservation ("64M", "128M", "256M", "512M", "1G") ["128M"]: 
Creating Application: OK 
Would you like to bind any services to 'lamernews'? [yN]: y 
The following system services are available 
1: mongodb 
2: mysql 
3: neo4j 
4: postgresql 
5: redis 
Please select one you wish to provision: 5 
Specify the name of the service ["redis-52216"]: 
Creating Service: OK 
Binding Service [redis-52216]: OK 
Uploading Application: 
Checking for available resources: OK 
Processing resources: OK 
Packing application: OK 
Uploading (1K): OK 
Push Status: OK 
Staging Application: OK 
Starting Application: OK 

Looks like the app deployed successfully.   Lamernews uses Redis to store comments, so let’s comment on a post and verify it stores successfully.

Now I press “Send comment”, and it looks like my comment was applied. Let’s take a look at the lamernews code that initializes the Redis connection:

 
RedisHost = "127.0.0.1" 
RedisPort = 10000 
$r = Redis.new(:host => RedisHost, :port => RedisPort) if !$r 

As you can see, the code is attempting to connect to Redis on localhost, however it worked just fine when we deployed to Cloud Foundry.  How is this possible?  Cloud Foundry will automatically detect initialization of several popular clients anywhere in your code and swap out your connection parameters for those of a service bound to your application. Read on to find out more about how this works!

Auto-reconfiguration for Sinatra

Your application consists of business logic and interaction with services such as database and messaging.  In a Sinatra application, you may initialize these services in Sinatra::Base#configure(). However, this is certainly not a requirement.  You are free to initialize your services wherever you want, perhaps lazily in response to a request. Additionally, there are several different client libraries you can use for connection to data and messaging services (ActiveRecord, DataMapper, Mongo Ruby Driver, MongoMapper, etc). For example, consider the following code that creates a Redis client:

 
require 'redis' 
module Demo 
class App < Sinatra::Base 
configure do 
  redis = Redis.new(:host => '127.0.0.1', :port => '6379') 
end 
... 
end 
end 

We can make one easy observation: The Redis host and port point to a server on localhost.  When you push this application to Cloud Foundry and bind a Redis service, the URL for that service is not going to be 127.0.0.1:6379!  So without an additional mechanism, such application will fail on startup.  This is where the auto-reconfiguration mechanism comes into play.  The auto-reconfiguration mechanism leverages Ruby metaprogramming to intercept the Redis initialization and replace the connection parameters with those of the Redis service bound to the application.  The result is that the user application works in local deployment and in Cloud Foundry without any change. When your Sinatra application is staged during the deployment process, Cloud Foundry will make two modifications:

1. It will add an additional cf-autoconfig gem to your Bundle
2. It will wrap the execution of your main Sinatra file (e.g. app.rb) in an auto_stage.rb file that ensures that all dynamic class modification is done prior to application execution.

Auto-reconfiguration for Rails

Cloud Foundry already provides auto-reconfiguration of your database in Rails by modifying the production settings in your config/database.yml during staging.  We have now added auto-reconfiguration of Mongo, Redis, and Rabbit clients as well. For example, you may have the following in config/initializers/redis.rb:

 
$redis = Redis.new(:host => '127.0.0.1', :port => 6379, :password => 'mypass') 

Once again, we can see that the Redis host and port point to a server on localhost.  Cloud Foundry will automatically replace these localhost connection parameters with those of the Redis service bound to your application. While it’s fairly common to put these types of connections in a Rails Initializer File, auto-reconfiguration should work just as well if you create the connection somewhere else within your application. When your Rails application is staged during the deployment process, Cloud Foundry will make two modifications:

1. It will add an additional cf-autoconfig gem to your Bundle
2. It will add an Initializer file to config/initializers that ensures that all dynamic class modification is done prior to loading other Initializers (and thus before your application executes).

Supported Clients

The following table shows the supported clients for auto-reconfiguration.

In some cases, you don’t need to be using these clients directly. For example, the popular object mapper for Mongo, 

mongo_mapper, uses the Mongo Ruby Driver.  Therefore, if your application uses mongo_mapper, Cloud Foundry can auto-reconfigure it to connect to your Mongo service. Note that the mysql and postgresql gems are listed as Sinatra only.  This is because we auto-reconfigure relational database connections in Rails without metaprogramming, by modifying your production database settings in your database.yml file.

Under the Hood

As mentioned, we leverage Ruby metaprogramming to intercept a common set of method calls that create connections.  We then replace the connection parameters with those of the service bound to your application.  We do this with some well-known metaprogramming patterns: Open Class or Class Extension and Around Alias.  These are described thoroughly in the excellent book Metaprogramming Ruby. Here is an example from our Redis auto-reconfiguration support:

 
require 'cfruntime/properties' 
module AutoReconfiguration 
module Redis 
def self.included( base ) 
  base.send( :alias_method, :original_initialize, :initialize) 
  base.send( :alias_method, :initialize, :initialize_with_cf ) 
end 
def initialize_with_cf(options = {}) 
  service_props = CFRuntime::CloudApp.service_props('redis') 
  cfoptions = options 
  cfoptions[:host] = service_props[:host] 
  cfoptions[:port] = service_props[:port] 
  cfoptions[:password] = service_props[:password] 
  original_initialize cfoptions 
end 
end 
end 
require 'redis' 
class Redis 
  include AutoReconfiguration::Redis
end 

The code starts by opening the Redis class and adding the methods defined in our AutoReconfiguration::Redis module.  The first method, self.included, sets up an Around Alias that directs all of your calls to Redis.new to the new initialize_with_cf method.  This method utilizes our cf-runtime gem library look up the Redis service connection properties and then calls the original Redis initialize method with the changed parameters. We use a similar approach for each supported client.  Feel free to browse the code in our github repo for more details.  Let us know if there are other clients or hook points we should support (feel free to submit a pull request!).

Limitations

Auto-reconfiguration of services work only if there is exactly one service of a given service type.  For example, you may bind only one relational database service (MySQL or Postgres) to an application. If an application doesn’t follow these limitations, auto-reconfiguration will not take place.  In those cases, you can still take advantage of the cf-runtime gem described in the next blog post to manually configure service access. The auto-reconfiguration mechanism expects typical Ruby applications.  If your application configuration is complex, it may not work.  In those cases, you can opt out of auto-reconfiguration as we will describe next.

Opting out of auto-reconfiguration

There may be situations where you will like to opt out of auto-reconfiguration.  Cloud Foundry offers a few ways to opt out of the auto-reconfiguration mechanism.

1. Create a file in your Sinatra or Rails application called “config/cloudfoundry.yml”. Add the entry “autoconfig: false”.
2. Include the ‘cf-runtime’ gem in you Gemfile. Cloud Foundry uses this mechanism to opt out, since applications either want to have the auto-reconfiguration behavior or simply take control over service creation completely. We do not see the value in auto-reconfiguring some services and manually configuring others. If you are using cf-runtime and not using Bundler, you can still opt-out of auto-reconfiguration by creating the cloudfoundry.yml file as described above.

Conclusion

Auto-reconfiguration in Cloud Foundry is a wonderful way to get started deploying your Rails and Sinatra apps quickly.  As your application matures or makes use of multiple services, you may need finer control of your service connections.  In the next blog in this series, Thomas Risberg will explain how to use the new cf-runtime gem for simplified connections to Cloud Foundry services.

– Jennifer Hickey, The Cloud Foundry Team

Don’t have a Cloud Foundry account yet?  Sign up for free today