Cloud Foundry Blog

About Maria Shaldibina


Cloud Foundry Now Supports Rails Assets Pipeline

Rails version 3.1 introduced the assets pipeline as a way to properly organize and serve static assets. We are adding assets pipeline support for Rails applications deployed on  CloudFoundry.com. Assets pipeline includes compression, ERB evaluation, Javascript and CSS processing using Rails 3.1 Sprockets library. In development mode, assets precompilation is not required – Rails turns on live compilation if it can’t find precompiled assets. In production mode, live compilation is disabled by default. One of the ways to make this work on Cloud Foundry is to bundle assets with your application before you push. Now CloudFoundry.com also provides assets precompilation as one of the steps of application pre-processing.

Bundle assets locally

Rails provides a rake task to compile and bundle assets with your app. You can prepare them locally for production by running:

RAILS_ENV=production bundle exec rake assets:precompile

This task discovers all the assets included in the application, pre-compiles them and saves compiled versions in public/assets directory by default as well as manifest.yml which contains the list of compiled assets with their digest.

Assets compilation on Cloud Foundry

In order to run this in CloudFoundry.com we had to implement some additional pre-processing when the application is staged. First, Cloud Foundry assumes that assets were compiled locally via an assets manifest. If the assets manifest is not found, Cloud Foundry runs the “assets:precompile” rake task during staging. Later, when the application is being run on the DEA (the container for Cloud Foundry apps) we have all the assets. These can be statically served with the digest and other magic that Rails adds to these files.

Live compilation

One potential problem can occur during application initialization. The precompile rake task runs a complete initialization of the Rails application. This might trigger some of the initialization procedures and require service connections and environment checks that are unavailable during staging. You can turn this off by adding a configuration option in application.rb:

config.assets.initialize_on_precompile = false

If the assets:precompile task fails, Cloud Foundry makes use of live compilation mode which is the alternative to assets precompilation. In this mode assets are compiled when they are loaded for the first time. This is done by injecting a plugin to Rails application that forces live compilation process.

Rails.application.config.assets.compile = true

Example

Let’s create a simple Rails application that makes use of CoffeeScript.

rails new coffee-app
cd coffee-app
rails generate controller Hello index

Let’s add some CoffeeScript to app/assets/javascripts/hello.js.coffee file:

 
name = "CoffeeScript" alert "Hello from #{ name }!"

If we start our application with “rails server” and browse,

http://localhost:3000/hello/index

we would see “Hello from CoffeeScript!”. This works in development mode without the need for assets precompilation. In production, Rails serves only precompiled static assets and does not fall back to assets pipeline which is disabled in production.rb:

config.assets.compile = false

With the support of assets precompilation we can push this application to CloudFoundry.com without any modifications.

vmc push coffee-app

If we go to http://coffee-app.cloudfoundry.com we would see our popup message as expected. By taking advantage of these features, your Rails 3.1+ applications will properly precompile all your SASS, Coffee Script and ERB assets as intended.

- Maria Shaldibina The Cloud Foundry Team

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Recent Changes in Node.js Modules Support

Since Cloud Foundry introduced npm support, we added several improvements that make deploying Node.js applications with dependencies easier, faster and more transparent to developers.

Easy development

When we introduced npm support in Node.js, in addition to generating “npm-shrinkwrap.json” file with locked down dependencies you were required to remove the “node_modules” folder. If this was too disruptive, you could create an “cloudfoundry.json” file with an “ignoreNodeModules” property set to ‘true’.

This wasn’t an obvious solution. So we decided to change it. Detection of native modules is now done automatically by analyzing module contents. To lock down module versions, you can simply use the “npm shrinkwrap” command and your application is ready to be pushed with or without bundled modules. With this change, we have deprecated ignoreNodeModules property.

If you want to store your application in a git repository without node modules, the next time someone clones it, he/she can just push it to CloudFoundry.com without any modifications.

Fast deployment

In addition to  npm, we also improved our cache system. There are two levels of caching now — fetched and compiled modules. All modules that have been fetched from the npm registry are cached locally and pulled from that cache from that point on. If a module contains native code, it gets compiled on Cloud Foundry and the compiled version gets cached as well.

Based on the information from npm-shrinkwrap.json, Cloud Foundry checks if the module has been provided with the application itself. If it is there, and contains native code, it rebuilds it as needed. If the module hasn’t been provided with the app, Cloud Foundry checks the local cache for the given module, and if it needs compilation, provides the cached version.

As module contents can be modified by users, we are using hash of module files as a cache key.

Local modules

There may be situations when you need to customize an existing node module or move a part of your application as a reusable module. In this case, you would want your local module to be used without any modifications and be rebuilt, should it contain native code.

With automatic detection of native code – such modules will be rebuilt on Cloud Foundry and cached, based on their contents for your next push.

More informative logging

In addition to these node improvements, you can now see a detailed log of what’s going on in log files. You can get log messages by running:

vmc logs application-name

We can see if npm support is enabled (npm-shrinkwrap.json was provided) and which node version we are using:

Installing dependencies. Node version 0.6.8

For each node module we can see its installation status:

Installing express@2.5.11 from local path

In case of a failure we can see an error message and/or npm error output.:

Failed installing bcrypt@0.5.0: Node version requirement >= 0.6.0 is not compatible with the current node version 0.4.12

Node-gyp

Modules using node-gyp for the installation process are now supported on CloudFoundry.com for node08 runtime. For node06 and node04 runtimes, Cloud Foundry falls back to node-waf if the module provides the wscript file.

Git URLs

Another recent improvement is that we also support git URLs in npm-shrinkwrap.json. They can be specified in “from” or “version” fields according to npm requirements and both methods are supported. We have a separate cache for git modules and they also get compiled on Cloud Foundry if they have native extensions.

Example

In the previous blog post on auto-reconfiguration we used a Node app, “calipso”,  and performed additional steps for npm support. Let’s see how pushing calipso works now.

First, we clone calipso source from github:

$ git clone git://github.com/cliftonc/calipso.git

Then, we install node modules and lock them down.

$ npm install
$ npm shrinkwrap

Now, we can push our application.

$ vmc push calipso-app --runtime=node08

As we push we add mongodb service and set memory to 256M.

Would you like to deploy from the current directory? [Yn]:
Detected a Node.js Application, is this correct? [Yn]:
Application Deployed URL [calipso-app.cloudfoundry.com]:
Memory reservation (128M, 256M, 512M, 1G, 2G) [64M]: 256M
How many instances? [1]:
Bind existing services to 'calipso-app'? [yN]:
Create services to bind to 'calipso-app'? [yN]: y
  1: blob
  2: mongodb
  3: mysql
  4: postgresql
  5: rabbitmq
  6: redis
What kind of service?: 2
Specify the name of the service [mongodb-62f4d]:
Create another? [yN]:
Would you like to save this configuration? [yN]:
Creating Application: OK
Creating Service [mongodb-62f4d]: OK
Binding Service [mongodb-62f4d]: OK
Uploading Application:
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (82K): OK
Push Status: OK
Staging Application 'calipso-app': OK
Starting Application 'calipso-app': OK

That’s it! We didn’t need to configure our application at all. In addition,  notice that staging process was faster and the preparation was effortless. When we look at the logs we can see how everything went. Try it and see how easy it is to manage your dependencies in Node.js applications now.

- Maria Shaldibina
The Cloud Foundry Team

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

New Runtime Module for Node.js Applications

In the previous blog post, Cloud Foundry Now Supports Auto-Reconfiguration for Node.js Applications, we saw that Node.js applications deployed to CloudFoundry.com can be automatically reconfigured to connect to Cloud Foundry services. However, there may be situations where you want to opt-out of that feature to have finer control over service connections or to overcome its limitations. In those cases, your applications would be required to parse JSON-formatted environment variables to perform that same function. While this is not overly complex given that JSON is trivial to parse with JavaScript, you will be required to understand the environment variable names and their payload schema. The new cf-runtime module introduced in this blog simplifies this by providing a way to obtain application information and service connection objects. This module moves Cloud Foundry’s Node.js support forward to the match the support for Java and Ruby applications.

Installation

Cf-runtime is available in the npm registry and can be easily installed with the Node Package Manager (npm). Run the following command in the base directory of your Node.js application.

npm install cf-runtime

Usage

This node module provides access to two types of objects. The first is a preconfigured object named 

CloudApp that contains application information. This includes the application’s host and port configured by Cloud Foundry, list of services bound to the application and their properties. Additionally, each service that is bound to the application can be accessed via Client object (i.e. RedisClient, MysqlClient). This object provides a convenient way to obtain the corresponding service connection with just a single function call. You can either create a service connection by the name used to create the service instance or by providing a general service name (e.g. redis or mongo) if there is only one service of this type that is bound to your application. This function may also accept additional parameters depending on the node module it uses (see details below in Service Clients section).

var cf = require('cf-runtime') 
var app = cf.CloudApp // Check if application is running in Cloud Foundry 
app.runningInCloud // Get application properties 
app.host app.port // Get the list of application service names 
app.serviceNames // Obtain connection properties for single service of type Redis 
app.serviceProps.redis // Obtain connection properties for service named 'redis-service-name' 
app.serviceProps['redis-service-name'] // Obtain the list of service names of specific type 
app.serviceNamesOfType.redis // Check if service of the given type is available 
cf.RedisClient !== undefined // Connect to a single service of type Redis 
var redisClient = cf.RedisClient.create() // Connect to redis service named 'redis-service-name' 
var redisClient = cf.RedisClient.createFromSvc('redis-service-name') 

Service Properties All services have the following common properties:

  • name: specific name of the service
  • label: name of service type, for example “redis”, “mysql”
  • version: software version of the service type
  • host
  • port
  • username
  • password
  • url: service connection url Additionally, PostgreSQL, MySQL and Redis include this service property:

  • database: the name of the database that is provided by the service RabbitMQ provides access to these additional properties:

  • vhost: the name of the virtual host MongoDB provides access to these additional properties:

  • db: the database name

Service Clients

The following table shows the available methods and parameters for each service type:

Node module Returns Functions Parameters
MongoDB
mongodb null cf.MongoClient.create([options], callback) cf.MongoClient.createFromSvc(name, [options], callback) name- the name of a service bound to the appoptions- optional {object} non-connection related optionscallback – {function} connection callback
MySQL
mysql Mysql client instance cf.MysqlClient.create([options]) cf.MysqlClient.createFromSvc(name, [options]) name- the name of a service bound to the appoptions – optional {object} non-connection related options
PostgreSQL
pg {boolean} cf.PGClient.create(callback) cf.PGClient.createFromSvc(name, callback) name- the name of a service bound to the appcallback – {function} connection callback
RabbitMQ
amqp AMQP client instance cf.AMQPClient.create([implOptions]) cf.AMQPClient.createFromSvc(name, [implOptions]) name- the name of a service bound to the appimplOptions – optional {object} non-connection related implementation options
Redis
redis Redis client instance cf.RedisClient.create([options]) cf.RedisClient.createFromSvc(name, [options]) name- the name of a service bound to the appoptions – optional {object} non-connection related options

Summary

The main purpose of cf-runtime is to make your Node.js applications understand their cloud better, retrieve the environment properties, find the available services, and connect to them easily. If you are writing Node.js applications, cf-runtime just made it easier to deploy your applications to Cloud Foundry.

- Maria Shaldibina The Cloud Foundry Team Don’t have a Cloud Foundry account yet?  Sign up for free today

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Cloud Foundry Now Supports Auto-Reconfiguration for Node.js Applications

Cloud Foundry has long supported auto-reconfiguration for Spring and Ruby applications. Now we are pleased to add auto-reconfiguration support for Node.js applications as well. Deploying Node.js applications to Cloud Foundry previously required parsing of environmental variables and overwriting server and service connection function calls to use Cloud Foundry specific parameters. This approach was not intuitive to developers who just started to use Cloud Foundry to deploy their applications. They would need to consult the documentation and figure out what port and host they need to connect to. Moreover, if an application uses services, developers would need to configure their applications to use the proper service connection parameters.

Auto-Reconfiguration in Action

Let’s look at the basic Node.js application. We are going to take some sample code from the

Node.js official website homepage and save it to a file called app.js:

var http = require('http'); 
http.createServer(function (req, res) {res.writeHead(200, {'Content-Type': 'text/plain'}); res.end('Hello World\n'); }).listen(1337, '127.0.0.1'); console.log('Server running at http://127.0.0.1:1337/');

As we can see, this code sets up your server to listen on your local port 1337. What if we now push this application to CloudFoundry.com ‘as-is’?

$ vmc push example-app
Would you like to deploy from the current directory? [Yn]:
Detected a Node.js Application, is this correct? [Yn]:
Application Deployed URL [example-app.cloudfoundry.com]:
Memory reservation (128M, 256M, 512M, 1G, 2G) [64M]:
How many instances? [1]:
Bind existing services to 'example-app'? [yN]:
Create services to bind to 'example-app'? [yN]:
Would you like to save this configuration? [yN]:
Creating Application: OK
Uploading Application:
 Checking for available resources: OK
 Packing application: OK
 Uploading (0K): OK   
 Push Status: OK
Staging Application 'example-app': OK                                           
Starting Application 'example-app': OK                                          

$ curl example-app.cloudfoundry.com
Hello World

We can see that the application is up and running. But how is this possible if we didn’t configure it to listen on a Cloud Foundry application-specific port? This is when auto-reconfiguration comes into play. It automatically detects and modifies server and service connection parameters, so that the application can run and connect to Cloud Foundry services without manually specifying configuration values. As a result, an application that is developed and tested locally can work seamlessly on CloudFoundry.com without any code changes. This was only a basic example of auto-reconfiguration in action. Let’s take a look at a more complex application that needs a database service to run. We are going to create our application using the content management system 

Calipso. It is based on the Express framework and uses the MongoDB database. First, we pull Calipso source from Github and install its dependencies. As Calipso depends on a native module, bcrypt, we should use Cloud Foundry’s npm support feature that recently became available. Following that blog post on npm support we create npm_shrinkwrap.json and set “ignoreNodeModules” in cloudfoundry.json. That’s it! Our application is ready to be deployed to CloudFoundry.com. As we deploy the application, we will be creating and binding a MongoDB service to the application.

$ vmc push calipso-app --runtime=node06
Would you like to deploy from the current directory? [Yn]:
Detected a Node.js Application, is this correct? [Yn]:
Application Deployed URL [calipso-app.cloudfoundry.com]:
Memory reservation (128M, 256M, 512M, 1G, 2G) [64M]: 128M
How many instances? [1]:
Bind existing services to 'calipso-app'? [yN]:
Create services to bind to 'calipso-app'? [yN]: y
1: mongodb
2: mysql
3: postgresql
4: rabbitmq
5: redis
6: vblob
What kind of service?: 1
Specify the name of the service [mongodb-c88a9]:
Create another? [yN]:
Would you like to save this configuration? [yN]:
Creating Application: OK
Creating Service [mongodb-c88a9]: OK
Binding Service [mongodb-c88a9]: OK
Uploading Application:
 Checking for available resources: OK
 Processing resources: OK
 Packing application: OK
 Uploading (95K): OK   
Push Status: OK
Staging Application 'calipso-app': OK                                           
Starting Application 'calipso-app': OK

As you can see from the output, the application was deployed successfully. If we go to its homepage we can see a welcome message from Calipso where we confirm that we are “awesome”!

Now we can follow the installation wizard steps. With auto-reconfiguration it means that we can just use any value, including the default, for the database setup. After the database is set up, we are ready to create a new article on our blog. And we can see that the connection to the data service is functioning, as the new article is published to our blog: To recap, we downloaded the application source, set up its dependencies, and deployed it to CloudFoundry.com using default connection parameters. The result is a working application. Let’s look now at the technical details on how this was accomplished.

Under the Hood

When your application is staged during the deployment process, Cloud Foundry makes two modifications:

  • Add a cf-autoconfig node module to the application
  • Preload the cf-autoconfig module while bootstrapping your application

The cf-autoconfig module uses the Node.js caching mechanism for module loading. Once a module is loaded, it is cached and requiring the same module elsewhere will take advantage of the cached code. The cf-autoconfig module searches for popular modules node.js applications use for connecting to services. It loads them before application code to redefine functions that connect to a service. Each modification replaces the original connection parameters (host, port, credentials, etc.) with equivalent parameters associated with a matching Cloud Foundry service bound to the application. With this arrangement in place, when application code subsequently loads the same module, attempts to connect to a service will yield a connection to an appropriate Cloud Foundry service. For an example, let’s see how it redefines the connect function of the MongoDB node module:

 
if ("connect" in moduleData) { 
  var oldConnect = moduleData.connect; 
  var oldConnectProto = moduleData.connect.prototype; 
  moduleData.connect = function () { 
    var args = Array.prototype.slice.call(arguments); 
    args[0] = props.url; 
    return oldConnect.apply(this, args); 
  }; 
  moduleData.connect.prototype = oldConnectProto; 
} 

Other functions are redefined the same way.

Take a look at the cf-autoconfig module’s source code on Github, and feel free to provide feedback or even a pull request.

Supported Modules

The following is the list of supported modules:

According to search.npmjs.org, most Node.js applications and other modules are dependent on the modules listed above. By providing support for these popular modules, any other modules that use them to form the database connection layer will inherit the benefit of auto-reconfiguration.

Limitations

Auto-reconfiguration of services works only under the following conditions:

  • You are only using one service of a given type. For example, only one mysql or one redis service.
  • You are using service node module from the list of supported modules above, or one that is based on a supported node module for service connections.
  • Your application does not use cf-runtime or cf-autoconfig node modules directly.
  • Your application is a typical Node.js application. (For a complex application you may want to consider opting-out of auto-reconfiguration and using the cf-runtime node module instead, which will be described in the next blog post in this series.)

Opting Out of Auto-Reconfiguration

Auto-reconfiguration can be turned off by providing cloudfoundry.json file in application base folder with the option “cfAutoconfig” set as false.

{ “cfAutoconfig” : false }

In addition, as mentioned above, auto-reconfiguration will not work if the application is using the cf-runtime node module.

Summary

Using auto-reconfiguration is a great way to quickly start deploying Node.js applications to Cloud Foundry. As your application grows and demands more precise control over its services you may need to consider using the cf-runtime node module to get easy access to application properties and services. In the next blog post we are going to show you how to use the cf-runtime node module to simplify connections to Cloud Foundry services.

- Maria Shaldibina The Cloud Foundry Team Don’t have a Cloud Foundry account yet?  Sign up for free today

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email

Cloud Foundry Supports Node.js Modules with NPM

Update: Recent Changes in Node.js Modules Support

We are pleased to announce support for npm (Node Package Manager) which manages Node.js application module dependencies on CloudFoundry.com. The popularity of Node.js can be partially attributed to its strong ecosystem that has created modules for practically any programming task–from database access to payment processing. At present, there are over 10,000 node modules listed on search.nodejs.org. Any cloud that aspires to provide good support for Node.js needs to simplify the task of using node modules. With the addition of npm support, Cloud Foundry now makes it easier for you to manage modules for node applications. The normal process of creating a Node.js application requires local installation of node modules (typically in the node_modules folder inside the application), essentially making the application self-contained. You can deploy this application locally or push all its contents, including the node modules, to CloudFoundry.com. This works fine as long as the app didn’t need a node module with native dependencies, which must be compiled specifically for the target platform. The npm support described in this blog obviates the need to package module dependencies with the app and makes it possible to use modules with native dependencies.

Requirements and Limitations

In order to use the npm feature, you would need to follow a few simple steps:

  • The application must provide an npm-shrinkwrap.json file. This file can be generated with the npm shrinkwrap command and reflects the whole tree of dependencies with fixed versions. This file is a snapshot of application dependencies in the same way that Gemfile.lock serves Ruby applications. This guarantees that Cloud Foundry will provide the exact same node module versions, which avoids incompatibility issues and follows the same dependencies hierarchy.
  • By default, Cloud Foundry uses node modules if the application includes them. More concretely, if an application has a node_modules folder in its base folder, then Cloud Foundry will simply use the modules in that folder. To overwrite the existing modules in the node_modules folder, you would provide a configuration file, cloudfoundry.json, with the “ignoreNodeModules” option set to true. If the above conditions are met, Cloud Foundry will install node modules to the application during staging based on the module versions, application runtime version and npm version. If a required node module does not work with the node engine that the application is running on, the module will not be installed. Currently, in order to push the application with the node engine version 0.6.x, it needs to be explicitly specified during application push.
$ vmc push app-name --runtime=node06

If a module installation fails for some reason, Cloud Foundry will use the node module provided by the user. Besides the compatibility requirement outlined above, there may be other possible reasons for installation failures. For example, when a node module of a given version is not found in the npm registry. When an error occurs, information about the installation process along with any npm error messages can be found in the application logs.

$ vmc logs app-name
...
Installing dependencies. Node version 0.6.8
Failed getting the requested package: bcrypt@0.4.5

Note that although npm supports git URLs specified manually after

npm-shrinkwrap.json was generated, Cloud Foundry currently does not support git URLs.

Native node modules Cloud Foundry npm support also allows deployment of Node.js applications with native module dependencies. Many popular node modules such as

bcrypt, fibers and nowjs have native code that depends on the platform. So locally installed node modules that were pushed along with the application would raise platform incompatibility errors often manifested as “invalid ELF header” errors during execution. Now, if an application meets the above requirements for npm support, compiled native dependencies will be provided by Cloud Foundry to the application during an application push.

Example of npm support in action

Let’s see how npm support works. Here we have an application with the main file app.js:

var app = require("express").createServer(); 
var bcrypt = require("bcrypt"); 
app.get("/", function(req, res) { 
  var salt = bcrypt.genSaltSync(10); 
  var hash = bcrypt.hashSync("B4c0/\/", salt); 
  if (bcrypt.compareSync("B4c0/\/", hash)) 
    res.send("World is safe!"); 
  else 
    res.send("World is in trouble!"); 
}); 
var port = process.env.VCAP_APP_PORT || 3000; 
app.listen(port); 

And package.json that specifies our dependencies. In this example we use express, which depends on other node modules and native dependency, bcrypt.

 
{ 
  "name" : "node-dependencies-example", 
  "version" : "0.1.0", 
  "dependencies" : { 
    "express" : "2.5.x", 
    "bcrypt" : "0.5.x" 
  } 
}

We first verified that the application is working locally with installed module versions. We can then run the ”npm shrinkwrap” command in the application’s base directory to generate the npm-shrinkwrap.json file:

  
{ 
  "name": "node-dependencies-example", 
  "version": "0.1.0", 
  "dependencies": { 
    "express": { 
      "version": "2.5.9", 
      "dependencies": { 
        "connect": { 
          "version": "1.8.7", 
          "dependencies": { "formidable": { "version": "1.0.9" } } 
        }, 
        "mime": { "version": "1.2.4" }, 
        "qs": { "version": "0.4.2" }, 
        "mkdirp": { "version": "0.3.0" } 
      } 
    }, 
    "bcrypt": { "version": "0.5.0" } 
  } 
}

We can now push the application without the node_modules folder, or we can provide a cloudfoundry.json file in the application’s base directory, which sets ignoreNodeModules to true to bypass the existing node modules.

 
{ "ignoreNodeModules" : true }

If we push our application now, Cloud Foundry will install node modules into the application that were compiled on Cloud Foundry’s platform.

$ vmc push node-dependencies-example --runtime=node06

Note that bcrypt has different APIs for different node engines, so we have to explicitly specify the required runtime. Finally, if we go to a browser we would see the display, “World is safe!”. Node dependencies example app screenshot

To summarize, npm support is a long-awaited feature that allows you to use the full variety of node modules available today, greatly enhancing your application.

- Maria Shaldibina The Cloud Foundry Team

Don’t have a Cloud Foundry account yet?  Sign up for free today

Facebook Twitter Linkedin Digg Delicious Reddit Stumbleupon Email