Creating a Vamos Grid

In a previous article we added an application to Vamos so lets move to the next step of creating a Vamos grid.

First, a Vamos grid is an association of applications, libraries, properties and a list of hosts they are to be deployed to. We frequently think of a Vamos grid as an environment.

Lets create a grid, add an application, a property and a repo:

 $ vamos grid Jupiter_dev create
 $ vamos grid Jupiter_dev addapp JupiterServer 1.0.1
 $ vamos grid Jupiter_dev addproperty jupiter_http_port 8091
 $ vamos grid Jupiter_dev addrepo arepo-vm-barcelona-alpha

Lets look at a grid I prepared earlier:

 $ vamos grid Jupiter_dev info
 Gridname       : Jupiter_dev - For Jupiter developer testing purposes
 Edit status    : Unfrozen
 Change status  : Unchanged
 Owner group    : developers
 Release group  : developers
 Applications   : JupiterServer             1.0.32     dev        Linux
                : JupiterUI                 1.0.17     prod       Windows
                : PythonDjango              1.7.1      prod       ALL
                : PythonGoogleFinance       0.7.0      prod       ALL
                : PythonLinux               3.4.lin32  prod       Linux64
                : PythonRequests            2.5.1      prod       ALL
                : PythonRestFramework       3.0.1      prod       ALL
                : PythonWindows             3.4.win32  prod       Windows64
 Properties     : jupiter_http_host = 192.168.56.101
                : jupiter_http_port = 8091
                : jupiter_window_title = JupiterUI-Version:1.0.17-Build:#73
 Repos          : arepo-nathan-pc           ALL        Vamos      Windows    Windows64
                : arepo-vm-barcelona-alpha  ALL        Linux      Linux64    Vamos

Here we have a collection of applications including python libraries, python itself and the 2 Jupiter applications. There are three environment specific properties being set and two repos one Linux host and one Windows client.

When we deploy this grid Vamos will deliver the applications and properties to both of the repositories. So providing the versions of application binaries to use and the runtime environment including environment specific properties.

This is explained in more detail here and in this YouTube demo http://bitlq.com/R0Joq

Auto increment application versioning - MAJOR.MINOR.PATCH

We noticed that many teams implement a mechanism to increment their application version. Frequently, a script is used to look at the previous version used and increment. With Vamos Deploy this is built-in.
Vamos recommends the use of semantic versioning for your application versions.To use, simply embed '++' in the version pattern to indicate if MAJOR, MINOR or PATCH is to be incremented.

To increment the patch level from 1.0.0 use '1.0.++' as in example below:

 $ vamos application JupiterServer 1.0.++ create --directory=$HOME/JupiterServer
Successfully registered application JupiterServer 1.0.1 and copied to repo at /vamos/myrepo/applications/JupiterServer/1.0.1

To increment the minor level from 1.0.0 use '1.++' as in example below:

 $ vamos application JupiterServer 1.++ create --directory=$HOME/JupiterServer
Successfully registered application JupiterServer 1.1.0 and copied to repo at /vamos/myrepo/applications/JupiterServer/1.1.0

To increment the major level from 1.0.0 use '++' as in example below:

 $ vamos application JupiterServer ++ create --directory=$HOME/JupiterServer
Successfully registered application JupiterServer 2.0.0 and copied to repo at /vamos/myrepo/applications/JupiterServer/2.0.0

Other styles of versioning can be used but semantic versioning is the style we have implemented with an auto increment mechanism. This can be changed, by developing a new auto increment plugin for Vamos Deploy.

Adding a new application to Vamos Deploy

This post will show you how to use the CLI to add a new application to Vamos Deploy.

Let's assume that you have your application code in an SCM (git, svn etc..) and you have the ability to build any parts, that you need, into the artifacts that will be deployed. 

What directory structure is best to use for our package? With Vamos you are free to decide on this yourself. The only rule is that you have a directory called 'bin'. Vamos will give all contents execute permissions.

 $ vamos application JupiterServer 1.0.0 create --directory=$HOME/JupiterServer

The above command will create the application in the Vamos database and copy the code to the local repository. It will instantly be useable on this host. File permissions will be set to be read-only with executable permissions being given to all contents of the 'bin' directory. The local repository will be at the following location:

 /vamos/<our department>/applications/JupiterServer/1.0.0/  

The status of this application will be 'local' and it only exists locally on the host we created it on. We can commit this application to the Vamos filestore so it can be used by other hosts. This will also change the status to 'dev':

 $ vamos application JupiterServer 1.0.0 commit  

We need to add a subscription to the application. More details on this in another post but they are used to specify what operating system applications were built on and hence what this application is compatible with. For now let's set to ALL to indicate it can run on any o/s:

 $ vamos application JupiterServer 1.0.0 addsub ALL  

Let's now see the details of what we have created:

 $ vamos application JupiterServer 1.0.0 info  
 Application   : JupiterServer 1.0.0  
 Status        : dev  
 Owner group   : developers  
 Subscription  : ALL  

The owner group will be set to the default group that your userid was registered with. In this case the user is in the developers group. Vamos uses a simple group ownership model for all objects it creates. If you created it then your group owns it and your group can do whatever it wants with it.

What do you need to learn in order to use Vamos Deploy?

No Git knowledge is needed, no Ruby or Python knowledge is needed, no YAML knowledge is needed. No writing of modules is required. Anyone can learn to use Vamos Deploy in 15 mins.

At Vamos Deploy we don't think you should have to learn a new language to be able to deploy your applications. We think a deployment tool should be simple to use which makes it easy to learn. 

Creating great software is hard so give yourself a break and make your deployments as easy as possible.   

Vamos Deploy consists of: 

• Vamos CLI which is designed for you to embed into your own scripts or simple call on the command line
• Vamos Dashboard is a web based application to view data about applications that have been deployed via Vamos.
• Vamos Jump which runs on clients desktops to facilitate the starting of your application UIs.

First Steps in Automation

If you have got this far you are probably already thinking that your application deployments needs improvement. Maybe you have one or more of the following deployment 'smells':

• home grown 'release' scripts that differ between applications
• manually scp/sftp/rsync software
• manually edit property files in production
• complex release notes

These can result in slow, unreliable, unrepeatable deployments which will not scale. But where do we start in transforming a release process? The key to an effective release process is consistency. Before we get to discussing Vamos Deploy there are areas that can be tackled in preparation which will reap rewards later on.

Packaging:

Vamos does not enforce a packaging methodology, its very open. The only rule is that there needs to be a ‘bin’ directory and all of its contents will be granted execute permissions. You have the flexibility to build, as you require, on top of this single rule. Though we do recommend setting a basic standard layout so most packages have the same look and feel.

Naming conventions, with a little thought, can make life much easier in the long run. A common mistake is to embed the management structure rather than the business into the name. The management structure can change frequently so can make naming obsolete.

A good naming standard would contain the business name, the application suite, application name and application instance. You should end up with names in the region of 10-15 chars in length. CamelCase is a good idea also.

Version numbers used for applications should follow the semantic versioning pattern of MAJOR.MINOR.PATCH. See http://semver.org/

If you have packages that are very large they will take a long time to deploy. If you change just 5% of a large package then 95% will be redeployed for no gain. So, try to split up large packages into smaller ones. We will use a Vamos Grid to link them together. As a rule of thumb, a package of more than 250Mb sounds large.

How many is the right number of packages? This varies hugely but a good rule of thumb is to have between 10 and 50. More packages, will mean more to manage. Its a balancing act between speed and manageability.

One idea is to split packages, instead of by function, by the frequency of change. Package independently, parts that change frequently. Then your deployment is fast and you have few moving parts.

Runtime:

Standardisation of runtime environments can bring huge benefits. Most of the next points sound obvious and simple, which they are, but it’s amazing how frequently it’s not done.

All software you need should be deployed. Don’t rely on any software being present on a host! Don’t rely on someone else to put it there for you! It's worth listing what you rely on so later you can create them all as Vamos applications.

All applications should be good citizens. Applications should be able to run alongside any other application with no conflict. It shouldn’t need a dedicated user to run.

The scripts used to start and stop your application should be shipped with your application. No ‘support’ scripts should be held locally. They should all be held with your source code and deployed in the same manner.

Make sure when your application is running on a host that it is easily identifiable from the process list. Makes support much easier.

Select a standard area for placing the logs of your application. This should be agreed so all applications supported by one team use a standard directory layout. Make the log filename unique so you can identify the instance running.

Use a dedicated scheduler to start and stop your application using the scripts you deploy. It should be the only tool that does this in a production environment. The schedule needs to be tested before it’s used in production!

Keep all your environments clean. There no excuse for untidiness. Have an automated job that zips old logfiles and delete them after a configurable number of days. Remove all old core files. Only keep files on your system that you need - put the rest in the trash.

Application configuration:

A simple approach that works well in most cases is to categorise your properties into:

• Properties that need to change for each environment: Add these properties to a grid. Then use Vamos templatisation to inject them into property files for your application to read.

• Properties that change across all runtime environments: These can be added to property files and deployed with your application. A code change will be needed to change them.

• Properties that need to be secret: A common way is to use UNIX permissions to secure a password file owned by the runtime user. We prefer to use RSA to secure passwords. Anyone with the public key can encrypt but only those with the private can decrypt.

Do not have property files that are stored in an environment and changed locally. This is a recipe for disaster!
 

Finally...

Now that we have made you aware of some preparation steps we can move onto how to use Vamos Deploy. Starting with the next entry.