Thursday 5 May 2016

Keeping Secrets Secret with Vamos Deploy

How should you distribute application configuration that needs to be kept private? This is easy with Vamos Deploy.

Applications generally need configuration in order to do their job in the environment they used. Some configuration can be viewed publicly and some needs to be kept secret. Vamos Deploy caters for the distribution of both. Here we will show how to use Vamos Deploy and OpenSSL to distribute a secret.

Our secret will be the phrase 'Hello, World.' for this simple example. We have created a grid called Neptune which encapsulates just one application (called secrets) which will use our secret phrase.

$ vamos grid Neptune info
Gridname       : Neptune
Edit status    : Unfrozen
Change status  : Unchanged
Owner group    : vamos
Release group  : vamos
Applications   : secrets                   1.0.0      uat      Linux
Repos          : arepo-centos7-alpha       Linux

Lets encrypt our secret phrase and add to the grid. First we generate RSA public and private keys. Then use openssl to create a pem file which we then use to encrypt the secret.

$ ssh-keygen -t rsa -b 2048 -C "nathancope@vamosdeploy.com”
$ openssl rsa -in ~/.ssh/id_rsa -pubout > ~/.ssh/id_rsa.pub.pem
$ secret=$(echo Hello, World. \
         | openssl rsautl -encrypt -pubin -inkey ~/.ssh/id_rsa.pub.pem \
         | base64 -w0) 

Now we add 'secret' to our grid.

$ vamos grid Neptune addproperty secret_phrase $secret

$ vamos grid Netune info
Gridname       : Neptune
Edit status    : Unfrozen
Change status  : Unchanged
Owner group    : vamos
Release group  : vamos
Applications   : moon                   1.0.0      uat      Linux
Properties     : secret_phrase = BarhaR4OSDuX6H90ZfTxYPNPs93yik6hPeasvVYK5O3xPaTeur2kqRBOT+HHmTVIJIx8LEPtG5EatVfUat9dcVagECgc1rWaT5blNlRN6f3iRuARs9NJpDAHACqt1+i1lVGri8vXCFzvoTjnY8UEvw6pfSM2gMsNgeXQ8QTkmEkehPiCqbOfFRB+tqMbPKQV0maQXWFlGFvPoWcQaQYsJ0hKhsLbPDtaK5GM3BUgml08PZKicySPOVTB4wKBbMcqkK43L4bHQjMXkyPT2ZTsCKs5AoJyp+gb4/sqhx0sSI/3Xf986OWKyZYJ0BBAlnUZwffahfSxkec0ghTc6juTCA==
Repos          : arepo-centos7-alpha       Linux  

Now, what is public is the encrypted version of our secret phrase. Anyone that has the public key can encrypt but only people with the private key can decrypt. Oh the power of RSA!

Now to decrypt and print the secret phrase. Our application 'moon' uses openssl to decrypt using the private key:

$ /vamos/ARepo/grids/Neptune/moon/bin/show_secrets.sh 
Hello, World. 

Lets have a look at the code of show_secrets.sh:

#! /bin/bash

# This is a really useful technique that we use alot in Vamos Deploy
_whence="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
_props_dir="$( cd $_whence/../../properties && pwd )"

# This file is always present in this location and contains the properties for this grid
source $_props_dir/grid.properties

# This is just for this example. We don't recommend displaying the secret :)
echo -n "The secret message is : "
echo $secret_phrase | base64 -d | openssl rsautl -decrypt -inkey ~/.ssh/id_rsa

How would we set this up in a live system? The public key (id_rsa.pub.pem) can be displayed on a wiki or live on a public drive. The corresponding private key (id_rsa) needs to be installed (in the .ssh directory) only on the hosts that need to decrypt secrets. You can use these pair of keys to distribute as many secrets on as many grids on as many hosts as you want.

Wednesday 23 March 2016

Encapsulation with Vamos Deploy

Vamos Deploy encapsulates an application with it's binaries, all it's dependant libraries and it's runtime configuration. The 'capsule' is a self-contained runtime instance of an application which can be deployed to multiple hosts with a single command. Capsules do not conflict with each other and can happy exist side-by-side. There are no hard walls though, we rely on users abiding by the rules and good neighbourly conduct. In Vamos jargon, we call an encapsulated application a 'grid'.

On each host there can be one or more local repository. Once registered, a repository can be added to any number of grids. When a grid is 'deployed' each local repository, associated with that grid, is updated with all the required software packages and configuration. The application is ready to go! Takes just 2 commands:



$ vamos grid JupiterDev1 addrepo myrepo-myhostname
$ vamos grid JupiterDev1 deploy

All software and configuration is stored within the local repository, which on Linux is a directory structure under /vamos. Software is not scattered around the filesystems it's neatly contained in a dependable location, irrespective of the operating system.

In a future post we will examine how an application accesses its dependancies and runtime configuration from within the local repository.

Get in contact to see a demo of how this works : http://bitlq.com/5WLqU

Wednesday 16 March 2016

Using Vamos Deploy

No Git knowledge is needed, no Ruby or Python knowledge is needed, no YAML knowledge is needed. No writing of modules is required. Anyone can learn to use Vamos Deploy in 15 mins.

With Vamos Deploy you don't need to learn a new language to deploy your applications. We think a deployment tool should be simple to use which makes it easy for you to learn.

Vamos Deploy has a simple CLI which is not overbloated:

Wednesday 9 March 2016

Nine bullet points on how Vamos Deploy automates the delivery of software

  • Vamos Deploy encapsulates an application with it's binaries, all it's dependant libraries and it's runtime configuration.

  • A capsule Vamos can be deployed to multiple hosts with a single command.

  • Multiple capsules can exist and function on the same host with no conflicts.

  • Each application has a promotional lifecycle. It passes through a series of testing tollgates on it's path to production.

  • A strong segregation of duties prevents dev teams from deploying to production.

  • Reduces handover effort between dev and ops. Dev teams configure what is going to be deployed and Ops take responsibility of the orchestration (or how it gets released).

  • No scripting is required just use the simple CLI.

  • Everything is audited, all data is visible, it can integrate with other systems and its easy to use.

  • Read more details and see our website.


Please get in contact if you are thinking of having painless software deployments.


Monday 22 February 2016

Promoting the safe use of open source software

No company still downloads OSS from the internet and uses it in production with no checks, do they? No one would ever download pre-built OSS and use it. Would they?

Of course, you download the source code (so it can be scanned for vulnerabilities) and build it yourself so it's open and transparent what version of what software you are using. Not doing this opens your company up to a potential security nightmare.

Imagine, after a production outage, tracing the culprit to the unchecked use of OSS. Perhaps the OSS binary is not what you thought it was and/or it has a virus built into it. You don't know unless you build it yourself from source.

The use of OSS is growing year on year. The IDC says that OSS is 30% or more of the code at major Global 2000 organizations. But, surprisingly, the use of OSS is frequently uncontolled and unregulated. Do you know what OSS your company is using? Is it controlled?

Typically, even if the binaries are built from source, we don't really know which applications in which environments are using what. Something that is frequently neglected is the tracking of software installed on each server which can be used to quickly assess impact when vulnerable OSS is disclosed.

With Vamos Deploy all dependancies are transparent and visible. When a library is reported as being vulnerable you can assess your exposure by asking Vamos Deploy which applications use it and which servers this OSS is installed in. This is irrespective of OS type (Linux, Windows) or the language used (Java, C++, Python etc).

Vamos Deploy encourages the regulated, safe use of OSS. As the saying goes, "Lets be careful out there!"

Friday 12 February 2016

The down-to-Earth view of application deployment


View of the Earth from the International Space Station. Thanks to Nasa.

One area that I have been concentrating on for a while now is the automated delivery of software that actually works. This has recently become a hot topic with Continuous Integration and Continuous Delivery being discussed. The DevOps banner is being used for wider discussions which include the breaking down of barriers between teams to increase co-operation. This is all good stuff and a step in the right direction in an area which has long been neglected. Companies are now starting to invest in how they deliver their software. But they may just be saving up problems for the future.

Configuration management and release orchestration

The new technologies (Ansible, Chef, Puppet) in this space have the ability to drastically reduce the amount of time it takes to deliver a 'working' host. Many a time I've witnessed a host ping-ponging between Sys Admins and Dev teams before it's fit for use. The Dev teams don't specify exactly what they need and the System Admins don't automate it for future use. With DevOps we break this cycle and bring the efficiencies of automation to OS builds.

But these technologies leave fundamental gaps in the delivery of application software. Let's think this through a bit :

I have an application called Saturn that:

  • comprises of 10 different processes that each serve a different purpose;
  • each process has a different binary package;
  • uses Java 1.6 and has a list of dependant libraries;
  • is not resource intensive so its processes can co-habitate with other applications on a host and;
  • has configuration that needs to differ between environments.

We don't have the luxury of a dedicated host so Saturn shares hosts with other applications. In fact, in Dev and UAT we have multiple instances of Saturn running and even many different versions of Saturn running on the same host. If we are to share it's essential for the application to be a good neighbour so it doesn't have to be isolated. But without the right tools, sharing of resources can cause conflicts. Here is why:

There is another team that are building Pluto and they are using Java 1.8. This is one of the applications that share hosts with Saturn. Instantly we see a conflict in the version of Java used. We could hardcode the Java version we use but that reduces our application portability.

We then discover that Saturn and Pluto use different versions of the same Java libraries. How do we cope with that? We could embed these libraries in our application but that seems rather clumsy, encourages duplication and reduces visibility of usage. This is a wider problem than just Java. There are two other teams that own Jupiter and Mars which are written in Python. They have very similar problems with the use of conflicting versions of Python and Python libraries. These problems stem from the reliance on the underlying OS build to fulfill runtime dependancies.

When resource sharing we need a tool that prevents dependancy conflicts.

There are similar conflicts when we look at the application binaries. We are making our lives unneccessarily difficult if we don't have tooling that provides functionality to:

  • easily manage multiple versions of the application software on a single host;
  • select/update binary versions used in each environment;
  • give transparancy and visibility of environment and configuration settings and;
  • automate the removal of unused software for effective disk management.

Storing your application binaries and libraries in a repository makes perfect sense. Yum is frequently used to hold RPMs that are built containing the application software. But this is for RedHat-based Linux distros; what about Debian-based distros and of course Windows and OS X? This is highly restrictive! We need a repository that can store software for all flavours of OS.

We need our repository to have functionality to:

  • be OS and packaging standard independant
  • distinguish between production and non-production packages
  • retain meta data about each package:
      • when was it created and by whom
      • what level of testing it has had
      • which environments its used in
      • which hosts it exists on
  • be immutable so you know it's not been tampered with
  • restrict who can add or remove packages

Configuration management tools can make a big difference in release orchestration. Releases often comprise of many steps that need to be performed in a certain order over a number of different hosts. Software deployment is usually an essential part but there are frequently many other steps involved (ie process stop/start, database updates, batch scheduler changes etc).

There is however the temptation to mix up what is being released with how it is being released. The release runbook (sometimes called recipe or playbook) should contain all the steps involved and their dependancies. If, in the runbook, we start to include what is actually changing (ie versions of packages to install or remove) then it would need modifying for each release performed and needs to be tested, just like code. To lessen the testing burden it seems logical to separate what is being released from how it gets released. But what tools do we need to do this?

Using a configuration manager to quickly set the standard OS build and to orchestrate a release saves a lot of time and money and gives welcome consistency. But if your application is part of a larger business where resources are shared and is going to be changing frequently, a configuration manager has fundamental gaps. We need something more.

Logical or pseudo containers

Vamos Deploy has the functionality to encapsulate the components of an application with all its dependancies, libraries, binaries and configuration files so they can be deployed as one to any number of local Vamos repositories irrespective of the underlying OS. We call this grouping a Vamos Grid.

Deploying your applications via Vamos Deploy:

  • frees you from relying on software that is installed as part of the underlying OS. This saves you from the conflicts that causes;
  • enables multiple instances of the same application can be running on the same kernel that can be independantly re-configured or updated;
  • enables many instances of different applications to happily and easily co-exist on the same host with different configurations;
  • enables applications to be promoted once they pass different stages of testing. This test-promote lifecycle keeps non-production quality code well away from the production domain;
  • brings the benefit of a full application lifecycle. The detection and removal, from local repositories, of application packages that are no longer used is running from day 1;
  • allows you to update your development environment with exact copies of software used in production in a single command. You can even look back at a fixed point in time and replicate how production looked;
  • avoids duplication due to the Vamos Grids sharing the underlying application software and libraries in the local repositories.

Vamos Deploy's promise is to distribute all the software components and configuration to the local Vamos repository on each host or client that you have told it to. This can be initiated from a configuration management tool as part of release orchestration. Vamos Deploy is responsible for what is being released and the configuration manager is responsible for how it gets released.

With Vamos Deploy you don't need to learn YAML, Ruby or Python. There is a simple Vamos CLI, which can be called from any configuration manager or embedded in any self-service environment manager. There is no need for lengthy training sessions on cutting-edge technologies and radically changing your deployment methodology.

Vamos Deploy is a down-to-Earth tool that takes the headaches out of managing the lifecycle of an application and it's deployments.

If you think that this product could help you then let me know. Or, if you have questions, don't hestitate to ask.