Presentation of my hosting infrastructure

Hello world,

Today I’d like to present you what I did for hosting the projects I am working on. I started to work on it one year ago and after endless refactoring I finally ended up making something I am quite proud and even maybe something other people will be able to reuse.

As you know, my very goal is the promotion of open-source softwares to people without technical knowledge. Either for Wikicompare or the community verticalization of Odoo, I want everyone to be able to easily create their own instances simply by filling a form on a website and for this I needed powerful, flexible infrastructure and something I’ll not spend too much time on.

Flexible, that’s the word. Wikicompare is based on Drupal and the community verticalization on Odoo. Moreover, I want to easily host any open-source application I’ll want to promote later.
This means completely different infrastructures and problematics and I iterated numerous time to make sure everything was flexible enough. Today I can already host Odoo, Drupal, WordPress and Seafile as availables applications and postgres/mysql/shinken/postfix/bind/piwik for the internal components. Now that the core is done, it take me only one day to add a new application.

The heart of my infrastructure is what I call an orchestror (can’t find any good English word for “chef d’orchestre” and I don’t like “conductor” please help me), I mean here the system which will contain the list of all servers/domains/databases managed by the infrastructure, call the executables on the servers when we launch an action and provide a backoffice for easily maintaining this list/launch actions.

Currently, I feel that all hosting companies developed their own scripts to make the orchestror, I guess they see it as a competitive advantage but this not my point of view. In my case hosting is only a cost and that’s why I open-sourced everything I did on this topic.

Tell me if I’m wrong, I tried to search if any open-source orchestror project already exist and I weren’t able to find any good enough and especially flexible enough.
Chef/puppet/salt aren’t designed to easily create and manage custom field/objects, and same for OpenStack I think. I tried OpenStack one whole day and it looked too complex for me and I wasn’t sure there was a way to create a backoffice with it, I think it will a be a good sub-component but not for the orchestror itself.
On the next part on this post I’ll present my orchestror, tell me if you can do everything I do with any existing open-source project, my goal is not to duplicate again the open-source effort, it’s the contrary.

So, I said I need the flexibility of specific development for the orchestror, with me this means odoo module. So yes my orchestror is some odoo modules you’ll find here : https://github.com/clouder-community/clouder. You can install them on an openerp V7 if you want to see the whole interface.

I think today it’s critical to separate each component on a server (separate each odoo install for each customers for example). Some may think about virtual machine but it cost too much CPU/RAM, you can’t have hundred of VM on a server, and so my preference goes to containers.
I’m not the only one, Docker is all the rage everywhere currently, and many odoo hosting company I spoke to use OpenVZ extensively since a lot time. I decided to go with docker, and my orchestror is designed around it.

I also currently use shinken for supervision, piwik for analytics, bind for DNS, postfix for sending mail/fetchmail and nginx for proxy.

Ok let’s show you some screenshot of the interface now, here is the main menu :

Main page

On the left menu you’ll find all concepts managed by the orchestror. Base, services and containers are the main ones.

Containers of courses represents the list of all docker containers.

Container list

You can for each one specify the application to host inside the container and the image version to use, the ports open and the volumes containing critical files. Container form

Services represent each install of the application inside a container (for example several odoo install on different ports inside the same container). Here you can specify the service name, the container it belong and the version of the application to use. A database user will automatically be created in postgres or in mysql. Service list Service form

Finally, base represent both a database and a domain. It is linked to one url which shall then be interpreted by the service. Here you can specify the domain name, the service used, the users credentials and the SSL certificate. It will automatically set the DNS, create the database, set supervision and configure the proxy. Main page

Base form

I think the three concepts container/server/base are keys to have the flexibility we need. With this you can do multi-base hosting like Odoo SA on few services, or on contrary create one container for each customer to ensure security. Multi-services is also really interesting to easily duplicate the production environment into a test one before a migration.

You can also of course manage the list of your physical server and domains you own
Server Domains

Saves are listed inside the orchestror, so you can easily restore them.Save list Save form

The list of docker images are also stored here. For each image you’ll be able to specify the ports, the volumes and the dockerfile used to build it.

When you configure the image, you just have to click on the build button, a new version will be created and you’ll be able to use it in the container list. Image

It’s the same for applications, you’ll be able for each application to manage the default image to use, the time between autosaves, the other applications they are linked to and the code used to build the application version.

The build method differ for each application type, for example for Odoo I used the anybox recipes while for Drupal I use drush files. Application

I think this conclude the presentation of my hosting infrastructure. There is still a lot to do to be state of the art but I truly believe that the foundation are solid and flexible enough.

I think there is a lot of dispersion of the open-source community on the hosting question, and even in the odoo community itself. If what I did seems good enough for you don’t hesitate to contact me so we can build a community around it.

A simple system with preconfigured profiles for every open-source application that every association/company in the world can use, in order to provide professional hosting to every people without technical knowledge. I truly believe this is what we need to promote open-source, maybe it can be this, or anything else, but we really need this.

EDIT :

With the many questions I received after this post, I moved the github repo into
https://github.com/clouder-community/clouder
and I created a mailing-list :
https://launchpad.net/~odoo-vertical-hosting.
Feels free to subscribe!