Presentation of my hosting infrastructure

Hello world,

Today I’d like to present you what I did for hosting the projects I am working on. I started to work on it one year ago and after endless refactoring I finally ended up making something I am quite proud and even maybe something other people will be able to reuse.

As you know, my very goal is the promotion of open-source softwares to people without technical knowledge. Either for Wikicompare or the community verticalization of Odoo, I want everyone to be able to easily create their own instances simply by filling a form on a website and for this I needed powerful, flexible infrastructure and something I’ll not spend too much time on.

Flexible, that’s the word. Wikicompare is based on Drupal and the community verticalization on Odoo. Moreover, I want to easily host any open-source application I’ll want to promote later.
This means completely different infrastructures and problematics and I iterated numerous time to make sure everything was flexible enough. Today I can already host Odoo, Drupal, WordPress and Seafile as availables applications and postgres/mysql/shinken/postfix/bind/piwik for the internal components. Now that the core is done, it take me only one day to add a new application.

The heart of my infrastructure is what I call an orchestror (can’t find any good English word for “chef d’orchestre” and I don’t like “conductor” please help me), I mean here the system which will contain the list of all servers/domains/databases managed by the infrastructure, call the executables on the servers when we launch an action and provide a backoffice for easily maintaining this list/launch actions.

Currently, I feel that all hosting companies developed their own scripts to make the orchestror, I guess they see it as a competitive advantage but this not my point of view. In my case hosting is only a cost and that’s why I open-sourced everything I did on this topic.

Tell me if I’m wrong, I tried to search if any open-source orchestror project already exist and I weren’t able to find any good enough and especially flexible enough.
Chef/puppet/salt aren’t designed to easily create and manage custom field/objects, and same for OpenStack I think. I tried OpenStack one whole day and it looked too complex for me and I wasn’t sure there was a way to create a backoffice with it, I think it will a be a good sub-component but not for the orchestror itself.
On the next part on this post I’ll present my orchestror, tell me if you can do everything I do with any existing open-source project, my goal is not to duplicate again the open-source effort, it’s the contrary.

So, I said I need the flexibility of specific development for the orchestror, with me this means odoo module. So yes my orchestror is some odoo modules you’ll find here : https://github.com/clouder-community/clouder. You can install them on an openerp V7 if you want to see the whole interface.

I think today it’s critical to separate each component on a server (separate each odoo install for each customers for example). Some may think about virtual machine but it cost too much CPU/RAM, you can’t have hundred of VM on a server, and so my preference goes to containers.
I’m not the only one, Docker is all the rage everywhere currently, and many odoo hosting company I spoke to use OpenVZ extensively since a lot time. I decided to go with docker, and my orchestror is designed around it.

I also currently use shinken for supervision, piwik for analytics, bind for DNS, postfix for sending mail/fetchmail and nginx for proxy.

Ok let’s show you some screenshot of the interface now, here is the main menu :

Main page

On the left menu you’ll find all concepts managed by the orchestror. Base, services and containers are the main ones.

Containers of courses represents the list of all docker containers.

Container list

You can for each one specify the application to host inside the container and the image version to use, the ports open and the volumes containing critical files. Container form

Services represent each install of the application inside a container (for example several odoo install on different ports inside the same container). Here you can specify the service name, the container it belong and the version of the application to use. A database user will automatically be created in postgres or in mysql. Service list Service form

Finally, base represent both a database and a domain. It is linked to one url which shall then be interpreted by the service. Here you can specify the domain name, the service used, the users credentials and the SSL certificate. It will automatically set the DNS, create the database, set supervision and configure the proxy. Main page

Base form

I think the three concepts container/server/base are keys to have the flexibility we need. With this you can do multi-base hosting like Odoo SA on few services, or on contrary create one container for each customer to ensure security. Multi-services is also really interesting to easily duplicate the production environment into a test one before a migration.

You can also of course manage the list of your physical server and domains you own
Server Domains

Saves are listed inside the orchestror, so you can easily restore them.Save list Save form

The list of docker images are also stored here. For each image you’ll be able to specify the ports, the volumes and the dockerfile used to build it.

When you configure the image, you just have to click on the build button, a new version will be created and you’ll be able to use it in the container list. Image

It’s the same for applications, you’ll be able for each application to manage the default image to use, the time between autosaves, the other applications they are linked to and the code used to build the application version.

The build method differ for each application type, for example for Odoo I used the anybox recipes while for Drupal I use drush files. Application

I think this conclude the presentation of my hosting infrastructure. There is still a lot to do to be state of the art but I truly believe that the foundation are solid and flexible enough.

I think there is a lot of dispersion of the open-source community on the hosting question, and even in the odoo community itself. If what I did seems good enough for you don’t hesitate to contact me so we can build a community around it.

A simple system with preconfigured profiles for every open-source application that every association/company in the world can use, in order to provide professional hosting to every people without technical knowledge. I truly believe this is what we need to promote open-source, maybe it can be this, or anything else, but we really need this.

EDIT :

With the many questions I received after this post, I moved the github repo into
https://github.com/clouder-community/clouder
and I created a mailing-list :
https://launchpad.net/~odoo-vertical-hosting.
Feels free to subscribe!

4 thoughts on “Presentation of my hosting infrastructure”

  1. Hi Yannick
    I’m quite like-minded and welcome your idea! However, I’d like to promote discussion about some design choices in this quickly evolving ecosystem. I think it’s crucial to make the right call on future developments. I’ve some opinion about this, I’d like to share. Ideally, I would prefer a http://www.discourse.org/ about it ;-) Mailinglists are somewhat shady.

    An architechture role model would help to understand your framework better. However from what I read, it is a backend/orchestrator. As the “orchestrator-based-on-odoo” part seems to me like an escpecially bad call on the future dynamics of the docker ecosystem (see “Docker Compose”, “Docker Machine”, “Docker Swarm”, “Deis”, “Mesosphere”), I tried to dig in further into the source. And indeed, I fear, that doing things the odoo/flexible way, is too monolythic to survive (as a concept).

    I think, when we want to make the right calls on deploying across infrastructure, we need to think about service oriented architecture, but distributed.

    I honestly think from within odoo-community we should only think about a connector. If the vision is brighter, probably grouping efforts with like-minded peers in the respective communities (mesos, deis, flynn, etcd, fleet) would be more ressource efficient.

    From an odoo point of view a connector (rather than an orchestrator – or “scheduler”, how it is also called) is also the right scope, because orchestrating infrastructure is non of odoo-communities core competencies – and should probably not be. So the make-or-buy decision should go for buying a scheduler (e.g. marathon, fleet) and use their respective platforms (as platfrom paradgim is changing from Single-OS [“ubuntu”] to distributed [“CoreOS”, “Mesos”]).

    I’m not sure if you have thaught in your code base about the rudiments of an interface between the connector / scheduler paradigm, but if so, I would be interested in promoting efforts.

    Let’s say a odoo-mesosphere connector or a odoo-deis connector.

    1. Hello David thank you for your insight and welcome in my blog.

      I have to say, I’m really not a expert about Docker. I know how to use it, but I don’t know about everything happening around it (still, I do know it’s the right technology to use currently and I know it’s power). I’m discovering many tools in your comment, thank you for this.

      You may think the usually “Why did he reinvented the wheels?” so I think I shall answer this question. The fact is, first I needed something to organize my infrastructure and I wanted something fully automated because I knew I’ll later have little time to care about hosting. When a customer want a new Drupal or Odoo instance, I only want to fill a form and it automatically create everything needed (including configuring bind/shinken/piwik/postgres/postfix needed to link to the container).
      Second, honestly when I started I didn’t knew what I wanted to do, I just tried to do my best, think about every possible case and when I finished I just thought “Hey not bad, maybe it can interest other people”. Just to say that at first, I took some time to see if some software did everything I needed, and when I found nothing I started developing it because I really needed it and had no other choice.

      Again, I value this kind of discussion because I know I don’t know all tools and maybe I miss one which does everything the odoo orchestrator do. If you know one, then I’ll drop immediately this and start using it, but I don’t think so and here is my arguments :
      -The most important, the graphical interface. For me, the odoo orchestrator is only a front-end for docker. The python code which make the system calls must be kept to the bare minimum and we shall delegate as much as possible to other applications (like Docker, puppet, whatever). I think Odoo is a great tool for this, because storing the list of the 100 databases you host for your customers is according to me an administrative task, and Odoo is for me THE tool for managing administrative process. Not to mention that it allow invoicing with the account module and have all the access right needed to open restricted access to customers so they can manage their own containers.

      -The flexibility, the reason why admin sys use so much bash script is because every company has his own problematics. Odoo is where flexibility can happen, calling this specific command after this application build, adding this field because you need to keep an information etc… This way we have the flexibility for our process while the subcomponent like Docker take care of the complex stuff under it.
      It’s just a front, I just decided to use Odoo instead of using a CMS, so we can more easily share the modules and because I think Odoo is more appropriate.

      -I don’t think any tools can currently correctly manage the concepts of multi-container, multi-service and multi-base. It’s important concepts because, even if an application like postgres only need the multi-container concept, Odoo and Drupal need the three (multi-container for customers who want ssh access, multi-service for separating production and test services, and multi-base if we want to host multi-base like Odoo SA). I found that theses three concepts are enough to manage all type of applications.

      -Which tool can correctly manage links today? As example, the fact that a new entry will be added in piwik when I deploy a new odoo instance. It seems Docker Compose can be use to bundle container together but I don’t want to deploy a new piwik each time I deploy an odoo instance, I want him to use the piwik I specified in the form. Of course if Docker Compose do everything I do with my links, I’ll adapt the interface to stick with it and use it as subcomponent.

      Again, I made this as an amator, I’m saying it with no shame, but I still don’t think any tool implement all theses concepts yet. I (and probably others) need theses four things, if a tool can answer all theses points then I’ll agree with you. Please share your thoughts.

      Finally a word about the future, I’m planning to provide odoo orchestrator hosting (odoo orchestrators which the hosting is managed by an odoo orchestrator, inception yeah!) this way I shall be able to provide some bitnami-like service (but full open-source).

      Again, thank you for your interesting comment, looking forward to see your thoughts.

Leave a Reply to Rui Andrada Cancel reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>