Metadata-Version: 1.0
Name: collective.hostout
Version: 1.0a4
Summary: collective.hostout will help you control multiple application environments
    with the minimum amount of effort. You can manage local, staging and deployment environments
    with one easy tool.
Home-page: https://svn.plone.org/svn/collective/collective.hostout
Author: Dylan Jay
Author-email: software@pretaweb.com
License: GPL
Description: Hostout - standardized deployment of buildout_ based applications with Fabric_
        ==============================================================================
        
        Hostout gives you:
        
        - the ability to configure your Farbic_ commands from within buildout_
        - a framework for integrating different Fabric_ scripts via setup tools packages
        - an out of the box deployment command for buildout_ based applications
        - plugins to integrate deployment with supervisord and libcloud
        
        Hostout is built around two ideas :-
        
        1. Sharing Fabric_ command configuration amongst developers in a team
        so where and how your applications is deployed becomes configuration, not
        documentation. Deployment then becomes a single action by any member of the team.
        
        2. Sharing fabric scripts via PyPi so we don't have to reinvent ways
        to deploy or manage hosted applications
        
        To contribute :-
        
        - Code repository: http://github.com/collective/collective.hostout
        - Questions and comments to http://github.com/collective/collective.hostout/issues
        - Report bugs at http://github.com/collective/collective.hostout/issues
        
        .. contents::
        
        
        Overview
        ********
        
        Hostout is a framework for managing remote buildouts via fabric scripts. It
        includes many helpful built-in commands to package, deploy and bootstrap a
        remote server with based on your local buildout.
        
        You don't need to learn Fabric_ to use hostout but you will need to learn buildout_.
        The good news is that many buildouts and snippets already exist for building django,
        pylons, pyramid, plone, zope, varnish, apache, haproxy or whichever server side
        technology you want to deploy.
        
        Users of hostout will be interested in
        
        - Intergrating your fabfiles into buildout, or
        - Deploying your buildout using hostout
        
        
        Hostout deploy
        **************
        
        Hostout deploy is a built-in Fabric command that packages your buildout and
        any development eggs you might have, copies them to the server, prepares
        the server to run and then runs buildout remotely for you. This makes it simple
        to deploy your application.
        
        Development buildout
        --------------------
        
        For example, let's say we had the worlds simplest wsgi application ::
        
        from webob import Request, Response
        
        def MainFactory(global_config, **local_conf):
        return MainApplication()
        
        class MainApplication(object):
        """An endpoint"""
        
        def __call__(self, environ, start_response):
        request = Request(environ)
        response = Response("Powered by collective.hostout!")
        return response(environ, start_response)
        
        We keep this is a package in ``src/hellowsgi``.
        We will create a buildout cfg file called ``base.cfg`` ::
        
        [buildout]
        parts = demo pasterini
        develop =
        src/hellowsgi
        
        [demo]
        recipe=zc.recipe.egg
        eggs =
        PasteScript
        hellowsgi
        
        [pasterini]
        recipe = collective.recipe.template
        output = parts/demo/paster.ini
        port = 8080
        input = inline:
        [server:main]
        use = egg:Paste#http
        host = 0.0.0.0
        port = ${:port}
        
        [pipeline:main]
        pipeline =
        app
        
        [app:app]
        use = egg:hellowsgi#main
        
        Once we bootstrap and build this::
        
        $> python bootstrap.py -c base.cfg
        $> bin/buildout -c base.cfg
        
        we have a working wsgi app if you run ::
        
        $> bin/paster serve parts/demo/paster.ini
        
        Production buildout
        -------------------
        
        Next you will create a "production buildout" which extends your base.cfg.
        This might contain parts to install webservers, databases, caching servers etc.
        
        Our prod.cfg is very simple ::
        
        [buildout]
        extends = base.cfg
        parts += supervisor
        
        [supervisor]
        recipe=collective.recipe.supervisor
        programs=
        10 demo ${buildout:directory}/bin/paster [serve ${pasterini:outout}] ${buildout:directory} true
        
        [pasterini]
        port = 80
        
        
        Deployment buildout
        -------------------
        
        Now create a 3rd buildout file, called ``buildout.cfg``. This will be our development/deployment
        buildout ::
        
        [buildout]
        extends = base.cfg
        parts += host1
        
        [host1]
        recipe = collective.hostout
        host = myhost.com
        hostos = ubuntu
        user = myusername
        path = /var/buildout/demo
        buildout = prod.cfg
        post-commands = bin/supervisord
        
        This buildout part will install a script which will deploy prod.cfg
        along with hellowsgi to remote path /var/buildout/demo on our server myhost.com ::
        
        $> bin/buildout
        Develop: '.../example'
        Uninstalling host1.
        Installing demo.
        Installing host1.
        
        As part of the buildout process hostout will automatically determine the
        versions of all the eggs in your development buildout in a file
        called ``hostoutversions.cfg`` and will pin them for
        you during deployment. This ensures that the production buildout will
        be running the same software as you have tested locally. Remember to
        manually version pin any additional eggs you use in your ``prod.cfg``
        as these will not be pinned for you.
        
        Running hostout deploy for the first time
        -----------------------------------------
        
        The ``bin/hostout`` command takes three kinds of parameters, ::
        
        hostout [hostname(s)] [commands] [command arguments]
        
        in our case we will run ::
        
        $> bin/hostout host1 deploy
        
        The first thing will do is ask you your password and attempt to login in to your
        server. It will then look for ``/var/buildout/demo/bin/buildout`` and when it doesn't
        find it it will automatically run another hostout command called ``bootstrap``.
        
        Bootstrap is further broken down into three commands, bootstrap_users_,
        bootstrap_python_ and bootstrap_buildout_. These will create an additional user
        to build and run your application, install basic system packages needed to
        run buildout and install buildout into your remote path. It will attempt to
        detect which version of linux your server is running to os python, but if this
        fails it will attempt to compile python from source. The version of python used
        will match the major version of python which your development buildout uses.
        
        Deploying and re-deploying
        --------------------------
        
        Once hostout bootstrap has ensured a working remote buildout, deployment will continue
        by running the following commands:
        
        1. "uploadeggs": Any develop eggs are released as eggs and uploaded to the server. These will be
        uploaded directly into the buildout's buildout-cache/downloads/dist directory which buildout
        uses to find packages before looking up the package index. It's very important your development
        packages package properly by including all the relevant files. The easiest way to do this
        is by using source control, checking in all your source files and installing the relevant
        setuptools plugin for your source control. e.g. for git do "easy_install setuptools-git".
        
        Tip: An excellent tool for this is `mkrelease <http://pypi.python.org/pypi/jarn.mkrelease>`_. Highly recommended!
        
        2. "uploadbuildout": The relevant .cfg files and any files/directories in the "include"
        parameter are synced to the remote server.
        
        3. "buildout": The uploaded production buildout is run on the remote server.
        
        If you continue to develop your application you can run ``hostout deploy`` each time
        and it will only upload the eggs that have changed and buildout will only reinstall
        changed parts of the buildout.
        
        In our example above deployment would look something like this ::
        
        $> bin/hostout host1 deploy
        running clean
        ...
        creating '...example-0.0.0dev_....egg' and adding '...' to it
        ...
        Hostout: Running command 'predeploy' from 'collective.hostout'
        ...
        Hostout: Running command 'uploadeggs' from 'collective.hostout'
        Hostout: Preparing eggs for transport
        Hostout: Develop egg src/demo changed. Releasing with hash ...
        Hostout: Eggs to transport:
        demo = 0.0.0dev-...
        Hostout: Wrote versions to host1.cfg
        ...
        Hostout: Running command 'uploadbuildout' from 'collective.hostout'
        ...
        Hostout: Running command 'buildout' from 'collective/hostout'
        ...
        Hostout: Running command 'postdeploy' from 'collective/hostout'
        ...
        
        Now if you visit myhost.com you will see your web application shared with the world
        
        Other built-in Commands
        ***********************
        
        Hostout comes with a set of helpful commands. You can show this list by
        not specifying any command at all. The list of commands will vary depending
        on what fabfiles your hostout references. ::
        
        $> bin/hostout host1
        cmdline is: bin/hostout host1 [host2...] [all] cmd1 [cmd2...] [arg1 arg2...]
        Valid commands are:
        bootstrap        : Install python and users needed to run buildout
        buildout         : Run the buildout on the remote server
        deploy           : predeploy, uploadeggs, uploadbuildout, buildout and then postdeploy
        postdeploy       : Perform any final plugin tasks
        predeploy        : Install buildout and its dependencies if needed. Hookpoint for plugins
        setupusers       : create buildout and the effective user and allow hostout access
        setowners        : Ensure ownership and permissions are correct on buildout and cache
        run              : Execute cmd on remote as login user
        sudo             : Execute cmd on remote as root user
        uploadbuildout   : Upload buildout pinned to local picked versions + uploaded eggs
        uploadeggs       : Any develop eggs are released as eggs and uploaded to the server
        
        
        The run command is helpful to run quick remote commands as the buildout user on the remote host ::
        
        $> bin/hostout host1 run pwd
        Hostout: Running command 'run' from collective.hostout
        Logging into the following hosts as root:
        127.0.0.1
        [127.0.0.1] run: sh -c "cd /var/host1 && pwd"
        [127.0.0.1] out: ...
        Done.
        
        We can also use our login user and password to run quick sudo commands ::
        
        $> bin/hostout host1 sudo cat /etc/hosts
        Hostout: Running command 'sudo' from collective.hostout
        Logging into the following hosts as root:
        127.0.0.1
        [127.0.0.1] run: sh -c "cd /var/host1 && cat/etc/hosts"
        [127.0.0.1] out: ...
        Done.
        
        
        Detailed Hostout Options
        ************************
        
        Basic Options
        -------------
        
        host
        the IP or hostname of the host to deploy to. by default it will connect to port 22 using ssh.
        You can override the port by using hostname:port
        
        user
        The user which hostout will attempt to login to your host as. Will read a users ssh config to get a default.
        
        password
        The password for the login user. If not given then hostout will ask each time.
        
        identity-file
        A public key for the login user.
        
        extends
        Specifies another part which contains defaults for this hostout
        
        fabfiles
        Path to fabric files that contain commands which can then be called from the hostout
        script. Commands can access hostout options via hostout.options from the fabric environment.
        
        
        Deploy options
        --------------
        
        buildout
        The configuration file you which to build on the remote host. Note this doesn't have
        to be the same .cfg as the hostout section is in but the versions of the eggs will be determined
        from the buildout with the hostout section in. Defaults to buildout.cfg
        
        
        path
        The absolute path on the remote host where the buildout will be created.
        Defaults to ~${hostout:effective-user}/buildout
        
        pre-commands
        A series of shell commands executed as root before the buildout is run. You can use this
        to shut down your application. If these commands fail they will be ignored.
        
        post-commands
        A series of shell commands executed as root after the buildout is run. You can use this
        to startup your application. If these commands fail they will be ignored.
        
        sudo-parts
        Buildout parts which will be installed after the main buildout has been run. These will be run
        as root.
        
        parts
        Runs the buildout with a parts value equal to this
        
        include
        Additional configuration files or directories needed to run this buildout
        
        buildout-cache
        If you want to override the default location for the buildout-cache on the host
        
        python-version
        The version of python to install during bootstrapping. Defaults to version
        used in the local buildout.
        
        hostos
        Over which platform specific bootstrap_python command is called. For instance
        if hostos=redhat, bootstrap_python_redhat will be called to use "yum" to
        install python and other developer tools. This paramter is also used in
        hostout.cloud_ to pick which VM to create.
        
        
        Users and logins
        ----------------
        
        The bootstrap_users command is called as part of the bootstrap process which is called if no buildout has
        already been bootstraped on the remote server. This command will login using "user"
        (the user should have sudo rights) and create two additional users and a group which joins them.
        
        effective-user
        This user will own the buildouts var files. This allows the application to write to database files
        in the var directory but not be allowed to write to any other part of the buildout code.
        
        buildout-user
        The user which will own the buildout files. During bootstrap this user will be created and be given a ssh key
        such that hostout can login and run buildout using this account.
        
        buildout-group
        A group which will own the buildout files including the var files. This group is created if needed in the bootstrap_users
        command.
        
        In addition the private key will be read from the location "identity_file" and be used to create
        a password-less login for the "buildout-user" account by copying the public key into the "authorized_keys"
        file of the buildout_user account. If no file exists for "identity_file" a DSA private key is created for you
        in the file "${hostname}_key" in the buildout directory.
        During a normal deployment all steps are run as the buildout-user so there is no need to use the "user" account
        and therefore supply a password. The exception to this is if you specify "pre-deploy", "post-deploy" or "sudo-parts" steps
        or have to bootstrap the server. These require the use of the sudo-capable "user" account.
        If you'd like to share the ability to deploy your application with others, one way to do this is to simply
        checkin the private key file specified by "identity_file" along with your buildout. If you do share deployment,
        remember to pin your eggs in your buildout so the result is consistent no matter where  it is deployed from. One trick
        you can use to achieve this is to add "hostoutversions.cfg" to the "extends" of your buildout and commit
        "hostoutversions.cfg" to your source control as well.
        
        
        
        Integrating Fabric into buildout
        ********************************
        
        Hostout uses fabric files. Fabric is an easy way to write python that
        calls commands on a host over ssh.
        
        
        Here is a basic fabfile which will echo two variables on the remote server.
        
        
        >>> write('fabfile.py',"""
        ...
        ... from fabric import api
        ... from fabric.api import run
        ...
        ... def echo(cmdline1):
        ...    option1 = api.env.option1
        ...    run("echo '%s %s'" % (option1, cmdline1) )
        ...
        ... """)
        
        Using hostout we can predefine some of the fabric scripts parameters
        as well as install the fabric runner. Each hostout part in your buildout.cfg
        represents a connection to a server at a given path.
        
        >>> write('buildout.cfg',
        ... """
        ... [buildout]
        ... parts = host1
        ...
        ... [host1]
        ... recipe = collective.hostout
        ... host = 127.0.0.1:10022
        ... fabfiles = fabfile.py
        ... option1 = buildout
        ... user = root
        ... password = root
        ... path = /var/host1
        ...
        ... """ )
        
        If you don't include your password you will be prompted for it later.
        
        When we run buildout a special fabric runner will be installed called bin/hostout
        
        >>> print system('bin/buildout -N')
        Installing host1.
        Generated script '/sample-buildout/bin/hostout'.
        
        
        >>> print system('bin/hostout')
        cmdline is: bin/hostout host1 [host2...] [all] cmd1 [cmd2...] [arg1 arg2...]
        Valid hosts are: host1
        
        We can run our fabfile by providing the
        
        - host which refers to the part name in buildout.cfg.
        
        - command which refers to the method name in the fabfile
        
        - any other options we want to pass to the command
        
        Note: We can run multiple commands on one or more hosts using a single commandline.
        
        In our example
        
        >>> print system('bin/hostout host1 echo "is cool"')
        Hostout: Running command 'echo' from 'fabfile.py'
        Logging into the following hosts as root:
        127.0.0.1
        [127.0.0.1] run: echo 'cd /var/host1 && buildout is cool'
        [127.0.0.1] out: ...
        Done.
        
        Note that we combined information from our buildout with
        commandline paramaters to determine the exact command sent
        to our server.
        
        Making a hostout plugin
        -----------------------
        
        It can be very helpful to package up our fabfiles for others to use.
        
        Hostout Plugins are eggs with three parts :-
        
        1. Fabric script
        
        2. A zc.buildout recipe to initialise the parameters of the fabric file commands
        
        3. Entry points for both the recipe and the fabric scripts
        
        >>>    entry_points = {'zc.buildout': ['default = hostout.myplugin:Recipe',],
        ...                    'fabric': ['fabfile = hostout.myplugin.fabfile']
        ...                    },
        
        Once packaged and released others can add your plugin to their hostout e.g.
        
        >>> write('buildout.cfg',
        ... """
        ... [buildout]
        ... parts = host1
        ...
        ... [host1]
        ... recipe = collective.hostout
        ... extends = hostout.myplugin
        ... param1 = blah
        ... """ )
        
        >>> print system('bin/buildout')
        
        >>> print system('bin/hostout host1')
        cmdline is: bin/hostout host1 [host2...] [all] cmd1 [cmd2...] [arg1 arg2...]
        Valid commands are:
        ...
        mycommand        : example of command from hostout.myplugin
        
        
        #TODO Example of echo plugin
        
        
        Using fabric plugins
        --------------------
        
        You use commands others have made via the extends option.
        Name a buildout recipe egg in the extends option and buildout will download
        and merge any fabfiles and other configuration options from that recipe into
        your current hostout configuration.  The following are examples of built-in
        plugins.  Others are available on pypi.
        
        hostout.cloud_
        Will create VM instances automatically for you on many popular hosting services such
        as Amazon, Rackspace and Slicehost
        
        hostout.supervisor_
        Will stop a supervisor before buildout is run and restart it afterwards. It provides
        some short commands to quickly manage your applications from your hostout
        commandline
        
        
        .. _hostout.cloud: http://pypi.python.org/pypi/hostout.cloud
        .. _hostout.supervisor: http://pypi.python.org/pypi/hostout.supervisor
        
        
        
        
        Why hostout
        ***********
        
        Managing multiple environments can be a real pain and a barrier to development.
        Hostout puts all of the settings for all of your environments in an easy-to-manage format.
        
        Compared to
        
        SilverLining
        Hostout allows you to deploy many different kinds of applications instead of just wsgi based
        python apps. Buildout lets you define the installation of alomost any kind of application.
        
        Puppet
        TODO
        
        mr.awesome
        TODO
        
        Fabric
        TODO
        
        Egg Proxies
        TODO
        
        
        
        Using hostout with a python2.4 buildout
        ***************************************
        
        Hostout itself requires python2.6. However it is possible to use hostout with
        a buildout that requires python 2.4 by using buildout's support for different
        python interpretters.
        
        >>> write('buildout.cfg',
        ... """
        ... [buildout]
        ... parts = host1
        ...
        ... [host1]
        ... recipe = collective.hostout
        ... host = 127.0.0.1:10022
        ... python = python26
        ...
        ... [python26]
        ... executalble = /path/to/your/python2.6/binary
        ...
        ... """ )
        
        or alternatively if you don't want to use your local python you can get buildoit to
        build it for you.
        
        
        >>> write('buildout.cfg',
        ... """
        ... [buildout]
        ... parts = host1
        ...
        ... [host1]
        ... recipe = collective.hostout
        ... host = 127.0.0.1:10022
        ... python = python26
        ...
        ... [python26]
        ... recipe = zc.recipe.cmmi
        ... url = http://www.python.org/ftp/python/2.6.1/Python-2.6.1.tgz
        ... executable = ${buildout:directory}/parts/python/bin/python2.6
        ... extra_options=
        ...    --enable-unicode=ucs4
        ...    --with-threads
        ...    --with-readline
        ...
        ... """ )
        
        
        
        Credits
        *******
        
        Dylan Jay ( software at pretaweb_ dot com )
        
        
        .. _recipe: http://pypi.python.org/pypi/zc.buildout#recipes
        .. _fabric: http://fabfile.org
        .. _Fabric: http://fabfile.org
        .. _buildout: http://www.buildout.org
        .. _pretaweb: http://www.pretaweb.com
        
        
        
        
        Release History
        ***************
        
        
        1.0a4 (2010-01-06)
        ----------
        - using decorators for setting user used for command
        - cleaned up user adding code. renamed to bootstrap_users
        - BUG: two hostout parts with different extends= values won't create two separate command stacks
        but instead one combined one.
        - BUG: extends was global to all hostouts [Fulvio Casali]
        - merged hostout.ubuntu and hostout.redhat back into core. These are now deprecated.
        - broke bootstrap step into bootstrap_users bootstrap_python bootstrap_buildout
        - will install python from source if no specific os bootstrap_python available.
        - change default buildout path to /var/buildout/PARTNAME
        - hellowsgi demo included in code make for simple tests
        - os specific bootstrap installs much less. only what's needed for basic buildout
        - move to the github collective.
        - new "shell" parameter to specify which shell you want commands executed as.
        - improvements to setowners including suid for directories.
        - changes to make more freebsd friendly (Tom Lazar)
        - setowners on every deployment
        
        1.0a3 (??)
        ----------
        - now using entrypoints for fabfiles
        
        1.0a2 (2010-05-27)
        ------------------
        - plugins can now call commands in other plugins
        - use buildout-user and buildout-group to increase security of hosted code
        - added setowners command to reset permissions
        - added setaccess command to setup passwordless access
        - moved hostout.ubuntu and hostout.supervisor to seperate plugins
        - introduced initcommand to set user to particular plugin
        - identify-file default to hostname_key so unique to host
        - usign sdist zips instead of eggs to make cross platform
        - python2.6 compatibility fixes
        
        1.0a1 (2010-02-14)
        ------------------
        - switched to python2.6 and fabric0.9
        
        
        0.9.4 (2009-01-31)
        ------------------
        - commandline changed to allow multsite and arguments
        - mr.developer plugin
        - supervisor plugin
        - documentation clearer and more details
        - ssh tests now working
        - Nice listing of available commands
        
        0.9.3
        -----
        - Fixed how extends works
        - fabric commands have hostout in environment rather than argument
        - new command to reset permissions
        - fab_hosts etc is set for all commands
        - supervisor recipe now uses fabfile
        
        0.9.2 (2009-09-26)
        ------------------
        - can extend from recipe directly instead of a buildout part
        - supervisor plugin recipe
        - fixed version pinning for recipes
        
        
        0.9.1 (2009-09-17)
        ------------------
        - fix bug in lowering case of package names in versions
        
        
        0.9 (2009-09-15)
        ----------------
        - new commandline to allow deploying to multiple hosts at once
        - ability to take defaults from another part using 'extends' option
        - fabfiles option to create your own hostout commands
        - moved hostout cfg generation to deployment time
        - egg releases hash of contents as version numbers and won't be built or uploaded unless changed
        - new parts option to only install specified parts
        - many option names simplified (backwards compatibility maintained)
        - pre-commmands now runs before buildout initialisation
        - buildout now run as 'effective-user' rather than root. pre and post commands still run as root
        - eggs released directly to the download cache rather than a seperate dist directory
        - version recognition only done once for all hostout parts
        
        
        0.1.3 (2009-05-06)
        ------------------
        
        - Fixed getting fabfile as a resource when packaged as an egg
        - Allow for specifying extra configuration not in the buildout files
        
        0.1.2 (2009-04-24)
        ------------------
        
        Initial release. Basic uploading of eggs and running of remote buildout.
        
        
Keywords: buildout,fabric,deploy,deployment,server,plone,django,host,hosting
Platform: UNKNOWN
Classifier: Framework :: Buildout
Classifier: Development Status :: 3 - Alpha
Classifier: Topic :: Utilities
