If you are going to operate a cloud at scale then you most certainly need to use some ways to automate the configuration and other tasks like provisioning when you spin up cloud infrastructure. One of my favorite open source projects for doing this is Puppet sponsored by Puppetlabs.

Puppet is a client-server application that allows you define roles and configurations for infrastructure in puppet manifests then deploy those manifests across numerous servers simultaneously. In addition Puppet has another project, the Marionette Collective (MCollective), which allows you to automate common tasks and works in conjunction with puppet.

One of the best ways to get up to speed on using puppet is to attend a Puppet Camp. The next one will be hosted inAtlanta on February 3rd.

“Puppet Camp is a community oriented gathering of Puppet users and developers. You’ll have the opportunity to network with a diverse group of Puppet users, benefit from insightful lectures delivered by prominent community members, and be able to share experiences and discuss potential implementations of Puppet during our attendee generated breakout sessions.”

If you can’t attend this session there are many other Puppet Camps being held worldwide listed on the PuppetLabs website.

This is a presentation originally given at SCALE 9x and revised for the  Indiana Linuxfest 3/26/2011.

My slides from my Crash Course in Cloud Computing talk from SCALE 9x, it was my first talk on the subject and based on some feedback I think I am going to add some additional information. On the open source tools I think I am going to add RunDeck and MCollective for orchestration.

This Friday at SCaLE 9x in Los Angeles, CA there will be a special build an open source cloud day teaching users how to use technologies from Cloud.com, OpenStack, Opscode and Zenoss to deploy, configure, manage and monitor infrastructure-as-a-service using open source software.

Here’s an overview of the program:

“Build a Cloud Day” will be dedicated to teaching users how to build and manage a cloud computing environment using free and open source software. The program is designed to expose attendees to the concepts and best practices around deploying cloud computing infrastructure. Attendees should expect to learn how to deploy a cloud computing environment using Cloudstack CE. In addition Build a Cloud Day attendees will learn about tools( Opscode Chef and Zenoss Core) that can be used to handle the dynamics of a cloud computing environment including automatic provisioning and configuration and monitoring.

9:00 – 9:15: Welcome – Mark Hinkle, VP of Community, Cloud.com

We’ll kick off the day reviewing the program with attendees and gathering information so we can tailor the program to best address the needs and experience level of the audience.

9:15 – 11:15: Build Your Private Cloud – Cloud.com

Using CloudStack Community Edition (CE), free and open source cloud computing software to build a private cloud. During the training attendees will be instructed on how to install Cloudstack CE to manage virtual infrastructure in a private cloud computing configuration. At the conclusion of the Build a Private section users will have the knowledge needed to create a simple private cloud network.

Cloudstack CE is an open source Infrastructure-as-a-Service (IaaS) software platform available under the GPLv3 license, which enables users to build, manage and deploy compute cloud environments. The community edition is based on the latest, leading edge features and bits that the Cloud.com team of engineers are working on and is supported by both our engineers and our open source community.

11:15-11:45: Openstack – Rackspace

OpenStack Deployment and Integration Engineer Jordan Rinke will give a brief overview of the Openstack project.

OpenStack is a collection of open source technologies delivering a
massively scalable cloud operating system. OpenStack is currently
developing two interrelated projects: OpenStack Compute and OpenStack Object Storage.
OpenStack Compute is software to provision and manage large groups of
virtual private servers, and OpenStack Object Storage is software for
creating redundant, scalable object storage using clusters of commodity
servers to store terabytes or even petabytes of data. OpenStack is sponsored by Rackspace, a leader in web hosting and managed infrastructure.

11:45 – 12:45: Lunch – Catered

We will bring in a simple catered lunch (We’ll do our best to accomadate all dietary restrictions).

12:45 – 2:45: Automatic Configuration of Your Cloud – Opscode

This training section will cover deploying cloud infrastructure automatically using Opscode Chef. We will be cover installation basics of Chef clients and working with a Chef server. Other topics will include: creating Chef repositories, writing cookbooks and advanced uses of the command line utility Knife.

Chef is a systems integration framework, built to bring the benefits of automated configuration management to your entire infrastructure. With Chef you can:

• Manage your servers by writing code, not by running commands.
• Dynamically integrate your applications, databases and other infrastructure.
• Easily configure applications that require knowledge about your entire infrastructure (“What systems are running my application?”, “What is the current master database server?”)

3:00 – 5:00: Monitoring Your Cloud – Zenoss

A representative from the Zenoss Core project will provide training on how to monitor a cloud computing environment. The presentation will give an overview of the Zenoss Core management platform. Attendees will also learn how to install Zenoss Core, add devices and send alerts based on conditions within your cloud computing infrastructure.

Zenoss Core is an open source IT monitoring product that delivers the functionality to effectively manage the configuration, health, performance of networks, servers and applications through a single, integrated software package.

All devices in the environment can be modeled – networks, servers, software, and applications – as well as custom devices such as power supplies and temperature sensors. The model provides logical and physical grouping allowing you to map devices to business systems, locations, and responsible people. The model is populated by the auto-discovery process, supplemented by the web services API, XML import/export, or manual user input.Collections of monitoring templates are called ZenPacks, and more than 100 are available.

5:00 -5:10 Special Guest – Arista Networks

Cloud architectures have changed the design requirements for high-performance networking infrastructure and a 100% Linux based, fully extensible networking solution can bring your Cloud to the next level. Arista Networks gives a 10-minute overview of the EOS Extensible Operating Systems and reviews some of the tricks of the trade that can make deploying, managing and maintaining a best-in-class network infrastructure easier than ever before.

Technorati Tags: cloud computing, cloud.com, opensource, OpenStack, Opscode

In celebration of System Administrator Appreciation Day I published a story at Linux.com called Open Source Toolchains for Systems Administrators, here is an excerpt:

“Software developers are very familiar with toolchains, series of programs where the output of one program forms the input for the next. A free software example would be using the GNU Emacs editor, the GNU bin-utils and the GNU Compiler Collection (GCC) to write a program. Software developers frequently create programs and subroutines that are used in other programs rather than recoding the same process over and over again.

Just as software developers have different task-specific tools to make up software tool chains (e.g. editors, compilers, build scripts), systems administrators can use tool chains made up of tools used to automate management functions and maintenance of Linux servers.”

I think it’s a good description of how to automate Linux server administration, read it and let me know.

Linux.com: Open Source Toolchains for Systems Administrators

Technorati Tags: automation, Cfengine, Chef, Linux, Monitoring, opensource, Opscode, provisioning, Puppet, Systems Management, Zabbix, Zenoss

There’s a real explosion of cloud platforms and management tools, it seems you can’t swing a dead cat without hitting one these days. In the commercial proprietary solutions space you have – CA’s 3Terra AppLogic, Enomaly, Nimbula, RightScale. In open source there are Eucalyptus, Cloud.com, Open Nebula and Ubuntu Enterprise Cloud. There are a bunch more that I failed to mention. It makes you wonder do we really need another one? How much different can they be? I am not sure but the newest one appears to be rather significant.

Today Rackspace has thrown their hat in the ring with their new OpenStack initiative in collaboration with NASA — as in rocket scientists, smartest guys in the world. Unlike Amazon’s EC2 which preaches open APIs, Rackspace is working to develop an open source platform that compliments their hosted cloud offering. They also have a strong open source partner in NASA who has been working on their own cloud computing platform, NASA Nebula. NASA Nebula will now become the cornerstone for the OpenStack initiative.

The goal of OpenStack is to allow any organization to create and offer cloud computing capabilities using open source software running on standard hardware. The project boasts both a compute and storage component. OpenStack Compute is software for automatically creating and managing large groups of virtual private servers and is available as a developer’s preview with a release target of October. OpenStack Storage is software for creating redundant, scalable object storage using clusters of commodity servers to store terabytes or even petabytes of data. Also available as a developer preview the OpenStack Storage project expects to release a production ready version in mid-September.

Adding the Rackspace hosting model to a strong open source project makes this approach to cloud computing especially interesting. Giving private cloud users a logical migration path to public cloud use. The question effect will this initiative have to truly drive open cloud computing standards.

OpenStack, A Foundation for Hybrid Clouds?

This initiative while founded on open source is not necessarily the cure for lock-in but it does go much farther than anyone else offering a fully accessible reference architecture available as open source. The closest comparison I see is Eucalyptus that is mimicks the Amazon EC2 cloud compute architecture (though not Amazon S3), though Amazon and Eucalyptus don’t seem to share a commonly agreed upon road map but rather a leader-follower relationship.

OpenStack’s formula is more coordinated and with a respectable user to champion it, NASA. The U.S space agency has one of the most compelling publicly documented private cloud computing stories.  NASA has gone so far as to package their solution in small footprint shipping containers to distribute among NASA research centers. These portable data centers are a model that many organizations looking to build private clouds are watching with interest.

I like that private clouds built on the OpenStack reference architecture should be fully compatible with Rackspace hosting services. Giving users the choice to run their own cloud or host or adopt a hybrid model. It’s not unlike open source adoption models were users download a free software version that has compatibility with a commercially supported version.  Plus this is not Rackspace’s only foray into open source distributed computing as they support the Apache-hosted Cassandra project, a highly scalable distributed database, and have been showing their support at numerous cloud and open source events.

The OpenStack Web Interface

The strong message accompanying the launch is one of open standards and prevention of cloud lock-in. Lew Moorman, President, Cloud and CSO at Rackspace states this clearly that OpenStack wants to prevent vendor lock-in:

“We are founding the OpenStack initiative to help drive industry standards, prevent vendor lock-in and generally increase the velocity of innovation in cloud technologies.”

This is not unlike VMware who echoed that sentiment with an announcement this spring to collaborate with Google AppEngine.  VMware’s CTO Steve Herrod stated that they too were committed to open standards and preventing login:

“Our shared vision is to make it easy to build, run, and manage applications for the cloud, and to do so in a way that makes the applications portable across clouds. The rich applications should be able to run in an enterprise’s private cloud, on Google’s AppEngine, or on other public clouds committed to similar openness.”

So with all this openness and commitment to open standards is the ability to move from cloud to cloud seamlessly just around the corner? This remains to be seen it but the initiatives all seem to be well-intentioned and moving in the right direction.

Does Open Source Prevent Cloud Lock-in?

I don’t know whether Rackspace’s OpenStack will truly prevent cloud lock-in but it does seem to be well-intentioned. Though I  believe the following things need to happen to insure cloud lock-in doesn’t become a rampant problem:

• Virtualization Portability – At a very simple level users need to be able to move from virtualization technologies including those hosted in the cloud need to be able to migrate seamlessly, that includes VMs running in VMware , Xen, HyperV and KVM. Then once in the cloud they need to be able to move across clouds both public and private unencumbered — Amazon, Rackspace, Eucalyptus, Ubuntu Enterprise Cloud and others. Adoption of a widespread virtualization standard like Open Virtualization Format (OVF) could help (OpenStack does seem to already support OVF, a good sign).
• Data Portability – Just as compute environments move so should data, but not only move but be accessible across network and cloud infrastructures with high fidelity.  Security of that data goes without saying but adds another layer of complexity.
• Cross Environment Tools – Finally, the tools to managing these environments need to manage both cloud and legacy architectures to insure that  the management of these new computing paradigms don’t make things even more complicated.

I hope OpenStack helps drive this vision. However to deliver on the true vision of true portability across cloud platforms other cloud providers and vendors other than Rackspace will have to participate.

Related Articles

• Rackspace’s Open Stack Makes Cloud Storage Personalized (fastcompany.com)
• NASA gives OpenStack instant credibility (zdnet.com)
• RackSpace’s OpenStack Targets Cloud Lock-in (pcworld.com)
• Rackspace Launches OpenStack – Open Source Cloud Computing Software (openstack.org)
• Robert Scoble: Rackspace ends cloud lock-in (scobleizer.com)
• OpenStack: Rackspace and NASA Nebula Join Forces for Open Cloud Ecosystem (readwriteweb.com)
• OpenStack: An Open Source Cloud Project Emerges (gigaom.com)
• OpenStack – an open source cloud platform (redmonk.com)
• Rackspace Launches OpenStack To Open Source The Cloud (laughingsquid.com)
• Rackspace, NASA launch OpenStack: Can it prevent cloud lock-in? (zdnet.com)
• Rackspace and The OpenStack Project (thetechscoop.net)
• NASA and Rackspace open source cloud fluffer (go.theregister.com)
• Announcing Open Stack – Open Cloud Computing Standards (blog.zenoss.com)
• Opening the Rackspace Cloud (rackspacecloud.com)

Technorati Tags: cloud computing, data, lockin, NASA, nebula, Open Source, OpenStack Compute, OpenStack Storage, ovf, platform, portability, Rackspace, VMware

For the last few years I have had an interest in configuration management of IT infrastructure. While by no means an expert I have a considerable amount of experience with the problems associated with mass server configuration and have come to believe it also one of the most under-served disciplines in systems management.

In a previous life I had an operations role maintaining primarily Linux servers and other open source infrastructure. In 2006 I worked on launching the open source NetDirector project, a graphical tool for configuring open source infrastructure like Apache, Samba, LDAP and NFS servers. During that time the challenges of maintaining server configurations started to really come to light for me.

The Challenges of Server Configuration

A large part of configuring infrastructure is repetitive and time consuming. Many sysadmins rely on their own scripts to help manage the process but it’s still a one-off for each administrator and their individual knowledge seldom if ever gets institutionalized throughout the entire organization.

Having a framework for maintaining configuration data is important. Configuration scripts are often authored as shell scripts or Perl or even Python but are seldom maintained or used beyond the original author. Some technologies use plugins that could be used to abstract configuration variables, users and systems then shared among users of the technology but plugins are often specific to a given technology and non-transferrable.

Despite the breadth of this problem there are relatively few solutions that can easily be consumed by medium-sized enterprises outside of large management suites available from the Big Four (HP, IBM, CA and BMC). The independent software vendors have all been consumed by bigger less focused organizations including Opsware (acquired by HP), Bladelogic (acquired by BMC) and Configuresoft (acquired by by VMware). In my opinion there is no real leader in this space.

In a conversation with Opscode CEO, Jesse Robbins, he shared his experience maintaining availability for web properties at Amazon.com. As a top ops guy and “master of disaster” at Amazon.com he had no access to these tools, they were simply sold in a way that was inconsistent with the way he evaluated and consumed products and services.  Opscode, a relatively new company develops the open source Chef project, which automates IT management via a client-server platform.

Opscode’s approach to server configuration challenges is to use recipes written in Ruby, the chef domain specific language(DSL). Then these cookbooks can be executed securely by the Chef client-server architecture and finally Chef is available as open source software to download, use and redistribute. In a nutshell Opscode met the following criteria is what interested me about their technology which is relatively easy to use, share configuration recipes and consume.

Chef, The Open Source Project

Chef is a systems integration framework released under the Apache License Version 2.0.  Chef, can manage servers by writing code in Ruby stored in configuration recipes called  cookbooks. Chef can integrate with existing infrastructure like LDAP via libraries using arbitrary Ruby code, either to extend Chef’s or to implement custom classes. Users can also configure applications that have dependencies on other parts of the infrastructure like databases and discern that information via the Chef server. However, I like Robbins’ description of Chef — sysadmin robot performing configuration tasks automatically and much more quickly than a single admin could ever hope to.

Though Chef was only released on January 15^th , 2009 it has gotten rapid adoption and gained a large number of contributors. According to the Opscode wiki there are 157 approved contributors to Opscode projects and well over 20 companies. Beyond that the #chef IRC channel is typically attended by over 100 users and Opscode staff, signs of a healthy, growing open source community.

Opscode, The Platform

The Opscode Platform is the commercial offering from Opscode Inc. It is a centrally managed data store hosted by Opscode into which servers publish data such as IP addresses, loaded kernel modules, OS versions and more delivered as infrastructrue-as-a-service (IaaS). This data on the Opscode Platform can be accessed and becomes useful in the following ways:

• Search-based Automation: All the data collected by the Opscode platform is indexed and searchable. Users can dynamically query this data from within Chef recipes to configure services that require complex configuration.
• Role-based Access Control: The data index is has an access control system enabling administrators to centrally manage the level of infrastructure access.
• Portability: The data stored on the Opscode Platform serves as a virtual blueprint of a given infrastructure, making it much easier to create perfect clones of a production environment.

The Opscode Platform is in a free beta release for the next 60 days. After the trial period, participants can manage up to 20 nodes on the Platform for $50 per month and $5 per month for each additional node. Pricing and availability information is available on their website.

Opscode, The Company

Opscode was founded in 2008 by Jesse Robbins and Adam Jacob, both experienced web operations leaders. Since then they have recruited a top notch team including cloud and systems management expert John Willis, Adrian Cole leader of the jclouds project and Christopher Brown a Founding Member, Architect, and Lead Developer for Amazon.com’s Elastic Compute Cloud (EC2).

Opscode today announced that they closed an $11 million Series B round of funding. The round was led by Battery Ventures (who also was an investor in BladLogic before it was acquired by BMC)  and includes a follow-on investment from Draper Fisher Jurvetson(DFJ) whose other open source investments include SugarCRM and Fonality. DFJ led Opscode’s Series A round of funding of $2.5 million, bringing the total amount raised for the company to $13.5 million, a sizable amount of capital to bring this technology to market.

Opscode is also seeing good adoption of Chef, not only do they have a few thousand active users on their wiki plus chef is currently in production at numerous top websites, including 37Signals, Etsy, IGN Entertainment, Scribd, and Wikia. Not only are web jockeys using chef but other large infrastructure providers are contributing to the project. Engine Yard, Rackspace, RightScale and the Springsource division of VMware have signed on to contribute to the project. They are even being very public about it as seen in this endorsement:

“We are excited about the open source contributions the Springsource Division of VMware has made to Opscode Chef.” said Javier Soltero, CTO of Springsource Management Products at VMware. “Chef is an important tool for automating infrastructure management and we look forward to its continued growth and success.”

Making Sysadmins into Superheroes

Opscode Chef is a hugely powerful tool that can greatly amplify the knowledge an effectiveness of systems administrators by automating a significant number of their maintenance tasks, improving their productivity and allowing them to focus on higher value tasks. Not only does Chef provide a framework for building systems but repairing them, keeping availability high and time to resolution low. This gives IT professionals a lot of leverage in getting their tasks done, allowing them to solve a problem once and then automate the process going forward. In other words Chef can turn systems administrators into super heroes by vastly improving their productivity and overall quality fo service.

The need that Opscode addresses can be filled to some degree through other software. Cfengine, Puppet and bcfg2 are all open source software solutions that address server configuration needs and have been around for some time. As mentioned above there is also large management suites that handle the same problems though they are expensive and have their own limitations. What is unique about Opscode approach is that they offer a robust, fully featured software platform as open source and a commercial offering that has full compatibility with the open source project.

This is somewhat unique as many commercial open source projects have a specific feature set that are only available to their enterprise customers. In a conversation with Opscode VP of Service, John M. Willis we discussed those users who are not interested in the Opscode platform but still want commercial support. He said that Opscode is building a select high-quality partner network that can handle these requests. Most recently Opscode has announced a partnership with DTO Solutions who employ members of the Control Tier project and is a big proponent of the DevOps approach to infrastructure management. Other partners will soon be onboard as well.

I am very much a fan of Opscode and there approach though my description probably doesn’t do it justice. Theoretically, a systems administrator who successfully implements automation tools such as Opscode can improve not only their productivity but the uptime of servers by improving speed to resolution for outages.  Opscode has a great opportunity thanks to a talented team, a novel go-to-market plan and a real need for these types of tools among IT professionals.

Related Articles

• The Origins of Amazon’s Cloud Computing (GigaOm.com)
• Chef Wiki (opscode.com)
• Understanding cloud and devops–part 1 (news.cnet.com)
• 3 Companies That Tackle Complexity in the Cloud (readwriteweb.com)
• Opscode Unveils Funding, Hosted Platform (datacenterknowledge.com)
• Opscode Gets $11M to Take on IBM and HP Management Software (gigaom.com)

Technorati Tags: Cfengine, Chef, Configuration Management, Iaas, Open Source, Opscode, PaaS, Puppet, Systems Management

I saw this presentation linked from my new co-worker  Josh’s blog, it’s Simon Sinek’s TED talk on How Great Leaders Inspire Action. He uses Apple as an example of a success and Tivo as an example of a failure. Sinek asserts that leading companies lead because of the why they do what they do not the what or the how. His golden circle theory says that most companies start by talking about the what they do, followed by how they do it and then talk about the why. Great companies and leaders flip that using the example of Apple’s why, “Think Different” and Martin Luther King’s “I have a dream” not “I have a plan”.

I have always believed that the why is most important part and have been recently struck by Tony Hseih’s, CEO of Zappos, assertion that delivering happiness (the why of Zappos) was the catalyst for their success and their eventual $1.2 billion dollar acquisition by Amazon. It’s what separated them from the thousands of other online retailers.

I really like Sinek’s talk and have listened to it many times over the last week.  It also reaffirms what I believe about open source– the reason open source software development works is because people believe in the way that it’s developed. Agree or not I think the talk is very good and worth the listen.

It was a lot of fun to be a guest on Twit.tv’s FLOSS Weekly with Randal Schwartz and Simon Phipps this week. My partner in crime, Zenoss Community Manager Matt Ray, and I gave them the lowdown on Zenoss and had a great time talking to the two excellent co-hosts.

If you are podcast listener and like open source I highly recommend FLOSS Weekly and all their episodes. You can listen to the Matt and talk about Zenoss on episode 124 from their site.

Zenoss Core Dashboard

This week I am very excited because we released Zenoss Core 2.5.2 with a cool new feature,  Xen hypervisor monitoring.

I am very happy to see Zenoss put this out as open source software, because it’s the beginning of what could be a great piece of software. Now that this is out there for anyone to use I expect to get lots of feedback to extend and improve it.

This extension to Zenoss Core discovers guests on Xen para-virtualized hosts and provides monitoring of performance and availability via SSH. While the Xen hosts run on physical servers, the virtual guests are listed per host and linked back to any discovered instances on the network. Admins can quickly find the associated hosts and guests and monitor their Xen virtual infrastructure along side their entire physical, virtual, and cloud-based IT environment through a single interface.

I really think this is the beginning of a lot of virtualization and cloud monitoring capabilities from Zenoss Core. In recent weeks we got a couple neat extensions for virtualization and cloud monitoring including Amazon Web Services,  Google App Engine, libvirt (a virtualization library favored by Red Hat) and Ganglia integration.

Technorati Tags: Hypervisor, Monitoring, Xen, Zenoss Core