Here are my slides from today’s talk at LinuxCon on “Open Source Toolchains for Cloud Computing“. The talk focused on what open source tools could be used to create automation for managing cloud computing. I really enjoyed giving the talk and the discussion afterward, it seems that the reality among attendees was that they did very little to automate management but all had a great interest in doing so. I also enjoyed hearing from a Big 4 systems management user who said clearly that their multi-million dollar system didn’t appear to have the same capabilities as the free software solutions I mentioned.

Technorati Tags: agile IT, automation, cloud computing, devops, Open Source

In celebration of System Administrator Appreciation Day I published a story at Linux.com called Open Source Toolchains for Systems Administrators, here is an excerpt:

“Software developers are very familiar with toolchains, series of programs where the output of one program forms the input for the next. A free software example would be using the GNU Emacs editor, the GNU bin-utils and the GNU Compiler Collection (GCC) to write a program. Software developers frequently create programs and subroutines that are used in other programs rather than recoding the same process over and over again.

Just as software developers have different task-specific tools to make up software tool chains (e.g. editors, compilers, build scripts), systems administrators can use tool chains made up of tools used to automate management functions and maintenance of Linux servers.”

I think it’s a good description of how to automate Linux server administration, read it and let me know.

Linux.com: Open Source Toolchains for Systems Administrators

Technorati Tags: automation, Cfengine, Chef, Linux, Monitoring, opensource, Opscode, provisioning, Puppet, Systems Management, Zabbix, Zenoss

There’s a real explosion of cloud platforms and management tools, it seems you can’t swing a dead cat without hitting one these days. In the commercial proprietary solutions space you have – CA’s 3Terra AppLogic, Enomaly, Nimbula, RightScale. In open source there are Eucalyptus, Cloud.com, Open Nebula and Ubuntu Enterprise Cloud. There are a bunch more that I failed to mention. It makes you wonder do we really need another one? How much different can they be? I am not sure but the newest one appears to be rather significant.

Today Rackspace has thrown their hat in the ring with their new OpenStack initiative in collaboration with NASA — as in rocket scientists, smartest guys in the world. Unlike Amazon’s EC2 which preaches open APIs, Rackspace is working to develop an open source platform that compliments their hosted cloud offering. They also have a strong open source partner in NASA who has been working on their own cloud computing platform, NASA Nebula. NASA Nebula will now become the cornerstone for the OpenStack initiative.

The goal of OpenStack is to allow any organization to create and offer cloud computing capabilities using open source software running on standard hardware. The project boasts both a compute and storage component. OpenStack Compute is software for automatically creating and managing large groups of virtual private servers and is available as a developer’s preview with a release target of October. OpenStack Storage is software for creating redundant, scalable object storage using clusters of commodity servers to store terabytes or even petabytes of data. Also available as a developer preview the OpenStack Storage project expects to release a production ready version in mid-September.

Adding the Rackspace hosting model to a strong open source project makes this approach to cloud computing especially interesting. Giving private cloud users a logical migration path to public cloud use. The question effect will this initiative have to truly drive open cloud computing standards.

OpenStack, A Foundation for Hybrid Clouds?

This initiative while founded on open source is not necessarily the cure for lock-in but it does go much farther than anyone else offering a fully accessible reference architecture available as open source. The closest comparison I see is Eucalyptus that is mimicks the Amazon EC2 cloud compute architecture (though not Amazon S3), though Amazon and Eucalyptus don’t seem to share a commonly agreed upon road map but rather a leader-follower relationship.

OpenStack’s formula is more coordinated and with a respectable user to champion it, NASA. The U.S space agency has one of the most compelling publicly documented private cloud computing stories.  NASA has gone so far as to package their solution in small footprint shipping containers to distribute among NASA research centers. These portable data centers are a model that many organizations looking to build private clouds are watching with interest.

I like that private clouds built on the OpenStack reference architecture should be fully compatible with Rackspace hosting services. Giving users the choice to run their own cloud or host or adopt a hybrid model. It’s not unlike open source adoption models were users download a free software version that has compatibility with a commercially supported version.  Plus this is not Rackspace’s only foray into open source distributed computing as they support the Apache-hosted Cassandra project, a highly scalable distributed database, and have been showing their support at numerous cloud and open source events.

The OpenStack Web Interface

The strong message accompanying the launch is one of open standards and prevention of cloud lock-in. Lew Moorman, President, Cloud and CSO at Rackspace states this clearly that OpenStack wants to prevent vendor lock-in:

“We are founding the OpenStack initiative to help drive industry standards, prevent vendor lock-in and generally increase the velocity of innovation in cloud technologies.”

This is not unlike VMware who echoed that sentiment with an announcement this spring to collaborate with Google AppEngine.  VMware’s CTO Steve Herrod stated that they too were committed to open standards and preventing login:

“Our shared vision is to make it easy to build, run, and manage applications for the cloud, and to do so in a way that makes the applications portable across clouds. The rich applications should be able to run in an enterprise’s private cloud, on Google’s AppEngine, or on other public clouds committed to similar openness.”

So with all this openness and commitment to open standards is the ability to move from cloud to cloud seamlessly just around the corner? This remains to be seen it but the initiatives all seem to be well-intentioned and moving in the right direction.

Does Open Source Prevent Cloud Lock-in?

I don’t know whether Rackspace’s OpenStack will truly prevent cloud lock-in but it does seem to be well-intentioned. Though I  believe the following things need to happen to insure cloud lock-in doesn’t become a rampant problem:

• Virtualization Portability – At a very simple level users need to be able to move from virtualization technologies including those hosted in the cloud need to be able to migrate seamlessly, that includes VMs running in VMware , Xen, HyperV and KVM. Then once in the cloud they need to be able to move across clouds both public and private unencumbered — Amazon, Rackspace, Eucalyptus, Ubuntu Enterprise Cloud and others. Adoption of a widespread virtualization standard like Open Virtualization Format (OVF) could help (OpenStack does seem to already support OVF, a good sign).
• Data Portability – Just as compute environments move so should data, but not only move but be accessible across network and cloud infrastructures with high fidelity.  Security of that data goes without saying but adds another layer of complexity.
• Cross Environment Tools – Finally, the tools to managing these environments need to manage both cloud and legacy architectures to insure that  the management of these new computing paradigms don’t make things even more complicated.

I hope OpenStack helps drive this vision. However to deliver on the true vision of true portability across cloud platforms other cloud providers and vendors other than Rackspace will have to participate.

Related Articles

• Rackspace’s Open Stack Makes Cloud Storage Personalized (fastcompany.com)
• NASA gives OpenStack instant credibility (zdnet.com)
• RackSpace’s OpenStack Targets Cloud Lock-in (pcworld.com)
• Rackspace Launches OpenStack – Open Source Cloud Computing Software (openstack.org)
• Robert Scoble: Rackspace ends cloud lock-in (scobleizer.com)
• OpenStack: Rackspace and NASA Nebula Join Forces for Open Cloud Ecosystem (readwriteweb.com)
• OpenStack: An Open Source Cloud Project Emerges (gigaom.com)
• OpenStack – an open source cloud platform (redmonk.com)
• Rackspace Launches OpenStack To Open Source The Cloud (laughingsquid.com)
• Rackspace, NASA launch OpenStack: Can it prevent cloud lock-in? (zdnet.com)
• Rackspace and The OpenStack Project (thetechscoop.net)
• NASA and Rackspace open source cloud fluffer (go.theregister.com)
• Announcing Open Stack – Open Cloud Computing Standards (blog.zenoss.com)
• Opening the Rackspace Cloud (rackspacecloud.com)

Technorati Tags: cloud computing, data, lockin, NASA, nebula, Open Source, OpenStack Compute, OpenStack Storage, ovf, platform, portability, Rackspace, VMware

For the last few years I have had an interest in configuration management of IT infrastructure. While by no means an expert I have a considerable amount of experience with the problems associated with mass server configuration and have come to believe it also one of the most under-served disciplines in systems management.

In a previous life I had an operations role maintaining primarily Linux servers and other open source infrastructure. In 2006 I worked on launching the open source NetDirector project, a graphical tool for configuring open source infrastructure like Apache, Samba, LDAP and NFS servers. During that time the challenges of maintaining server configurations started to really come to light for me.

The Challenges of Server Configuration

A large part of configuring infrastructure is repetitive and time consuming. Many sysadmins rely on their own scripts to help manage the process but it’s still a one-off for each administrator and their individual knowledge seldom if ever gets institutionalized throughout the entire organization.

Having a framework for maintaining configuration data is important. Configuration scripts are often authored as shell scripts or Perl or even Python but are seldom maintained or used beyond the original author. Some technologies use plugins that could be used to abstract configuration variables, users and systems then shared among users of the technology but plugins are often specific to a given technology and non-transferrable.

Despite the breadth of this problem there are relatively few solutions that can easily be consumed by medium-sized enterprises outside of large management suites available from the Big Four (HP, IBM, CA and BMC). The independent software vendors have all been consumed by bigger less focused organizations including Opsware (acquired by HP), Bladelogic (acquired by BMC) and Configuresoft (acquired by by VMware). In my opinion there is no real leader in this space.

In a conversation with Opscode CEO, Jesse Robbins, he shared his experience maintaining availability for web properties at Amazon.com. As a top ops guy and “master of disaster” at Amazon.com he had no access to these tools, they were simply sold in a way that was inconsistent with the way he evaluated and consumed products and services.  Opscode, a relatively new company develops the open source Chef project, which automates IT management via a client-server platform.

Opscode’s approach to server configuration challenges is to use recipes written in Ruby, the chef domain specific language(DSL). Then these cookbooks can be executed securely by the Chef client-server architecture and finally Chef is available as open source software to download, use and redistribute. In a nutshell Opscode met the following criteria is what interested me about their technology which is relatively easy to use, share configuration recipes and consume.

Chef, The Open Source Project

Chef is a systems integration framework released under the Apache License Version 2.0.  Chef, can manage servers by writing code in Ruby stored in configuration recipes called  cookbooks. Chef can integrate with existing infrastructure like LDAP via libraries using arbitrary Ruby code, either to extend Chef’s or to implement custom classes. Users can also configure applications that have dependencies on other parts of the infrastructure like databases and discern that information via the Chef server. However, I like Robbins’ description of Chef — sysadmin robot performing configuration tasks automatically and much more quickly than a single admin could ever hope to.

Though Chef was only released on January 15^th , 2009 it has gotten rapid adoption and gained a large number of contributors. According to the Opscode wiki there are 157 approved contributors to Opscode projects and well over 20 companies. Beyond that the #chef IRC channel is typically attended by over 100 users and Opscode staff, signs of a healthy, growing open source community.

Opscode, The Platform

The Opscode Platform is the commercial offering from Opscode Inc. It is a centrally managed data store hosted by Opscode into which servers publish data such as IP addresses, loaded kernel modules, OS versions and more delivered as infrastructrue-as-a-service (IaaS). This data on the Opscode Platform can be accessed and becomes useful in the following ways:

• Search-based Automation: All the data collected by the Opscode platform is indexed and searchable. Users can dynamically query this data from within Chef recipes to configure services that require complex configuration.
• Role-based Access Control: The data index is has an access control system enabling administrators to centrally manage the level of infrastructure access.
• Portability: The data stored on the Opscode Platform serves as a virtual blueprint of a given infrastructure, making it much easier to create perfect clones of a production environment.

The Opscode Platform is in a free beta release for the next 60 days. After the trial period, participants can manage up to 20 nodes on the Platform for $50 per month and $5 per month for each additional node. Pricing and availability information is available on their website.

Opscode, The Company

Opscode was founded in 2008 by Jesse Robbins and Adam Jacob, both experienced web operations leaders. Since then they have recruited a top notch team including cloud and systems management expert John Willis, Adrian Cole leader of the jclouds project and Christopher Brown a Founding Member, Architect, and Lead Developer for Amazon.com’s Elastic Compute Cloud (EC2).

Opscode today announced that they closed an $11 million Series B round of funding. The round was led by Battery Ventures (who also was an investor in BladLogic before it was acquired by BMC)  and includes a follow-on investment from Draper Fisher Jurvetson(DFJ) whose other open source investments include SugarCRM and Fonality. DFJ led Opscode’s Series A round of funding of $2.5 million, bringing the total amount raised for the company to $13.5 million, a sizable amount of capital to bring this technology to market.

Opscode is also seeing good adoption of Chef, not only do they have a few thousand active users on their wiki plus chef is currently in production at numerous top websites, including 37Signals, Etsy, IGN Entertainment, Scribd, and Wikia. Not only are web jockeys using chef but other large infrastructure providers are contributing to the project. Engine Yard, Rackspace, RightScale and the Springsource division of VMware have signed on to contribute to the project. They are even being very public about it as seen in this endorsement:

“We are excited about the open source contributions the Springsource Division of VMware has made to Opscode Chef.” said Javier Soltero, CTO of Springsource Management Products at VMware. “Chef is an important tool for automating infrastructure management and we look forward to its continued growth and success.”

Making Sysadmins into Superheroes

Opscode Chef is a hugely powerful tool that can greatly amplify the knowledge an effectiveness of systems administrators by automating a significant number of their maintenance tasks, improving their productivity and allowing them to focus on higher value tasks. Not only does Chef provide a framework for building systems but repairing them, keeping availability high and time to resolution low. This gives IT professionals a lot of leverage in getting their tasks done, allowing them to solve a problem once and then automate the process going forward. In other words Chef can turn systems administrators into super heroes by vastly improving their productivity and overall quality fo service.

The need that Opscode addresses can be filled to some degree through other software. Cfengine, Puppet and bcfg2 are all open source software solutions that address server configuration needs and have been around for some time. As mentioned above there is also large management suites that handle the same problems though they are expensive and have their own limitations. What is unique about Opscode approach is that they offer a robust, fully featured software platform as open source and a commercial offering that has full compatibility with the open source project.

This is somewhat unique as many commercial open source projects have a specific feature set that are only available to their enterprise customers. In a conversation with Opscode VP of Service, John M. Willis we discussed those users who are not interested in the Opscode platform but still want commercial support. He said that Opscode is building a select high-quality partner network that can handle these requests. Most recently Opscode has announced a partnership with DTO Solutions who employ members of the Control Tier project and is a big proponent of the DevOps approach to infrastructure management. Other partners will soon be onboard as well.

I am very much a fan of Opscode and there approach though my description probably doesn’t do it justice. Theoretically, a systems administrator who successfully implements automation tools such as Opscode can improve not only their productivity but the uptime of servers by improving speed to resolution for outages.  Opscode has a great opportunity thanks to a talented team, a novel go-to-market plan and a real need for these types of tools among IT professionals.

Related Articles

• The Origins of Amazon’s Cloud Computing (GigaOm.com)
• Chef Wiki (opscode.com)
• Understanding cloud and devops–part 1 (news.cnet.com)
• 3 Companies That Tackle Complexity in the Cloud (readwriteweb.com)
• Opscode Unveils Funding, Hosted Platform (datacenterknowledge.com)
• Opscode Gets $11M to Take on IBM and HP Management Software (gigaom.com)

Technorati Tags: Cfengine, Chef, Configuration Management, Iaas, Open Source, Opscode, PaaS, Puppet, Systems Management

I saw this presentation linked from my new co-worker  Josh’s blog, it’s Simon Sinek’s TED talk on How Great Leaders Inspire Action. He uses Apple as an example of a success and Tivo as an example of a failure. Sinek asserts that leading companies lead because of the why they do what they do not the what or the how. His golden circle theory says that most companies start by talking about the what they do, followed by how they do it and then talk about the why. Great companies and leaders flip that using the example of Apple’s why, “Think Different” and Martin Luther King’s “I have a dream” not “I have a plan”.

I have always believed that the why is most important part and have been recently struck by Tony Hseih’s, CEO of Zappos, assertion that delivering happiness (the why of Zappos) was the catalyst for their success and their eventual $1.2 billion dollar acquisition by Amazon. It’s what separated them from the thousands of other online retailers.

I really like Sinek’s talk and have listened to it many times over the last week.  It also reaffirms what I believe about open source– the reason open source software development works is because people believe in the way that it’s developed. Agree or not I think the talk is very good and worth the listen.

It was a lot of fun to be a guest on Twit.tv’s FLOSS Weekly with Randal Schwartz and Simon Phipps this week. My partner in crime, Zenoss Community Manager Matt Ray, and I gave them the lowdown on Zenoss and had a great time talking to the two excellent co-hosts.

If you are podcast listener and like open source I highly recommend FLOSS Weekly and all their episodes. You can listen to the Matt and talk about Zenoss on episode 124 from their site.

Interesting tidbit from the folks at Geeknet (They are the owners of  SourceForge, Slashdot and Ohloh). OSS projects hosted on SourceForge are increasingly becoming more platform agnostic. Since early 2005, the percentage of projects that are platform agnostic has increased from approximately 50 percent to nearly 70 percent as of Q3 2009.

Here are some of the numbers:

• Amount of OSS Windows compatible projects has grown from 72 percent in 2005 to 82 percent in 2009. (This is approximately 350,000 of the total 433,000 OSS projects)
• Windows is the only operating system compatible with the top 10 all-time most downloaded projects on SourceForge
• 23 of the top 25 projects run on Windows and 14 of them run only on Windows

Related Articles

• Geeknet at OSBC 2010 (port25.technet.com)

Technorati Tags: Geeknet, Microsoft Windows, Open Source, Operating system

Zenoss Core Dashboard

This week I am very excited because we released Zenoss Core 2.5.2 with a cool new feature,  Xen hypervisor monitoring.

I am very happy to see Zenoss put this out as open source software, because it’s the beginning of what could be a great piece of software. Now that this is out there for anyone to use I expect to get lots of feedback to extend and improve it.

This extension to Zenoss Core discovers guests on Xen para-virtualized hosts and provides monitoring of performance and availability via SSH. While the Xen hosts run on physical servers, the virtual guests are listed per host and linked back to any discovered instances on the network. Admins can quickly find the associated hosts and guests and monitor their Xen virtual infrastructure along side their entire physical, virtual, and cloud-based IT environment through a single interface.

I really think this is the beginning of a lot of virtualization and cloud monitoring capabilities from Zenoss Core. In recent weeks we got a couple neat extensions for virtualization and cloud monitoring including Amazon Web Services,  Google App Engine, libvirt (a virtualization library favored by Red Hat) and Ganglia integration.

Technorati Tags: Hypervisor, Monitoring, Xen, Zenoss Core

On February 6, the White House Open Government Initiative launched a government wide public participation opportunity.  As part of the Open Government Directive issued in early December, every major agency published an open government website.

In response the Open Source for America (OSFA) initiative has submitted their:  “Proposed Guidelines for Open Government Plans” to the Obama Administration for consideration. These plans are meant to encourage the government to embrace the open source ideals of participation, collaboration and transparency.

The proposal notes that there is, ” ….a tremendous opportunity to capture the innovation and ingenuity of government employees, who have the means to create their own tools to make themselves more effective, rather than waiting for a cumbersome and unresponsive procurement process. Open source software is, in fact, the most concrete form of participation available to the government’s constituents and its employees”

To get involved the Administration has invited public comments on each agency’s open government page through March 19, 2010. Open source advocates are encouraged to visit these agencies listed on the White House website and share their opinion.

Technorati Tags: government, Open Source, United States

Rube Goldberg would be proud of this machine.

Technorati Tags: cadbury egg, funny