Luis's Blog

Today's Grid forecast: "Partly Cloudy"

I've read a number of discussions recently debating whether cloud computing will supplant grid computing as the preferred deployment environment for complex computation needs. Notwithstanding the clear value that cloud computing environments provide, I firmly believe that for various reasons (architecture, security, bandwidth etc.), enterprises will not move all of their infrastructure to cloud environments any time soon. Rather, they're likely to create hybrid or "partly cloudy" environments in which some computations will remain local and others will be sent to the cloud.

In fact, we're working with a client at this very moment on creating such a hybrid environment. The client wants to retain some core applications and computation within their local environment, but wants to leverage Amazon's Elastic Compute Cloud (EC2) for their larger calculation needs. In this case, however, simply having the raw compute resources available to them isn't sufficient, as they need to manage and coordinate the calculation tasks that are submitted to the cloud. The EC2 APIs and utilities enable you to create, manage and interact with virtual machine images, but does not provide facilities for tying these images together into a coordinated computation platform.

This is where the Grid comes in. By deploying virtual machine images containing Grid nodes, the cloud becomes an extension of the local compute grid that can be activated and deactivated on an as-needed basis. This hybrid approach enables enterprises to retain tight control of the management, policy and security related components of the grid by running them on local infrastructure, and at the same time, lets the Grid farm out compute tasks to the cloud grid nodes.

The exciting part is that this scenario works today. Our client is building a partly cloudy deployment environment using GridServer and Amazon EC2, and these products work seamlessly right out of the box. The Amazon Machine Images (AMIs) are GridServer engine nodes. GridServer even enables these images to declare on which EC2 Instance Type the node is running so that you can create policies that select the appropriate instance type (i.e. size) for any given compute task. Once the AMIs are created, and the required NAT and security configurations are defined, the GridServer broker can assign tasks to any of the cloud images just as it would for local images.

Taking this a step further, by using GridServer extension mechanisms and the EC2 API, the grid can be configured to register, unregister, run and terminate AMIs automatically as needed. In doing so, the client will be able to let the grid decide when new AMIs are needed rather than preallocating EC2 instances (and paying for them!) in anticipation of need. This policy-based management of cloud instances is one of the key features that makes this partly cloudy approach exciting. It's hard to believe I'm promoting a partly cloudy forecast, especially given the weather in here in Boston lately. But in this case I say bring on the Clouds!

It's a data center, not a development center

Last week, I listened to Donna Scott of Gartner describe some interesting poll results from the Gartner 2007 Data Center conference. The point that struck me most was that the largest percentage of respondents said their application teams, rather than IT operations teams, are responsible for deploying their applications to the data center. I'm not really surprised, given the complexity of deploying .NET and Java applications, which are the technologies used by over 70% of the respondents. But I can't help but wonder how much development productivity and operations staff utilization is being lost in these cases. How can application teams maintain a steady deployment pipeline of business application functionality if developers must change hats to deploy new or updated applications, manage deployment environments and respond to deployment and configuration changes? How can the IT operations staff efficiently manage their data centers when they must rely on and wait for the development team to solve their operational issues?

I'm sure some enterprises would say this is just how IT works. I say it doesn't have to. Certainly these teams have to work well together. But try as they might, if the application platforms don't support automated deployment facilities, then the teams will continue to fuss over fragile scripts that are developed and managed by the development team and are often poorly understood by the IT ops team. Furthermore, beyond initial deployment, runtime factors such as user load, scheduled operations and business priorities will require further changes to the application deployment. .NET and Java are great application development platforms, but they are lacking when it comes to simple, automated management in production environments.

There are some excellent products on the market that address portions of this problem. Server provisioning products from the likes of BladeLogic or Opsware (HP) ease the process of getting applications and updates deployed to the data center. Configuration management tools from mValent or Symantec bring much needed repeatability and control to the configuration and release management processes. But server provisioning and configuration management don't simplify the deployment changes that are required to address runtime factors in real time. Instead, more fragile scripts are developed, more developers are brought in to manage them and more IT staff is stuck fighting fire drills when expected and unexpected system usage strains the statically deployed applications.

Clearly enterprises must free their development teams from managing application deployments if they are to bring true agility to data center operations. The next step would be to replace this manual intervention with the appropriate data center automation products. We certainly would recommend a product like FabricServer that dynamically manages applications based on policy and demand, but enterprises should make their choice based on their business, operations and staffing requirements. The faster these changes are made, the faster the applications teams can get back to focusing on development, and IT ops teams to managing a robust real time infrastructure.

New Year, New Releases

In the same way that I avoid making New Year's resolutions that I'm likely to break, I'm going to forgo the ever present 'predictions for 2008' entry and kick off the year (and this blog) with something I know is happening: our new product releases of FabricServer 2.5 and Studio.  You're going to be hearing plenty of information about these product releases in the coming months, so I won't even attempt to give them full marketing coverage in this entry. 

However, I would like to focus on just how easy it is to enable applications for the grid with Studio, our new development environment.  As many of you know, enabling applications for a grid environment isn't always straightforward.  If you're designing a new application, you may have an easier time because you're building in distribution, aggregation, and service-orientation from the ground up.  But what about those legacy applications and platforms or COTS software?  They've already been architected and may or may not have these principles as design centers.   It is in these cases that Studio really shines.

Studio is our Eclipse-based development environment for packaging applications for the GridServer and FabricServer environments.  The Studio toolset is great when building applications from the ground up, as its editors and tools work right alongside your familiar Eclipse development environment.  When deploying existing applications, Studio's packaging tools drastically simplify the whole process.  Studio introspects your application installation and builds a map of the files and directories it finds there.  Using this map, it packages your application so that it can be distributed and installed on any supported target OS just as if it were installed natively. Further, when it's time to update or patch your application, Studio uses the map it has already built to determine what needs to be updated and repackaged.   Update packaging is done once, and the grid ensures that the update is distributed everywhere. 

The excitement about Studio is certainly spreading.  Our services team is already using Studio to help customers deploy GridServer and FabricServer faster and more accurately than ever.   Our development teams are using it in "skunk works" projects to enable new technologies and platforms for the grid (more to come on this...).  I'm even investigating enabling my product and project management software for grid deployment.   Sure this is a small step for me compared to the major deployments of our customers, but that's one step closer to getting my infrastructure operating in real time.  Not quite a New Year's resolution, but certainly a great way to kick off the year.