Something I saw or thought I saw

Last week we were on vacation in my favorite place in the universe, Austin, Texas. With three kids, you tend to get out of bed a bit earlier than you'd like. One early morning whilst coaxing my mind out of dreamland over a cup of joe, I found myself looking out the patio doors at a bird house.

The birdhouse is one of those iconic red "See Rock City" jobs. Anyway, I saw a small fledgling - some kind of finch I think - fall from the bird house to the earth below.

The bird was clearly dazed. Maybe even a bit sore. I watched the bird gather it's bird wits, dust itself off, and learn how to fly.


Food for thought.

POWER7 stats

I parsed and munged SpecJBB2005 data to get some interesting information about performance of various hardware platforms on JDK 1.6. A couple of things are interesting:

1) POWER7 is the mainstream performance champ. There are some oddball Nehalem contenders from Fujitsu and Cisco. In general though, the POWER 780 kicks some tail and the 750 isn't far behind. (There are no data from the 770.)

2) Performance data from one 780 running AIX is very different from another 780 running System i. This has to do with the quirky little JVM for System i, I think.

3) The top 16 performers on a BOPS/core basis are all running the IBM J9 JVM. JRockit doesn't show up until 17. Of the top 30, 28 use the J9 JVM. This tells me that IBM has really caught up with the BEA guys in terms of JDK performance.

Yay me!

My company was asked to pursue a new IBM certification focused on cloud computing topics. The new certification requires passing the 000-032 cert test. I passed on Sunday, 35 minutes after the test was made available to IMPACT attendees. (The test is not yet available on Prometric.)

That makes me the first person in the universe to have passed the exam. I am now a Certified Solution Adviser - Cloud Computing Architecture, v1. Yay.

The test covers a broad spectrum of cloud topics, from IBM's productized cloud offerings (Lotus Live, TPM, etc) as well as a bunch of content on SAAS, PAAS, and IAAS, cloud methodologies, and business benefits for public, private, and virtual private cloud topologies.

WXS In A Box?

The Zealot is out in Vegas for the annual IMPACT conference. One of the more interesting product announcements (other than WXS 7.1) is the DataPower XC10. It appears to be a product of the eXtreme Scale product dev team (thanks Joe Lea!).

Short version: the XC10 provides a pure-play implementation of WXS technology with 160 GB of storage. Reading between the lines, WXS disk offload clearly works, and this device will handle DynaCache, session cache, and "web cache" (including Hibernate near-line) for your web app server environment.

I'm impressed. Outlying questions:

1) Is this thing as hardened as the XA35-XM70 appliances? They do very different things, and WXS has always been Java-based. That's a far cry from a pure ASIC-implementation of XML acceleration.

2) Speaking of Java, does this thing use a near-realtime JVM? Deterministic performance (meaning predictable) is very much key for caching at the high end.

3) How competitive is the XC10 versus Azul's bad-boy Java appliances?

More to come.

The virtualization conundrum

In principle, it seems as though virtualization is a panacea for getting more workloads onto increasingly capable systems. The rise of multi-core processors does a couple of things:

1) Allows you to do more things with a smaller hardware footprint.

2) Can result in significant economies of scale if interoperating systems can "prefer" proximally close dependent systens. (Purty words for leverage a service on the same machine if given a choice between the local and a remote peer.)

3) Greatly and hugely inflate your software spend unless you take action to do things in a different way.

Hardware guys love to say, "Slam it on a VM and save." This can work, but not always. If your application is native to the OS (meaning not interpreted and not bytecode executed by a virtual machine), chances are good that your consolidation efforts will pay off if you make sure you aren't introducing I/O chokepoints.

Here's the rub: where Java is concerned, the operating system isn't all that interesting. The platform you need to virtualize is the platform running your code: Java. One more time: Java is the platform. The further that platform gets from the bare metal, the worse it runs. There are all sorts of bad things that can happen with Java and VMWare.

1) Scheduled schedulers

If VMWare is scheduling CPU time for operations, and the operating system is scheduling CPU time for threads, do they communicate to ensure that VMWare and the OS agree on which thread/operation are getting CPU time? Answer: no, not really. I've used examples of a big gear (VMWare) turning several little gears versus a planetary gear. Eyes usually glaze over. The point is the more threads you have in the OS, the more the OS has to coordinate with the VMWare scheduler that it can't really talk to. There is no real introspection between the hypervisor and the operating system, a situation made infinitely worse by the fact that the JVM itself is also trying to manage workload, schedule threads, and stop the world to clean up the garbage.

2) Non-deterministic performance

Deterministic: Events that have no random or probabilistic aspects but rather occur in a completely predictable fashion.

Non-determinstic: the opposite of deterministic.

Even with nearly real time Java, Java's performance is non-deterministic. The JVM will stop the world (perhaps not as much as it used to, but still...) to clean house. A work thread is a work thread, and mutators are mutators. All work is created equally in the eyes of the JVM, unless it's a housekeeping function. Furthermore, the JVM will almost certainly be in a state where it will have to wait on data from an EIS platform or some other dependent system. You don't want a Java thread in a wait state to pop and consume a CPU share when it's not doing anything relevant.

So, what does determinism/non-determinism have to do with virtualization? Everything. The situation with Java can be hard enough where the predictability of performance is concerned. Add to it the fact that VMWare might yank the rug out from under your guest to give some cycles to something else, and you have an incredibly volatile and decidedly unreliable environment. I have seen cases where a JVM entering into a GC run would peg the CPUs (which is normal), and cause VMWare to relocate that VM to another host. What should have taken 3-5 seconds, took 45 seconds thanks to VMotion. I have seen cases where a very low volume Cold Fusion environment running on JRun became completely unreliable when moved to VMWare. At issue was and is the fact that VMWare can't talk to the JVM to find out what's going on, and has (at best) very poor workload management capabilities. "Oooh, he looks busy. He must be important. Ya'll have to wait while I let this important guy finish doing his thing." Even worse: "Oooh, he looks busy. Unrelated guest, now it's your turn to clobber the CPU because I use a fair-share scheduler."

While I frequently slam VMWare, nothing said here is specific to VMWare. z/VM is in the exact same boat. HyperV is too. If your hypervisor can't talk to the platform hosting the workload, it will be ill-suited to allocating the hardware requirements for intelligently managing the workload. WebSphere Virtual Enterprise does a great job of bridging this gap. It gives you the ability to give the JVM information about the HW allocation for the dynamic partitions, VMs, and z/VM guests. This gives Java a fighting chance to not get over-extended.

It's not the same as the hypervisor providing reserve capacity or blocking some kind of hypervisor event (reallocation, for example) when Java gets busy, but this is a huge step forward. If you are running VMWare or z/VM and WebSphere, you need to look at adding WVE to your WAS environment.

PowerVM and Xen are a bit different. PowerVM, like it's Oracle-Borged Sun and HP counterparts, doesn't depend on a software hypervisor like VMWare, z/VM, and HyperV. Still, these systems have some form of DLPAR that can wreak havoc on Java. WVE is useful on PowerVM too. (I don't think the virtualization toolkit works on Solaris or HP/UX, but not a single one of my customers cares about those platforms.)

Xen is a whole other beast. It's kind of like OS WLM on steroids. I like it. It needs more granular prioritization, but I think Xen is a better virtualization approach than VMWare. Yeah, that's a lot like saying that a kiwi is a better fruit than an apple. That doesn't mean that kiwi pie is any good. They do different things. One is much better suited to managing the unique kind of workload presented by JEE application servers. The other is VMWare.

So in conclusion: Any thoughts you have about using VMWare to mitigate the costs incurred by multi-core x86 processors can be shortsighted. You will lose performance, and a lot of it. Consider Xen. Consider WVE. Consider both. Consider PowerVM and take advantage of the unique capabilities of the JVM for AIX that constantly interrogates the hypervisor to understand it's hardware allocation. POWER7 and AIX 6.1 are crazy powerful and the costs have never been lower.

There are better options than VMWare alone.

Webcast: Technically Speaking: Real World Elastic Scalability with WebSphere eXtreme Scale

I'm co-presenting next week with Joe Lea from IBM's WXS team....

================
March 30 Webcast: "Technically Speaking: Real World Elastic Scalability with WebSphere eXtreme Scale”
11:00 AM ET
This webcast session will demonstrate how innovative solutions built around the latest scaling technologies provide business value in the distributed caching space. You will learn about distributed caching architectural patterns as well as state of the art elastic scaling solutions in this space.

Speakers: Joseph Lea, WebSphere eXtreme Scale Product Manager, IBM; Scott Lewis, Software IT Architect, MSI System Integrators
================

Should be fun! The basic premise is how do we use WXS to build a data tier (or data presentation tier) that scales linearly? If we don't, we'll never build an application tier that does the same.

If you're interested, register below.
https://events.webdialogs.com/portal/wipevents/register.php?id=9339903393&l=en-US