Technology

I sat a on my fingers for all the weekend but now i can't take all this comments in the blogs and the mailing lists any longer. Ten years ago, i would have gone ballistic long ago... but the fuse got longer of the past few years. However i'm still able to detonate, however despite the undirected explosion 10 years ago, it's more directed today.

And this article is such a directed explosion. I worked a while on this article and this was one of the reasons why it was relatively silent on this blog the last few days. At first i didn't wanted to publish it, but some events of today led me to think otherwise. I don't know if this is a wise move, because the torch of this rant will burn down some beards. On the other side, i think the article is worth the publication, but you have to judge about that.

However i try to prevent me from exploding thus i used this weekend to play with my new toy (i purchased an iPad last week and it's really fscking cool, i just thought "Like Enterprise ... i just wouldn't gave them away like they do in TNG" ) and worked a lot on the planing on the modifications on my newly purchased building.

However the mail by Garrett d'Amore asking about the removal of SVM was a drop. It was one of those drops, that led to a severe spill-over. So this blog article will be a long rant. If you don't like rants, just skip this.

And keep in mind, it's a rant ... it's not meant to be fair or objective ... think of it as a way to write down my frustration about a lot of events in the last few weeks and especially in the last few days.

At the moment you read a lot about this X-Force report and Sun is said to keep more vulnerabilities unpatched. But More interesting than the number of unpatched number of patches is the number of "Percentage of Critical and High 2010 H1 Disclosures with no patch" on page 20 in this report.

1. Google: 33%
2. IBM: 29% (the owner of X-Force)
3. Oracle: 22%
4. Linux: 20%
5. Microsoft: 11%
6. Novell: 10%
7. Sun: 8%

Some code is so old that it predates the "invention" ( ) of free software by the FSF. Such an example is the RPC code provided by Sun to the world under a permissive license in 1984 and it was use in many implementations like the one used in Linux to provide NFS services. This was possible due to the licensing Sun choose at that time:

Sun RPC is a product of Sun Microsystems, Inc. and is provided for
unrestricted use provided that this legend is included on all tape
media and as a part of the software program in whole or part. Users
may copy or modify Sun RPC without charge, but are not authorized
to license or distribute it to anyone else except as part of a product or
program developed by the user.

However it isn't technically free software, as it was freeded before free software was formally defined ... and out of some strange reasons, people found the license now free enough. In some Linux distributions this situation was considered as a serious bug (I would file a bug against something different in regard of this, but that's a different story ) However due to whatever reasons, this particular issue wasn't resolved for years ...

As Tom Callaway wrote in a recent blog entry, this situation has been resolved now:

So, we restarted the effort with Oracle, and on August 18, 2010, Wim Coekaerts, on behalf of Oracle America, gave permission for the remaining files that we knew about under the Sun RPC license (netkit-rusers, krb5, and glibc) to be relicensed under the 3 clause BSD license.

The new "IBM Power Systems Performance Report POWER7, POWER6 and POWER5 results" holds an interesting piece of information. A reoccuring question be colleagues and befriended admins is the impact of LPARs to the performance. It looks like IBM needs the LPARS to get some speed out of their larger systems.

Just a few examples: When using a 795 with 4.25 GHz and 64 cores a configuration with 4 LPARS a 16 cores yield a relative performance of 926.28. The same system with just 1 LPAR with 64 processors yield a relative performance of 777.09. So leaving the scaling to the OS instead of dividing it into 4 small systems gives you just 83.89% of the performance. When using a 795 with 4.25 GHz and 64 cores a configuration with 8 LPARs with 16 cores each yields a rperf value of 1852,56. With 2 LPAR with 64 cores each you get 1554,18. Interestingly is 83,89% again.

At first i thought "16 cores are easily fitting on a processor book (with 4 procs each). A 64 cores LPAR has to use two processor books. So when you use a configuration larger than a processor book you will leave 16,11% on the way". But doing the same calculation with some other data showed otherwise. But the move from 32 to 64 core lpars just reduce the performance by 5,7 percent respectively 5,4 %. 32 cores fit on a processor book, too. Thus the difference should be similar to the 64 to 16 cores situation.
So my interpretation is a little bit different: The scalability of AIX seems to have sweet spot between 16 and 32 cores. I thought a moment about an intra-book bottleneck, but the CPUs on the book are fully meshed (1 hop from each CPU to every other), so i don't think it's a problem.



When you look into this chart (please click into the image for a larger version), you may find some interesting points. The light blue line is a hypothetical perfectly scaling 4.0 GHz Power7 in a 795. The data is based on rPerf number of 103.41 for an 8 core system(source: Page 20 of the document). Please look at the right side of the chart at 256 cores. You end up at 81,9% of the hypothetical performance when you use 64 core LPARs and at 86% of the hypothetical performance when using 32 core LPARS (the difference is interestingly pretty much the same as computed before for 64 cores instead). At 64 cores your load is distributed at 8 processors, thus just 2 processor board. Still there is an serious impact of almost 20%. Will be interesting to further dig into this topic.

However it's important to know, that the operating system is limited to 64 core SMP no matter how many cores are in the system by the LPARs configuration. So this numbers doesn't factor in scalability challenges of AIX above 64 cores as the os has not to scale above this point while generating this rperf numbers. The numbers for the large core number configurations are not single OS image numbers. Further penalties for the OS scaling comes on top. Furthermore this benchmark is a pure CPU/memory benchmark. As IBM explained in their own description of the benchmark, there is no I/O and no networking involved.

That said, a number of really interesting data points are missing in the pdf:

• The rperf number of a fully blown unpartitioned system
• The rperf number on a fully blown system with just one LPAR in the size of the complete system
• Somehow i have the impression, IBM is hiding something. When you look into the mentioned pdf there is SPEC number for an unpartitioned system, but no rperf for it. The existence of the SPEC numbers hints to the point, that they had indeed an OS that was able to scale to 256 cores. On the other side, there is no SPEC number for the partitioned systems, but the rperf numbers. Just call it a presentiment ...

Congratulations to IBM for the new lead in TPC-C. IBM has once again the lead in a benchmark that got meaningless a few years ago. Just before you ask, i wrote the same when Oracle was first. But somehow it looks like IBM wasn't able just to call it a day So they did a TPC-C benchmark run again and at the end they were able to yield 10,366,254 tpmC.

The result is indeed impressive. However there is are some key difference. The response times are vastly longer in the IBM result.

Response Times (in seconds)	90th %	Average	Maximum
New Order	2.1	1.137	24.041
Payment	2.1	1.138	21.293
Order-Status	2.06	1.095	20.169
Delivery (interactive)	1.64	0.749	17.953
Delivery (deferred)	0.95	0.42	2.48
Stock-Level	2.08	1.113	21.547
Menu	1.64	0.77	23.037

Now look at the response times at the Oracle result.

Response Time in Seconds	90 th %	Avg.	Max.
New-Order	0.170	0.168	5.885
Payment	0.160	0.156	5.758
Order-Status	0.150	0.150	5.433
Delivery (Interactive)	0.120	0.134	3.869
Delivery (Deferred)	0.040	0.021	2.839
Stock	0.210	0.182	4.796
Menu	0.120	0.136	4.474

With similar response times, the number of transaction is more in the range of 8,9 million as stated by the diagrams in the full disclosures.

This is the diagram from the IBM result:

Somewhere between 80% and 100% of the final result the reaction time explodes.

This is the matching result of the TPC-C result of Oracle:

There is no equivalent "explosion" in this result....


Furthermore i want to hint you on certain points in the configuration as stated by the full disclosure.

• This configuration has 48*380 GB SAS cards with battery backed write cache, thus 17,8 GiB battery protected write cache in total.
• The configuration used 3*224 SSDs summing up to 672 SSD. I would suspect that they use SSD with Sandforce 1500 (the same 177 GB like they used in the TPC-C that is documented at several places to be generated with Sandforce based SSD). This controller has an interesting capability. It's capable to do compress and some benchmarks have suggested, that the performance of this drive is quite different with compressible and less compressible data. It's would be an interesting point to research how compressible TPC-C data is.
• The SSD use MLC. The Sandforce Controller have some special mode of operation to enable the use of MLC to reach better durability, but this mode is based on compression, too. Interesting questions are: Are the SSD really capable to hold 3 years of TPC-C load due to the usage of MLC and what would be the impact on durability of less compressible data.
• The database is completely on the SSD. Just the database log is on disk. That's similar to the Oracle config
• The configuration of the database on the system is ... well ... interesting. They use a partitioned database and all this partition are bound to certain resource sets. So essentially they splitted the system in several ones, to be exact ... into 32 partitions per systems bound to a resource set each. With this amount it's ensured that all requests are CPU local factoring out the interconnect.
• The configuration doesn't provide any availability protection, but that's okay, because TPC-C doesn't mandate such. Just keep this in mind with the pricing and with the configuration. There are no mechanisms ensure availability and you can't transform it into an available configuration because the storage is direct attached to a single node (the storage is in the i/o drawers). The Oracle configuration is highly available by accident, as it uses shared storage and RAC

Conclusion

I think TPC-C is now really a corporate ego thing. We are in an arms race here. I'm really interested, how Oracle strikes back

Just found this article about a new SSD from Micron. This article reminded me of my thought that there should be a third category in SSD. In the harddisk market there are consumer drives, enterprise drives and near-line drives. A consumer drive is that stuff you include in your desktop that is designed to run for a limited number of hours a day and mostly sits idle in the computer casing. An enterprise drive is designed to to work 24/7 with being busy most of the time (80% for example). Nearline drives are the one that can run for 24/7 but they are just designed for 20% busy time. Of course you you could use any disk for each task, but it's a simple equation: You swap busy time for the lifetime of the drive. You can use your SATA home user drive for your database but expect to see your storage array very often.

That said, i think we need a similar distinction in the SSD area. At the moment there are just enthusiasts drives and enterprise drives. Out of a strange reason there are enthusiasts drives that got the enterprise moniker, but thats a different story. But there is something similar important to the busy time for SSD, even when they aren't susceptible to mechanic wear. It's something i would like to call "write time". It's specified as the amount of time the drive is writing data to the media. That's unimportant for rotating rust, but for Flash based solid state drive it's completely different.

Let's do some calculation: Microns new drive is said to deliver 275 Megabyte/s. 275 Megabyte 60 seconds 60 minutes 24 hours 365 days * 5 years equates to 40,3840095 Petabyte. However the endurance of the drive is specified to be 4.5 Petabyte in 5 years. Thus you could only write in roughly 1/10 of the time data onto the disk. It has an allowable write time of roughly 10%.

So that may not a problem for usage if normal filesystems are used, where read and write load occurs on the disk, but now think of the ZFS separated log ... this is a write-only device for almost all practical purposes. You never read from it in normal operation,so there are no reads to relief the drive from the write load.

So you should not size the amount of SSD disks not only for the expected write bandwidth, you should also take into consideration how this influences the lifetime of your drive and how long you expect the server to stay in operation. At 275 Megabytes per second you would brick a drive designed for 4.5 PB within five months. As well as you choose the right disk type by thinking about the busy time, you have to factor in the write time for your SSD.

Where should someone set the differences between the types? Enthusiast drives are easy to spot ... MLC drives and all drives without capacitors in case they have caches. Near-line and enterprise SSD? I would draw the line somewhere around 50%. When you can't get through the warranty time at max speed with 50% writes, it's a nearline drive. Above that it's an enterprise drive. But that's only my personal opinion and a first idea.

With this article i want to learn a thing or two and i want to start a discussion.

Whenever you express, that a mainframe hasn't no advantages over a high-end Unix system, the first comment or reply to a comment is "Oh, but IBM has Parallel Sysplex and delivers availability xyz can only dream of". Often the discussion ends there, because few people know, what this "Parallel Sysplex" is. I had to read into a lot of documentation to learn more about the Parallel Sysplex.

I've learned a few things in that: A sysplex doesn't seem to be much different from something like we simply call a cluster in Unix. A parallel sysplex is an extension of this and enables the system to act as one large system. However you can't consider it as a tool that is able to distribute arbitrary tasks on sub-process level onto several systems.

As far as i understand the concept of the Parallel Sysplex, it's not the way that any and every software is enabled to be a part of a parallel sysplex as it has to use certain facilities that make a parallel sysplex out of a sysplex. Your application have to use a thing called the coupling facility. And parallel sysplex exactly contributes this facility to the applications running on the mainframe.

Parallel Sysplex doesn't help you to make your zLinux more available. You have to work with the Linux cluster-frameworks to do so, albeit there seems to be some workarounds

I understand that the parallel sysplex makes DB2 on z/OS more scalable and available, but where is the difference to Oracle RAC on Solaris on a 2 big node cluster? I understand that a parallel sysplex makes CICS more available and scaleable, but where is that different for example from using Oracle Tuxedo in a highly available configuration with clustering support? I understand that that parallel sysplex can make storage more available, but how is that different from the Availability Suite with it's Remote mirror facility, from SamFS/QFS, from the global filesystem in Sun Cluster.

I know, all these things working differently, but that's not the point. I want to know, why some people think "Oracle can just dream of". What's so special about that ... the parallel sysplex is an ... well ... interesting architecture, however i fail to see any reason why it's special that everyone talking about mainframes is just pointing to it.

Perhaps it's just the way that mainframe people doesn't understand the open systems world to the same extend as open systems people understand the mainframe world: Barely to an extend at all. Additionally i have the impression that many services on a mainframe are so ubiquious in that world that they consider properties of this software components as properties of the mainframes. Another hypothesis is, that most mainframe guys doesn't know what's possible on Unix (real Unix, not the pengiun variant) and thus consider their technologies as superior. I talked to a mainframe guy on a conference some time ago and i had to explain that SunCluster has gone great lengths since Sun Cluster 2.

We talk about synchronous storage replication in Solaris for quite a time now (2000), service restarting isn't a problem since SMF. Instruction Retry on hardware basis is available since the introduction of the M-Series, the same for memory mirroring. Hot swap of I/O, memory and CPUS is possible since last century.

May be i just don't get get, but may you are able to help me: I want to postulate "There is nothing in a Parallel Sysplex that a equivalent configuation on the basis of an M9000 isn't capable to do". But i'm a open systems guy from the core, so i ask the mainframe aware people to explaon me, why Parallel Sysplex is considered as superior to Unix-HA clustering when used with applications developed for high availability. Please don't answer with "You have no idea, ParSys is so superior" ... i want some real reasons.

The ZFS in FreeBSD is now at version 15. In version 15 user/group space accounting (aka Quotas) was introduced into Solaris. However there are still missing features:

jmoekamp@hivemind:~# zpool upgrade -v
This system is currently running ZFS pool version 22.

The following versions are supported:

VER DESCRIPTION
--- --------------------------------------------------------
[...]
16 stmf property support
17 Triple-parity RAID-Z
18 Snapshot user holds
19 Log device removal
20 Compression using zle (zero-length encoding)
21 Deduplication
22 Received properties
23 slim ZIL


According to the Coraid website, NetApp has send a "nice" letter to Coraid, a vendor of an ZFS based storage appliance. NetApp should stop to bully around small vendors, get back to innovation and sort that thing out with someone in the same or a larger weight class, for example with Oracle. I can just explain this behaviour by increasing pressure in the market by ZFS.

PS: At the end the lawsuit is still valid, as far as i know most of the relevant patents has been invalidated ....

Okay ... i really thought that enforcing a certain brand of cables was a thing of the past. Apparently not as reported by @randybias on his Twitter timeline:

This is really a common problem as shown here or here. Especially the later one is showing the pain you could experience when you use those 10GbE direct attach cables . You have to think about this DAC like two hardwired transceivers). As both ends contain the transceiver logic it's easy to implement an ID rom in that cable and check it before bringing up the link.

I should note, that there is a standard for the interface that connects those DACs to the switches/NICs. It's called SFF 8431. But now lets look into reality: The Intel card of the user in the second links doesn't support the DAC cables from HP. HP seems to just support HP cables, not allowing anyone to use other cables by software. Switches from HP are equally rigid, they just support HP branded DACs and nothing else. Cisco Nexus switches seems to have at least a mechanism to allow unsupported DACs to work to get HP cables running . Randy wasn't able to use Arista DACs with HP Flex 10.

Welcome in the world of cable-locks, where the intersection of supported with one systems and supported with the other system may be empty I thought, standards were made for a reason.

There is an interesting article inIEEE Spectrum:"The Trouble With Multicore". David Patterson writes in this piece about the problem multitudes of cores are introducing to programming of such systems. Worth a read!

(click here to view)

Update: The same video at Youtube.

Kris hat einen tollen Artikel über relationale Algebra geschrieben. Hochinteressant und eine eindeutige Leseempfehlung: Was bedeutet eigentlich 'Relationale Algebra'? Insbesondere gefällt mir:

Das ist genug Mathematik. Wir wollen Informatik machen.

Das bedeutet als erstes einmal: Weg mit den doofen Unendlichkeiten. Komplexe, reelle, rationale, ja sogar natürliche Zahlen - das ist alles unendliches, nicht anfaßbares Gekasper von irgendwelchen Mathespinnern. Wir wollen Werte, die man anfassen und mit dem Debugger im Speicher sehen kann. Informatik macht Diskrete Mathematik, alles schön endlich. Da kann man eine Funktion dann auch schon mal als Wertetabelle statt als Rechenvorschrift definieren.

Und so landen wir bei den Tabellen.

A while ago i assumed (and some Sun colleagues with me) that the M9000 could give the IBM p595 quite a run in TPC-H. At that time I couldn't substantiate it as it isn't allowed to compare TPC-H benchmarks at different scaling factors (e.g. TPC-H@1TB and TPC-H@3TB). Sun hadn't a 3 TB result, and IBM hadn't a 1 TB result. That changed: Today the TPC.org published the TPC-H@3TB result for the M9000-32.

On 04/10/10 a Sun M9000-32 was reported to yield 188,229 QphH@3000GB with 32 processors, 128 cores, resulting in $23.99/QphH@3000GB. Same amount of memory in both systems. Just as a comparision: The p595 fully blown processor-wise with 32 processors and 64 cores was able to yield 156,537 QphH@3000GB end of last year.

Some comments:

• The Sun system is able to yield 20.246 percent more performance than the IBM system.
• The result is the best non-clustered TPC-H@3TB result.
• This benchmark uses the current top end machine of IBM in a fully blown config. The Sun machine has still some growth potential, as you you could still opt for the M9000-64 with 64 sockets.
• While using the same number of sockets, the p595 needs less cores for the same job. So the core-based pricing with Oracle would take a larger dent out of the budget with the Sun system than with the IBM system, when using the same db. However the p6 core has a multiplier of 1, whereas the SPARC64 core has a multiplier of 0.75. So it's 64 oracle cores with IBM versus 96 Oracle cores with Sun. This somewhat kills the $/perf metric, despite the point that the hardware is significantly cheaper. Would like to see a socket based licensing in the future
• The IBM result was done with Sybase IQ Single Application Server Edition v.15.1 ESD #1.2, a somewhat specialized database engine for reporting, analytics, and data warehousing. That's interesting as TPC-H is the decision support benchmark in the TPC benchmark. By the way: The limits of Sybase IQ SASE may explain why the data base license is really cheap compared to Oracle. The Sun result used Oracle Database 11g Release 2 Enterprise Edt. I would like to see a IBM result with a general purpose database, too.