mastermind's weblog

Benchmarking 8.4 - Chapter 2/bulk loading

nospam@example.com (Stefan Kaltenbrunner) — Tue, 16 Jun 2009 09:28:00 +0200

As promised in the previous post this is the second part in a series of testing/benchmarking 8.4 under various circumstances.
The topic in this post is bulk loading of data. Knowing about expected and theoretical bulk loading performance is a very important thing for an DBA. It affects not only data warehouse style operations but also plays an important part in disaster recovery scenarios because the total time it takes to restore your database from your backup is directly related to its bulk loading performance.

Continue reading "Benchmarking 8.4 - Chapter 2/bulk loading"

Benchmarking 8.4 - Chapter 1/Read-Only workloads

nospam@example.com (Stefan Kaltenbrunner) — Fri, 12 Jun 2009 22:21:00 +0200

Computing platforms are constantly changing, evolving and improving. This holds true for both the hardware and the software running on that hardware.
With PostgreSQL 8.4 just around the corner and a 2U Nehalem based IBM x3650M2 at my hands I thought I would do some benchmarking and see how well PostgreSQL does on that kind of hardware under a range of different workloads. This is planned as a series of posts starting with read-only benchmarking, followed up by bulk load testing and maybe some OLTP benchmarks as well.

Continue reading "Benchmarking 8.4 - Chapter 1/Read-Only workloads"

Nagios conference 2008

nospam@example.com (Stefan Kaltenbrunner) — Sat, 13 Sep 2008 11:36:00 +0200

Long time no post - but I just got back home from the Netways Nagios Conference 2008 where I was invited to give a talk on "PostgreSQL Monitoring - Introduction, Internals And Monitoring Strategies for postgresql.org" so I figured it is time to say something again ...

Continue reading "Nagios conference 2008"

fosdem - a huge success

nospam@example.com (Stefan Kaltenbrunner) — Mon, 25 Feb 2008 23:30:00 +0100

FOSDEM was a really nice and successful event for Opensource in general and especially PostgreSQL - we had a really impressive booth and a huge number of talks with a lot of attendances.
We also managed to get the initial board for the european user group voted and that will pave to wya to get the group set up as a legal entity within the next few months - YEAH!

Continue reading "fosdem - a huge success"

stefan: "you are becoming uncle"

nospam@example.com (Stefan Kaltenbrunner) — Fri, 08 Feb 2008 19:04:00 +0100

this feels strange - my sister just told me that she is expecting a baby and that I'm about to become uncle which is feeling really strange.
However I'm really really happy for Andrea and Harry (see my brothers blog for a picture of them) and I wish them the best.
It now looks that Harry will have to take on the burden of doing all the hard work on their new apartment - but I guess thats ok :-)

feature matrix - or "what is new in version XY?"

nospam@example.com (Stefan Kaltenbrunner) — Thu, 31 Jan 2008 22:33:00 +0100

PostgreSQL is a thriving and fast evolving project with regular releases that add new features, improve existing functionality and increase speed.
Most of those improvements are mentioned in a rather brief from in the release announcement/press release and or in a lengthy form in the release notes.
The new feature matrix on the main website is an attempt to provide a quick tabular overview about "what is new in release XY" starting from 7.3 to the about to be released 8.3.

Continue reading "feature matrix - or "what is new in version XY?""

8.3 vs. 8.2 - a simple benchmark

nospam@example.com (Stefan Kaltenbrunner) — Tue, 11 Dec 2007 22:33:00 +0100

With 8.3 just around the corner more and more people are actually starting to test 8.3 with their code base and wondering if it will be worth to switch/upgrade and so did I.
The following is not really an in-depth benchmark but meant as a simple testing of 8.3 on a very specific (but not uncommon) workload and with small set of different configuration parameters.

All the testing is done on a DL380 G5 with two Quadcore Intel Xeon CPUs (X5345 - so 2,33Ghz per core) 12GB of RAM and 6 disks in a RAID1+0 with 512MB BBWC on a Smartarray P400.
The box is running Debian/Etch AMD/64 with the debian supplied kernel (2.6.18 more or less) and a current -HEAD snapshot of PostgreSQL (more or less BETA4 code) compiled as a 64bit binary.

The benchmark database is initialized with a scaling factor of 100 (equals to 10M rows) which seems to be a reasonable size for a table in an OLTP style database) all testing was done with 100 clients and 100000 transactions/client which comes out to 10M transactions and an average runtime of about 1,5 hours.
The first test is a simple comparison of 8.2 vs 8.3B4 in both the default (ie. unchanged postgresql.conf) and one with a somewhat reasonable tuning:

checkpoint_segments=192
maintenance_work_mem=128MB
shared_buffers=1536MB
wal_buffers=1024kB
effective_cache_size=3084MB
filesystem(ext3) mounted with noatime

83b4shm.gif

what we can see here is that 8.3B4 is about 2.2x faster than 8.2 out-of-the-box for this very update heavy workload and is able to keep that advantage after similar tuning is applied to both instances.
The main reason for this rather large improvement is HOT. pgbench is one of the workloads that benefit most from it and a bit of preliminary testing with similiar real-life workloads (session tables or stuff like the SQL-Bayes support in Spamassassin) show equally impressive gains.

the second thing I tried to test was the effect of various shared_buffer settings on the transaction rate(the other parts of the configuration stayed the same as in the "tuned" variant above):

there are two main things to note here - one is the obvious one that the scaling on the x-axis is not linear.
The other one is that the best performance seems to be around 1,5GB of shared memory which is about the size of the database on-disk in 8.3. Higher settings to not help and in fact cause a slight performance degeneration.

All in all 8.3 seems to become a rather impressive release and as always proper tuning is very critical to getting the best out of (dedicated) database server instances.

the next step in postgresql evolution: a benchfarm ?

nospam@example.com (Stefan Kaltenbrunner) — Sat, 03 Nov 2007 10:57:00 +0100

Joshua Drake just announced that the first machine is finally available for a PostgreSQL Performance Test lab donated by a range of commercial sponsors.
Ever since Andrew Dunstan invented the buildfarm and seeing the smashing success it had on improving the PostgreSQL development process I have been toying with the idea of doing a similar thing on the performance front. What the past few releases have shown is that PostgreSQL is gaining more and more complex performance related features and planner improvements.
However some of this things change fundamental infrastructure in PostgreSQL (for example the outer join reordering stuff in 8.2) and while often causing dramatic improvements on the one side we also a fair number of regressions which largely went undetected during BETA and even the first few minor releases.
The current buildfarm setup is very good in covering a large range of platforms and architectures for testing portability and basic regression testing but due to its nature is not really suitable for benchmarking of any kind.

Some of the reasons why it is not so well suited for that purpose are:

a lot of the buildfarm clients are not really dedicated installations (or just VMs)
capabilities/resources on some of the boxes are quite restricted (lionfish for example has only a few hundred megabytes of disk space and 48MB of RAM available which is barely enough to get through the regression tests ...)
limited control over the change management on those boxes (updates/upgrades) which might make it difficult to compare historic results

What I would like to see in the future is a threefold approach:

We need continuous and long term benchmarking of at least -HEAD using defined benchmark tools (sysbench, marc wongs dbt implementations or Jans tpc-w stuff) done at least weekly if not daily so we can get an idea on how our overall performance improves (or degrades) on various workloads on a relative scale.
Tracking of query plans - the current regression test framework is designed to give very deterministic results and therefor uses a lot of ORDER BY and similar stuff get get consistent results. This however tends to hide subtle regressions in the planner for large and complex queries and on large databases (the typical regression database is very very small compared to what people use in the wild). What we need is a collection of real life test cases that load data and execute queries on them comparing the plans AND execution times over a long period of time on small and large datasets and notify on changes.
Dedicated boxes for individual hackers to work on new features or experimental patches

leaving a wonderful conference and getting home again ...

nospam@example.com (Stefan Kaltenbrunner) — Mon, 09 Jul 2007 19:33:28 +0200

pgday.it in Prato was a truly marvelous conference, but everything is eventually coming to an end and so did this conference. On sunday Gabriele and Andrea took us one a tour through Florence ending in a great dinner and a bit of a challenge to catch the last train^bus back to prato ...

Today we started at around 09:30 in the morning trying to get Magnus to the airport which turned into quite a long trip mostly due to the fact that andreas navigation system developed something that can only be described as "artificial (un)intelligence".

After we left the airport we thought we were good for going back up to Salzburg (and Munch) but the navigation system acted up again and caused so much confusion that we actually managed to get up the highway WITHOUT(!) getting a ticket from the toll station.
So just imagine what it might take to get off a highway with a fully automated toll collection system without a ticket and neither the remote assistance nor the one lone guy that was responsible for some 18+ exit gateways and not even they guy in the main office is capable of understanding a single word of german or english ...

After a somewhat interesting discussion which mainly consisted of well wild gestures and "i have no idea what you are talking about" faces we managed to get a form that probably claims (again all Italian) that we turned around on the highway or something like that - but well it seems we don't have to pay anything for "hacking" the italian highway system.

On the way back to the car from the autostrada office we met the guy manning the 18+ toll houses again telling us that there is a completely unlabeled underground crossing to the other side of the highway which would have been a good thing to to know earlier for well "making crossing a highway "easier".

pgday.it - beer, grappa, wine and fun #2

nospam@example.com (Stefan Kaltenbrunner) — Sat, 07 Jul 2007 18:33:00 +0200

So well - the official part of pgday.it is over now and today we had a lot of interesting talks from a number of international people covering everything from encoding(Susanne) related stuff over to full text search(Oleg) and PITR(Simon).
Josh did his keynote on new features in 8.3 which is something I'm a "bit" familiar with - but nevertheless it contained quite a lot of information especially the announcement of the official benchmarks sun did was very impressive and clearly how powerful and also cost effective PostgreSQL really is.
Again we got ton's of food and well there was also this guy doing interviews for a satellite TV station (boy do I hope that mine will not get broadcasted ...).
We will soon meet up at the bar of the prato linux usergroup for doing a bit of brainstorming and discussions on a possible european postgresql usergroup and afterwards we are going to have beer, grappa, wine and lot's and lot's of fun with our new Italian friends.

pgday.it - beer, grappa, wine and fun #1

nospam@example.com (Stefan Kaltenbrunner) — Fri, 06 Jul 2007 17:30:00 +0200

so pgday.it is in full swing, and it's a truly impressive conference. Andreas and I arrived after a 7h drive from Salzburg in Prato.
Andreas claims it took us that long due to the strict traffic limits in Austria (there is a 100km/h limit on the A10 from Salzburg down to Italy for "environment protection" reason) but I'm not entirely convinced this is true ;-)
Anyway - we barely managed to check-in at the hotel when we met David, Magnus and Josh who took us on some sort of "sightseeing tour" around prato. Which means that we basically had a [ guide with long hair and in short trousers taking us through half of the city (couple miles at least) looking for a place to get food and sit down in prato - well we didn't succeed until someone else (the "map guy" from [the http://www.prato.linux.it/ prato linux user group]) gave us the right hint.
Later on the Italian guys organized a very cool dinner at a restaurant serving typical local food - and we learned quite a lot about local eating and drinking habits (pictures courtesy of Andreas).
We ended the day at around 2 o'clock in the morning at the "New York" bar directly in the city of prato which is basically a float on the river after feeding fish and drinking beer ...

Today is the first day of the actual conference and one can only barely imagine the amount of work the organizers have put into this ...
The organization is very good (with a nice ) there is a lot of neat merchandising stuff available and I even got a pgday.it staff shirt for myself, and everything is free (including the fantastic food we are getting served).
The talks are pretty interesting too with real time translation of the talks into English (or Italian for the ones held in English) - though the schedule is a bit crammed and could take some additional breaks between the speakers.
If you want to see what's going on here simply look at the official website and listen to the live audio broadcasts.

new tech toy ...

nospam@example.com (Stefan Kaltenbrunner) — Tue, 22 May 2007 20:59:00 +0200

ok just bought me a new (and rather expensive) toy after some people said "ok if you buy one today I will buy you a game for that".

So I went to the electronic shop and bought one (including the optional remote and two games) but I guess that might have only been the start because now a nice new LCD television would suddenly make sense ...

weird animals on the postgresql buildfarm

nospam@example.com (Stefan Kaltenbrunner) — Sat, 19 May 2007 14:02:00 +0200

Today I spent some time on doing a bit of maintenance work on some of my buildfarm boxes and suddenly I thought it would nice to present some of the more weird ones to others.
Of course there is also magnus complaint about a lack of blog posts on planetpostgresql.org that I just had to react too ;-)

I have a total of 14 registered hosts on the buildfarm with nine of them actively reporting and four of them (those that I think are the most weird and interesting ones) are worth to get mentioned in a bit more detail:

lionfish:

Hardware: Cobalt Cube 2
CPU: 250Mhz MIPS in little endian mode
Bogomips: ~250
Memory: 48MB of RAM (+196MB of swap)
Disk: 4GB IDE
Time to complete a build farm run: ~5,5-6 hours(this makes lionfish by far the slowest box on the farm)
On the farm since: 2004
Operation System: Debian/Sarge 3.1

issues found by lionfish:

race condition in ALTER INDEX RENAME
crashes with the new multiple autovacuum worker code
buggy MIPS spinlock code

quagga:

Hardware: ALLNET6500 (identically to the Thecus 2100 with 256MB of RAM instead of 128MB)
CPU: Intel IOP 80219 ARMv5TE running at 600Mhz
Bogomips: ~250
Memory: 256MB DDR-SDRAM
Disk: 250GB SATA
Time to complete a build farm run: ~2,5 hours
On the buildfarm since: January 2007
Operating System: Debian/Etch 4.0

issues found by quagga:

tcl upstream bug on ARM and MIPS

spoonbill:

hardware: Sun Ultra 10 Workstation
CPU: 300Mhz UltraSPARC-IIi
Memory: 1GB
Disk: 40GB IDE
Time to complete a buildfarm run: ~1,5 hours
On the buildfarm since: at least autumn 2004
Operating System: OpenBSD 3.9/Sparc64 (upgraded a few times though)

issues found by spoonbill:

gcc(<3.4) optimizer bug on Sparc64 platforms
pg_database flushing race condition
portability issues in the regression test framework on BSD-platforms
configure portability fixes for OpenBSD
tsearch2 failures/crashes
helped diagnosing an OpenBSD libc bug
WAL page split patch use after free

sponge:

Hardware: IBM RS/6000 7046-B50
CPU: PowerPC 604e 375Mhz
Bogomips: ~41
Memory: 256MB
Disk: 18GB SCSI
Time to complete a buildfarm run: 1-1,5 hours
On the farm since: spring 2006
Operating System: Fedora Core 5/ppc

issues found by sponge:

regression test race condition
XML regression test problem

While hardware of that kind is not likely to be found in any serious or performance critical production use (at least I hope so!) this summary clearly shows the importance of the buildfarm as well the value of having not-so-mainstream boxes there :-)
I would be interested in getting details on other weird boxes people have on the farm or are are planning to add ...

PostgreSQL 8.3 patch status page

nospam@example.com (Stefan Kaltenbrunner) — Sat, 05 May 2007 17:28:00 +0200

In an apparent moment of brain fade I volunteered to add a 8.3 patch status page to the developer wiki.
It is synced up to all the outstanding patches that are actively considered for 8.3 and partly based on the Wishlist for 8.3 and the Wishlist for 8.4 I have been maintaining for a while now - and I would really like to get some feedback on both.
While I'm not convinced at all that a wiki is a proper place for keeping this kind of information it might be a start for looking into other more sophisticated tracking tools - at least if I manage to keep up with the review process for the 8.3 cycle :-)

"cheap" SAN gear

nospam@example.com (Stefan Kaltenbrunner) — Mon, 30 Apr 2007 16:51:00 +0200

The discussion on using SAN vs. DASD based storage is nearly a religious war(as can be seen in a lot of discussions on pgsql-performance) and in many ways similar to the infamous emacs vs. vi debate.
From personal experience I have found the IBM DS4300 and IBM DS4300 Turbo (basically the same as the DS4300 but with more memory/cache and a hefty markup in price) quite a reliable and basically maintenance free solution.

However - for some workloads those types of SAN are not really that appropriate. A DS4300(which is a now withdrawn from marketing) can do only a bit above 100MB/s of seq IO(nearly independent on the number of disks!) per controller(about 135MB/s if used together) which is really not much when one considers how fast modern hard drives are.

I recently got a SAN Array to play with that looks quite interesting since while expensive it still seems reasonably priced compared to what companies like IBM or others want for similar gear.

The array I got for testing is basically a non-branded LSI/Engenio 3994 with 16 2Gbit 10k 146FC drives and 2GB of battery backed cache per controller.
It is directly connected via two QLogic QLE2460 PCI-Express adapters to a HP DL380 G5 running CentOS 5 for testing.
The first impression of the array is a solid one - it looks very familiar for people that are used to the IBM DS4000 storage line and the Management GUI is basically the same (with an Engenio logo in place of the IBM one).
The controller chassis can hold 16 disks (up from 14 in the older designs) in 3U and the available expansion enclosures have the same capacity and dimensions (up to 6 are supported) and can be added online(untested!) without disruption to ongoing IO.

Due to the use of disks that are only capable of 2Gbit/s, the speed of the two drive channel loops is also limited to 2Gbit/s (using 4GBit FC drives it can be configured to use 4Gbit/s on the drive channels).
The following is not meant as a thorough benchmark of neither the array nor PostgreSQL but rather some ad-hoc testing and playing around to get some impression on the overall performance characteristics of the device and are done using ext3(I'm fully aware of the fact that other file systems - especially XFS - might provide noticeable better streaming performance, but I have a much higher level of trust in ext3 and that's my choice in production environments) in the default journaling mode.

In the following test(test case 1) we use two volume groups - each a RAID10 (8 disks) and a RAID0 in the OS and write cache mirroring between the controllers(keeps both controller caches in sync so in case one controller fails the other one can take over). To utilize both controllers the HBAs are set up so that controller A is using on and controller B the other.

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
convm004        16G 51121  98 188347  76 97426  28 58961  98 378240  38 732.5   2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                512 19599  94 257598 100  8272  27 19270  91 331205  99  4879  17
convm004,16G,51121,98,188347,76,97426,28,58961,98,378240,38,732.5,2,512,19599,94,257598,100,8272,27,19270,91,331205,99,4879,17

and the same with write mirroring disabled for both logical volumes (test case 2):

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
convm004        16G 51372  99 235020  96 122183  35 58880  98 369848  37 723.0   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                512 19888  95 256732  99 10037  32 19704  93 332286  99  5541  19
convm004,16G,51372,99,235020,96,122183,35,58880,98,369848,37,723.0,1,512,19888,95,256732,99,10037,32,19704,93,332286,99,5541,19

so write mirroring seems to have a 20% penalty for sequential writes and rewriting but not much impact for others - so it might be worth keeping it turned on due to the additional data integrity guarantees it provides .
It further seems that the device seems to be bottlenecked by the speed of the drive channels (there are two loops in the array head and half the drives are on the one and the other half on the other) due to the 2Gbit disks.
But it also shows that the devices seems to scale fairly well - until it hit's the bandwidth limit - at least for RAID10.

and now for comparison a test using only volume group and a single controller (test case 3):

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
convm004        16G 51346  98 134822  56 69414  17 58651  97 251779  23 758.8   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                512 20048  91 259694  99  5846  18 18707  84 338671  99  2935   9
convm004,16G,51346,98,134822,56,69414,17,58651,97,251779,23,758.8,1,512,20048,91,259694,99,5846,18,18707,84,338671,99,2935,9

so let's see what PostgreSQL is able to do in terms of sequential IO on such device:

bench=# select version();
                                                   version                                                    
--------------------------------------------------------------------------------------------------------------
 PostgreSQL 8.3devel on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.1.1 20070105 (Red Hat 4.1.1-52)
(1 row)

bench=#

simple sequential scan on a large table (pgbench schema generated with a scale of 10000) using only a single controller (same setup as in test case 3):

bench=# select count(1) from accounts;
   count    
------------
 1000000000
(1 row)

Time: 619865.939 ms
bench=# select pg_relation_size('accounts')/619::float;
     ?column?     
------------------
 216998258.558966
(1 row)

so we are getting about 215MB/s out of 250MB/s which looks ok.

so what happens with software raid 0 over two 8 disk RAID10 volume groups on different controllers (same setup as test case 1):

bench=# select count(1) from accounts;
   count    
------------
 1000000000
(1 row)

Time: 478785.617 ms
bench=# select pg_relation_size('accounts')/478::float;
     ?column?     
------------------
 281008205.121339
(1 row)

Time: 265.791 ms

so that is more interesting - it seems that PostgreSQL is getting CPU bottlenecked(the array/file system can do >370MB/s) here and those ~280MB/s are pretty much in line with what Luke usually quotes (PostgreSQL getting CPU bottlenecked at around 300MB/s even on very fast AMD Opteron based boxes).

for those curious here are some other random tests (uncommented so judge by yourself):

single raid 5 with 4 logical volumes (each 500GB) and software RAID0 in the OS - two volumes per channel

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
convm004        16G 50978  99 208098  82 89274  25 59058  98 236993  24 488.5   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                512 20860  94 258368  99  7770  25 21087  95 335918 100  4291  14
convm004,16G,50978,99,208098,82,89274,25,59058,98,236993,24,488.5,1,512,20860,94,258368,99,7770,25,21087,95,335918,100,4291,14

A single RAID5 array over all 16 disks and two identically sized logical volumes each around 1TB in size.

bonnie++:

on one LUN:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
convm004        16G 51245  98 121190  49 69406  17 56902  94 256111  22 840.9   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                512 20781  94 235507  91  7233  23 18685  84 338017  99  4125  14

using both LUNs and software RAID0:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
convm004        16G 51423  99 204881  84 83740  23 59040  98 232573  23 554.7   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                512 20303  93 259481  99  7230  23 20357  93 337312 100  3793  13
convm004,16G,51423,99,204881,84,83740,23,59040,98,232573,23,554.7,1,512,20303,93,259481,99,7230,23,20357,93,337312,100,3793,13

with disabled write cache mirroring:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
convm004        16G 51751  99 242637  97 105392  30 58859  98 235541  23 563.2   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                512 21034  95 255377  99  6485  21 19269  88 337095  99  4119  14
convm004,16G,51751,99,242637,97,105392,30,58859,98,235541,23,563.2,1,512,21034,95,255377,99,6485,21,19269,88,337095,99,4119,14