the next step in postgresql evolution: a benchfarm ?

Joshua Drake just announced that the first machine is finally available for a PostgreSQL Performance Test lab donated by a range of commercial sponsors.
Ever since Andrew Dunstan invented the buildfarm and seeing the smashing success it had on improving the PostgreSQL development process I have been toying with the idea of doing a similar thing on the performance front. What the past few releases have shown is that PostgreSQL is gaining more and more complex performance related features and planner improvements.
However some of this things change fundamental infrastructure in PostgreSQL (for example the outer join reordering stuff in 8.2) and while often causing dramatic improvements on the one side we also a fair number of regressions which largely went undetected during BETA and even the first few minor releases.
The current buildfarm setup is very good in covering a large range of platforms and architectures for testing portability and basic regression testing but due to its nature is not really suitable for benchmarking of any kind.

Some of the reasons why it is not so well suited for that purpose are:

• a lot of the buildfarm clients are not really dedicated installations (or just VMs)
• capabilities/resources on some of the boxes are quite restricted (lionfish for example has only a few hundred megabytes of disk space and 48MB of RAM available which is barely enough to get through the regression tests ...)
• limited control over the change management on those boxes (updates/upgrades) which might make it difficult to compare historic results

What I would like to see in the future is a threefold approach:

• We need continuous and long term benchmarking of at least -HEAD using defined benchmark tools (sysbench, marc wongs dbt implementations or Jans tpc-w stuff) done at least weekly if not daily so we can get an idea on how our overall performance improves (or degrades) on various workloads on a relative scale.
• Tracking of query plans - the current regression test framework is designed to give very deterministic results and therefor uses a lot of ORDER BY and similar stuff get get consistent results. This however tends to hide subtle regressions in the planner for large and complex queries and on large databases (the typical regression database is very very small compared to what people use in the wild). What we need is a collection of real life test cases that load data and execute queries on them comparing the plans AND execution times over a long period of time on small and large datasets and notify on changes.
• Dedicated boxes for individual hackers to work on new features or experimental patches