.NET runtime related issues

Testing .NET applications for performance is associated with few complexities that exist because of virtualization offered by .NET runtime:

1. MSIL code is JITted into native code on the fly.
2. There is a garbage collector (GC), that periodically interrupts all the running threads. I know this isn't fully true - but it's enough to mention this for the brief description.

Our performance tests take these factors into account:

1. We perform a warm-up pass of each test before actual measurement. This ensures all the tested code is already JITted on actual measurement. Warm-up pass is currently exactly the same as the actual test. But if actual test takes relatively long time (~ 1 minute; currently there are no such tests), we will consider making its warm-up pass up to 10 times shorter (for any tested ORM, of course).
2. We perform full garbage collection before each actual measurement. To ensure this is done, we run GC.GetTotalMemory(true) and GC.WaitForPendingFinalizers() in a loop with 100ms pauses for 5 times.

ORM related issues

We're aware that almost any ORM utilizes instance caching techniques. Such caches are normally bundled into Session-like objects. Since the goal of our performance tests is to compare raw ORM performance without any instance-level caching, we ensure they are unused by few simple rules:

1. Any test pass utilizes new Session-like object (DataContext descendant, etc.)
2. We never use the same instance of persistent entity in more than one basic tested operation. I.e. if there are 10K operations in test, they will involve 10K or more persistent entities.
3. Session creation time is excluded from measurement, if this is possible to achieve this for all tested ORM tools. Otherwise it is always included into the test.

Database related issues

1. Both tests and database server run on the same multicore PC and communicate via shared memory protocol, if other isn't explicitly mentioned. This minimizes the roundtrip time and thus maximizes the overhead from ORM usage.
2. To ensure the whole command sequence is actually sent to the database server and executed there, we include transaction commit time (i.e. invocation of transaction.Commit() - like method) into any measurement.
3. We use a table with tiny rows and single clustered primary index on performance tests combined with sequential reading to get peak row throughput. This ensures ORM will "flooded" by tiny pieces of data it must convert into objects, and thus the overhead of its usage will be maximal - as well as materialization rate.