How multiple enumeration is killing your performance in C# (#csharp #dev #performance)
Overview
In this post, we’ll dive into a common pitfall in C# development: multiple enumeration. We’ll explore what it means, why it can harm performance, and how you can avoid it with simple adjustments to your code.
Imagine you’re at an all-you-can-eat buffet, but every time you reach for dessert, you have to wait for the kitchen to cook it again. Annoying, right? That’s what multiple enumeration can feel like for your code—repeatedly processing a sequence when you really only need to do it once.
Refresher on IEnumerable vs List
In C#, an IEnumerable represents a sequence of elements that you can iterate over. It is the foundation for LINQ and deferred execution, meaning the actual data isn’t fetched or computed until you start iterating. This lazy nature can be a double-edged sword.
On the other hand, a List (or IList) is a collection that holds elements in memory. When you have a List, every item is already computed and stored, which means iterating over it is fast and predictable. Yes, yes, I know, in your application you have a list of 15 metric tons (or in freedom units: the weight of 15 beer trucks, 2 giraffes, and 3 steaks) of data, and loading everything into memory is not an option. But bear with me for a moment.
Ok, but what is multiple enumeration?
Multiple enumeration occurs when you iterate over the same IEnumerable sequence more than once. While this might seem
harmless, it can lead to several issues:
- Performance Hit: If your sequence is backed by a costly operation (like a database call or heavy computation), iterating multiple times means you’re paying that cost repeatedly.
- Side Effects: If the sequence’s iterator has side effects (e.g., logging, modifying state), you might get unintended consequences.
- Inconsistencies: With deferred execution, the underlying data may change between enumerations, leading to inconsistent results.
Practical Example
To illustrate this, imagine doing operations in a database, fetching products like the one bellow:
public record Product
{
public int Id { get; set; }
public string Name { get; set; }
public double Price { get; set; }
}
The example below we have a simple console application that is doings things we would usually do while working with a database. Just in a very very bad code.
What we’re doing is:
- Connecting to a SQLite database (in-memory);
- Seeding the database with some products;
- Fetching the products (aka: Creating an
IEnumerable<Product>); - Sorting the products by name and then by price;
- Increasing the price by 10%;
- Iterating over that list, and printing every product;
- Iterating over that list again, and printing every product.
I added a few Console.WriteLine to show you what is happening in the background.
using System.Data;
using Bogus;
using Dapper;
using Microsoft.Data.Sqlite;
using Multiple.Enumeration;
using var connection = GetDataBaseConnection();
SeedTestDatabase(connection);
Console.WriteLine("Messing around with the IEnumerable...");
var products = GetProducts(connection);
products = products.OrderBy(x =>
{
Console.WriteLine($"Sorting by Name: {x}");
return x.Name;
}).ThenBy(x =>
{
Console.WriteLine($"Sorting by Price: {x}");
return x.Price;
}).Select(x =>
{
Console.WriteLine($"Increasing price by 10%: {x}");
x.Price += x.Price * 0.1;
return x;
});
Console.WriteLine("Iterating over it...");
foreach (var product in products)
{
Console.WriteLine($"{product.Name} - {product.Price:C}");
}
Console.WriteLine("Again...");
foreach (var product in products)
{
Console.WriteLine($"{product.Name} - {product.Price:C}");
}
Want to run this code? You can check it here in my Github: Multiple Enumeration
Output
Messing around with the IEnumerable...
Iterating over it...
Fetched from DB: Product { Id = 1, Name = Gulgowski - Wisozk, Price = 51.07550073662538 }
Fetched from DB: Product { Id = 2, Name = Hilpert, Ondricka and Stanton, Price = 36.239341080004074 }
Fetched from DB: Product { Id = 3, Name = Cole, Block and Harber, Price = 88.1801337902502 }
Sorting by Name: Product { Id = 1, Name = Gulgowski - Wisozk, Price = 51.07550073662538 }
Sorting by Name: Product { Id = 2, Name = Hilpert, Ondricka and Stanton, Price = 36.239341080004074 }
Sorting by Name: Product { Id = 3, Name = Cole, Block and Harber, Price = 88.1801337902502 }
Sorting by Price: Product { Id = 1, Name = Gulgowski - Wisozk, Price = 51.07550073662538 }
Sorting by Price: Product { Id = 2, Name = Hilpert, Ondricka and Stanton, Price = 36.239341080004074 }
Sorting by Price: Product { Id = 3, Name = Cole, Block and Harber, Price = 88.1801337902502 }
Increasing price by 10%: Product { Id = 3, Name = Cole, Block and Harber, Price = 88.1801337902502 }
Cole, Block and Harber - $97.00
Increasing price by 10%: Product { Id = 1, Name = Gulgowski - Wisozk, Price = 51.07550073662538 }
Gulgowski - Wisozk - $56.18
Increasing price by 10%: Product { Id = 2, Name = Hilpert, Ondricka and Stanton, Price = 36.239341080004074 }
Hilpert, Ondricka and Stanton - $39.86
Again...
Fetched from DB: Product { Id = 1, Name = Gulgowski - Wisozk, Price = 51.07550073662538 }
Fetched from DB: Product { Id = 2, Name = Hilpert, Ondricka and Stanton, Price = 36.239341080004074 }
Fetched from DB: Product { Id = 3, Name = Cole, Block and Harber, Price = 88.1801337902502 }
Sorting by Name: Product { Id = 1, Name = Gulgowski - Wisozk, Price = 51.07550073662538 }
Sorting by Name: Product { Id = 2, Name = Hilpert, Ondricka and Stanton, Price = 36.239341080004074 }
Sorting by Name: Product { Id = 3, Name = Cole, Block and Harber, Price = 88.1801337902502 }
Sorting by Price: Product { Id = 1, Name = Gulgowski - Wisozk, Price = 51.07550073662538 }
Sorting by Price: Product { Id = 2, Name = Hilpert, Ondricka and Stanton, Price = 36.239341080004074 }
Sorting by Price: Product { Id = 3, Name = Cole, Block and Harber, Price = 88.1801337902502 }
Increasing price by 10%: Product { Id = 3, Name = Cole, Block and Harber, Price = 88.1801337902502 }
Cole, Block and Harber - $97.00
Increasing price by 10%: Product { Id = 1, Name = Gulgowski - Wisozk, Price = 51.07550073662538 }
Gulgowski - Wisozk - $56.18
Increasing price by 10%: Product { Id = 2, Name = Hilpert, Ondricka and Stanton, Price = 36.239341080004074 }
Hilpert, Ondricka and Stanton - $39.86
Process finished with exit code 0.
What is wrong with this result?
Looking at the output, you can clearly see that all the products were fetched from the database twice, and all the operations were also done twice. This is a clear example of multiple enumeration.
This is obviously an extreme example, but it illustrates the point. In a real-world scenario, you might not have a situation as bad as this (reaching out to the database twice), but this type of thing can easily kill the performance of your application.
How to avoid Multiple Enumerations
To dodge this performance pitfall, consider the following strategies:
- Materialize Your Sequence: Convert the IEnumerable to a List or an Array using .ToList() or .ToArray(). This forces immediate execution and stores the results in memory, so subsequent iterations are fast and free of side effects.
- Cache the Result: If you know you’ll need to enumerate a sequence multiple times, store the result in a variable instead of reusing the original query.
- Be Mindful with LINQ: Understand that some LINQ methods (like .Where(), .Select()) use deferred execution, and plan accordingly.
This is not an exhaustive list, but it should give you a good starting point. You need to take into account what your application needs and what is the best approach for your specific scenario.
Conclusion
Multiple enumeration is a subtle yet potent performance killer in C#. By understanding the nature of IEnumerable and being mindful of deferred execution, you can write more efficient, predictable, and maintainable code. So next time you’re working with LINQ queries, remember: one good enumeration is better than two!
Hope that helps!
References: