CSOM Deep Dive: Batching Requests for Performance

The Client-Side Object Model (CSOM) is how managed code talks to SharePoint from outside the farm — and increasingly from outside the network entirely, against SharePoint Online. Its defining characteristic is that nothing happens until you call ExecuteQuery. Understand that one fact and you can write CSOM that is both correct and fast; ignore it and you will write code that makes dozens of round-trips when one would do.

How CSOM actually works

Every CSOM call you make against a ClientContext is queued, not executed. You build up a batch of operations, then send them all to the server in a single request when you call ExecuteQuery() (or ExecuteQueryAsync()). The server processes the batch, returns the results in one response, and the client populates your objects.

This is why you must explicitly Load the properties you intend to read. The proxy doesn't fetch anything until you ask and then execute:

using (var ctx = new ClientContext("https://contoso.sharepoint.com/sites/ops"))
{
    ctx.Credentials = new SharePointOnlineCredentials(user, securePassword);

    Web web = ctx.Web;
    ctx.Load(web, w => w.Title, w => w.Url);   // queue: load two properties

    ctx.ExecuteQuery();                          // one round-trip

    Console.WriteLine(web.Title);                // now populated
}

The chatty anti-pattern

Here is the mistake that tanks performance — calling ExecuteQuery inside a loop:

// DON'T: one round-trip per item — slow and throttling-prone
foreach (var id in itemIds)
{
    ListItem item = list.GetItemById(id);
    ctx.Load(item);
    ctx.ExecuteQuery();          // N network calls!
    Console.WriteLine(item["Title"]);
}

Against SharePoint Online, with network latency of 50–150 ms per call, a thousand items means minutes of wall-clock time and a high chance of being throttled. Batch the loads instead and execute once:

// DO: queue all loads, execute one batch
var loaded = new List<ListItem>();
foreach (var id in itemIds)
{
    ListItem item = list.GetItemById(id);
    ctx.Load(item, i => i["Title"]);
    loaded.Add(item);
}
ctx.ExecuteQuery();              // one network call for the whole set

foreach (var item in loaded)
    Console.WriteLine(item["Title"]);

Shape your requests: load only what you need

By default Load on a collection returns every field of every item — a lot of wasted bytes. Use lambda projections and Include to request just the columns you want:

CamlQuery query = CamlQuery.CreateAllItemsQuery(2000);
ListItemCollection items = list.GetItems(query);

ctx.Load(items,
    coll => coll.Include(
        i => i.Id,
        i => i["Title"],
        i => i["Status"]));      // only these three fields come back

ctx.ExecuteQuery();

Paging large lists with ListItemCollectionPosition

SharePoint will not return an unbounded result set, and you should never ask it to. Use the row limit in your CAML and page with ListItemCollectionPosition until it comes back null:

ListItemCollectionPosition position = null;
do
{
    var camlQuery = new CamlQuery
    {
        ViewXml = "<View><RowLimit>500</RowLimit></View>",
        ListItemCollectionPosition = position
    };

    ListItemCollection items = list.GetItems(camlQuery);
    ctx.Load(items, i => i.ListItemCollectionPosition,
                    i => i.Include(x => x["Title"]));
    ctx.ExecuteQuery();

    foreach (var item in items)
        Process(item);

    position = items.ListItemCollectionPosition;   // null when done
}
while (position != null);

ConditionalScope: skip work that isn't needed

ConditionalScope lets the server evaluate a test and only run the queued operations if the condition holds — useful for "load this only if it exists" without a separate round-trip:

bool listExists;
var scope = new ConditionalScope(ctx,
    () => ctx.Web.Lists.GetByTitle("Contracts").ServerObjectIsNull.Value != true);

using (scope.StartScope())
{
    using (scope.StartIfTrue())
    {
        var list = ctx.Web.Lists.GetByTitle("Contracts");
        ctx.Load(list, l => l.ItemCount);
    }
}
ctx.ExecuteQuery();
listExists = scope.TestResult.Value;

Bulk writes and surviving throttling

The same batching rule applies to writes — set all your fields and call ExecuteQuery once per reasonable batch (a few hundred operations), not once per item. But against SharePoint Online you must also plan for throttling: the service returns HTTP 429 (Too Many Requests) or 503 with a Retry-After header when you push too hard.

The correct response is to honour Retry-After and back off exponentially — never hammer in a tight retry loop:

void ExecuteWithRetry(ClientContext ctx, int maxRetries = 5)
{
    int attempt = 0;
    int delay = 2000; // ms
    while (true)
    {
        try { ctx.ExecuteQuery(); return; }
        catch (WebException wex)
            when (IsThrottled(wex, out int retryAfter))
        {
            if (++attempt > maxRetries) throw;
            Thread.Sleep(retryAfter > 0 ? retryAfter * 1000 : delay);
            delay *= 2;   // exponential back-off
        }
    }
}

Takeaways

• Nothing executes until ExecuteQuery — batch aggressively, never call it in a loop.
• Load only the properties and fields you actually read.
• Page large lists with a row limit and ListItemCollectionPosition.
• Expect throttling on SharePoint Online; honour Retry-After with exponential back-off.

These habits scale from a quick console utility to a nightly sync job moving hundreds of thousands of items. The PnP Sites Core library (and later PnP PowerShell) bakes many of these patterns in for you — but knowing what they do under the hood is what lets you debug the day they don't.