gRPC client-side load balancing

[!INCLUDE]

By James Newton-King

Client-side load balancing is a feature that allows gRPC clients to distribute load optimally across available servers. This article discusses how to configure client-side load balancing to create scalable, high-performance gRPC apps in .NET.

Client-side load balancing requires:

• .NET 5 or later.
• Grpc.Net.Client version 2.45.0 or later.

Configure gRPC client-side load balancing

Client-side load balancing is configured when a channel is created. The two components to consider when using load balancing:

• The resolver, which resolves the addresses for the channel. Resolvers support getting addresses from an external source. This is also known as service discovery.
• The load balancer, which creates connections and picks the address that a gRPC call will use.

Built-in implementations of resolvers and load balancers are included in Grpc.Net.Client. Load balancing can also be extended by writing custom resolvers and load balancers.

Addresses, connections and other load balancing state is stored in a GrpcChannel instance. A channel must be reused when making gRPC calls for load balancing to work correctly.

[!NOTE] Some load balancing configuration uses dependency injection (DI). Apps that don't use DI can create a xref:Microsoft.Extensions.DependencyInjection.ServiceCollection instance.

If an app already has DI setup, like an ASP.NET Core website, then types should be registered with the existing DI instance. GrpcChannelOptions.ServiceProvider is configured by getting an xref:System.IServiceProvider from DI.

Configure resolver

The resolver is configured using the address a channel is created with. The URI scheme of the address specifies the resolver.

A channel doesn't directly call a URI that matches a resolver. Instead, a matching resolver is created and used to resolve the addresses.

For example, using GrpcChannel.ForAddress("dns:///my-example-host", new GrpcChannelOptions { Credentials = ChannelCredentials.Insecure }):

• The dns scheme maps to DnsResolverFactory. A new instance of a DNS resolver is created for the channel.
• The resolver makes a DNS query for my-example-host and gets two results: 127.0.0.100 and 127.0.0.101.
• The load balancer uses 127.0.0.100:80 and 127.0.0.101:80 to create connections and make gRPC calls.

DnsResolverFactory

The DnsResolverFactory creates a resolver designed to get addresses from an external source. DNS resolution is commonly used to load balance over pod instances that have a Kubernetes headless services.

var channel = GrpcChannel.ForAddress(
    "dns:///my-example-host",
    new GrpcChannelOptions { Credentials = ChannelCredentials.Insecure });
var client = new Greet.GreeterClient(channel);

var response = await client.SayHelloAsync(new HelloRequest { Name = "world" });

The preceding code:

• Configures the created channel with the address dns:///my-example-host.
  • The dns scheme maps to DnsResolverFactory.
  • my-example-host is the hostname to resolve.
  • No port is specified in the address, so gRPC calls are sent to port 80. This is the default port for unsecured channels. A port can optionally be specified after the hostname. For example, dns:///my-example-host:8080 configures gRPC calls to be sent to port 8080.
• Doesn't specify a load balancer. The channel defaults to a pick first load balancer.
• Starts the gRPC call SayHello:
  • DNS resolver gets addresses for the hostname my-example-host.
  • Pick first load balancer attempts to connect to one of the resolved addresses.
  • The call is sent to the first address the channel successfully connects to.

DNS address caching

Performance is important when load balancing. The latency of resolving addresses is eliminated from gRPC calls by caching the addresses. A resolver will be invoked when making the first gRPC call, and subsequent calls use the cache.

Addresses are automatically refreshed if a connection is interrupted. Refreshing is important in scenarios where addresses change at runtime. For example, in Kubernetes a restarted pod triggers the DNS resolver to refresh and get the pod's new address.

By default, a DNS resolver is refreshed if a connection is interrupted. The DNS resolver can also optionally refresh itself on a periodic interval. This can be useful for quickly detecting new pod instances.

services.AddSingleton<ResolverFactory>(
    sp => new DnsResolverFactory(refreshInterval: TimeSpan.FromSeconds(30)));

The preceding code creates a DnsResolverFactory with a refresh interval and registers it with dependency injection. For more information on using a custom-configured resolver, see Configure custom resolvers and load balancers.

StaticResolverFactory

A static resolver is provided by StaticResolverFactory. This resolver:

• Doesn't call an external source. Instead, the client app configures the addresses.
• Is designed for situations where an app already knows the addresses it calls.

var factory = new StaticResolverFactory(addr => new[]
{
    new BalancerAddress("localhost", 80),
    new BalancerAddress("localhost", 81)
});

var services = new ServiceCollection();
services.AddSingleton<ResolverFactory>(factory);

var channel = GrpcChannel.ForAddress(
    "static:///my-example-host",
    new GrpcChannelOptions
    {
        Credentials = ChannelCredentials.Insecure,
        ServiceProvider = services.BuildServiceProvider()
    });
var client = new Greet.GreeterClient(channel);

The preceding code:

• Creates a StaticResolverFactory. This factory knows about two addresses: localhost:80 and localhost:81.
• Registers the factory with dependency injection (DI).
• Configures the created channel with:
  • The address static:///my-example-host. The static scheme maps to a static resolver.
  • Sets GrpcChannelOptions.ServiceProvider with the DI service provider.

This example creates a new xref:Microsoft.Extensions.DependencyInjection.ServiceCollection for DI. Suppose an app already has DI setup, like an ASP.NET Core website. In that case, types should be registered with the existing DI instance. GrpcChannelOptions.ServiceProvider is configured by getting an xref:System.IServiceProvider from DI.

Configure load balancer

A load balancer is specified in a service config using the ServiceConfig.LoadBalancingConfigs collection. Two load balancers are built-in and map to load balancer config names:

service config is an abbreviation of service configuration and is represented by the ServiceConfig type. There are a couple of ways a channel can get a service config with a load balancer configured:

• An app can specify a service config when a channel is created using GrpcChannelOptions.ServiceConfig.
• Alternatively, a resolver can resolve a service config for a channel. This feature allows an external source to specify how its callers should perform load balancing. Whether a resolver supports resolving a service config is dependent on the resolver implementation. Disable this feature with GrpcChannelOptions.DisableResolverServiceConfig.
• If no service config is provided, or the service config doesn't have a load balancer configured, the channel defaults to PickFirstLoadBalancerFactory.

var channel = GrpcChannel.ForAddress(
    "dns:///my-example-host",
    new GrpcChannelOptions
    {
        Credentials = ChannelCredentials.Insecure,
        ServiceConfig = new ServiceConfig { LoadBalancingConfigs = { new RoundRobinConfig() } }
    });
var client = new Greet.GreeterClient(channel);

var response = await client.SayHelloAsync(new HelloRequest { Name = "world" });

The preceding code:

• Specifies a RoundRobinLoadBalancerFactory in the service config.
• Starts the gRPC call SayHello:
  • DnsResolverFactory creates a resolver that gets addresses for the hostname my-example-host.
  • Round-robin load balancer attempts to connect to all resolved addresses.
  • gRPC calls are distributed evenly using round-robin logic.

Configure channel credentials

A channel must know whether gRPC calls are sent using transport security. http and https are no longer part of the address, the scheme now specifies a resolver, so Credentials must be configured on channel options when using load balancing.

• ChannelCredentials.SecureSsl - gRPC calls are secured with Transport Layer Security (TLS). Equivalent to an https address.
• ChannelCredentials.Insecure - gRPC calls don't use transport security. Equivalent to an http address.

var channel = GrpcChannel.ForAddress(
    "dns:///my-example-host",
    new GrpcChannelOptions { Credentials = ChannelCredentials.Insecure });
var client = new Greet.GreeterClient(channel);

var response = await client.SayHelloAsync(new HelloRequest { Name = "world" });

Use load balancing with gRPC client factory

gRPC client factory can be configured to use load balancing:

var builder = WebApplication.CreateBuilder(args);

builder.Services
    .AddGrpcClient<Greeter.GreeterClient>(o =>
    {
        o.Address = new Uri("dns:///my-example-host");
    })
    .ConfigureChannel(o => o.Credentials = ChannelCredentials.Insecure);

builder.Services.AddSingleton<ResolverFactory>(
    sp => new DnsResolverFactory(refreshInterval: TimeSpan.FromSeconds(30)));

var app = builder.Build();

The preceding code:

• Configures the client with a load-balancing address.
• Specifies channel credentials.
• Registers DI types with the app's xref:Microsoft.Extensions.DependencyInjection.IServiceCollection.

Write custom resolvers and load balancers

Client-side load balancing is extensible:

• Implement Resolver to create a custom resolver and resolve addresses from a new data source.
• Implement LoadBalancer to create a custom load balancer with new load balancing behavior.

[!IMPORTANT] The APIs used to extend client-side load balancing are experimental. They can change without notice.

Create a custom resolver

A resolver:

• Implements Resolver and is created by a ResolverFactory. Create a custom resolver by implementing these types.
• Is responsible for resolving the addresses a load balancer uses.
• Can optionally provide a service configuration.

public class FileResolver : PollingResolver
{
    private readonly Uri _address;
    private readonly int _port;

    public FileResolver(Uri address, int defaultPort, ILoggerFactory loggerFactory)
        : base(loggerFactory)
    {
        _address = address;
        _port = defaultPort;
    }

    public override async Task ResolveAsync(CancellationToken cancellationToken)
    {
        // Load JSON from a file on disk and deserialize into endpoints.
        var jsonString = await File.ReadAllTextAsync(_address.LocalPath);
        var results = JsonSerializer.Deserialize<string[]>(jsonString);
        var addresses = results.Select(r => new BalancerAddress(r, _port)).ToArray();

        // Pass the results back to the channel.
        Listener(ResolverResult.ForResult(addresses));
    }
}

public class FileResolverFactory : ResolverFactory
{
    // Create a FileResolver when the URI has a 'file' scheme.
    public override string Name => "file";

    public override Resolver Create(ResolverOptions options)
    {
        return new FileResolver(options.Address, options.DefaultPort, options.LoggerFactory);
    }
}

In the preceding code:

• FileResolverFactory implements ResolverFactory. It maps to the file scheme and creates FileResolver instances.
• FileResolver implements PollingResolver. PollingResolver is an abstract base type that makes it easy to implement a resolver with asynchronous logic by overriding ResolveAsync.
• In ResolveAsync:
  • The file URI is converted to a local path. For example, file:///c:/addresses.json becomes c:\addresses.json.
  • JSON is loaded from disk and converted into a collection of addresses.
  • Listener is called with results to let the channel know that addresses are available.

Create a custom load balancer

A load balancer:

• Implements LoadBalancer and is created by a LoadBalancerFactory. Create a custom load balancer and factory by implementing these types.
• Is given addresses from a resolver and creates Subchannel instances.
• Tracks state about the connection and creates a SubchannelPicker. The channel internally uses the picker to pick addresses when making gRPC calls.

The SubchannelsLoadBalancer is:

• An abstract base class that implements LoadBalancer.
• Manages creating Subchannel instances from addresses.
• Makes it easy to implement a custom picking policy over a collection of subchannels.

public class RandomBalancer : SubchannelsLoadBalancer
{
    public RandomBalancer(IChannelControlHelper controller, ILoggerFactory loggerFactory)
        : base(controller, loggerFactory)
    {
    }

    protected override SubchannelPicker CreatePicker(List<Subchannel> readySubchannels)
    {
        return new RandomPicker(readySubchannels);
    }

    private class RandomPicker : SubchannelPicker
    {
        private readonly List<Subchannel> _subchannels;

        public RandomPicker(List<Subchannel> subchannels)
        {
            _subchannels = subchannels;
        }

        public override PickResult Pick(PickContext context)
        {
            // Pick a random subchannel.
            return PickResult.ForSubchannel(_subchannels[Random.Shared.Next(0, _subchannels.Count)]);
        }
    }
}

public class RandomBalancerFactory : LoadBalancerFactory
{
    // Create a RandomBalancer when the name is 'random'.
    public override string Name => "random";

    public override LoadBalancer Create(LoadBalancerOptions options)
    {
        return new RandomBalancer(options.Controller, options.LoggerFactory);
    }
}

In the preceding code:

• RandomBalancerFactory implements LoadBalancerFactory. It maps to the random policy name and creates RandomBalancer instances.
• RandomBalancer implements SubchannelsLoadBalancer. It creates a RandomPicker that randomly picks a subchannel.

Configure custom resolvers and load balancers

Custom resolvers and load balancers need to be registered with dependency injection (DI) when they are used. There are a couple of options:

• If an app is already using DI, such as an ASP.NET Core web app, they can be registered with the existing DI configuration. An xref:System.IServiceProvider can be resolved from DI and passed to the channel using GrpcChannelOptions.ServiceProvider.
• If an app isn't using DI then create:
  • A xref:Microsoft.Extensions.DependencyInjection.ServiceCollection with types registered with it.
  • A service provider using xref:Microsoft.Extensions.DependencyInjection.ServiceCollectionContainerBuilderExtensions.BuildServiceProvider*.

var services = new ServiceCollection();
services.AddSingleton<ResolverFactory, FileResolverFactory>();
services.AddSingleton<LoadBalancerFactory, RandomLoadBalancerFactory>();

var channel = GrpcChannel.ForAddress(
    "file:///c:/data/addresses.json",
    new GrpcChannelOptions
    {
        Credentials = ChannelCredentials.Insecure,
        ServiceConfig = new ServiceConfig { LoadBalancingConfigs = { new LoadBalancingConfig("random") } },
        ServiceProvider = services.BuildServiceProvider()
    });
var client = new Greet.GreeterClient(channel);

The preceding code:

• Creates a ServiceCollection and registers new resolver and load balancer implementations.
• Creates a channel configured to use the new implementations:
  • ServiceCollection is built into an IServiceProvider and set to GrpcChannelOptions.ServiceProvider.
  • Channel address is file:///c:/data/addresses.json. The file scheme maps to FileResolverFactory.
  • service config load balancer name is random. Maps to RandomLoadBalancerFactory.

Why load balancing is important

HTTP/2 multiplexes multiple calls on a single TCP connection. If gRPC and HTTP/2 are used with a network load balancer (NLB), the connection is forwarded to a server, and all gRPC calls are sent to that one server. The other server instances on the NLB are idle.

Network load balancers are a common solution for load balancing because they are fast and lightweight. For example, Kubernetes by default uses a network load balancer to balance connections between pod instances. However, network load balancers are not effective at distributing load when used with gRPC and HTTP/2.

Proxy or client-side load balancing?

gRPC and HTTP/2 can be effectively load balanced using either an application load balancer proxy or client-side load balancing. Both of these options allow individual gRPC calls to be distributed across available servers. Deciding between proxy and client-side load balancing is an architectural choice. There are pros and cons for each.

• Proxy: gRPC calls are sent to the proxy, the proxy makes a load balancing decision, and the gRPC call is sent on to the final endpoint. The proxy is responsible for knowing about endpoints. Using a proxy adds:

  • An additional network hop to gRPC calls.
  • Latency and consumes additional resources.
  • Proxy server must be setup and configured correctly.

• Client-side load balancing: The gRPC client makes a load balancing decision when a gRPC call is started. The gRPC call is sent directly to the final endpoint. When using client-side load balancing:

  • The client is responsible for knowing about available endpoints and making load balancing decisions.
  • Additional client configuration is required.
  • High-performance, load balanced gRPC calls eliminate the need for a proxy.

Additional resources

• xref:grpc/client