Active:Active

Overview

The Active:Active feature provides automatic failover and intelligent routing across multiple Redis deployments. It is built from several cooperating pieces:

1. Connecting to multiple servers or regions at once, giving a deployment redundant endpoints to fall back on.
2. Health checks that actively probe each endpoint on a timer to monitor its availability.
3. Circuit breakers that passively monitor availability from the observed success and failure of the traffic already flowing.
4. Automatic retries that let straightforward operations ride out a possibly-unstable connection — including silently transitioning to another endpoint during a failover, with no change to your calling code.

The features for Active:Active are available in the Availability sub-namespace:

using StackExchange.Redis;
using StackExchange.Redis.Availability;

The library automatically selects the best available endpoint based on:

1. Availability - Connected endpoints are always preferred over disconnected ones
2. Weight - User-defined preference values (higher is better)
3. Latency - Measured response times (lower is better)

This enables scenarios such as:

• Multi-datacenter deployments with automatic failover
• Geographic routing to the nearest Redis instance
• Graceful degradation during maintenance or outages
• Load distribution across multiple Redis clusters

Basic Usage

Connecting to Multiple Groups

To create an Active:Active connection, use ConnectionMultiplexer.ConnectGroupAsync() with an array of ConnectionGroupMember instances:

// Define your Redis endpoints
ConnectionGroupMember[] members = [
    new("us-east.redis.example.com:6379", name: "US East"),
    new("us-west.redis.example.com:6379", name: "US West"),
    new("eu-central.redis.example.com:6379", name: "EU Central")
];

// Connect to all members
await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members);

// Use the connection normally
var db = conn.GetDatabase();
await db.StringSetAsync("mykey", "myvalue");
var value = await db.StringGetAsync("mykey");

Using ConfigurationOptions

You can also use ConfigurationOptions for more advanced configuration:

var eastConfig = new ConfigurationOptions
{
    EndPoints = { "us-east-1.redis.example.com:6379", "us-east-2.redis.example.com:6379" },
    Password = "your-password",
    Ssl = true,
};

var westConfig = new ConfigurationOptions
{
    EndPoints = { "us-west-1.redis.example.com:6379", "us-west-2.redis.example.com:6379" },
    Password = "another-different-password",
    Ssl = true,
};

ConnectionGroupMember[] members = [
    new(eastConfig, name: "US East"),
    new(westConfig, name: "US West")
];

await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members);

Configuring Weights

Weights allow you to express preference for specific endpoints. Higher weights are preferred when multiple endpoints are available:

ConnectionGroupMember[] members = [
    new("local-dc.redis.example.com:6379") { Weight = 10 },    // Strongly preferred
    new("nearby-dc.redis.example.com:6379") { Weight = 5 },    // Moderately preferred
    new("remote-dc.redis.example.com:6379") { Weight = 1 }     // Fallback option
];

await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members);

Weights can be adjusted dynamically:

// Adjust weight based on runtime conditions
members[0].Weight = 1;  // Reduce preference for local DC
members[2].Weight = 10; // Increase preference for remote DC

Working with IDatabase

The IDatabase interface works transparently with Active:Active connections. All operations are automatically routed to the currently selected endpoint:

var db = conn.GetDatabase();

// String operations
await db.StringSetAsync("user:1:name", "Alice");
var name = await db.StringGetAsync("user:1:name");

// Hash operations
await db.HashSetAsync("user:1", new HashEntry[] {
    new("name", "Alice"),
    new("email", "[email protected]")
});

// List operations
await db.ListRightPushAsync("queue:tasks", "task1");
var task = await db.ListLeftPopAsync("queue:tasks");

// Set operations
await db.SetAddAsync("tags", new RedisValue[] { "redis", "cache", "database" });
var members = await db.SetMembersAsync("tags");

// Sorted set operations
await db.SortedSetAddAsync("leaderboard", "player1", 100);
var rank = await db.SortedSetRankAsync("leaderboard", "player1");

// Transactions
var tran = db.CreateTransaction();
var t1 = tran.StringSetAsync("key1", "value1");
var t2 = tran.StringSetAsync("key2", "value2");
if (await tran.ExecuteAsync())
{
    await t1;
    await t2;
}

// Batches
var batch = db.CreateBatch();
var b1 = batch.StringSetAsync("key1", "value1");
var b2 = batch.StringSetAsync("key2", "value2");
batch.Execute();
await Task.WhenAll(b1, b2);

Working with ISubscriber

Pub/Sub operations work across all connected endpoints. When you subscribe to a channel, the subscription is established against all endpoints (for immediate pickup during failover events), and received messages are filtered in the library so only the messages for the active endpoint are observed. Message publishing occurs only to the active endpoint. The effect of this is that pub/sub works transparently as though you were only talking to the active endpoint:

var subscriber = conn.GetSubscriber();

// Subscribe to a channel
await subscriber.SubscribeAsync(RedisChannel.Literal("notifications"), (channel, message) =>
{
    Console.WriteLine($"Received: {message}");
});

// Publish to a channel
await subscriber.PublishAsync(RedisChannel.Literal("notifications"), "Hello, World!");

// Pattern-based subscriptions
await subscriber.SubscribeAsync(RedisChannel.Pattern("events:*"), (channel, message) =>
{
    Console.WriteLine($"Event on {channel}: {message}");
});

// Unsubscribe
await subscriber.UnsubscribeAsync(RedisChannel.Literal("notifications"));

Note: When the active endpoint changes (due to failover), subscriptions are automatically re-established on the new endpoint.

Monitoring Connection Changes

You can monitor when the active connection changes using the ConnectionChanged event:

conn.ConnectionChanged += (sender, args) =>
{
    Console.WriteLine($"Connection changed: {args.Type}");
    Console.WriteLine($"Previous: {args.PreviousGroup?.Name ?? "(none)"}");
    Console.WriteLine($"Current: {args.Group.Name}");
};

Monitoring Member Status

Each ConnectionGroupMember provides status information:

foreach (var member in conn.GetMembers())
{
    Console.WriteLine($"{member.Name}:");
    Console.WriteLine($"  Connected: {member.IsConnected}");
    Console.WriteLine($"  Unhealthy: {member.IsUnhealthy}");
    Console.WriteLine($"  Weight: {member.Weight}");
    Console.WriteLine($"  Latency: {member.Latency}");
}

These are the same instances that were passed into ConnectGroupAsync.

Health Checks

The Active:Active feature includes configurable health checking to monitor the health of all endpoints and automatically route traffic away from unhealthy instances.

Basic Health Check Configuration

Health checks are configured globally for all members using the MultiGroupOptions parameter:

var healthCheck = new HealthCheck
{
    Interval = TimeSpan.FromSeconds(5),          // How often to check health
    ProbeCount = 3,                              // Maximum number of probe attempts per check
    ProbeTimeout = TimeSpan.FromSeconds(3),      // Timeout for each probe attempt
    ProbeInterval = TimeSpan.FromMilliseconds(500), // Delay between failed probes
    Probe = HealthCheckProbe.Ping,               // Which probe type to use
    ProbePolicy = HealthCheckProbePolicy.AllSuccess // Evaluation policy
};

var options = new MultiGroupOptions
{
    HealthCheck = healthCheck
};

ConnectionGroupMember[] members = [
    new("us-east.redis.example.com:6379", name: "US East") { Weight = 10 },
    new("us-west.redis.example.com:6379", name: "US West") { Weight = 5 }
];

await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members, options);

Using Default Health Checks

If you don't specify a health check, the system uses sensible defaults:

// Uses default health check settings
await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members);

// Equivalent to:
var options = new MultiGroupOptions
{
    HealthCheck = HealthCheck.Default
};
await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members, options);

You can also clone and customize the default:

var customHealthCheck = HealthCheck.Default.Clone();
customHealthCheck.Interval = TimeSpan.FromSeconds(15);
customHealthCheck.ProbeCount = 5;

var options = new MultiGroupOptions
{
    HealthCheck = customHealthCheck
};

Health Check Properties

The HealthCheck class provides several configurable properties:

Built-in Probes

StackExchange.Redis provides several built-in health check probes:

HealthCheckProbe.Ping

The simplest probe that executes a PING command against the server:

var healthCheck = new HealthCheck
{
    Probe = HealthCheckProbe.Ping
};

This is the default and recommended probe for most scenarios as it's lightweight and tests basic connectivity.

HealthCheckProbe.IsConnected

Checks the connection status without sending any commands:

var healthCheck = new HealthCheck
{
    Probe = HealthCheckProbe.IsConnected
};

This is even more lightweight than Ping but only verifies the socket connection, not Redis responsiveness.

HealthCheckProbe.StringSet

Performs a write operation to verify read/write capability:

var healthCheck = new HealthCheck
{
    Probe = HealthCheckProbe.StringSet
};

This probe writes a random value to a health check key and verifies it can be retrieved. It's more comprehensive but has higher overhead than Ping. Note that this probe automatically skips replica servers.

Health Check Policies

The probe policy determines how multiple probe results are evaluated to determine overall health:

HealthCheckProbePolicy.AnySuccess

The health check passes if any probe succeeds. This provides the most lenient evaluation:

var healthCheck = new HealthCheck
{
    ProbeCount = 3,
    ProbePolicy = HealthCheckProbePolicy.AnySuccess
};
// Healthy if 1 or more of 3 probes succeed

HealthCheckProbePolicy.AllSuccess (Default)

The health check passes only if all probes succeed. This provides the strictest evaluation:

var healthCheck = new HealthCheck
{
    ProbeCount = 3,
    ProbePolicy = HealthCheckProbePolicy.AllSuccess
};
// Healthy only if all 3 probes succeed

HealthCheckProbePolicy.MajoritySuccess

The health check passes if a majority of probes succeed:

var healthCheck = new HealthCheck
{
    ProbeCount = 3,
    ProbePolicy = HealthCheckProbePolicy.MajoritySuccess
};
// Healthy if 2 or more of 3 probes succeed

Health Check Behavior

When a health check fails for a member:

• The member is flagged unhealthy (IsUnhealthy); this is distinct from IsConnected (see Unhealthy State and Failback below)
• Traffic is automatically routed to other healthy members based on weight and latency
• The system continues to perform health checks on the unhealthy member
• Once the member recovers and passes health checks, traffic automatically resumes (subject to FailbackDelay)

Best Practices

1. Choose appropriate probe types: Use Ping for most scenarios; use StringSet when you need to verify write capability
2. Balance probe frequency: More frequent checks provide faster failover but increase load on your Redis servers
3. Match policy to requirements: Use AnySuccess for resilience, AllSuccess for strict validation, MajoritySuccess for balance
4. Increase probe count for critical systems: More probes with MajoritySuccess reduces false positives from transient failures
5. Set reasonable timeouts: Ensure ProbeTimeout accounts for network latency to your Redis servers
6. Consider replica behavior: Write-based probes automatically skip replicas to avoid false negatives

Circuit Breakers

Where health checks actively probe each member on a timer, a circuit breaker works passively: it observes the outcome of the normal traffic already flowing over a connection, and tears that connection down as soon as it decides the connection has become unstable. The two are complementary:

• A health check answers "is this member reachable?" by sending its own probes every Interval.
• A circuit breaker answers "is the traffic I'm already sending actually succeeding?" with no extra round-trips, reacting the instant real commands start failing.

When a circuit breaker trips, it shuts the underlying physical connection down with ConnectionFailureType.CircuitBreaker. The connection then reconnects as usual, and — in an Active:Active group — the health check and member-selection logic route traffic to other members until the affected member recovers. A circuit breaker is evaluated per connection, so its state is scoped to exactly the connection whose health it is measuring; a reconnect starts from a clean slate - but breaking the connection is sufficient for Active:Active to notice non-availability.

Configuring a Circuit Breaker for a Group

Circuit breakers are configured globally for all members via MultiGroupOptions, alongside the health check. The setting flows into every member connection:

var options = new MultiGroupOptions
{
    CircuitBreaker = CircuitBreaker.Default
};

ConnectionGroupMember[] members = [
    new("us-east.redis.example.com:6379", name: "US East") { Weight = 10 },
    new("us-west.redis.example.com:6379", name: "US West") { Weight = 5 }
];

await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members, options);

If you don't specify one, CircuitBreaker.Default is used automatically:

// these are equivalent
await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members);

var options = new MultiGroupOptions { CircuitBreaker = CircuitBreaker.Default };
await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members, options);

Tuning the Default Circuit Breaker

The default circuit breaker uses a rolling time-window: it counts successes and failures over a short window and trips once the failure rate crosses a threshold, provided enough failures have been seen to be statistically meaningful. Use CircuitBreaker.Builder to tune it:

var options = new MultiGroupOptions
{
    CircuitBreaker = new CircuitBreaker.Builder
    {
        FailureRateThreshold = 25,                       // trip above 25% failures
        MinimumNumberOfFailures = 100,                   // ...but only after 100 failures in the window
        MetricsWindowSize = TimeSpan.FromSeconds(5),     // rolling window to measure over
    }
};

A Builder converts implicitly to a CircuitBreaker, so it can be assigned directly as above; calling .Create() explicitly is equivalent.

Which faults count against the breaker is decided by classification, not by exception type: transient and connection-level errors (and timeouts) count as failures, while application-level errors — for example a RedisServerException for a bad command, or a WRONGTYPE — are treated as a success for circuit-breaking purposes, since they are not a sign of an unhealthy connection. This is derived from the fault's RedisErrorKind; to change what counts, override IsFailure(in FaultContext) on a custom accumulator (see Custom circuit breakers below).

Disabling the Circuit Breaker

Use CircuitBreaker.None to opt out entirely:

var options = new MultiGroupOptions
{
    CircuitBreaker = CircuitBreaker.None
};

Automatic Retries

Health checks and circuit breakers keep the group pointed at a healthy member; automatic retries deal with the individual operation that was in flight when something went wrong. Wrapping a database with WithRetry(...) returns a database that transparently re-issues failed operations according to a RetryPolicy — riding out transient faults, and (in an Active:Active group) following a failover across to another member, without the caller having to catch-and-retry by hand.

await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members);

// wrap the database once; reuse the wrapper like any other IDatabaseAsync
IDatabaseAsync db = conn.GetDatabase().WithRetry(new RetryPolicy());

// a transient fault (e.g. the active member briefly returning LOADING) is retried
// automatically; if the group fails over in the meantime, the retry lands on the new member
var value = await db.StringGetAsync("mykey");

You can call WithRetry on any database (IDatabase or IDatabaseAsync), but the wrapper it returns exposes only the async API — there is no synchronous form, since retrying may inherently have delays. It cannot wrap a batch or transaction, nor an already-retrying database.

RetryPolicy settings

RetryPolicy controls how many times, how often, and how far an operation is retried:

var policy = new RetryPolicy
{
    MaxAttempts = 5,
    RetryDelay = TimeSpan.FromMilliseconds(200),
    JitterMax = TimeSpan.FromMilliseconds(100),
};
IDatabaseAsync db = conn.GetDatabase().WithRetry(policy);

Only transient faults are retried — the same RedisErrorKind-based classification the default circuit breaker uses. An application-level error such as WRONGTYPE, or an unknown command, is not transient and is never retried, no matter what the policy says.

Which operations are safe to retry

Retrying is not free of consequence: replaying INCR after an ambiguous failure could double-count, whereas replaying GET is harmless; SET is "last wins", so: usally fine. Every command therefore carries a retry category describing its side-effects, and a policy only retries commands at or below its MaxCommandRetryCategory.

For the built-in typed methods (StringGet, StringSet, HashSet, ...) the library assigns the appropriate category automatically, so retries "just work" within the default policy.

Custom commands: Execute and ScriptEvaluate

The library cannot infer the side-effects of a command it doesn't recognise — and that includes arbitrary commands issued via Execute/ExecuteAsync, and Lua run via ScriptEvaluate/ScriptEvaluateAsync (whose effect depends entirely on the script). Such commands are therefore treated pessimistically: an uncategorised command defaults to CommandRetryNever and is not retried.

The categories, from safest to most dangerous, are:

When possible when using ad-hoc commands or script, callers should supply the most appropriate CommandRetry* category in the command's CommandFlags:

// an arbitrary read-only command: safe to retry
var result = await db.ExecuteAsync("LOLWUT", args: [], flags: CommandFlags.CommandRetryReadOnly);

// a Lua script that only reads: opt into retries
var value = await db.ScriptEvaluateAsync(
    "return redis.call('GET', KEYS[1])",
    keys: [key],
    flags: CommandFlags.CommandRetryReadOnly);

Choose the category honestly — it describes what a replay would do. If a retry could double-apply a side-effect, use CommandRetryWriteAccumulating (or leave it uncategorised) rather than claiming it's a read. Conversely, if you want more-side-effecting operations retried across the board, raise the policy's MaxCommandRetryCategory instead of tagging each call.

Unhealthy State and Failback

Each member tracks two independent pieces of state:

• IsConnected — the last observed connectivity of the underlying connection.
• IsUnhealthy — whether the member has been disabled by a failing health-check or a tripped circuit breaker.

A member is only eligible to be selected as the active member when it is connected and not unhealthy. Separating the two lets a member that is technically reconnected still be held out of rotation until we're confident it is stable again — which is what FailbackDelay controls.

How a member becomes unhealthy

• A health check returns Unhealthy for it.
• Its circuit breaker trips (ConnectionFailureType.CircuitBreaker). The failing member is flagged unhealthy immediately on the circuit-breaker fast-path, so traffic routes away without waiting for the next health-check tick.

Each time a member is (re)marked unhealthy, the time of that failure is recorded.

How a member becomes healthy again

An unhealthy member is cleared in one of three ways:

1. Automatically, once it passes a health check and its most recent failure is older than FailbackDelay (see below).
2. Explicitly, by calling member.ResetIsUnhealthy().
3. Implicitly, by calling IConnectionGroup.TryFailoverTo(member) — an explicit failover request always clears the target's unhealthy flag first, since the caller is deliberately asking for that member.

FailbackDelay

MultiGroupOptions.FailbackDelay is the interval a member must remain healthy — measured from its most recent failure — before it is automatically returned to rotation:

var options = new MultiGroupOptions
{
    FailbackDelay = TimeSpan.FromMinutes(2), // must be healthy for 2 minutes after its last failure
};

This guards against flapping: a member that is intermittently failing will keep pushing its "last failure" time forward, so it never satisfies the delay and stays out of rotation until it is genuinely stable.

Implementation note: the failback check is pure tick math on the wall clock — the last-failure time and the cutoff (UtcNow - FailbackDelay) are both compared as raw long UTC ticks, so DateTimeKind never enters into it. (DateTime.Ticks and TimeSpan.Ticks share the same 100 ns unit.)

Anti-flap tiebreak in selection

Member selection ranks candidates by connectivity, explicit override, weight, and latency. When two candidates are otherwise indistinguishable, selection prefers the member that is already active, rather than picking arbitrarily.

Manual Failover

In some scenarios, you may need to manually control which member is actively serving traffic, overriding the automatic selection based on weight and latency. The TryFailoverTo method allows you to explicitly switch to a specific member or restore automatic selection.

Basic Failover to a Specific Member

await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members);

// Get the members to find the one you want to fail over to
var groupMembers = conn.GetMembers();
var targetMember = groupMembers.FirstOrDefault(m => m.Name == "US West");

if (targetMember != null)
{
    // Attempt to fail over to the specified member
    bool success = conn.TryFailoverTo(targetMember);

    if (success)
    {
        Console.WriteLine($"Successfully failed over to {targetMember.Name}");
    }
    else
    {
        Console.WriteLine($"Failed to fail over to {targetMember.Name} (member may be disconnected)");
    }
}

Restore Automatic Selection

To remove an explicit failover and return to automatic member selection based on weight and latency:

// Pass null to remove the explicit failover
bool hadExplicitFailover = conn.TryFailoverTo(null);

if (hadExplicitFailover)
{
    Console.WriteLine("Removed explicit failover, now using automatic selection");
}
else
{
    Console.WriteLine("No explicit failover was active");
}

Failover Behavior

The TryFailoverTo method has the following behavior:

• Returns true: The failover was successful and the specified member is now active (or an explicit override was successfully removed)
• Returns false: The failover failed because:
  • The member is not connected
  • The member is not part of this connection group

When an explicit failover is active:

• The specified member will be preferred for all traffic
• Weight and latency are ignored for member selection
• If the explicitly selected member becomes unavailable, the system automatically falls back to other connected members
• Health checks continue to run on all members

Example: Maintenance Mode

This is particularly useful when performing maintenance on one region and you want to temporarily route all traffic to another:

var members = new ConnectionGroupMember[]
{
    new("us-east.redis.example.com:6379", name: "US East") { Weight = 100 },
    new("us-west.redis.example.com:6379", name: "US West") { Weight = 100 }
};

await using var conn = await ConnectionMultiplexer.ConnectGroupAsync(members);

// During maintenance on US East, explicitly route to US West
var westMember = conn.GetMembers().First(m => m.Name == "US West");
if (conn.TryFailoverTo(westMember))
{
    Console.WriteLine("Traffic now routed to US West for maintenance");
}

// ... perform maintenance on US East ...

// After maintenance, restore automatic selection
if (conn.TryFailoverTo(null))
{
    Console.WriteLine("Maintenance complete, automatic selection restored");
}

Example: Monitoring Failover Events

You can monitor when failovers occur using the ConnectionChanged event:

conn.ConnectionChanged += (sender, args) =>
{
    if (args.Type == GroupConnectionChangedEventArgs.ChangeType.ActiveChanged)
    {
        Console.WriteLine($"Active member changed from {args.PreviousGroup?.Name ?? "none"} to {args.Group.Name}");
    }
};

// Trigger an explicit failover
var member = conn.GetMembers().First(m => m.Name == "Backup");
conn.TryFailoverTo(member);
// Event will fire: "Active member changed from Primary to Backup"

Important Notes

1. Connection Required: You can only fail over to a member that is currently connected (IsConnected == true)
2. Temporary Override: The explicit failover persists until:
   • You call TryFailoverTo(null) to remove it
   • The connection group is disposed
   • The explicitly selected member becomes disconnected (automatic fallback occurs)
3. Not Persistent: Explicit failovers are not persisted across application restarts
4. Thread-Safe: TryFailoverTo is thread-safe and can be called concurrently with normal operations

Dynamic Member Management

You can add or remove members dynamically using the IConnectionGroup interface:

// Cast to IConnectionGroup to access dynamic member management
var group = (IConnectionGroup)conn;

// Add a new member at runtime
var newMember = new ConnectionGroupMember("new-dc.redis.example.com:6379", name: "New Datacenter")
{
    Weight = 5
};
await group.AddAsync(newMember);
Console.WriteLine($"Added {newMember.Name} to the group");

// Remove a member
var memberToRemove = members[2]; // Reference to an existing member
if (group.Remove(memberToRemove))
{
    Console.WriteLine($"Removed {memberToRemove.Name} from the group");
}
else
{
    Console.WriteLine($"Failed to remove {memberToRemove.Name} - member not found");
}

// Check current members
var currentMembers = group.GetMembers();
Console.WriteLine($"Current member count: {currentMembers.Length}");
foreach (var member in currentMembers)
{
    Console.WriteLine($"  - {member.Name} (Weight: {member.Weight}, Connected: {member.IsConnected})");
}

Adding Members During Maintenance

Add a new datacenter before removing an old one for zero-downtime migrations:

var group = (IConnectionGroup)conn;

// Add the new datacenter
var newDC = new ConnectionGroupMember("new-location.redis.example.com:6379", name: "New Location")
{
    Weight = 10 // High weight to prefer the new location
};
await group.AddAsync(newDC);

// Wait for the new member to be fully connected and healthy
await Task.Delay(TimeSpan.FromSeconds(5));

if (newDC.IsConnected)
{
    Console.WriteLine("New datacenter is online and healthy");

    // Reduce weight of old datacenter
    var oldDC = members[0];
    oldDC.Weight = 1;

    // Wait for traffic to shift
    await Task.Delay(TimeSpan.FromSeconds(10));

    // Remove the old datacenter
    if (group.Remove(oldDC))
    {
        Console.WriteLine("Old datacenter removed successfully");
    }
}

Advanced Customization

The building blocks above cover the common cases. The extension points below let you replace the default health-check, retry, and circuit-breaker behavior with your own logic; most applications will not need them.

Custom Health Check Probes

You can implement custom health check logic by extending HealthCheckProbe. Note that care must be used if the probe involves talking to data via a RedisKey, as on "cluster" configurations, it must be ensured that the key used resolves to the correct server; for this purpose, the server.InventKey method can be used:

public abstract class CustomProbe : HealthCheckProbe
{
    public override Task<HealthCheckResult> CheckHealthAsync(HealthCheck healthCheck, IServer server)
    {
        // create a random key that routes to the correct server, using
        // the specified prefix
        RedisKey key = server.InventKey("health-check/");
        // ...
    }
}

Or more conveniently, the key-specific KeyWriteHealthCheckProbe encapsulates this logic:

public class CustomWriteProbe : KeyWriteHealthCheckProbe
{
    public override async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheck healthCheck,
        IDatabaseAsync database,
        RedisKey key)
    {
        try
        {
            var value = Guid.NewGuid().ToString();
            await database.StringSetAsync(key, value, expiry: healthCheck.ProbeTimeout);
            bool isMatch = value == await database.StringGetAsync(key);

            return isMatch ? HealthCheckResult.Healthy : HealthCheckResult.Unhealthy;
        }
        catch
        {
            return HealthCheckResult.Unhealthy;
        }
    }
}

Custom Probe Policies

In addition to the inbuilt policies, custom policies can be implemented by extending HealthCheckProbePolicy. By checking the properties of the HealthCheckProbeContext parameter, your policy can make a determination about the health of the server - returning HealthCheckResult.Healthy or HealthCheckResult.Unhealthy as appropriate. If you return HealthCheckResult.Inconclusive, the health check will continue with additional probes.

Example: Require at Least N Successes

This example demonstrates a policy that requires at least a specified number of successful probes before declaring the endpoint healthy:

public class AtLeastPolicy(int requiredSuccesses) : HealthCheckProbePolicy
{
    public override HealthCheckResult Evaluate(in HealthCheckProbeContext context)
    {
        // Success if we have at least the required number of successful probes
        if (context.Success >= requiredSuccesses) return HealthCheckResult.Healthy;

        // If no more probes remaining, we haven't met our threshold; otherwise: keep trying
        return context.Remaining == 0 ? HealthCheckResult.Unhealthy : HealthCheckResult.Inconclusive;
    }
}

// Use the custom policy requiring at least 2 successes
var healthCheck = new HealthCheck
{
    ProbeCount = 5,  // Need enough probes to allow for the required successes
    ProbePolicy = new AtLeastPolicy(2)
};

var options = new MultiGroupOptions
{
    HealthCheck = healthCheck
};

This policy ensures that transient successes don't immediately mark an endpoint as healthy. It requires at least the specified number of successful probes, which provides better confidence in the endpoint's stability while still being more lenient than AllSuccess.

Custom Circuit Breakers

This is an advanced extension point; most applications should use CircuitBreaker.Default (optionally tuned via CircuitBreaker.Builder) or CircuitBreaker.None.

You can implement your own policy by extending CircuitBreaker and its Accumulator. CreateAccumulator() is called once per underlying connection, so each accumulator holds the state for a single connection. For every completed message the connection calls ObserveResult, passing a FaultContext that describes the outcome: IsFault indicates whether a fault occurred, with the associated Fault, ErrorKind and ConnectionFailureType available for inspection. IsHealthy() is consulted separately: return true while the connection should be considered healthy, or false to trip the breaker and tear the connection down.

By default only faults that IsFailure regards as genuine failures reach ObserveResult as failures (transient/connection errors, timeouts, and similar - classified from the fault's ErrorKind); everything else - including application-level errors such as a bad command - is passed as a success. Override IsFailure(in FaultContext) if you need different rules for what counts:

public sealed class ConsecutiveFailureBreaker(int limit) : CircuitBreaker
{
    public override Accumulator CreateAccumulator() => new Acc(limit);

    private sealed class Acc(int limit) : Accumulator
    {
        private int _consecutiveFailures;

        // a fault is present iff context.IsFault; a success resets the run
        public override void ObserveResult(in FaultContext context)
        {
            if (context.IsFault)
            {
                _consecutiveFailures++;
            }
            else
            {
                _consecutiveFailures = 0;
            }
        }

        // healthy until we hit the configured run of consecutive failures
        public override bool IsHealthy() => _consecutiveFailures < limit;

        // called if the connection wants to discard accumulated history
        public override void Reset() => _consecutiveFailures = 0;

        // (optional) override to change what counts as a failure; the default classifies from ErrorKind
        // protected override bool IsFailure(in FaultContext fault) => fault.IsFault;
    }
}

var options = new MultiGroupOptions
{
    CircuitBreaker = new ConsecutiveFailureBreaker(limit: 5)
};

Keep ObserveResult cheap and thread-safe: it runs on the hot path for every completed message, and may be called concurrently.

Custom Retry Policies

RetryPolicy is itself extensible: override CanRetry(in FaultContext fault) to make the retry decision yourself. It returns a RetryPolicyResult — None to give up, or a combination of SameServer and FailoverServer to indicate where a retry may be attempted. The FaultContext gives you the classified ErrorKind, the ConnectionFailureType, and the command Flags (including its retry category) to base the decision on:

public sealed class ReadOnlyOnlyRetryPolicy : RetryPolicy
{
    public override RetryPolicyResult CanRetry(in FaultContext fault)
    {
        // only ever retry pure reads, and only on the same server
        if (fault.ErrorKind == RedisErrorKind.Loading
            && (fault.Flags & CommandFlags.CommandRetryReadOnly) != 0)
        {
            return RetryPolicyResult.SameServer;
        }
        // note: base.CanRetry(fault) would apply the default logic
        return RetryPolicyResult.None;
    }
}

IDatabaseAsync db = conn.GetDatabase().WithRetry(new ReadOnlyOnlyRetryPolicy());