Leader election on k8s

The problem

Sometimes there are situations on which you have an application and you need to run multiple instances of it, however there is some particular task (or set of tasks) that you would like only one instance to execute. The reasons may be many:

• to ensure consistency
• to ensure order of processing
• to avoid concurrency
• to save resources
• to avoid redundancy and potential noise
• etc

If you find yourself facing such need, how do you address it? One option would be to segregate that task into a particular side-kick service, with a single instance and which single purpose would be to address that task (or all the tasks following same requirements). Although this may actually be an option in some particular cases, most of the times is just too cumbersome and unpractical.

So what is the alternative? Simple – a process of leader election.

The conceptual solution

Although it may sound somehow complex, leader election is quite a simple concept. There are several different implementations for it, but they all follow the exact same principle.

The idea is that you have several candidates (instances) and you subject them through a process of election, from which one (and only one) will gain the role of leader (the one who executes the task). Nothing more than this.

So how is such a process implemented? Usually resorting to some resource on which they will all try to put a lock on. Only one is capable of holding the lock and at a time and whoever does it is granted the role of leader. The underlying resource can be anything you can put a lock on, such as a file, a DB record, etc.

So what if the leader goes into a bad state? How do we ensure that the tasks are still run? Well, to avoid this type of issues usually there is an health check. What it does is to periodically revalidate that the leader is ok, ensuring someone is performing the tasks. If the leader doesn’t properly revalidate his role, the leadership will be revoked and a new election will occur, from which a new leader will raise.

Spring Cloud k8s Leader Election

On the particular context of running a Spring service on k8s, one practical solution for leader election is to resort to the Spring Cloud k8s Leader Election implementation.

In this implementation, the underlying resource is a k8s ConfigMap. The lock will be hold on it and the name of the pod, that is currently the leader, will figure as value for a key in it.

In order to use this, you just need to import the following dependency:

<dependency>
	<groupId>org.springframework.cloud</groupId>
	<artifactId>spring-cloud-kubernetes-fabric8-leader</artifactId>
</dependency>

Additionally to that, you can tweak any of the following properties:

Last thing you need to do is just listen to the events that grant or revoke the leadership and save the context that tells you whether or not you are currently the leader. You can then leverage on that context to decide whether or not you should run the task. Here is a simple example:

@Component
public class LeaderCandidate {
    private Context context;

    @EventListener
    public void onGrantedEvent(OnGrantedEvent event) {
        this.context = event.getContext();
    }

    @EventListener
    public void onRevokedEvent(OnRevokedEvent event) {
        this.context = event.getContext();
    }

    public boolean isLeader() {
        return context != null && context.isLeader();
    }
}

@Service
@RequiredArgsConstructor
public class TaskRunner {
    private final LeaderCandidate candidate;

    @Scheduled(fixedRateString = "PT30S")
    public void run() {
        if(candidate.isLeader()) {
            //run something
        }
    }
}

Quite simple, uh? Hope this helped! Let me know in the comment section about any doubts/suggestions you may have!