Creating a cronjob micro-service using Elixir

If you've ever needed to automate a task or ensure that an important job gets done on schedule, you've probably used a cronjob. Simply put, a cronjob is a tool that allows you to schedule tasks to run automatically at a predetermined time or interval. Whether it's sending out a daily report or backing up a database, cronjobs are a convenient way to automate repetitive or time-consuming tasks.

Recently, I was working on a side project where I wanted to explore the benefits and trade-offs of using multiple languages in a system. I decided to create a cronjob micro-service using Elixir, a functional programming language that is well-suited for building scalable and fault-tolerant systems.

In this blog post, I'll document my experience building the service, and share some of the benefits and challenges I encountered along the way. If you're interested in using Elixir to create your own cronjob service, or if you're just curious about what's involved, I hope you'll find this post helpful.

Why Elixir for a Cronjob Service?

When it came to choosing a language for my cronjob service, I knew I wanted something that was fast, scalable, and fault-tolerant. Elixir checked all of those boxes and then some.

For those unfamiliar with Elixir, it is a functional programming language that runs on the Erlang virtual machine. One of the key benefits of Elixir is its support for concurrency, which allows it to easily handle multiple tasks concurrently. This makes it well-suited for building scalable and fault-tolerant systems.

In addition to its good performance and concurrency support, I was also drawn to Elixir's functional nature. Elixir encourages a functional programming style, which can make it easier to reason about code and write tests. All of these factors made Elixir an appealing choice for my cronjob service.

Using Phoenix for the Web Interface

To build the web interface for my cronjob service, I decided to use the Phoenix framework. Phoenix is a popular web framework for Elixir that makes it easy to build scalable and reliable web applications. It offers a variety of features that made it a good fit for my cronjob service, including support for web sockets, channels, and live view.

One of the key benefits of Phoenix is its use of the actor model for concurrency. In Phoenix, each web request is handled by its own Elixir process, which makes it easy to scale the application by adding more processes. This makes Phoenix well-suited for building a cronjob service, which may need to handle a large number of concurrent tasks.

Overall Architecture

The overall architecture of my cronjob service is designed to be scalable and fault-tolerant. It is composed of multiple Elixir processes that communicate with each other using the actor model. Each process is responsible for a specific task, such as scheduling a job to run or executing a job.

To ensure that the service can recover from failures, I used Elixir's built-in process supervision to monitor the health of the service. If any process fails, the supervisor will restart it, ensuring that the service stays up and running.

The Role of the Database

In my cronjob service, I used a database to store information about the tasks that are scheduled to run and the status of those tasks. This made it easy to track the progress of the service and ensure that tasks were being run as expected.

Integrating a database with an Elixir-based service can sometimes be a challenge, but I found that Elixir's Ecto library made it relatively straightforward. Ecto is a database library for Elixir that provides a simple interface for querying and updating a database.

Setting up the Development Environment

Before I could start building my cronjob service, I needed to set up a development environment. This involved installing Elixir, the Phoenix framework, and any other dependencies that were needed.

If you're new to Elixir, the first step is to install the Elixir runtime and build tools. You can find instructions for installing Elixir on the Elixir website. Once Elixir is installed, you'll also need to install the Phoenix framework. You can do this by running the following command:

mix archive.install hex phx_new

Next, you'll need to set up a database for your cronjob service. I chose to use PostgreSQL, but you could use any database that is supported by Elixir's Ecto library. Once you have a database set up, you'll need to configure your development environment to use it. This typically involves creating a database and a user, and then updating your Phoenix configuration to use the correct database credentials.

Defining the Tasks

Once I had my development environment set up, I was ready to start defining the tasks that my cronjob service would run. I used Elixir's built-in scheduling functions, such as cron/4, to specify the schedule for each task. For example, if I wanted to run a task every hour, I would use the following code:

cron("0 * * * *", MyApp.TaskScheduler, :run_task, [])

In this example, the cron/4 function takes four arguments: a cron expression, a module, a function, and a list of arguments. The cron expression specifies the schedule for the task, and the module and function specify the code that should be run when the task is triggered.

Implementing the Tasks

Once I had defined the tasks that my cronjob service would run, I needed to implement the code that would actually perform the work. This involved writing Elixir functions that would be called by the cron/4 function when the tasks were triggered.

For example, let's say I had defined a task to send a daily report by email. The implementation of this task might look something like this:

defmodule MyApp.TaskScheduler do
  def run_task do
    # Generate the report
    report = generate_report()

    # Send the report by email
    send_email(report)
  end

  defp generate_report do
    # code to generate the report goes here
  end

  defp send_email(report) do
    # code to send the email goes here
  end
end

In this example, the run_task/0 function is the entry point for the task. It calls the generate_report/0 and send_email/1 functions to perform the work of generating and sending the report.

Configuring the Cronjob Service

Once I had implemented the tasks that my cronjob service would run, I needed to configure the service to run on a predetermined schedule. To do this, I used the Phoenix framework to set up routes and controllers for the service.

For example, let's say I wanted to create a web interface for my cronjob service that would allow users to view and manage the tasks that were scheduled to run.

To create a web interface for my cronjob service, I used the Phoenix framework to set up routes and controllers. For example, I might create a route like this:

scope "/tasks", MyApp do
  pipe_through :api
  resources "/", TaskController
end

This route would allow users to access the tasks resource at the /tasks URL. I could then create a TaskController to handle requests to this resource.

To make it easy for users to view and manage the tasks that were scheduled to run, I used Phoenix's live view feature. Live view allows you to build real-time, interactive interfaces with minimal coding. For example, I might create a live view like this:

defmodule MyApp.TaskLive do
  use Phoenix.LiveView
  
  def mount(_params, _session, socket) do
    tasks = fetch_tasks()
    {:ok, assign(socket, tasks: tasks)}
  end
  
  def render(assigns) do
    # code to render the live view goes here
  end
  
  def handle_event("add_task", %{"name" => name, "schedule" => schedule}, socket) do
    # code to handle the "add_task" event goes here
  end
  
  defp fetch_tasks do
    # code to fetch the tasks from the database goes here
  end
end

In this example, the mount/3 function is called when the live view is first rendered. It fetches the tasks from the database and assigns them to the tasks variable. The render/1 function is then called to render the live view, and the handle_event/3 function is called to handle events that are sent from the client (such as an "add_task" event).

Using live view made it easy for me to create a real-time, interactive interface for my cronjob service. Users could view and manage the tasks that were scheduled to run, and they could see the changes in real time as they were made.

External communication with the service

Here is an example of how I used a message queue (in this case, RabbitMQ) to communicate with the cronjob service:

First, I needed to set up a RabbitMQ server and install the amqp library, which is an Elixir client library for RabbitMQ. You can do this by adding the following dependencies to your mix.exs file:

defp deps do
  [
    {:amqp, "~> 3.0"}
  ]
end

Next, I needed to create a connection to the RabbitMQ server and set up a channel for sending and receiving messages. We can do this in our application's startup code:

def start(_type, _args) do
  # Connect to the RabbitMQ server
  {:ok, conn} = AMQP.Connection.open(...)

  # Open a channel
  {:ok, chan} = AMQP.Channel.open(conn)

  # Set up queues and exchanges
  AMQP.Queue.declare(chan, "tasks", durable: true)
  AMQP.Exchange.direct(chan, "tasks", durable: true)
  AMQP.Queue.bind(chan, "tasks", "tasks", "")

  # Start the task scheduler process
  TaskScheduler.start_link(chan)
  
    # Start the web server
  {:ok, _pid} = Phoenix.Server.start_link(...)
end

With the connection and channel set up, we can start using RabbitMQ to send and receive messages. Here is an example of how one might use it in the TaskScheduler process:

defmodule TaskScheduler do
  use GenServer

  def start_link(chan) do
    GenServer.start_link(__MODULE__, chan, name: __MODULE__)
  end

  def init(chan) do
    # Set up the queue and exchange
    {:ok, _queue} = AMQP.Queue.declare(chan, "", exclusive: true)
    AMQP.Queue.bind(chan, "", "tasks", "")

    # Set up a consumer to receive messages from the queue
    AMQP.Basic.consume(chan, "", fn(payload, _metadata, _ack) ->
      # Parse the message and schedule the task
      {:ok, %{"name" => name, "schedule" => schedule}} = Jason.decode(payload)
      schedule_task(name, schedule)

      # Acknowledge the message
      AMQP.Basic.ack(chan, _ack)
    end)

    # Return the initial state
    {:ok, chan}
  end

  def schedule_task(name, schedule) do
    # Code to schedule the task goes here
  end
end

This TaskScheduler process uses RabbitMQ to set up a consumer that listens for messages on the "tasks" exchange. When a message is received, it parses the message and calls the schedule_task/2 function to schedule the task.

To send a message to the cronjob service, you can use the AMQP.Basic.publish/4 function. For example:

def create_task(name, schedule) do
  # Connect to the RabbitMQ server
  {:ok, conn} = AMQP.Connection.open(...)

  # Open a channel
  {:ok, chan} = AMQP.Channel.open(conn)

  # Encode the message
  message = Jason.encode(%{"name" => name, "schedule" => schedule})

  
  # Publish the message to the "tasks" exchange
  AMQP.Basic.publish(chan, "tasks", "", message)
  
  # Close the channel and connection
  AMQP.Channel.close(chan)
  AMQP.Connection.close(conn)
end

Conclusion

Creating a cronjob micro-service using Elixir was a fun and interesting project. I enjoyed the process of setting up a development environment, defining and implementing tasks, and configuring the service to run on a predetermined schedule.

One of the biggest challenges I faced was getting used to the syntax and concepts of Elixir, which was new to me. However, once I got the hang of it, I found that Elixir was a powerful and expressive language that made it easy to build the cronjob service.

If you're interested in creating your own cronjob service using Elixir, I recommend checking out the following resources:

The Elixir documentation: https://elixir-lang.org/docs/stable/elixir/
The Phoenix framework documentation: https://hexdocs.pm/phoenix/index.html
The Ecto library documentation: https://hexdocs.pm/ecto/index.html

I hope this blog post has been helpful and gives you an idea of what's involved in creating a cronjob service using Elixir. If you have any questions or comments, I'd love to hear from you!

If you find anything wrong, or anything that needs correction, please feel free to leave a comment and let me know and I will make sure to check it out and address it.

Ciao 👋🏾

References

1. https://github.com/quantum-elixir/quantum-core

2. https://www.phoenixframework.org/

3. https://blog.kalvad.com/write-your-own-cron-with-with-elixir/

4. https://wrgoldstein.github.io/2017/02/20/phoenix-rabbitmq.html