Running Hangfire in an Azure Continuous WebJob

A few years back I was tasked with implementing a solution for running both on-demand and scheduled background tasks, and to do so in a manner that was reliable, with little impact to the performance of our web applications, and with a UI to allow our operations team to monitor scheduled, executing and completed tasks. I also needed to make sure this solution could easily run in Azure.

I remembered reading an older blog post by Scott Hanselman from 2014 describing several possible solutions, and when I returned to the post I rediscovered Hangfire by Sergey Odinokov. Hangfire turned out to be the perfect solution. It met all of our needs, and then some. We were able to run on-demand tasks in the background, schedule tasks, and embed the out-of-the-box web dashboard in one of our ASP.NET web applications. There were also several data storage options to choose from (I used the SQL Server nuget).

The best part of the solution I implemented was deploying the Hangfire server as an Azure Continuous WebJob. All of the tutorials I found only showed how to run the Hangfire server within an ASP.NET web application. We had some concerns, especially because we were considering running a number of long running tasks. But what if we could run Hangfire within it’s own app service in Azure? In the early years of our web application where we have light traffic, we can deploy the WebJob to an app service in the same app service plan as our web application. But we also have the option to deploy the WebJob to it’s own app service, possibly running under its own app service plan where we are able to scale independently from our web application.

In this tutorial, I will show you how to quickly get Hangfire up and running in an Azure WebJob.

Source

The source code for this project can be found here: https://github.com/tedterrill/HangfireInAzure

Create a New WebJob Project

I started by creating a new Azure WebJob project in Visual Studio (I’m using Visual Studio 2019). I named my project HangfireInAzure. In this sample project, I targeted .NET Framework 4.72, but it is important to note you can target .NET Core and the code will be the same. At the time of writing this, I ran the following NuGet package installs/updates in the order below.

• Newtonsoft.Json 12.02
• MIcrosoft.Azure.WebJobs.Core 3.0.13
• Microsoft.Azure.WebJobs 3.0.13
• WindowsAzure.Storage 9.3.1
• Microsoft.Azure.WebJobs.Host.Storage 3.0.13
• Microsoft.Azure.WebJobs.Extensions 3.0.2
• Microsoft.Azure.WebJobs.Extensions.Storage 3.0.8
• Microsoft.Extensions.DependencyInjection 3.0.0
• Microsoft.Extensions.Logging 3.0.0
• Microsoft.Extensions.Logging.Console 3.0.0
• Hangfire 1.7.6

Configure Data Store

Hangfire requires data storage. I used Sql Server. Note: you can also find NuGet packages for Redis, Mongo, PostgreSql, and MySql, as well as Sql Server with MSMQ. If you are using Sql Server like me, open the app.config file and add a connection string for your database. While you are in there, add “UseDevelopmentStorage=true” for the AzureWebJobsDashboard and AzureWebJobsStorage storage account connection strings.

...
  <connectionStrings>
    <add name="Hangfire" providerName="System.Data.SqlClient" connectionString="Server=<SERVER>;Database=<DATABASE>;User ID=<USERID>;Password=<PASSWORD>;Enlist=True" />
    <!-- For local execution, the value can be set either in this config file or through environment variables -->
    <add name="AzureWebJobsDashboard" connectionString="UseDevelopmentStorage=true" />
    <add name="AzureWebJobsStorage" connectionString="UseDevelopmentStorage=true" />
  </connectionStrings>
...

Configure the WebJob Host

The boilerplate code that the project starts with in Program.cs applies to WebJobs SDK 1.0. Because we updated to version 3.0, we need to rewrite this. Remove the code from Program.cs and replace with the following.

using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using System;
using System.Threading;
using System.Threading.Tasks;

namespace HangfireInAzure {
    // To learn more about Microsoft Azure WebJobs SDK, please see https://go.microsoft.com/fwlink/?LinkID=320976
    public class Program {
        public static async Task Main() {
            var hostBuilder = new HostBuilder()
                              .ConfigureWebJobs(webJobsBuilder => {
                                  webJobsBuilder.AddAzureStorageCoreServices();
                                  webJobsBuilder.AddAzureStorage();
                              })
                              .ConfigureLogging((context, loggingBuilder) => {
                                  // add your logging here - we'll use a simple console logger for this demo
                                  loggingBuilder.SetMinimumLevel(LogLevel.Debug);
                                  loggingBuilder.AddConsole();
                              })
                              .ConfigureServices((context, services) => {
                                  services
                                      .AddSingleton(context.Configuration);
                              })
                              .UseConsoleLifetime();

            using (var host = hostBuilder.Build()) {
                var logger = host.Services.GetService<ILogger<Program>>();

                try {
#if DEBUG
                    Environment.SetEnvironmentVariable("WEBJOBS_SHUTDOWN_FILE", "c:\\temp\\WebJobsShutdown");
#endif
                    
                    var watcher           = new WebJobsShutdownWatcher();
                    var cancellationToken = watcher.Token;

                    logger.LogInformation("Starting the host");
                    await host.StartAsync(cancellationToken);
                    var jobHost = host.Services.GetService<IJobHost>();

                    logger.LogInformation("Starting the Hangfire server");
                    await jobHost
                          .CallAsync(nameof(HangfireServer.RunServer),
                                     cancellationToken: cancellationToken)
                          .ContinueWith(result => {
                              logger.LogInformation(
                                  $"The job host stopped with state: {result.Status}");
                          }, cancellationToken);
                }
                catch (Exception ex) {
                    logger.LogError(ex.Message);
                }
            }
        }
    }
}

In the code above, we are configuring and creating the webjob host using the HostBuilder. It is in here that we configure a logger and register any services. In the webjob host, we are invoking a function responsible for creating the Hangfire server. Note, before starting the WebJob host and the Hangfire JobHost, we create a WebJobsShutdownWatcher and retrieve its CancellationToken. This token will be cancelled by the watcher when the WebJob is shut down in Azure. We will use it later in our code.

Create the Hangfire Server

Create a file called HangfireServer.cs and add the code below. This class contains the function that the WebJob will invoke from Program.cs. It is here that we create and configure the Hangfire server. This method will take a cancellation token (the one from the watcher). We also register all of our dependencies here, like the logger and the jobs.

using Hangfire;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;

namespace HangfireInAzure {
    public class HangfireServer {
        private static void ConfigureServices(IServiceCollection services) {
            services.AddLogging(loggingBuilder => { loggingBuilder.AddConsole(); })
                    .Configure<LoggerFilterOptions>(options => options.MinLevel = LogLevel.Debug)
                    // register our job activator

                    // register tasks/jobs here


                ;
        }

        [NoAutomaticTrigger]
        [FunctionName(nameof(RunServer))]
        public async Task RunServer(ILogger logger,
                                    CancellationToken cancellationToken) {
            try {
                var serviceCollection = new ServiceCollection();
                ConfigureServices(serviceCollection);

                // create service provider
                var serviceProvider =
                    serviceCollection.BuildServiceProvider(new ServiceProviderOptions {
                        ValidateOnBuild = true
                    });

                logger.LogInformation("Configuring Hangfire...");

                GlobalConfiguration.Configuration
                                   .UseColouredConsoleLogProvider(Hangfire.Logging.LogLevel.Info)
                                   // set up our storage method - this demo uses SQL server
                                   .UseSqlServerStorage("Hangfire")
                                   // get the job activator
                                   .UseActivator(serviceProvider.GetService<JobActivator>())
                    ;

                using (new BackgroundJobServer(
                    new BackgroundJobServerOptions {
                        WorkerCount               = 2,
                        HeartbeatInterval         = TimeSpan.FromSeconds(10),
                        SchedulePollingInterval   = TimeSpan.FromSeconds(10),
                        CancellationCheckInterval = TimeSpan.FromSeconds(10)
                    })) {
                    
                    logger.LogInformation(
                        "The Hangfire server has started.");
                    
                    var tasks = new List<Task> {
                        // Put some code here to add jobs to the queue. This is just for testing. You will not do this in a real app
                        JobScheduler.RunAsync(logger, serviceProvider, cancellationToken),
                        AwaitCancellation(logger, cancellationToken)
                    };

                    await Task.WhenAll(tasks);
                }
            }
            catch (OperationCanceledException oce) {
                logger.LogWarning($"Cancellation was raised while running server: {oce.Message}", oce);
            }
            catch (Exception ex) {
                logger.LogError($"Error occurred while running server: {ex.Message}", ex);
                throw;
            }
        }

        private static async Task AwaitCancellation(ILogger logger, CancellationToken cancellationToken) {
            while (!cancellationToken.IsCancellationRequested) {
                await Task.Delay(30000, cancellationToken);
            }
        }
    }
}

For my Hangfire server, I have a custom job activator responsible for instantiating jobs to be performed. I inject the ServicesCollection into the activator, thus requiring registration of all of the jobs and their dependencies. It is possible to use the DI library of your choice, but that is beyond the scope of this tutorial.

using Hangfire;
using Microsoft.Extensions.DependencyInjection;
using System;
using System.Linq;

namespace HangfireInAzure {
    public class HangfireJobActivator : JobActivator {
        private readonly IServiceProvider _serviceProvider;

        public HangfireJobActivator(IServiceProvider serviceProvider) {
            _serviceProvider = serviceProvider;
        }

        public override object ActivateJob(Type type) {
            var instance = _serviceProvider.GetService(type);

            if (instance == null && type.GetInterfaces().Any())
                instance = _serviceProvider.GetService(type.GetInterfaces().FirstOrDefault());

            return instance;
        }

        public override JobActivatorScope BeginScope(JobActivatorContext context) {
            return new ServiceProviderScope(_serviceProvider, context);
        }
    }

    internal class ServiceProviderScope : JobActivatorScope {
        private readonly JobActivatorContext _context;
        private readonly IServiceScope       _scope;

        public ServiceProviderScope(IServiceProvider serviceProvider, JobActivatorContext context) {
            _context = context;
            _scope   = serviceProvider.CreateScope();
        }

        public override object Resolve(Type type) {
            var instance = _scope.ServiceProvider.GetService(type);

            if (instance == null && type.GetInterfaces().Any())
                instance = _scope.ServiceProvider.GetService(type.GetInterfaces().FirstOrDefault());

            return instance;
        }

        public override void DisposeScope() {
            _scope?.Dispose();
        }
    }
}

Go back to the Hangfire service configuration method and register the activator.

        private static void ConfigureServices(IServiceCollection services) {
            services.AddLogging(loggingBuilder => { loggingBuilder.AddConsole(); })
                    .Configure<LoggerFilterOptions>(options => options.MinLevel = LogLevel.Debug)
                    // register our job activator
                    .AddSingleton<JobActivator>(s => new HangfireJobActivator(s))
                    // register tasks/jobs here


                ;
        }

Add Some Jobs

For this tutorial, there are two sample jobs. The first is a simple job that writes a message to the console.

using Microsoft.Extensions.Logging;
using System;
using System.Threading;

namespace HangfireInAzure {
    public class MySimpleJob {
        private readonly ILogger<MySimpleJob> _logger;

        public MySimpleJob(ILogger<MySimpleJob> logger) {
            _logger = logger;
        }

        public void DoWork(string work, CancellationToken cancellationToken) {
            _logger.LogDebug($"[{DateTime.Now:s}] {work}");
        }
    }
}

The second job simulates a long-running task and allows us to test graceful shutdown of the WebJob. Note how this job takes a cancellation token and will exit the loop when cancellation is triggered. Hangfire will inject its own token into the job and will cancel when the Hangfire server is disposed.

using Microsoft.Extensions.Logging;
using System;
using System.Threading;

namespace HangfireInAzure {
    public class MyLongRunningJob {
        private readonly ILogger<MyLongRunningJob> _logger;

        public MyLongRunningJob(ILogger<MyLongRunningJob> logger) {
            _logger = logger;
        }

        public void DoWork(string work, CancellationToken cancellationToken) {
            while (!cancellationToken.IsCancellationRequested) {
                _logger.LogDebug($"[{DateTime.Now:s}] Long running task is doing work: {work}");
                Thread.Sleep(TimeSpan.FromSeconds(12));
            }

            _logger.LogWarning("Long running task was cancelled.");
        }
    }
}

Go back to the Hangfire service configuration method and register the job types.

...
        private static void ConfigureServices(IServiceCollection services) {
            services.AddLogging(loggingBuilder => { loggingBuilder.AddConsole(); })
                    .Configure<LoggerFilterOptions>(options => options.MinLevel = LogLevel.Debug)
                    // register our job activator
                    .AddSingleton<JobActivator>(s => new HangfireJobActivator(s))
                    // register tasks/jobs here
                    .AddScoped<MySimpleJob>()
                    .AddScoped<MyLongRunningJob>()
                ;
        }
...

Now you need to create some jobs to run after the Hangfire server starts. For the sake of this tutorial, I am creating and adding jobs to the queue immediately after the server starts. In reality, you will more likely have other applications (web app, function app) that are responsible for adding jobs to the Hangfire queue.

using Hangfire;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using System;
using System.Threading;
using System.Threading.Tasks;

namespace HangfireInAzure {
    public class JobScheduler {
        public static async Task RunAsync(ILogger           logger,
                                          IServiceProvider  serviceProvider,
                                          CancellationToken cancellationToken) {

            RunLongRunningJob(serviceProvider);

            while (!cancellationToken.IsCancellationRequested) {
                RunSomeJobs(serviceProvider);
                await Task.Delay(30000, cancellationToken);
            }
        }

        private static void RunSomeJobs(IServiceProvider serviceProvider) {
            var job = serviceProvider.GetService<MySimpleJob>();

            // Queue up a job to run immediately
            BackgroundJob.Enqueue(() =>
                                      job.DoWork(
                                          "Doing some work right away",
                                          CancellationToken.None) // use 'None' and hangfire will inject a cancellation token
            );


            // Schedule a job to run 10 seconds later
            var delay = TimeSpan.FromSeconds(10);
            var runAt = new DateTimeOffset(DateTime.Now.Add(delay));
            BackgroundJob.Schedule(() =>
                                       job.DoWork(
                                           $"This work is being scheduled for {runAt}",
                                           CancellationToken.None),
                                   runAt
            );
        }

        private static void RunLongRunningJob(IServiceProvider serviceProvider) {
            var longRunningJob = serviceProvider.GetService<MyLongRunningJob>();

            BackgroundJob.Enqueue(() =>
                                      longRunningJob.DoWork(
                                          "Staying busy", CancellationToken.None)
            );
        }
    }
}

Now that you have code to create some jobs, hook it up to your Hangfire server by adding the line below to HangfireServer.cs. Like I said above, you will not do this in your real application. This is merely to test the Hangfire server for this tutorial.

...

                // Put some code here to add jobs to the queue. This is just for testing. You will not do this in a real app
                JobScheduler.RunAsync(logger, serviceProvider, cancellationToken);

...

Now it’s time to fire it up. You can debug a WebJob easily through Visual Studio, so just press F5 to get started. You will see something like this:

The logging should give you an idea of what is happening. There are 3 jobs running here. A simple job that is “Doing some work right away”, another simple job that is scheduled for 10 seconds later, and a long running job that writes a message every 10 seconds.

WebJob Shutdown

You may have noticed the line of code below in Program.cs.

...

#if DEBUG
                    Environment.SetEnvironmentVariable("WEBJOBS_SHUTDOWN_FILE", "c:\\temp\\WebJobsShutdown");
#endif

...

This environment variable tells the WebJobsShutdownWatcher the directory to monitor. This is automatically handled for us when the WebJob is running in an Azure app service. (Read here for a description of how this works in Azure). For testing, we can specify a local directory and then add a file to this directory to trigger a shutdown. After starting up the program and letting it run for a few minutes, go ahead and add a file (any file with any name, doesn’t matter) to the specified directory and you will see the following logging in the console.

Warning! If you stop the debugger before triggering a WebJob shutdown, the long running job will not complete. The next time you start the debugger, Hangfire will identify this incomplete job and start it again, along with the new long running job that gets created upon startup. Keep this in mind if you begin to see multiple logs show up from the long running job.

If this is your first time using Hangfire, make sure you check out the documentation to find out what is possible and where to go from here.