Hangfire Documentation, Exercises of Logic

Download Hangfire Documentation and more Exercises Logic in PDF only on Docsity! Hangfire Documentation Release 1.7 Hangfire OÜ Aug 26, 2022 CHAPTER 1 Overview Hangfire allows you to kick off method calls outside of the request processing pipeline in a very easy, but reliable way. These method invocations are performed in a background thread and called background jobs. From the 10.000-feet view the library consists of three main components: client, storage and server. Here is a small diagram that describes the main processes in Hangfire: 1 Hangfire Documentation, Release 1.7 2 Chapter 1. Overview CHAPTER 2 Requirements Hangfire is not tied to the specific .NET application type. You can use it in ASP.NET web applications, non-ASP.NET web applications, in console applications or Windows services. Here are the requirements: • .NET Framework 4.5 • Persistent storage (listed below) • Newtonsoft.Json library 5.0.1 3 Hangfire Documentation, Release 1.7 6 Chapter 3. Client CHAPTER 4 Job Storage Hangfire keeps background jobs and other information that relates to the processing inside a persistent storage. Per- sistence helps background jobs to survive on application restarts, server reboots, etc. This is the main distinction between performing background jobs using CLR’s Thread Pool and Hangfire. Different storage backends are sup- ported: • SQL Azure, SQL Server 2008 R2 (and later of any edition, including Express) • Redis SQL Server storage can be empowered with MSMQ or RabbitMQ to lower the processing latency. GlobalConfiguration.Configuration.UseSqlServerStorage("db_connection"); 7 Hangfire Documentation, Release 1.7 8 Chapter 4. Job Storage CHAPTER 6 Table of Contents 6.1 Getting Started 6.1.1 Requirements Hangfire works with the majority of .NET platforms: .NET Framework 4.5 or later, .NET Core 1.0 or later, or any platform compatible with .NET Standard 1.3. You can integrate it with almost any application framework, including ASP.NET, ASP.NET Core, Console applications, Windows Services, WCF, as well as community-driven frameworks like Nancy or ServiceStack. 6.1.2 Storage Storage is a place where Hangfire keeps all the information related to background job processing. All the details like types, method names, arguments, etc. are serialized and placed into storage, no data is kept in a process’ memory. The storage subsystem is abstracted in Hangfire well enough to be implemented for RDBMS and NoSQL solutions. This is the main decision you must make, and the only configuration required before start using the framework. The following example shows how to configure Hangfire with a SQL Server database. Please note that connection string may vary, depending on your environment. GlobalConfiguration.Configuration .UseSqlServerStorage(@"Server=.\SQLEXPRESS; Database=Hangfire.Sample; Integrated →˓Security=True"); 6.1.3 Client The Client is responsible for creating background jobs and saving them into Storage. Background job is a unit of work that should be performed outside of the current execution context, e.g. in background thread, other process, or even on different server – all is possible with Hangfire, even with no additional configuration. 11 Hangfire Documentation, Release 1.7 BackgroundJob.Enqueue(() => Console.WriteLine("Hello, world!")); Please note this is not a delegate, it’s an expression tree. Instead of calling the method immediately, Hangfire serializes the type (System.Console), method name (WriteLine, with all the parameter types to identify it later), and all the given arguments, and places it to Storage. 6.1.4 Server Hangfire Server processes background jobs by querying the Storage. Roughly speaking, it’s a set of background threads that listen to the Storage for new background jobs, and perform them by de-serializing type, method and arguments. You can place this background job server in any process you want, including dangerous ones like ASP.NET – even if you terminate a process, your background jobs will be retried automatically after restart. So in a basic configuration for a web application, you don’t need to use Windows Services for background processing anymore. using (new BackgroundJobServer()) { Console.ReadLine(); } 6.1.5 Installation Hangfire is distributed as a couple of NuGet packages, starting from the primary one, Hangfire.Core, that contains all the primary classes as well as abstractions. Other packages like Hangfire.SqlServer provide features or abstraction implementations. To start using Hangfire, install the primary package and choose one of the available storages. After the release of Visual Studio 2017, a completely new way of installing NuGet packages appeared. So I give up listing all the ways of installing a NuGet package, and fallback to the one available almost everywhere using the dotnet app. dotnet add package Hangfire.Core dotnet add package Hangfire.SqlServer 6.1.6 Configuration Configuration is performed using the GlobalConfiguration class. Its Configuration property provides a lot of extension methods, both from Hangfire.Core, as well as other packages. If you install a new package, don’t hesitate to check whether there are new extension methods. GlobalConfiguration.Configuration .SetDataCompatibilityLevel(CompatibilityLevel.Version_170) .UseSimpleAssemblyNameTypeSerializer() .UseRecommendedSerializerSettings() .UseSqlServerStorage("Database=Hangfire.Sample; Integrated Security=True;", new →˓SqlServerStorageOptions { CommandBatchMaxTimeout = TimeSpan.FromMinutes(5), SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5), QueuePollInterval = TimeSpan.Zero, UseRecommendedIsolationLevel = true }) (continues on next page) 12 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 (continued from previous page) .UseBatches() .UsePerformanceCounters(); Method calls can be chained, so there’s no need to use the class name again and again. Global configuration is made for simplicity, almost every class of Hangfire allows you to specify overrides for storage, filters, etc. In ASP.NET Core environments global configuration class is hidden inside the AddHangfire method. 6.1.7 Usage Here are all the Hangfire components in action, as a fully working sample that prints the “Hello, world!” message from a background thread. You can comment the lines related to server, and run the program several times – all the background jobs will be processed as soon as you uncomment the lines again. using System; using Hangfire; using Hangfire.SqlServer; namespace ConsoleApplication2 { class Program { static void Main() { GlobalConfiguration.Configuration .SetDataCompatibilityLevel(CompatibilityLevel.Version_170) .UseColouredConsoleLogProvider() .UseSimpleAssemblyNameTypeSerializer() .UseRecommendedSerializerSettings() .UseSqlServerStorage("Database=Hangfire.Sample; Integrated →˓Security=True;", new SqlServerStorageOptions { CommandBatchMaxTimeout = TimeSpan.FromMinutes(5), SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5), QueuePollInterval = TimeSpan.Zero, UseRecommendedIsolationLevel = true }); BackgroundJob.Enqueue(() => Console.WriteLine("Hello, world!")); using (var server = new BackgroundJobServer()) { Console.ReadLine(); } } } } ASP.NET Applications You can place the background processing in an ASP.NET application without using additional processes like Windows Services. Hangfire’s code is ready for unexpected process terminations, application pool recycles and restarts during the deployment process. Since persistent storages are used, you’ll not lose any background job. 6.1. Getting Started 13 Hangfire Documentation, Release 1.7 (continued from previous page) .UseSimpleAssemblyNameTypeSerializer() .UseRecommendedSerializerSettings() .UseSqlServerStorage("Server=.\\SQLEXPRESS; Database=HangfireTest; →˓Integrated Security=True;", new SqlServerStorageOptions { CommandBatchMaxTimeout = TimeSpan.FromMinutes(5), SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5), QueuePollInterval = TimeSpan.Zero, UseRecommendedIsolationLevel = true, DisableGlobalLocks = true }); yield return new BackgroundJobServer(); } protected void Application_Start() { AreaRegistration.RegisterAllAreas(); RouteConfig.RegisterRoutes(RouteTable.Routes); HangfireAspNet.Use(GetHangfireServers); // Let's also create a sample background job BackgroundJob.Enqueue(() => Debug.WriteLine("Hello world from Hangfire!")); } } You might also need to disable OWIN’s Startup class detection, when using initialization based on Global.asax. cs file. The problem is Hangfire.AspNet package depends on Microsoft.Owin.SystemWeb package, and it requires OWIN Startup class to be present in your web application. If the following exception appears, just disable the automatic startup in your web.config file as should below. EntryPointNotFoundException: The following errors occurred while attempting to load →˓the app. - No assembly found containing an OwinStartupAttribute. - No assembly found containing a Startup or [AssemblyName].Startup class. <!-- web.config --> <appSettings> <add key="webpages:Version" value="3.0.0.0" /> <add key="webpages:Enabled" value="false" /> <add key="ClientValidationEnabled" value="true" /> <add key="UnobtrusiveJavaScriptEnabled" value="true" /> <add key="owin:AutomaticAppStartup" value="false"/> </appSettings> Running Application Run your application in the Debug mode by pressing F5 (this is required to see the output of the Debug.WriteLine method). Then check the Output window for the following message to see whether background processing has started successfully. 16 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 When application is started, open the following URL (assuming your app is running on the 5000 port) to access to the Hangfire Dashboard interface. As we can see, our background job was completed successfully. Startup class is required for Dashboard UI Please note, Dashboard UI is available only if you were using the Startup class to configure Hangfire. http://<your-web-app>/hangfire That’s all, now you are ready to create other background jobs! 6.1. Getting Started 17 Hangfire Documentation, Release 1.7 ASP.NET Core Applications Before we start with our tutorial, we need to have a working ASP.NET Core application. This documentation is devoted to Hangfire, please, read the official ASP.NET Core Documentation to learn the details on how to create and initialize a new web application: Getting Started and Tutorials. Installing Hangfire Hangfire is available as a set of NuGet packages, so you need to add them to the *.csproj file by adding new PackageReference tags as below. Please note that versions in the code snippet below may be outdated, so use versions from the following badges. They are updated in real-time. <ItemGroup> <PackageReference Include="Microsoft.AspNetCore.App" /> <PackageReference Include="Hangfire.Core" Version="1.7.*" /> <PackageReference Include="Hangfire.SqlServer" Version="1.7.*" /> <PackageReference Include="Hangfire.AspNetCore" Version="1.7.*" /> </ItemGroup> Creating a database As you can see from the snippet above, we’ll be using SQL Server as a job storage in this article. Before configuring Hangfire, you’ll need to create a database for it, or use an existing one. Configuration strings below point to the HangfireTest database living in the SQLEXPRESS instance on a local machine. You can use SQL Server Management Studio or any other way to execute the following SQL command. If you are using an other database name or instance, ensure you’ve changed the connection strings when configuring Hangfire during the next steps. CREATE DATABASE [HangfireTest] GO Configuring Hangfire We’ll start our configuration process with defining a configuration string for the Hangfire.SqlServer package. Consider you have an sqlexpress named instance running on localhost, and just created the “HangfireTest” database. The current user should be able to create tables, to allow automatic migrations to do their job. Also, the Hangfire.AspNetCore package has a logging integration with ASP.NET Core applications. Hangfire’s log messages are sometimes very important and help to diagnose different issues. Information level allows to see how Hangfire is working, and Warning and higher log levels help to investigate problems. Configuring Settings Open the appsettings.json file, and add the highlighted lines from the following snippet. { "ConnectionStrings": { "HangfireConnection": "Server=.\\sqlexpress;Database=HangfireTest;Integrated →˓Security=SSPI;" (continues on next page) 18 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 info: Hangfire.SqlServer.SqlServerStorage[0] Start installing Hangfire SQL objects... Hangfire SQL objects installed. Using job storage: 'SQL Server: .\@AspNetCoreTest' Using the following options for SQL Server job storage: Queue poll interval: 00:00:15. info: Hangfire.BackgroundJobServer[0] Starting Hangfire Server... Using the following options for Hangfire Server: Worker count: 20 Listening queues: 'default' Shutdown timeout: 00:00:15 Schedule polling interval: 00:00:15 These lines contain messages regarding SQL Server Job Storage that is used to persist background jobs, and the Background Job Server, which is processing all the background jobs. The following message should also appear, since we created background job, whose only behavior is to write a message to the console. Hello world from Hangfire! When the application has started, open the following URL (assuming your app is running on the 5000 port), to access to the Hangfire Dashboard interface. As we can see, our background job was completed successfully. http://localhost:5000/hangfire When you finished working with the application, press the Ctrl+C in your console window to stop the application. The following message should appear telling you that background processing server was stopped gracefully. 6.1. Getting Started 21 Hangfire Documentation, Release 1.7 info: Hangfire.BackgroundJobServer[0] Hangfire Server stopped. You can also kill your process, but in this case some background jobs may be delayed in invocation. 6.2 Configuration Starting from version 1.4, GlobalConfiguration class is the preferred way to configure Hangfire. This is an entry point for a couple of methods, including ones from third-party storage implementations or other extensions. The usage is simple, just include Hangfire namespace in your application initialization class and discover extension methods for the GlobalConfiguration.Configuration property. For example, in ASP.NET applications, you can place initialization logic to the Global.asax.cs file: using Hangfire; public class MvcApplication : System.Web.HttpApplication { protected void Application_Start() { // Storage is the only thing required for basic configuration. // Just discover what configuration options do you have. GlobalConfiguration.Configuration .UseSqlServerStorage("<name or connection string>"); //.UseActivator(...) //.UseLogProvider(...) } } For OWIN-based applications (ASP.NET MVC, Nancy, ServiceStack, FubuMVC, etc.), place the configuration lines to the OWIN Startup class. using Hangfire; [assembly: OwinStartup(typeof(Startup))] public class Startup { public void Configuration(IAppBuilder app) { GlobalConfiguration.Configuration.UseSqlServerStorage("<name or connection →˓string>"); } } For other applications, place it somewhere before calling other Hangfire methods. 6.2.1 Using Dashboard Hangfire Dashboard is a place where you could find all the information about your background jobs. It is written as an OWIN middleware (if you are not familiar with OWIN, don’t worry), so you can plug it into your ASP.NET, ASP.NET MVC, Nancy, ServiceStack application as well as use OWIN Self-Host feature to host Dashboard inside console applications or in Windows Services. 22 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 • Adding Dashboard (OWIN) • Configuring Authorization • Read-only view • Change URL Mapping • Change Back to site Link • Multiple Dashboards Adding Dashboard (OWIN) Additional package required for ASP.NET + IIS Before moving to the next steps, ensure you have Microsoft.Owin.Host.SystemWeb package installed, otherwise you’ll have different strange problems with the Dashboard. OWIN Startup class is intended to keep web application bootstrap logic in a single place. In Visual Studio 2013 you can add it by right clicking on the project and choosing the Add / OWIN Startup Class menu item. If you have Visual Studio 2012 or earlier, just create a regular class in the root folder of your application, name it Startup and place the following contents: using Hangfire; using Microsoft.Owin; using Owin; [assembly: OwinStartup(typeof(MyWebApplication.Startup))] (continues on next page) 6.2. Configuration 23 Hangfire Documentation, Release 1.7 Change Back to site Link By default, Back to site link (top-right corner of Dashboard) leads you to the root URL of your application. In order to change it, use the DashboardOptions class. // Change `Back to site` link URL var options = new DashboardOptions { AppPath = "http://your-app.net" }; // Make `Back to site` link working for subfolder applications var options = new DashboardOptions { AppPath = VirtualPathUtility.ToAbsolute("~") }; app.UseHangfireDashboard("/hangfire", options); Multiple Dashboards You can also map multiple dashboards that show information about different storages. var storage1 = new SqlServerStorage("Connection1"); var storage2 = new SqlServerStorage("Connection2"); app.UseHangfireDashboard("/hangfire1", new DashboardOptions(), storage1); app.UseHangfireDashboard("/hangfire2", new DashboardOptions(), storage2); 6.2.2 Using SQL Server SQL Server is the default storage for Hangfire – it is well known to many .NET developers and used in many project environments. It may be interesting that in the early stage of Hangfire development, Redis was used to store informa- tion about jobs, and SQL Server storage implementation was inspired by that NoSql solution. But back to the SQL Server. . . SQL Server storage implementation is available through the Hangfire.SqlServer NuGet package. To install it, type the following command in your NuGet Package Console window: Install-Package Hangfire.SqlServer This package is a dependency of the Hangfire’s bootstrapper package Hangfire, so if you installed it, you don’t need to install the Hangfire.SqlServer separately – it was already added to your project. Supported database engines Microsoft SQL Server 2008R2 (any edition, including LocalDB) and later, Microsoft SQL Azure. Snapshot isolation is not supported! Applies only to Hangfire < 1.5.9: Ensure your database doesn’t use the snapshot isolation level, and the READ_COMMITTED_SNAPSHOT option (another name is Is Read Committed Snapshot On) is disabled. Otherwise some of your background jobs will not be processed. Configuration The package provides extension methods for GlobalConfiguration class. Choose either a connection string to your SQL Server or a connection string name, if you have it. 26 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 GlobalConfiguration.Configuration // Use connection string name defined in `web.config` or `app.config` .UseSqlServerStorage("db_connection") // Use custom connection string .UseSqlServerStorage(@"Server=.\sqlexpress; Database=Hangfire; Integrated →˓Security=SSPI;"); Starting from version 1.7.0 it is recommended to set the following options for new installations (for existing ones, please see Upgrading to Hangfire 1.7). These settings will be turned on by default in 2.0, but meanwhile we should preserve backward compatibility. GlobalConfiguration.Configuration .UseSqlServerStorage("db_connection", new SqlServerStorageOptions { CommandBatchMaxTimeout = TimeSpan.FromMinutes(5), SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5), QueuePollInterval = TimeSpan.Zero, UseRecommendedIsolationLevel = true, DisableGlobalLocks = true // Migration to Schema 7 is required }); Installing objects Hangfire leverages a couple of tables and indexes to persist background jobs and other information related to the processing: Some of these tables are used for the core functionality, others fulfill the extensibility needs (making possible to write extensions without changing the underlying schema). Advanced objects like stored procedures, triggers and so on are not used to keep things as simple as possible and allow the library to be used with SQL Azure. SQL Server objects are installed automatically from the SqlServerStorage constructor by executing statements described in the Install.sql file (which is located under the tools folder in the NuGet package). Which contains the migration script, so new versions of Hangfire with schema changes can be installed seamlessly, without your intervention. If you want to install objects manually, or integrate it with your existing migration subsystem, pass your decision through the SQL Server storage options: 6.2. Configuration 27 Hangfire Documentation, Release 1.7 var options = new SqlServerStorageOptions { PrepareSchemaIfNecessary = false }; GlobalConfiguration.Configuration.UseSqlServerStorage("<name or connection string>", →˓options); You can isolate HangFire database access to just the HangFire schema. You need to create a separate HangFire user and grant the user access only to the HangFire schema. The HangFire user will only be able to alter the HangFire schema. Below is an example of using a contained database user for HangFire. The HangFire user has least privileges required but still allows it to upgrade the schema correctly in the future. CREATE USER [HangFire] WITH PASSWORD = 'strong_password_for_hangfire' GO IF NOT EXISTS (SELECT 1 FROM sys.schemas WHERE [name] = 'HangFire') EXEC ('CREATE →˓SCHEMA [HangFire]') GO ALTER AUTHORIZATION ON SCHEMA::[HangFire] TO [HangFire] GO GRANT CREATE TABLE TO [HangFire] GO Configuring the Polling Interval One of the main disadvantage of raw SQL Server job storage implementation – it uses the polling technique to fetch new jobs. Starting from Hangfire 1.7.0 it’s possible to use TimeSpan.Zero as a polling interval, when SlidingInvisibilityTimeout option is set. var options = new SqlServerStorageOptions { SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5), QueuePollInterval = TimeSpan.Zero }; GlobalConfiguration.Configuration.UseSqlServerStorage("<name or connection string>", →˓options); This is the recommended value in that version, but you can decrease the polling interval if your background jobs can tolerate additional delay before the invocation. 6.2.3 Using SQL Server with MSMQ This extension will be deprecated soon Hangfire.SqlServer 1.7.X versions have long-polling feature implemented when using the recommended settings with the latest schema version. Additional technology like MSMQ complicates the application infrastructure, additional storage like MSMQ brings consistency issues on data loss, and MSMQ itself is a complicated technology, especially with DTC transactions. Therefore, the number of disadvantages outweigh all the advantages of using this extension. 28 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 Please, see the downloads page to obtain latest version of Redis. If you unfamiliar with this great storage, please see its documentation. Binaries for Windows are available through NuGet (32-bit, 64-bit) and Chocolatey galleries (64-bit package only). Limitations Multiple Redis endpoints are only supported in Redis Cluster configuration starting from Hangfire.Pro.Redis 2.1.0. You can’t use multiple detached masters or Redis Sentinel configurations. Redis Configuration Please read the official Redis documentation to learn how to configure it, especially Redis Persistence and Redis Administration sections to get started with the fundamentals. The following options should be configured to run your background jobs smoothly. Ensure the following options are configured These values are default for on-premise Redis installations, but other environments may have different defaults, for example Azure Redis Cache and AWS ElastiCache use non-compatible settings by default. # Hangfire neither expect that non-expired keys are deleted, # nor expiring keys are evicted before the expiration time. maxmemory-policy noeviction # Non-zero value cause long-running background jobs to be # processed multiple times due to connection being closed. # ONLY FOR Hangfire.Pro.Redis 1.X! timeout 0 6.2. Configuration 31 Hangfire Documentation, Release 1.7 If you are planning to use the Redis ACL feature, below you can find a minimal supported set of rules you can specify to use Redis as a job storage. They restrict keyspace for regular and pub/sub commands to the default prefix used by Hangfire (hangfire:) and declare a minimal set of Redis commands used by Hangfire.Pro.Redis. resetkeys ~hangfire:* resetchannels &hangfire:* nocommands +info +ping +echo +select →˓+cluster +time +@read +@write +@set +@sortedset +@list +@hash +@string +@pubsub →˓+@transaction +@scripting Hangfire.Pro.Redis 2.x Redis 2.6.12 is required Installation Ensure that you have configured the private Hangfire Pro NuGet feed as written here, and use your favorite NuGet client to install the Hangfire.Pro.Redis package: PM> Install-Package Hangfire.Pro.Redis If your project targets .NET Core, just add a dependency in your *.csproj file: <ItemGroup> <PackageReference Include="Hangfire.Pro.Redis" Version="2.8.2" /> </ItemGroup> Configuration After installing the package, a couple of the UseRedisStorage extension method overloads will be available for the IGlobalConfiguration interface. They allow you to configure Redis job storage, using both configuration string and Hangfire-specific options. Connection string The basic one is the following, will connect to the Redis on localhost using the default port, database and options: GlobalConfiguration.Configuration.UseRedisStorage(); For ASP.NET Core projects, call the UseRedisStorage method from the AddHangfire method delegate: services.AddHangfire(configuration => configuration.UseRedisStorage()); You can customize the connection string using the StackExchange.Redis’ configuration string format. Please read their documentation for details. The values for the following options have their own defaults in Hangfire, but can be overridden in the connection string: Option Default syncTimeout 30000 allowAdmin true 32 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 GlobalConfiguration.Configuration .UseRedisStorage("contoso5.redis.cache.windows.net,abortConnect=false,ssl=true, →˓password=..."); Redis Cluster support You can use a single endpoint to connect to a Redis cluster, Hangfire will detect other instances automatically by querying the node configuration. However, it’s better to pass multiple endpoints in order to mitigate connectivity issues, when some of endpoints aren’t available, e.g. during the failover process. Since Hangfire requires transactions, and Redis doesn’t support ones that span multiple hash slots, you also need to configure the prefix to assign it to the same hash tag: GlobalConfiguration.Configuration.UseRedisStorage( "localhost:6379,localhost:6380,localhost:6381", new RedisStorageOptions { Prefix = "{hangfire-1}:" }); This will bind all the keys to a single Redis instance. To be able to fully utilize your Redis cluster, consider using multiple JobStorage instances and leveraging some load-balancing technique (round-robin is enough for the most cases). To do so, pick different hash tags for different storages and ensure they are using hash slots that live on different masters by using commands CLUSTER NODES and CLUSTER KEYSLOT. Passing options You can also pass the Hangfire-specific options for Redis storage by using the RedisStorageOptions class instances: var options = new RedisStorageOptions { Prefix = "hangfire:app1:" }; GlobalConfiguration.Configuration.UseRedisStorage("localhost", options); The following options are available for configuration: 6.2. Configuration 33 Hangfire Documentation, Release 1.7 Listing 2: appsettings.json { "Logging": { "LogLevel": { "Default": "Warning", "Hangfire": "Information" } } } Once integration is complete, please refer to the official Logging in ASP.NET Core article to learn how to configure the logging subsystem of .NET Core itself. .NET Framework If your application uses one of the following libraries, no manual action is required in the most cases. Hangfire knows about these loggers and uses reflection to determine the first available one (in the order defined below) and to call the corresponding methods when logging. And since reflection is used, there are no unnecessary package or assembly references. 1. Serilog 2. NLog 3. Log4Net 4. EntLib Logging 5. Loupe 6. Elmah Automatic wiring works correctly when your project references only a single logging package. Also, due to breaking changes (rare enough in the packages above), it’s possible that wiring doesn’t succeed. And to explicitly tell Hangfire what package to use to avoid the ambiguity, you can call one of the following methods (last invocation wins). GlobalConfiguration.Configuration .UseSerilogLogProvider() .UseNLogLogProvider() .UseLog4NetLogProvider() .UseEntLibLogProvider() .UseLoupeLogProvider() .UseElmahLogProvider(); If your project doesn’t have the required references when calling these methods, you may get a run-time exception. Of course if you don’t have any logging package installed or didn’t configure it properly, Hangfire will not log anything, falling back to the internal NoOpLogger class. So it’s a great time to install one, for example Serilog, as it’s the most simple logging package to set up. Console logger For simple applications you can use the built-in console log provider, please see the following snippet to learn how to activate it. But please ensure you aren’t using it in production environments, because this logger may produce unwanted blocks, since global lock is obtained each time we are writing a message to ensure the colors are correct. 36 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 GlobalConfiguration.Configuration.UseColouredConsoleLogProvider(); Using a custom logger If your application uses another logging library that’s not listed above, you can implement your own logging adapter. Please see the following snippet to learn how to do this – all you need is to implement some interfaces and register the resulting log provider in a global configuration instance. using Hangfire.Logging; public class CustomLogger : ILog { public string Name { get; set; } public bool Log(LogLevel logLevel, Func<string> messageFunc, Exception exception →˓= null) { if (messageFunc == null) { // Before calling a method with an actual message, LogLib first probes // whether the corresponding log level is enabled by passing a `null` // messageFunc instance. return logLevel > LogLevel.Info; } // Writing a message somewhere, make sure you also include the exception →˓parameter, // because it usually contain valuable information, but it can be `null` for →˓regular // messages. Console.WriteLine(String.Format("{0}: {1} {2} {3}", logLevel, Name, →˓messageFunc(), exception)); // Telling LibLog the message was successfully logged. return true; } } public class CustomLogProvider : ILogProvider { public ILog GetLogger(string name) { // Logger name usually contains the full name of a type that uses it, // e.g. "Hangfire.Server.RecurringJobScheduler". It's used to know the // context of this or that message and for filtering purposes. return new CustomLogger { Name = name }; } } After implementing the interfaces above, call the following method: GlobalConfiguration.Configuration.UseLogProvider(new CustomLogProvider()); 6.2. Configuration 37 Hangfire Documentation, Release 1.7 Log level description There are the following semantics behind each log level. Please take into account that some logging libraries may have slightly other names for these levels, but usually they are almost the same. If you are looking for a good candidate for the minimal log level configuration in your application, choose the LogLevel.Info. Level Description TraceThese messages are for debugging Hangfire itself to see what events happened and what conditional branches taken. DebugUse this level to know why background processing does not work for you. There are no message count thresholds for this level, so you can use it when something is going wrong. But expect much higher number of messages, comparing to the next levels. Info This is the recommended minimal level to log from, to ensure everything is working as expected. Processing server is usually using this level to notify about start and stop events – perhaps the most important ones, because inactive server doesn’t process anything. Starting from this level, Hangfire tries to log as few messages as possible to not to harm your logging subsystem. Warn Background processing may be delayed due to some reason. You can take the corresponding action to minimize the delay, but there will be yet another automatic retry attempt anyway. ErrorBackground process or job is unable to perform its work due to some external error which lasts for a long time. Usually a message with this level is logged only after a bunch of retry attempts to ensure you don’t get messages on transient errors or network blips. Also usually you don’t need to restart the processing server after resolving the cause of these messages, because yet another attempt will be made automatically after some delay. FatalCurrent processing server will not process background jobs anymore, and manual intervention is required. This log level is almost unused in Hangfire, because there are retries almost everywhere, except in the retry logic itself. Theoretically, ThreadAbortException may cause a fatal error, but only if it’s thrown in a bad place – usually thread aborts are being reset automatically. Please also keep in mind that we can’t log anything if process is died unexpectedly. 6.3 Background Methods Background jobs in Hangfire look like regular method calls. Most of its interfaces are using expression trees to define what method should be called and with what arguments. And background jobs can use both instance and static method calls as in the following example. BackgroundJob.Enqueue<IEmailSender>(x => x.Send("hangfire@example.com")); BackgroundJob.Enqueue(() => Console.WriteLine("Hello, world!")); These lines use expression trees – not delegates like Action or Func<T>. And unlike usual method invocations, they are supposed to be executed asynchronously and even outside of the current process. So the purpose of the method calls above is to collect and serialize the following information. • Type name, including namespace and assembly. • Method name and its parameter types. • Argument values. Serialization is performed by the Newtonsoft.Json package and resulting JSON, that looks like in the following snippet, is persisted in a storage making it available for other processes. As we can see everything is passed by value, so heavy data structures will also be serialized and consume a lot of bytes in our storage. 38 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 Performing recurrent tasks Recurring job registration is almost as simple as background job registration – you need to write a single line of code, but you also need to specify an identifier you can use to refer to your job later. The call to AddOrUpdate method will create a new recurring job or update existing job with the same identifier. RecurringJob.AddOrUpdate("easyjob", () => Console.Write("Easy!"), Cron.Daily); This line creates a new entry in persistent storage. A special component in Hangfire Server (see Processing background jobs) checks the recurring jobs on a minute-based interval and then enqueues them as fire-and-forget jobs. This enables you to track them as usual. Make sure your app is always running Your Hangfire Server instance should be always on to perform scheduling and processing logic. If you perform the processing inside an ASP.NET application, please also read the Making ASP.NET application always running chapter. The Cron class contains different methods and overloads to run jobs on a minute, hourly, daily, weekly, monthly and yearly basis. You can also use CRON expressions to specify a more complex schedule: RecurringJob.AddOrUpdate("powerfuljob", () => Console.Write("Powerful!"), "0 12 * */2 →˓"); Identifiers should be unique Use unique identifiers for each recurring job, otherwise you’ll end with a single job. Identifiers may be case sensitive Recurring job identifier may be case sensitive in some storage implementations. Manipulating recurring jobs You can remove an existing recurring job by calling the RemoveIfExists method. It does not throw an exception when there is no such recurring job. RecurringJob.RemoveIfExists("some-id"); To run a recurring job now, call the Triggermethod. The information about triggered invocation will not be recorded in the recurring job itself, and its next execution time will not be recalculated from this running. For example, if you have a weekly job that runs on Wednesday, and you manually trigger it on Friday it will run on the following Wednesday. RecurringJob.Trigger("some-id"); The RecurringJob class is a facade for the RecurringJobManager class. If you want some more power and responsibility, consider using it: var manager = new RecurringJobManager(); manager.AddOrUpdate("some-id", Job.FromExpression(() => Method()), Cron.Yearly()); 6.3. Background Methods 41 Hangfire Documentation, Release 1.7 Passing arguments You can pass additional data to your background jobs as a regular method arguments. I’ll write the following line once again (hope it hasn’t bothered you): BackgroundJob.Enqueue(() => Console.WriteLine("Hello, {0}!", "world")); As in a regular method call, these arguments will be available for the Console.WriteLine method during the performance of the background job. But since they are marshaled through process boundaries, they are serialized. The awesome Newtonsoft.Json package is used to serialize arguments into JSON strings (since version 1.1.0). So you can use almost any type as a parameter; including arrays, collections and custom objects. Please see corresponding documentation for more details. Reference parameters are not supported You can not pass arguments to parameters by reference – ref and out keywords are not supported. Since arguments are serialized, consider their values carefully as they can blow up your job storage. Most of the time it is more efficient to store concrete values in an application database and pass their identifiers only to your background jobs. Remember that background jobs may be processed days or weeks after they were enqueued. If you use data that is subject to change in your arguments, it may become stale – database records may be deleted, the text of an article may be changed, etc. Plan for these changes and design your background jobs accordingly. Passing dependencies In almost every job you’ll want to use other classes of your application to perform different work and keep your code clean and simple. Let’s call these classes as dependencies. How to pass these dependencies to methods that will be called in background? When you are calling static methods in background, you are restricted only to the static context of your application, and this requires you to use the following patterns of obtaining dependencies: • Manual dependency instantiation through the new operator • Service location • Abstract factories or builders • Singletons However, all of these patterns greatly complicate the unit testability aspect of your application. To fight with this issue, Hangfire allows you to call instance methods in background. Consider you have the following class that uses some kind of DbContext to access the database, and EmailService to send emails. public class EmailSender { public void Send(int userId, string message) { var dbContext = new DbContext(); var emailService = new EmailService(); // Some processing logic } } 42 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 To call the Send method in background, use the following override of the Enqueue method (other methods of BackgroundJob class provide such overloads as well): BackgroundJob.Enqueue<EmailSender>(x => x.Send(13, "Hello!")); When a worker determines that it needs to call an instance method, it creates the instance of a given class first using the current JobActivator class instance. By default, it uses the Activator.CreateInstance method that can create an instance of your class using its default constructor, so let’s add it: public class EmailSender { private IDbContext _dbContext; private IEmailService _emailService; public EmailSender() { _dbContext = new DbContext(); _emailService = new EmailService(); } // ... } If you want the class to be ready for unit testing, consider to add constructor overload, because the default activator can not create instance of class that has no default constructor: public class EmailSender { // ... public EmailSender() : this(new DbContext(), new EmailService()) { } internal EmailSender(IDbContext dbContext, IEmailService emailService) { _dbContext = dbContext; _emailService = emailService; } } If you are using IoC containers, such as Autofac, Ninject, SimpleInjector and so on, you can remove the default constructor. To learn how to do this, proceed to the next section. Using IoC containers As I said in the previous section Hangfire uses the JobActivator class to instantiate the target types before invoking instance methods. You can override its behavior to perform more complex logic on a type instantiation. For example, you can tell it to use IoC container that is used in your project: public class ContainerJobActivator : JobActivator { private IContainer _container; public ContainerJobActivator(IContainer container) (continues on next page) 6.3. Background Methods 43 Hangfire Documentation, Release 1.7 Hangfire takes care of passing a proper non-null instance of IJobCancellationToken during the job execution at runtime. Writing unit tests I will not tell you anything related to unit testing background methods, because Hangfire does not add any specific changes to them (except IJobCancellationToken interface parameter). Use your favourite tools and write unit tests for them as usual. This section describes how to test that background jobs were created. All the code examples use the static BackgroundJob class to tell you how to do this or that stuff, because it is simple for demonstrational purposes. But when you want to test a method that invokes static methods, it becomes a pain. But don’t worry – the BackgroundJob class is just a facade for the IBackgroundJobClient interface and its default implementation – BackgroundJobClient class. If you want to write unit tests, use them. For example, consider the following controller that is used to enqueue background jobs: public class HomeController : Controller { private readonly IBackgroundJobClient _jobClient; // For ASP.NET MVC public HomeController() : this(new BackgroundJobClient()) { } // For unit tests public HomeController(IBackgroundJobClient jobClient) { _jobClient = jobClient; } public ActionResult Create(Comment comment) { ... _jobClient.Enqueue(() => CheckForSpam(comment.Id)); ... } } Simple, yeah. Now you can use any mocking framework, for example, Moq to provide mocks and check the invo- cations. The IBackgroundJobClient interface provides only one method for creating a background job – the Create method, that takes a Job class instance, that represents the information about the invocation, and a IState interface implementation to know the creating job’s state. [TestMethod] public void CheckForSpamJob_ShouldBeEnqueued() { // Arrange var client = new Mock<IBackgroundJobClient>(); var controller = new HomeController(client.Object); var comment = CreateComment(); // Act controller.Create(comment); (continues on next page) 46 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 (continued from previous page) // Assert client.Verify(x => x.Create( It.Is<Job>(job => job.Method.Name == "CheckForSpam" && job.Args[0] == comment. →˓Id), It.IsAny<EnqueuedState>()); } Note: job.Method property points only to background job’s method information. If you also want to check a type name, use the job.Type property. Using Batches Pro Only This feature is a part of Hangfire Pro package set Batches allow you to create a bunch of background jobs atomically. This means that if there was an exception during the creation of background jobs, none of them will be processed. Consider you want to send 1000 emails to your clients, and they really want to receive these emails. Here is the old way: for (var i = 0; i < 1000; i++) { BackgroundJob.Enqueue(() => SendEmail(i)); // What to do on exception? } But what if storage become unavailable on i == 500? 500 emails may be already sent, because worker threads will pick up and process jobs once they created. If you re-execute this code, some of your clients may receive annoying duplicates. So if you want to handle this correctly, you should write more code to track what emails were sent. But here is a much simpler method: BatchJob.StartNew(x => { for (var i = 0; i < 1000; i++) { x.Enqueue(() => SendEmail(i)); } }); In case of exception, you may show an error to a user, and simply ask to retry her action after some minutes. No other code required! Installation Batches are available in the Hangfire.Pro package, and you can install it using NuGet Package Manager Console window as usually: PM> Install-Package Hangfire.Pro 6.3. Background Methods 47 Hangfire Documentation, Release 1.7 Batches require to add some additional job filters, some new pages to the Dashboard, and some new navigation menu items. But thanks to the new GlobalConfiguration class, it is now as simple as a one method call: GlobalConfiguration.Configuration.UseBatches(); Limited storage support Only official Hangfire.InMemory, Hangfire.SqlServer and Hangfire.Pro.Redis job storage implementations are currently supported. There is nothing special for batches, but some new storage methods should be implemented. Configuration The default batch job expiration/retention time if the batch succeeds is 7 days, but you can configure it when calling the UseBatches method: GlobalConfiguration.Configuration.UseBatches(TimeSpan.FromDays(2)); Chaining Batches Continuations allow you to chain multiple batches together. They will be executed once all background jobs of a parent batch finished. Consider the previous example where you have 1000 emails to send. If you want to make final action after sending, just add a continuation: var id1 = BatchJob.StartNew(/* for (var i = 0; i < 1000... */); var id2 = BatchJob.ContinueBatchWith(id1, x => { x.Enqueue(() => MarkCampaignFinished()); x.Enqueue(() => NotifyAdministrator()); }); So batches and batch continuations allow you to define workflows and configure what actions will be executed in parallel. This is very useful for heavy computational methods as they can be distributed to different machines. Complex Workflows Create action does not restrict you to create jobs only in Enqueued state. You can schedule jobs to execute later, add continuations, add continuations to continuations, etc.. var batchId = BatchJob.StartNew(x => { x.Enqueue(() => Console.Write("1a... ")); var id1 = x.Schedule(() => Console.Write("1b... "), TimeSpan.FromSeconds(1)); var id2 = x.ContinueJobWith(id1, () => Console.Write("2... ")); x.ContinueJobWith(id2, () => Console.Write("3... ")); }); BatchJob.ContinueBatchWith(batchId, x => { x.Enqueue(() => Console.WriteLine("4...")); }); 48 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 More Continuations Since version 2.0 it’s possible to continue batch with a regular background job without creating a batch that consists only of a single background job. Unfortunately we can’t add extension methods for static classes, so let’s create a client first. var backgroundJob = new BackgroundJobClient(); var batchId = BatchJob.StartNew(/* ... */); backgroundJob.ContinueBatchWith(batchId, () => Console.WriteLine("Continuation")); You can use the new feature in other way, and create batch continuations for regular background jobs. So you are free to define workflows, where synchronous actions are continued by a group of parallel work, and then continue back to a synchronous method. var jobId = BackgroundJob.Enqueue(() => Console.WriteLine("Antecedent")); BatchJob.ContinueJobWith(jobId, batch => batch.Enqueue(() => Console.WriteLine( →˓"Continuation"))); Cancellation of a Batch If you want to stop a batch with millions of background jobs from being executed, not a problem, you can call the Cancel method, or click the corresponding button in dashboard. var batchId = BatchJob.StartNew(/* a lot of jobs */); BatchJob.Cancel(batchId); This method does not iterate through all the jobs, it simply sets a property of a batch. When a background job is about to execute, job filter checks for a batch status, and move a job to the Deleted state, if a batch has cancelled. 6.4 Background Processing 6.4.1 Processing background jobs Hangfire Server part is responsible for background job processing. The Server does not depend on ASP.NET and can be started anywhere, from a console application to Microsoft Azure Worker Role. Single API for all applications is exposed through the BackgroundJobServer class: // Create an instance of Hangfire Server and start it. // Please look at ctor overrides for advanced options like // explicit job storage instance. var server = new BackgroundJobServer(); // Wait for graceful server shutdown. server.Dispose(); Always dispose your background server Call the Dispose method whenever possible to have graceful shutdown features working. 6.4. Background Processing 51 Hangfire Documentation, Release 1.7 Hangfire Server consists of different components that are doing different work: workers listen to queue and process jobs, recurring scheduler enqueues recurring jobs, schedule poller enqueues delayed jobs, expire manager removes obsolete jobs and keeps the storage as clean as possible, etc. You can turn off the processing If you don’t want to process background jobs in a specific application instance, just don’t create an instance of the BackgroundJobServer class. The Dispose method is a blocking one, it waits until all the components prepare for shutdown (for example, workers will place back interrupted jobs to their queues). So, we can talk about graceful shutdown only after waiting for all the components. Strictly saying, you aren’t required to invoke the Dispose method. Hangfire can handle even unexpected process terminations, and will retry interrupted jobs automatically. However it is better to control the exit points in your methods by using cancellation tokens. 6.4.2 Processing jobs in a web application Ability to process background jobs directly in web applications is a primary goal of Hangfire. No external application like Windows Service or console application is required for running background jobs, however you will be able to change your decision later if you really need it. So, you can postpone architecture decisions that complicate things. Since Hangfire does not have any specific dependencies and does not depend on System.Web, it can be used together with any web framework for .NET: • ASP.NET WebForms • ASP.NET MVC • ASP.NET WebApi • ASP.NET vNext (through the app.UseOwin method) • Other OWIN-based web frameworks (Nancy, Simple.Web) • Other non-OWIN based web frameworks (ServiceStack) Using BackgroundJobServer class The basic way (but not the simplest – see the next section) to start using Hangfire in a web framework is to use host-agnostic BackgroundJobServer class that was described in the previous chapter and call its Start and Dispose method in corresponding places. Dispose the server instance when possible In some web application frameworks it may be unclear when to call the Dispose method. If it is really impossible, you can omit this call as described here (but you’ll lose the graceful shutdown feature). For example, in ASP.NET applications the best place for start/dispose method invocations is the global.asax.cs file: using System; using System.Web; using Hangfire; (continues on next page) 52 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 (continued from previous page) namespace WebApplication1 { public class Global : HttpApplication { private BackgroundJobServer _backgroundJobServer; protected void Application_Start(object sender, EventArgs e) { GlobalConfiguration.Configuration .UseSqlServerStorage("DbConnection"); _backgroundJobServer = new BackgroundJobServer(); } protected void Application_End(object sender, EventArgs e) { _backgroundJobServer.Dispose(); } } } Using OWIN extension methods Hangfire also provides a dashboard that is implemented on top of OWIN pipeline to process requests. If you have simple set-up and want to keep Hangfire initialization logic in one place, consider using Hangfire’s extension methods for OWIN’s IAppBuilder interface: Install Microsoft.Owin.Host.SystemWeb for ASP.NET + IIS If you are using OWIN extension methods for ASP.NET application hosted in IIS, ensure you have Microsoft. Owin.Host.SystemWeb package installed. Otherwise some features like graceful shutdown feature will not work for you. If you installed Hangfire through the Hangfire package, this dependency is already installed. public class Startup { public void Configuration(IAppBuilder app) { app.UseHangfireServer(); } } This line creates a new instance of the BackgroundJobServer class automatically, calls the Start method and registers method Dispose invocation on application shutdown. The latter is implemented using a CancellationToken instance stored in the host.OnAppDisposing environment key. 6.4.3 Processing jobs in a console application To start using Hangfire in a console application, you’ll need to install Hangfire packages to your console application first. So, use your Package Manager Console window to install it: 6.4. Background Processing 53 Hangfire Documentation, Release 1.7 Then build your project, install your Windows Service and run it. If it fails, try look at your Windows Event Viewer for recent exceptions. installutil <yourproject>.exe 6.4.5 Dealing with exceptions Bad things happen. Any method can throw different types of exceptions. These exceptions can be caused either by programming errors that require you to re-deploy the application, or transient errors, that can be fixed without additional deployment. Hangfire handles all exceptions occurred both in internal (belonging to Hangfire itself), and external methods (jobs, filters and so on), so it will not bring down the whole application. All internal exceptions are logged (so, don’t forget to enable logging) and the worst case they can lead – background processing will be stopped after 10 retry attempts with increasing delay modifier. When Hangfire encounters external exception that occurred during the job performance, it will automatically try to change its state to the Failed one, and you always can find this job in the Monitor UI (it will not be expired unless you delete it explicitly). 56 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 In the previous paragraph I said that Hangfire will try to change its state to failed, because state transition is one of places, where job filters can intercept and change the initial pipeline. And the AutomaticRetryAttribute class is one of them, that schedules the failed job to be automatically retried after increasing delay. This filter is applied globally to all methods and have 10 retry attempts by default. So, your methods will be retried in case of exception automatically, and you receive warning log messages on every failed attempt. If retry attempts exceeded their maximum, the job will be move to the Failed state (with an error log message), and you will be able to retry it manually. If you don’t want a job to be retried, place an explicit attribute with 0 maximum retry attempts value: [AutomaticRetry(Attempts = 0)] public void BackgroundMethod() { } Use the same way to limit the number of attempts to the different value. If you want to change the default global value, add a new global filter: GlobalJobFilters.Filters.Add(new AutomaticRetryAttribute { Attempts = 5 }); If you are using ASP.NET Core you can use the IServiceCollection extension method AddHangfire. Note that Ad- dHangfire uses the GlobalJobFilter instance and therefore dependencies should be Transient or Singleton. services.AddHangfire((provider, configuration) => { configuration.UseFilter(provider.GetRequiredService<AutomaticRetryAttribute>()); } 6.4.6 Tracking the progress There are two ways to implement this task: polling and pushing. Polling is easier to understand, but server push is a more comfortable way, because it helps you to avoid unnecessary calls to server. Plus, SignalR greatly simplifies the latter task. I’ll show you a simple example, where client only needs to check for a job completion. You can see the full sample in Hangfire.Highlighter project. Highlighter has the following background job that calls an external web service to highlight code snippets: 6.4. Background Processing 57 Hangfire Documentation, Release 1.7 public void Highlight(int snippetId) { var snippet = _dbContext.CodeSnippets.Find(snippetId); if (snippet == null) return; snippet.HighlightedCode = HighlightSource(snippet.SourceCode); snippet.HighlightedAt = DateTime.UtcNow; _dbContext.SaveChanges(); } Polling for a job status When can we say that this job is incomplete? When the HighlightedCode property value is null. When can we say it was completed? When the specified property has value – this example is simple enough. So, when we are rendering the code snippet that is not highlighted yet, we need to render a JavaScript that makes ajax calls with some interval to some controller action that returns the job status (completed or not) until the job was finished. public ActionResult CheckHighlighted(int snippetId) { var snippet = _db.Snippets.Find(snippetId); return snippet.HighlightedCode == null ? new HttpStatusCodeResult(HttpStatusCode.NoContent) : Content(snippet.HighlightedCode); } When code snippet becomes highlighted, we can stop the polling and show the highlighted code. But if you want to track progress of your job, you need to perform extra steps: • Add a column Status to the snippets table. • Update this column during background work. • Check this column in polling action. But there is a better way. Using server push with SignalR Why do we need to poll our server? It can say when the snippet becomes highlighted itself. And SignalR, an awesome library to perform server push, will help us. If you don’t know about this library, look at it, and you’ll love it. Really. I don’t want to include all the code snippets here (you can look at the sources of this sample). I’ll show you only the two changes that you need, and they are incredibly simple. First, you need to add a hub: public class SnippetHub : Hub { public async Task Subscribe(int snippetId) { await Groups.Add(Context.ConnectionId, GetGroup(snippetId)); // When a user subscribes a snippet that was already (continues on next page) 58 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 6.4.9 Running multiple server instances Obsolete since 1.5 You aren’t required to have additional configuration to support multiple background processing servers in the same process since Hangfire 1.5, just skip the article. Server identifiers are now generated using GUIDs, so all the instance names are unique. It is possible to run multiple server instances inside a process, machine, or on several machines at the same time. Each server use distributed locks to perform the coordination logic. Each Hangfire Server has a unique identifier that consists of two parts to provide default values for the cases written above. The last part is a process id to handle multiple servers on the same machine. The former part is the server name, that defaults to a machine name, to handle uniqueness for different machines. Examples: server1:9853, server1:4531, server2:6742. Since the defaults values provide uniqueness only on a process level, you should handle it manually if you want to run different server instances inside the same process: var options = new BackgroundJobServerOptions { ServerName = String.Format( "{0}.{1}", Environment.MachineName, Guid.NewGuid().ToString()) }; var server = new BackgroundJobServer(options); // or app.UseHangfireServer(options); 6.4.10 Configuring Job Queues Hangfire can process multiple queues. If you want to prioritize your jobs, or split the processing across your servers (some processes for the archive queue, others for the images queue, etc), you can tell Hangfire about your decisions. To place a job into a different queue, use the QueueAttribute class on your method: [Queue("alpha")] public void SomeMethod() { } BackgroundJob.Enqueue(() => SomeMethod()); Queue name argument formatting The Queue name argument must consist of lowercase letters, digits, underscore, and dash (since 1.7.6) characters only. To begin processing multiple queues, you need to update your BackgroundJobServer configuration. var options = new BackgroundJobServerOptions { (continues on next page) 6.4. Background Processing 61 Hangfire Documentation, Release 1.7 (continued from previous page) Queues = new[] { "alpha", "beta", "default" } }; app.UseHangfireServer(options); // or using (new BackgroundJobServer(options)) { /* ... */ } Processing order Queues are run in the order that depends on the concrete storage implementation. For example, when we are us- ing Hangfire.SqlServer the order is defined by alphanumeric order and array index is ignored. When using Hang- fire.Pro.Redis package, array index is important and queues with a lower index will be processed first. The example above shows a generic approach, where workers will fetch jobs from the alpha queue first, beta second, and then from the default queue, regardless of an implementation. ASP.NET Core For ASP.NET Core, define the queues array with services.AddHangfireServer in Startup.cs: public void ConfigureServices(IServiceCollection services) { // Add the processing server as IHostedService services.AddHangfireServer(options => { options.Queues = new[] { "alpha", "beta", "default" }; }); } 6.4.11 Concurrency & Rate Limiting Note: Hangfire.Throttling package is a part of Hangfire.Ace extensibility set and available on the private NuGet feed. Hangfire.Throttling package contains advanced types and methods to apply concurrency and rate limits directly to our background jobs without touching any logic related to queues, workers, servers or using additional services. So we can control how many particular background jobs are running at the same point of time or within a specific time window. Throttling is performed asynchronously by rescheduling jobs to a later time or deleting them when throttling condition is met, depending on the configured behavior. And while throttled jobs are waiting for their turn, our workers are free to process other enqueued background jobs. The primary focus of this package is to provide a simpler way of reducing the load on external resources. Databases or third-party services affected by background jobs may suffer from additional concurrency, causing increased latency and error rates. While standard solution to use different queues with a constrained number of workers works well enough, it requires additional infrastructure planning and may lead to underutilization. And throttling primitives are much easier to use for this purpose. Everything works on a best-effort basis 62 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 While it can be possible to use this package to enforce proper synchronization and concurrency control over back- ground jobs, it’s very hard to achieve it due to the complexity of distributed processing. There are a lot of things to consider, including appropriate storage configuration, and a single mistake will ruin everything. Throttlers apply only to different background jobs, and there’s no reliable way to prevent multiple ex- ecutions of the same background job other than by using transactions in background job method itself. DisableConcurrentExecution may help a bit by narrowing the safety violation surface, but it heavily re- lies on an active connection, which may be broken (and lock is released) without any notification for our background job. Hangfire.Throttling provides the following primitives, all of them are implemented as regular state changing filters that run when a worker is starting or completing a background job. They form two groups, depending on their acquire and release behavior. Concurrency Limiters • Mutexes – allow only a single background job to be running concurrently. • Semaphores – limit how many background jobs are allowed to run concurrently. Rate Limiters • Fixed Window Counters – limit how many job executions are allowed within a given fixed time interval. • Sliding Window Counters – limit how many job executions are allowed to run within any specific time interval. • Dynamic Window Counters – allow to create window counters dynamically depending on job arguments. Requirements Supported only for Hangfire.SqlServer (better to use 1.7) and Hangfire.Pro.Redis (recommended 2.4.0) storages. Community storage support will be denoted later after defining correctness conditions for storages. Installation The package is available on a private Hangfire.Ace NuGet feed (that’s different from Hangfire.Pro one), please see the Downloads page to learn how to use it. After registering the private feed, we can install the Hangfire. Throttling package by editing our .csproj file for new project types: <PackageReference Include="Hangfire.Throttling" Version="1.0.*" /> Alternatively we can use Package Manager Console window to install it using the Install-Package command as shown below. Install-Package Hangfire.Throttling Configuration The only configuration method required for throttlers is the IGlobalConfiguration.UseThrottling exten- sion method. If we don’t call this method, every background job decorated with any throttling filter will be eventually moved to the failed state. GlobalConfiguration.Configuration .UseXXXStorage() .UseThrottling(); 6.4. Background Processing 63 Hangfire Documentation, Release 1.7 [Mutex("orders:{0}")] [Throttling(StrictMode = true)] Task ProcessOrderAsync(long orderId); In either mode, throttler’s release and background job’s state transition performed in the same transaction. Concurrency Limiters Mutexes Mutex prevents concurrent execution of multiple background jobs that share the same resource identifier. Unlike other primitives, they are created dynamically so we don’t need to use IThrottlingManager to create them first. All we need is to decorate our background job methods with the MutexAttribute filter and define what resource identifier should be used. [Mutex("my-resource")] public void MyMethod() { // ... } When we create multiple background jobs based on this method, they will be executed one after another on a best- effort basis with the limitations described below. If there’s a background job protected by a mutex currently executing, other executions will be throttled (rescheduled by default a minute later), allowing a worker to process other jobs without waiting. Mutex doesn’t prevent simultaneous execution of the same background job As there are no reliable automatic failure detectors in distributed systems, it is possible that the same job is being processed on different workers in some corner cases. Unlike OS-based mutexes, mutexes in this package don’t protect from this behavior so develop accordingly. DisableConcurrentExecution filter may reduce the probability of violation of this safety property, but the only way to guarantee it is to use transactions or CAS-based operations in our background jobs to make them idem- potent. If a background job protected by a mutex is unexpectedly terminated, it will simply re-enter the mutex. It will be held until background job is moved to the final state (Succeeded, Deleted, but not Failed). We can also create multiple background job methods that share the same resource identifier, and mutual exclusive behavior will span all of them, regardless of the method name. [Mutex("my-resource")] public void FirstMethod() { /* ... */ } [Mutex("my-resource")] public void SecondMethod() { /* ... */ } Since mutexes are created dynamically, it’s possible to use a dynamic resource identifier based on background job arguments. To define it, we should use String.Format-like templates, and during invocation all the placeholders will be replaced with actual job arguments. But ensure everything is lower-cased and contains only alphanumeric characters with limited punctuation – no rules except maximum length and case insensitivity is enforced, but it’s better to keep identifiers simple. 66 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 Maximal length of resource identifiers is 100 characters Please keep this in mind especially when using dynamic resource identifiers. [Mutex("orders:{0}")] public void ProcessOrder(long orderId) { /* ... */ } [Mutex("newsletters:{0}:{1}")] public void ProcessNewsletter(int tenantId, long newsletterId) { /* ... */ } Throttling Batches By default background job id is used to identify the current owner of a particular mutex. But since version 1.3 it is possible to use any custom value from a given job parameter. With this feature we can throttle entire batches, since we can pass the BatchId job parameter that’s used to store the batch identifier. To accomplish this, we need to create two empty methods with ThrottlerMode.Acquire and ThrottlerMode.Release semantics that will acquire and release a mutex: [Mutex("orders-api", Mode = ThrottlerMode.Acquire, ParameterName = "BatchId")] public static void StartBatch() { /* Doesn't do anything */ } [Mutex("orders-api", Mode = ThrottlerMode.Release, ParameterName = "BatchId")] public static void CompleteBatch() { /* Doesn't do anything */ } And then create a batch as a chain of continuations, starting with the StartBatch method and ending with the CompleteBatch method. Please note that the last method is created with the BatchContinuationOptions. OnAnyFinishedState option to release the throttler even if some of our background jobs completed non- successfully (deleted, for example). BatchJob.StartNew(batch => { var startId = batch.Enqueue(() => StartBatch()); var bodyId = batch.ContinueJobWith(startId, nestedBatch => { for (var i = 0; i < 5; i++) nestedBatch.Enqueue(() => Thread.Sleep(5000)); }); batch.ContinueBatchWith( bodyId, () => CompleteBatch(), options: BatchContinuationOptions.OnAnyFinishedState); }); In this case batch identifier will be used as an owner, and entire batch will be protected by a mutex, preventing other batches from running simultaneously. Semaphores Semaphore limits concurrent execution of multiple background jobs to a certain maximum number. Unlike mutexes, semaphores should be created first using the IThrottlingManager interface with the maximum number of con- current background jobs allowed. The AddOrUpdateSemaphore method is idempotent, so we can safely place it in the application initialization logic. 6.4. Background Processing 67 Hangfire Documentation, Release 1.7 IThrottlingManager manager = new ThrottlingManager(); manager.AddOrUpdateSemaphore("newsletter", new SemaphoreOptions(maxCount: 100)); We can also call this method on an already existing semaphore, and in this case the maximum number of jobs will be updated. If background jobs that use this semaphore are currently executing, there may be temporary violation that will eventually be fixed. So if the number of background jobs is higher than the new maxCount value, no exception will be thrown, but new background jobs will be unable to acquire a semaphore. And when all of those background jobs finished, maxCount value will be satisfied. We should place the SemaphoreAttribute filter on a background job method and provide a correct resource identifier to link it with an existing semaphore. If semaphore with the given resource identifier doesn’t exist or was removed, an exception will be thrown at run-time, and background job will be moved to the Failed state. [Semaphore("newsletter")] public void SendNewsletter() { /* ... */ } Multiple executions of the same background job count as 1 As with mutexes, multiple invocations of the same background job aren’t respected and counted as 1. So actually it’s possible that more than the given count of background job methods are running concurrently. As before, we can use DisableConcurrentExecution to reduce the probability of this event, but we should be prepared for this anyway. As with mutexes, we can apply the SemaphoreAttribute with the same resource identifier to multiple back- ground job methods, and all of them will respect the behavior of a given semaphore. However dynamic resource identifiers based on arguments aren’t allowed for semaphores as they are required to be created first. [Semaphore("newsletter")] public void SendMonthlyNewsletter() { /* ... */ } [Semaphore("newsletter")] public void SendDailyNewsletter() { /* ... */ } Unused semaphore can be removed in the following way. Please note that if there are any associated background jobs are still running, an InvalidOperationException will be thrown (see Removing Attributes to avoid this scenario). This method is idempotent, and will simply succeed without performing anything when the corresponding semaphore doesn’t exist. manager.RemoveSemaphoreIfExists("newsletter"); Rate Limiters Fixed Window Counters Fixed window counters limit the number of background job executions allowed to run in a specific fixed time window. The entire time line is divided into static intervals of a predefined length, regardless of actual job execution times (unlike in Sliding Window Counters). Fixed window is required to be created first and we can do this in the following way. First, we need to pick some resource identifier unique for our application that will be used later when applying an attribute. Then specify the upper limit as well as the length of an interval (minimum 1 second) via the options. 68 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 strategies. With all of these features we can get some kind of fair processing, where one participant can’t capture all the available resources that’s especially useful for multi-tenant applications. DynamicWindowAttribute filter is responsible for this kind of throttling, and along with setting a resource identifier we need to specify the window format with String.Format-like placeholders (as in Mutexes) that will be converted into dynamic window identifiers at run-time based on job arguments. Maximal length of resource identifiers is 100 characters Please keep this in mind especially when using dynamic resource identifiers. [DynamicWindow("newsletter", "tenant:{0}")] public void SendNewsletter(long tenantId, string template) { /* ... */ } Dynamic Fixed Windows The following code snippet demonstrates the simplest form of a dynamic window counter. Since there’s a single bucket, it will create a fixed window of one-hour length with maximum 4 executions per each tenant. There will be up to 1000 fixed windows to not to blow up the data structure’s size. IThrottlingManager manager = new ThrottlingManager(); manager.AddOrUpdateDynamicWindow("newsletter", new DynamicWindowOptions( limit: 4, interval: TimeSpan.FromHours(1), buckets: 1)); Dynamic Sliding Windows If we increase the number of buckets, we’ll be able to use sliding windows instead with the given number of buckets. Limitations are the same as in sliding windows, so minimum bucket length is 1 second. As with fixed windows, there will be up to 1000 sliding windows to keep the size under control. manager.AddOrUpdateDynamicWindow("newsletter", new DynamicWindowOptions( limit: 4, interval: TimeSpan.FromHours(1), buckets: 60)); Limiting the Capacity Capacity allows us to control how many fixed or sliding sub-windows will be created dynamically. After running the following sample, there will be maximum 5 sub-windows limited to 4 executions. This is useful in scenarios when we don’t want a particular background job to take all the available resources. manager.AddOrUpdateDynamicWindow("newsletter", new DynamicWindowOptions( capacity: 20, limit: 4, interval: TimeSpan.FromHours(1), buckets: 60)); 6.4. Background Processing 71 Hangfire Documentation, Release 1.7 Rebalancing Limits When the capacity is set, we can also define dynamic limits for individual sub-windows in the following way. When rebalancing is enabled, individual limits depend on a number of active sub-windows and the capacity. manager.AddOrUpdateDynamicWindow("newsletter", new DynamicWindowOptions( capacity: 20, minLimit: 2, maxLimit: 20, interval: TimeSpan.FromHours(1), buckets: 60)); So in the example above, if there are background jobs only for a single tenant, they will be performed at full speed, 20 per hour. But if other participant is trying to enter, existing ones will be limited in the following way. • 1 participant: 20 per hour • 2 participants: 10 per hour for each • 3 participants: 7 per hour for 2 of them, and 6 per hour for the last • 4 participants: 5 per hour for each • . . . • 10 participants: 2 per hour for each Removing the Throttling As with other rate limiters, you can just remove the DynamicWindow attributes from your methods and call the following methods. There’s no need to change the mode to Release as with Concurrency Limiters, since no logic is running on background job completion. manager.RemoveDynamicWindowIfExists("newsletter"); 6.5 Best Practices Background job processing can differ a lot from a regular method invocation. This guide will help you keep back- ground processing running smoothly and efficiently. The information given here is based off of this blog post. 6.5.1 Make job arguments small and simple Method invocation (i.e. a job) is serialized during the background job creation process. Arguments are converted into JSON strings using the TypeConverter class. If you have complex entities and/or large objects; including arrays, it is better to place them into a database, and then pass only their identities to the background job. Instead of doing this: public void Method(Entity entity) { } Consider doing this: public void Method(int entityId) { } 72 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 6.5.2 Make your background methods reentrant Reentrancy means that a method can be interrupted in the middle of its execution and then safely called again. The interruption can be caused by many different things (i.e. exceptions, server shut-down), and Hangfire will attempt to retry processing many times. You can have many problems, if you don’t prepare your jobs to be reentrant. For example, if you are using an email sending background job and experience an error with your SMTP service, you can end with multiple emails sent to the addressee. Instead of doing this: public void Method() { _emailService.Send("person@example.com", "Hello!"); } Consider doing this: public void Method(int deliveryId) { if (_emailService.IsNotDelivered(deliveryId)) { _emailService.Send("person@example.com", "Hello!"); _emailService.SetDelivered(deliveryId); } } To be continued. . . 6.6 Deployment to Production 6.6.1 Making ASP.NET application always running By default, Hangfire Server instance in a web application will not be started until the first user hits your site. Even more, there are some events that will bring your web application down after some time (I’m talking about Idle Timeout and different app pool recycling events). In these cases your recurring tasks and delayed jobs will not be enqueued, and enqueued jobs will not be processed. This is particularly true for smaller sites, as there may be long periods of user inactivity. But if you are running critical jobs, you should ensure that your Hangfire Server instance is always running to guarantee the in-time background job processing. On-Premise applications For web applications running on servers under your control, either physical or virtual, you can use the auto-start feature of IIS 7.5 shipped with Windows Server 2008 R2. Full setup requires the following steps to be done: 1. Enable automatic start-up for Windows Process Activation (WAS) and World Wide Web Publishing (W3SVC) services (enabled by default). 2. Configure Automatic Startup for an Application pool (enabled by default). 3. Enable Always Running Mode for Application pool and configure Auto-start feature as written below. 6.6. Deployment to Production 73 Hangfire Documentation, Release 1.7 <applicationPools> <add name="MyAppWorkerProcess" managedRuntimeVersion="v4.0" startMode= →˓"AlwaysRunning" /> </applicationPools> <!-- ... --> <sites> <site name="MySite" id="1"> <application path="/" serviceAutoStartEnabled="true" serviceAutoStartProvider="ApplicationPreload" /> </site> </sites> <!-- Just AFTER closing the `sites` element AND AFTER `webLimits` tag --> <serviceAutoStartProviders> <add name="ApplicationPreload" type="WebApplication1.ApplicationPreload, →˓WebApplication1" /> </serviceAutoStartProviders> Note that for the last entry, WebApplication1.ApplicationPreload is the full name of a class in your application that implements IProcessHostPreloadClient and WebApplication1 is the name of your ap- plication’s library. You can read more about this here. There is no need to set IdleTimeout to zero – when Application pool’s start mode is set to AlwaysRunning, idle timeout does not work anymore. Ensuring auto-start feature is working If something went wrong. . . If your app won’t load after these changes made, check your Windows Event Log by opening Control Panel → Administrative Tools → Event Viewer. Then open Windows Logs → Application and look for a recent error records. The simplest method - recycle your Application pool, wait for 5 minutes, then go to the Hangfire Dashboard UI and check that current Hangfire Server instance was started 5 minutes ago. If you have problems – don’t hesitate to ask them on forum. Azure web applications Enabling always running feature for application hosted in Microsoft Azure is simpler a bit: just turn on the Always On switch on the Configuration page and save settings. This setting does not work for free sites. If nothing works for you. . . . . . because you are using shared hosting, free Azure website or something else (btw, can you tell me your configuration in this case?), then you can use the following ways to ensure that Hangfire Server is always running: 76 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 1. Use separate process to handle background jobs either on the same, or dedicated host. 2. Make HTTP requests to your website on a recurring basis by external tool (for example, Pingdom). 3. Do you know any other ways? Let me know! Making ASP.NET Core application always running on IIS Follow these directions in IIS: 1. Set application pool under which the application runs to: a. .NET CLR version: .NET CLR Version v4.0.30319 i. Normally, for a .NET Core app, you’d use No managed code, but if you do that, the application preload option won’t work. b. Managed pipeline mode: Integrated 2. Right-click on the same application pool and select “Advanced Settings”. Update the following values: a. Set start mode to “Always Running”. i. Setting the start mode to “Always Running” tells IIS to start a worker process for your application right away, without waiting for the initial request. b. Set Idle Time-Out (minutes) to 0. 3. Open Advanced Settings on your application: 6.6. Deployment to Production 77 Hangfire Documentation, Release 1.7 4. Set Preload Enabled = True: 5. Go to the Configuration Editor on your app, and navigate to system.webServer/applicationInitialization. Set the following settings: a. doAppInitAfterRestart: True b. Open up the Collection. . . ellipsis. On the next window, click Add and enter the following: i. hostName: {URL host for your Hangfire application} ii. initializationPage: {path for your Hangfire dashboard, like /hangfire} You can check this page for more documentation about it. 6.6.2 Using performance counters Pro Only This feature is a part of Hangfire Pro package set Performance Counters is a standard way to measure different application metrics on a Windows platform. This package enables Hangfire to publish performance counters so you can track them using different tools, including Performance Monitor, Nagios, New Relic and others. 78 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 (continued from previous page) public void OnPerforming(PerformingContext context) { Logger.InfoFormat("Starting to perform job `{0}`", context.BackgroundJob.Id); } public void OnPerformed(PerformedContext context) { Logger.InfoFormat("Job `{0}` has been performed", context.BackgroundJob.Id); } public void OnStateElection(ElectStateContext context) { var failedState = context.CandidateState as FailedState; if (failedState != null) { Logger.WarnFormat( "Job `{0}` has been failed due to an exception `{1}`", context.BackgroundJob.Id, failedState.Exception); } } public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction →˓transaction) { Logger.InfoFormat( "Job `{0}` state was changed from `{1}` to `{2}`", context.BackgroundJob.Id, context.OldStateName, context.NewState.Name); } public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction →˓transaction) { Logger.InfoFormat( "Job `{0}` state `{1}` was unapplied.", context.BackgroundJob.Id, context.OldStateName); } } Apply it Like ASP.NET filters, you can apply filters on method, class and globally: [LogEverything] public class EmailService { [LogEverything] public static void Send() { } } GlobalJobFilters.Filters.Add(new LogEverythingAttribute()); If your service is resolved via interface, then the filter should be applied to your interface instead of class: 6.7. Extensibility 81 Hangfire Documentation, Release 1.7 [LogEverything] public interface IEmailService { [LogEverything] void Send() { } } 6.8 Tutorials 6.8.1 Sending Mail in Background with ASP.NET MVC Table of Contents • Installing Postal • Further considerations • Installing Hangfire • Automatic retries • Logging • Fix-deploy-retry • Preserving current culture Let’s start with a simple example: you are building your own blog using ASP.NET MVC and want to receive an email notification about each posted comment. We will use the simple but awesome Postal library to send emails. Tip: I’ve prepared a simple application that has only comments list, you can download its sources to start work on tutorial. You already have a controller action that creates a new comment, and want to add the notification feature. // ~/HomeController.cs [HttpPost] public ActionResult Create(Comment model) { if (ModelState.IsValid) { _db.Comments.Add(model); _db.SaveChanges(); } return RedirectToAction("Index"); } Installing Postal First, install the Postal NuGet package: 82 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 Install-Package Postal.Mvc5 Then, create ~/Models/NewCommentEmail.cs file with the following contents: using Postal; namespace Hangfire.Mailer.Models { public class NewCommentEmail : Email { public string To { get; set; } public string UserName { get; set; } public string Comment { get; set; } } } Create a corresponding template for this email by adding the ~/Views/Emails/NewComment.cshtml file: @model Hangfire.Mailer.Models.NewCommentEmail To: @Model.To From: mailer@example.com Subject: New comment posted Hello, There is a new comment from @Model.UserName: @Model.Comment <3 And call Postal to send email notification from the Create controller action: [HttpPost] public ActionResult Create(Comment model) { if (ModelState.IsValid) { _db.Comments.Add(model); _db.SaveChanges(); var email = new NewCommentEmail { To = "yourmail@example.com", UserName = model.UserName, Comment = model.Text }; email.Send(); } return RedirectToAction("Index"); } Then configure the delivery method in the web.config file (by default, tutorial source code uses C:\Temp directory to store outgoing mail): 6.8. Tutorials 83 Hangfire Documentation, Release 1.7 Warning: Emails now are sent outside of request processing pipeline. As of Postal 1.0.0, there are the following limitations: you can not use layouts for your views, you MUST use Model and not ViewBag, embedding images is not supported either. That’s all! Try to create some comments and see the C:\Temp path. You also can check your background jobs at http://<your-app>/hangfire. If you have any questions, you are welcome to use the comments form below. Note: If you experience assembly load exceptions, please, please delete the following sections from the web. config file (I forgot to do this, but don’t want to re-create the repository): <dependentAssembly> <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture= →˓"neutral" /> <bindingRedirect oldVersion="0.0.0.0-6.0.0.0" newVersion="6.0.0.0" /> </dependentAssembly> <dependentAssembly> <assemblyIdentity name="Common.Logging" publicKeyToken="af08829b84f0328e" culture= →˓"neutral" /> <bindingRedirect oldVersion="0.0.0.0-2.2.0.0" newVersion="2.2.0.0" /> </dependentAssembly> Automatic retries When the emailService.Sendmethod throws an exception, Hangfire will retry it automatically after a delay (that is increased with each attempt). The retry attempt count is limited (10 by default), but you can increase it. Just apply the AutomaticRetryAttribute to the NotifyNewComment method: [AutomaticRetry( Attempts = 20 )] public static void NotifyNewComment(int commentId) { /* ... */ } Logging You can log cases when the maximum number of retry attempts has been exceeded. Try to create the following class: public class LogFailureAttribute : JobFilterAttribute, IApplyStateFilter { private static readonly ILog Logger = LogProvider.GetCurrentClassLogger(); public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction →˓transaction) { var failedState = context.NewState as FailedState; if (failedState != null) { Logger.ErrorException( String.Format("Background job #{0} was failed with an exception.", →˓context.JobId), failedState.Exception); (continues on next page) 86 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 (continued from previous page) } } public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction →˓transaction) { } } And add it: Either globally by calling the following method at application start: public void Configuration(IAppBuilder app) { GlobalConfiguration.Configuration .UseSqlServerStorage( "MailerDb", new SqlServerStorageOptions { QueuePollInterval = TimeSpan.FromSeconds(1) →˓}) .UseFilter(new LogFailureAttribute()); app.UseHangfireDashboard(); app.UseHangfireServer(); } Or locally by applying the attribute to a method: [LogFailure] public static void NotifyNewComment(int commentId) { /* ... */ } You can see the logging is working when you add a new breakpoint in LogFailureAttribute class inside method On- StateApplied If you like to use any of common logger and you do not need to do anything. Let’s take NLog as an example. Install NLog (current version: 4.2.3) Install-Package NLog Add a new Nlog.config file into the root of the project. <?xml version="1.0" encoding="utf-8" ?> <nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" autoReload="true" throwExceptions="false"> <variable name="appName" value="HangFire.Mailer" /> <targets async="true"> <target xsi:type="File" name="default" layout="${longdate} - ${level:uppercase=true}: ${message}${onexception:$ →˓{newline}EXCEPTION\: ${exception:format=ToString}}" (continues on next page) 6.8. Tutorials 87 Hangfire Documentation, Release 1.7 (continued from previous page) fileName="${specialfolder:ApplicationData}\${appName}\Debug.log" keepFileOpen="false" archiveFileName="${specialfolder:ApplicationData}\${appName}\Debug_$ →˓{shortdate}.{##}.log" archiveNumbering="Sequence" archiveEvery="Day" maxArchiveFiles="30" /> <target xsi:type="EventLog" name="eventlog" source="${appName}" layout="${message}${newline}${exception:format=ToString}"/> </targets> <rules> <logger name="*" writeTo="default" minlevel="Info" /> <logger name="*" writeTo="eventlog" minlevel="Error" /> </rules> </nlog> run application and new log file could be find on cd %appdata%HangFire.MailerDebug.log Fix-deploy-retry If you made a mistake in your NotifyNewComment method, you can fix it and restart the failed background job via the web interface. Try it: // Break background job by setting null to emailService: EmailService emailService = null; Compile a project, add a comment and go to the web interface by typing http://<your-app>/hangfire. Exceed all automatic attempts, then fix the job, restart the application, and click the Retry button on the Failed jobs page. Preserving current culture If you set a custom culture for your requests, Hangfire will store and set it during the performance of the background job. Try the following: // HomeController/Create action Thread.CurrentThread.CurrentCulture = CultureInfo.GetCultureInfo("es-ES"); BackgroundJob.Enqueue(() => NotifyNewComment(model.Id)); And check it inside the background job: public static void NotifyNewComment(int commentId) { var currentCultureName = Thread.CurrentThread.CurrentCulture.Name; if (currentCultureName != "es-ES") { throw new InvalidOperationException(String.Format("Current culture is {0}", →˓currentCultureName)); } // ... 88 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 (continued from previous page) } We have a controller with a single action. To test that our application is working, scaffold an empty view for Index action. The view scaffolding process also adds additional components to the project, like Bootstrap, jQuery, etc. After these steps my solution looks like: 6.8. Tutorials 91 Hangfire Documentation, Release 1.7 Let’s test the initial setup of our application. Press the F5 key to start debugging and wait for your browser. If you encounter exceptions or don’t see the default page, try to reproduce all the given steps, see the tutorial sources or ask a question in the comments below. Defining a model We should use a persistent storage to preserve snippets after application restarts. So, we’ll use SQL Server 2008 Express (or later) as a relational storage, and Entity Framework to access the data of our application. Installing Entity Framework Open the Package Manager Console window and type: 92 Chapter 6. Table of Contents Hangfire Documentation, Release 1.7 Install-Package EntityFramework After the package installed, create a new class in the Models folder and name it HighlighterDbContext: // ~/Models/HighlighterDbContext.cs using System.Data.Entity; namespace Hangfire.Highlighter.Models { public class HighlighterDbContext : DbContext { public HighlighterDbContext() : base("HighlighterDb") { } } } Please note, that we are using undefined yet connection string name HighlighterDb. So, lets add it to the web. config file just after the </configSections> tag: <connectionStrings> <add name="HighlighterDb" connectionString="Server=.\sqlexpress; Database=Hangfire. →˓Highlighter; Trusted_Connection=True;" providerName="System.Data.SqlClient" /> </connectionStrings> Then enable Entity Framework Code First Migrations by typing in your Package Manager Console window the following command: Enable-Migrations Adding code snippet model It’s time to add the most valuable class in the application. Create the CodeSnippet class in the Models folder with the following code: // ~/Models/CodeSnippet.cs using System; using System.ComponentModel.DataAnnotations; using System.Web.Mvc; namespace Hangfire.Highlighter.Models { public class CodeSnippet { public int Id { get; set; } [Required, AllowHtml, Display(Name = "C# source")] public string SourceCode { get; set; } public string HighlightedCode { get; set; } public DateTime CreatedAt { get; set; } public DateTime? HighlightedAt { get; set; } } } 6.8. Tutorials 93