---
title: System Schedules and the Scheduler
description: "Almost everything RackHelm does on its own — raising renewal invoices, charging stored cards, suspending unpaid services, pulling support email into tickets, reconciling domains at the registrar, refreshing exchange rates — is a **system schedule**. Each one is a row…"
url: "https://rackhelm.com/documentation/system-schedules/"
updated: "2026-09-24"
category: System Administration
---

# System Schedules and the Scheduler

Almost everything RackHelm does on its own — raising renewal invoices, charging stored cards, suspending unpaid services, pulling support email into tickets, reconciling domains at the registrar, refreshing exchange rates — is a **system schedule**. Each one is a row you can enable, retime, tune and run on demand from the admin area. This article explains how the scheduler is driven, what each seeded schedule does, what the **Run now** button really does, and the options that most often get set wrong.

## Before you start

> **Warning:** Every seeded schedule except **Refresh Exchange Rates** ships **disabled**. A fresh install raises no invoices, takes no payments and reads no support mail until you enable the ones you want. This is deliberate — an install that started billing before you had finished configuring it would be worse — but it does mean "nothing happens" on day one is expected.

> **Note:** **Refresh Exchange Rates** is the one exception: it is seeded enabled, because a currency whose "Auto-update rate" box is ticked but whose rate never moves is a silent pricing error. Until you flag a currency for auto-update the job returns without making any HTTP call at all.

> **Note:** Editing schedules requires the `manage_schedules` permission, which is deliberately withheld from the built-in Administrator role — only a Super Admin holds it until you grant it on the Roles screen. The dashboard's schedule-status panel requires `view_schedules`, which Administrator does hold.

> **Warning:** The **Schedules** menu in the admin sidebar is wrapped in a check for `manage_settings`, and the **Queue Monitor** entry inside it in a check for `manage_roles`. Neither of those keys exists in the portal's permission list, so neither can ever be granted to a role, and a check against an undefined permission passes only for a Super Admin — who bypasses every gate anyway. In practice the sidebar menu is Super-Admin-only. A staff member you have granted `manage_schedules` can still reach a schedule by clicking its name in the dashboard's **System schedule status** table, or by going to `/admin/system-schedules` directly. Grant that permission first: an Administrator holds `view_schedules` but not `manage_schedules`, so the dashboard links give them a 403.

## How the scheduler is driven

RackHelm does not poll on its own. A single system cron entry must run the framework scheduler once a minute; that process reads your enabled schedule rows, works out which are due this minute, and queues their jobs. Without it, nothing on this screen ever fires.

```bash
* * * * * cd /var/www/rackhelm && php artisan schedule:run >> /dev/null 2>&1
```

The automated installer writes exactly that line to `/etc/cron.d/rackhelm`, running as the web-server user, so a scripted install already has it. Confirm it with:

```bash
cat /etc/cron.d/rackhelm
```

On a manual install, or if that file is missing, add the entry to the crontab of the user that owns the application files (commonly `www-data`, `nginx` on RHEL):

```bash
sudo crontab -u www-data -e
```

The jobs it queues are then executed by your queue workers, so both pieces must be running. See [Queue Workers, Horizon and the Scheduler](/docs/queue-workers-and-scheduler/).

> **Note:** Only an explicit list of job classes may be launched from these rows, and the same list guards the **Run now** button. If a row's job class is not on it — because it was edited directly in the database, for instance — the job is refused and a critical line is written to the application log rather than being run. A row whose class no longer exists at all is skipped with a `Scheduler: Job class does not exist` warning.

## Finding and editing a schedule

Expand **Schedules** in the sidebar: every schedule is listed there by name and links straight to its own editor. The dashboard also carries a **System schedule status** table showing each job's state, cron expression, last run and next run, and a **Scheduler** line in the health strip that reads **On time** or **N overdue** alongside the number of active schedules.

![The System schedule status table on the admin dashboard listing each job with its state, frequency, last run and next run](/wp-content/uploads/rackhelm-docs/system-schedule-status.png)
The dashboard table is the quickest way to see which schedules are enabled and whether they are running on time. Each job name links to its editor.

The editor header repeats **Last run** and **Next run** for that one schedule — **Next run** reads *Disabled* when the row is off — and carries the **Run now** button. Below it is one short form:

| Field | What it does |
| --- | --- |
| **Name** | Required. The label used in the sidebar, the dashboard and any summary email. |
| **Description** | Read-only. The seeded explanation of what the job does; the editor displays it but does not let you change it. |
| **Enabled** | Whether the schedule is registered with the scheduler at all. |
| **Summary Email** | Optional address that receives the job's console output after each run. Must be a valid address. |
| **Days Offsets (comma separated)** | The job's day-offset option. Only some jobs read it — see the table below. |
| **Enable File Logging** | Writes this job's progress and errors to the standard application log. Off by default to keep the log readable. |
| **Minute / Hour / Day / Month / Weekday** | The five cron fields. All are required. |

The five cron fields are assembled into one expression and validated before saving. An invalid combination is rejected with "The schedule fields do not form a valid cron expression." That check matters: a malformed expression is parsed while the scheduler runs, outside any error handling, so a single bad row would otherwise abort the entire run — invoicing, suspensions and inbound mail all at once.

> **Note:** **Days Offsets** and **Enable File Logging** are the only options the editor exposes. Some rows carry further options that exist only in the database — the registrar sweep's four section switches, for example. Saving from this screen merges your changes into the row's existing options rather than replacing them, so those hidden settings survive a save; there is simply no admin control for them.

> **Note:** A row's job class is not editable from this screen, on purpose: it is the string the scheduler turns into an object. Moving a schedule onto a different job is done with a seeder, never by hand.

> **Tip:** All schedules run on UTC. If you are in the UK, a job set to `0 0` runs at midnight GMT and 01:00 BST. Pick times where the hour boundary does not matter.

## Running a schedule now

The editor has a **Run now** button, for the cases where waiting for the next cron window is not an option: proving a newly configured mail route works, forcing an invoice run after fixing a due date, or re-driving a sweep after a registrar's API came back.

![The schedule editor header showing Last run, Next run and the green Run now button](/wp-content/uploads/rackhelm-docs/system-schedule-edit-run-now.png)
Run now sits above the form and has its own confirmation, so pressing it never saves unrelated edits.

What it does, precisely:

- It asks "Queue this job to run now? It will start as soon as a queue worker is free", then **queues** the job. It does not run it inside the web request — several of these jobs walk every due invoice or poll a remote panel, and a page timeout is the wrong place to discover that.
- It builds the same object the scheduler would have built: same class, same options payload, so your **Enable File Logging** flag still applies.
- It ignores **Enabled**. "The cron is off but I need this once, now" is the main reason the button exists.
- It does **not** update **Last run**. Only a successful scheduled run stamps that; all a click proves is that the job was enqueued.
- If no Horizon master is running it still queues the job but warns you: "No queue worker appears to be running, so it will sit in the queue until one starts."

> **Note:** There is a one-minute cooldown per schedule and a rate limit of twelve runs a minute per user. A second click inside the minute is refused with "That job was queued less than a minute ago — give it a moment before running it again." Each of these jobs is a sweep over live billing or provisioning data, so the button is deliberately not a way to stack them up.

If the button is greyed out, the row's job class is not on the scheduler allowlist — hover it for the class name. That is almost always a schedule left pointing at a job that was merged away; see the registrar sweep below.

## The seeded schedules

| Schedule | What it does | Default timing |
| --- | --- | --- |
| **Provision Orders** | Scans for order items that are paid but still awaiting provisioning and queues each one against its product's active integration. This is the sweeper that also retries stalled provisioning. | Every minute |
| **Generate Invoices** | Creates renewal invoices for orders whose due date falls inside the advance window. Reads **Days Offsets** — see the trap below. | Daily at 00:00 |
| **Process Due Invoices** | Finds pending invoices due today or earlier and queues an automatic payment attempt against the customer's stored payment method. | Daily at 00:00 |
| **Email to Tickets** | Polls every active email route (IMAP, POP3 or Microsoft Graph) and turns incoming mail into new tickets or replies on existing ones. | Every 5 minutes |
| **Process Cancellations** | Terminates order items pending cancellation once their paid-up-to date has passed, then voids or reduces any pending invoices covering them. | Daily at 00:00 |
| **Notify Expiring Cards** | Notifies customers whose stored card is due to expire. | Daily at 00:00 |
| **Domain Updater** | The single registrar sweep, across every configured registrar: reconciles in-flight transfers, refreshes the provider domain cache, detects domains transferred away, and warms the per-domain details and DNS cache for the customer area. See below. | Hourly, on the hour (`0 * * * *`) |
| **Invoice Reminders** | Sends reminders for unpaid invoices at the offsets in **Days Offsets**. Negative values are days *before* the due date, positive values days *after*. Seeded as `-1,1,7`. | Daily at 00:00 |
| **Suspend Overdue Services** | Suspends order items and opens a ticket when their invoice is still unpaid this many days after the due date. Reads a single number from **Days Offsets**; seeded as `7`, and falls back to 3 if the value is missing or zero. | Daily at 00:00 |
| **Terminate Overdue Services** | Terminates order items whose invoice is still unpaid this many days after the due date. Reads a single number from **Days Offsets**; seeded as `14`, defaulting to 7. | Daily at 00:00 |
| **Apply Late Fees** | Applies a late fee to invoices still unpaid after the grace period. | Daily at 00:00 |
| **Auto Close Tickets** | Closes tickets awaiting a customer reply after a period of inactivity, optionally warning the customer first. | Daily at 00:00 |
| **Mass Mailer** | Sends any email campaign whose scheduled send time has passed, building its recipient list from the campaign's segment rules. | Daily at 00:00 |
| **Refresh Exchange Rates** | Refreshes the rate of every currency flagged auto-update from European Central Bank reference rates (`api.frankfurter.app`). The base currency and manually-rated currencies are never touched. **Seeded enabled.** | Hourly at :17 (`17 * * * *`) |

### Which schedules actually read Days Offsets

The field appears on every schedule but only four jobs read it: **Generate Invoices**, **Invoice Reminders**, **Suspend Overdue Services** and **Terminate Overdue Services**. Setting it on any other schedule has no effect.

Three cases are worth calling out:

- **Apply Late Fees** ignores it. The fee amount, type and grace period come from the billing cycle price, then the product, then built-in defaults (disabled, 10%, 7 days' grace). Configure late fees on the product or cycle, not here.
- **Auto Close Tickets** ignores it too. It reads its own option keys, which no admin screen writes, so it runs on its defaults: close after 7 days of customer inactivity, no advance warning.
- **Domain Updater**, **Refresh Exchange Rates**, **Mass Mailer**, **Provision Orders**, **Process Due Invoices**, **Process Cancellations** and **Notify Expiring Cards** ignore it. Some of them carry a leftover `days` value in their row; it is never read.

## The Generate Invoices "days" trap

> **Warning:** If you save **Generate Invoices** with **Days Offsets** empty, the option is stored as an empty list rather than as "unset" — so the built-in fallback never applies, the advance window collapses to zero days, and renewal invoices are only raised on the exact due date. Customers get no notice, automatic payment has no lead time, and nothing errors. Keep the field populated.

The job takes the *largest* value in the list as its advance window: with `14,7,1,-1,-7,-14` it raises invoices up to 14 days before the due date. The negative entries are carried for the reminder cadence and do not shorten that window. The fallback that only applies when the key is absent entirely comes from `INVOICE_SCHEDULE_DAYS` in `.env`, defaulting to `14,7,1,-1,-7,-14`.

To recover from an emptied field, open **Generate Invoices**, set **Days Offsets** back to:

```ini
14,7,1,-1,-7,-14
```

and save. The next run picks up every order that should already have been invoiced. To see what that run will do before it happens:

```bash
php artisan test:invoice-schedule-dry-run
```

> **Note:** That preview only lists orders *no item of which has ever been invoiced*, while the job itself asks whether an item is invoiced for *this* due date. A service that has already renewed once never appears in the preview, however due it is — so treat an empty preview as "no new signups to bill", not as "nothing will be billed".

## The hourly registrar sweep

**Domain Updater** is now one sweep across every configured registrar rather than one job per registrar. In a single hourly run it:

1. reconciles domains with a transfer in progress, writing the real expiry back onto the order item and its order once the transfer completes;
2. refreshes the provider domain cache from each registrar's domain list;
3. detects domains that have been transferred away and cancels the order items behind them;
4. warms the per-domain details and DNS cache the customer area reads.

Transfers are done first, because that is the cheapest and most time-sensitive section: a slow or broken registrar API further down the list cannot delay it.

The run carries a soft time budget — 180 seconds by default. When it is spent, the sweep stops cleanly *between* registrars rather than being killed mid-write by the worker timeout, logs "Stopping before <next step>: Ns spent of a 180s budget. The remaining work resumes on the next run", and picks up where it left off an hour later. A killed run would cost a failed-jobs row and skip every later section for good, so stopping early is the better outcome.

> **Note:** Each of the four sections can be switched off independently through the row's options (`sync_transfers`, `sync_registrars`, `detect_transferred_out`, `warm_cache`), along with `cache_ttl_minutes` (90), `max_seconds` (180) and `stale_floor_hours` (168). All default on. None of them has an admin control — they live in the schedule row itself. Keep `cache_ttl_minutes` above the gap between runs, or warmed entries expire mid-hour and customer page loads refill them one live registrar call at a time.

## Recommended starting point

1. Confirm the cron entry exists and queue workers are running.
2. Enable **Provision Orders** — nothing provisions automatically without it.
3. Enable **Generate Invoices** with **Days Offsets** set, then **Process Due Invoices**.
4. Enable **Invoice Reminders**, and **Email to Tickets** if you use inbound mail.
5. Enable **Domain Updater** if you sell domains — transfers do not complete in the portal without it.
6. Leave **Suspend Overdue Services** and **Terminate Overdue Services** until you are confident the billing chain is correct — they act on live customer services.
7. Check the dashboard's schedule table the next day: each enabled row should show a recent **Last run**.

## Troubleshooting

| Symptom | Cause and fix |
| --- | --- |
| Every schedule shows **Last run: Never** | The per-minute cron entry is missing or running as the wrong user. Check `/etc/cron.d/rackhelm`, and `crontab -l` for the application's user. |
| One schedule never runs while others do | It is disabled, or its cron expression does not match the times you expect. The dashboard shows the expression and the computed next run. |
| Schedules fire but nothing happens | They queue jobs; a worker executes them. Check the queue monitor for a backlog or failed jobs. |
| **Run now** says the job was queued, but **Last run** does not change | Expected. Only a successful scheduled run stamps **Last run**; a manual run only proves the job was enqueued. Watch the queue monitor, or turn on **Enable File Logging**. |
| "No queue worker appears to be running" | The job is on the queue but nothing is consuming it. Start Horizon; the job runs as soon as a worker is free. |
| Renewal invoices only appear on the due date | The **Days Offsets** trap above. Restore the list. |
| Reminders go out at the wrong time relative to the due date | Offsets are signed: negative is before the due date, positive is after. |
| Domain transfers never complete in the portal | **Domain Updater** is disabled, or still points at the deleted per-registrar job. Enable it and run the repoint seeder. |
| The registrar sweep only ever gets part-way | It is hitting its 180-second budget and resuming next hour — look for the "Stopping before …" line. That is by design; if it never catches up, one registrar's API is slow enough to be worth investigating. |
| The whole scheduler stopped after a schedule was edited in the database | An invalid cron expression. The admin form refuses those, and the scheduler now skips them with a warning in the log, but a row edited directly can still be wrong. Fix the five fields and save from the admin screen. |
| You want to see what a job did | Turn on **Enable File Logging** for that schedule and read the application log, or set a **Summary Email**. |
| A job failed and nobody noticed | Route the **Scheduled job failed** event to an address or a chat channel — see [Admin Notifications](/docs/admin-notifications/). |