Cron Jobs in Production Best Practices for Reliable Scheduled Tasks

Cron Jobs in Production Best Practices

Running cron jobs in production requires more care than development. This guide covers best practices for reliability.

1. Make Jobs Idempotent

An idempotent job produces the same result whether it runs once or multiple times:

// WRONG   sends duplicate emails if run twice
cron.schedule("0 9 * * *", async () => {
  const users = await User.find();
  for (const user of users) {
    await sendEmail(user.email, "dailyDigest", {});
  }
});

// RIGHT   track sent emails
cron.schedule("0 9 * * *", async () => {
  const today = new Date().toISOString().split("T")[0];
  const alreadySent = await DigestLog.findOne({ date: today });
  if (alreadySent) {
    console.log("Digest already sent today, skipping");
    return;
  }
  
  const users = await User.find();
  for (const user of users) {
    await sendEmail(user.email, "dailyDigest", {});
  }
  
  await DigestLog.create({ date: today, sentAt: new Date(), count: users.length });
});

2. Handle Timezones

// WRONG   uses server timezone (might be UTC on EC2)
cron.schedule("0 9 * * *", () => {
  console.log("9 AM server time");
});

// RIGHT   specify timezone
cron.schedule("0 9 * * *", () => {
  console.log("9 AM IST");
}, {
  timezone: "Asia/Kolkata"
});

// For per-user timezone (send at 9 AM user's local time)
cron.schedule("0 * * * *", async () => {
  const currentHourUTC = new Date().getUTCHours();
  
  // Find users whose local time is 9 AM
  const users = await User.find({
    timezone: { $exists: true }
  });
  
  for (const user of users) {
    const userLocalHour = getLocalHour(currentHourUTC, user.timezone);
    if (userLocalHour === 9) {
      await sendEmail(user.email, "dailyDigest", {});
    }
  }
});

3. Set Timeouts

const runWithTimeout = async (jobFn, timeoutMs = 300000) => {
  return Promise.race([
    jobFn(),
    new Promise((_, reject) => 
      setTimeout(() => reject(new Error("Job timed out")), timeoutMs)
    )
  ]);
};

cron.schedule("0 0 * * *", () => {
  runWithTimeout(runCleanup, 300000) // 5 minute timeout
    .then(() => console.log("Cleanup done"))
    .catch(err => {
      console.error("Cleanup failed:", err.message);
      sendAlert(err);
    });
});

4. Use Distributed Locks (For Multiple Instances)

If you run multiple Node.js instances, prevent all of them from running the same cron job:

const runWithLock = async (lockKey, jobFn, ttl = 300) => {
  // Try to acquire lock
  const acquired = await redis.set(lockKey, "locked", "NX", "EX", ttl);
  if (!acquired) {
    console.log("Another instance is running this job");
    return;
  }
  
  try {
    await jobFn();
  } finally {
    await redis.del(lockKey);
  }
};

cron.schedule("0 0 * * *", () => {
  runWithLock("cron:daily-cleanup", runCleanup);
});

5. Gradual Rollout for New Jobs

// Start with a subset of users
cron.schedule("0 9 * * *", async () => {
  // First week: 10% of users
  const users = await User.find({
    emailBounced: false,
    _id: { $mod: [10, 0] } // First 10%
  });
  
  for (const user of users) {
    await sendEmail(user.email, "dailyDigest", {});
  }
});

6. Monitor and Alert

// Set up CloudWatch metrics
const { CloudWatch } = require("@aws-sdk/client-cloudwatch");
const cloudwatch = new CloudWatch({ region: "us-east-1" });

const putMetric = async (jobName, status) => {
  await cloudwatch.putMetricData({
    Namespace: "DevTinder/Cron",
    MetricData: [{
      MetricName: "JobExecution",
      Dimensions: [
        { Name: "JobName", Value: jobName },
        { Name: "Status", Value: status }
      ],
      Value: 1,
      Unit: "Count"
    }]
  });
};

cron.schedule("0 0 * * *", async () => {
  try {
    await runCleanup();
    await putMetric("daily-cleanup", "success");
  } catch (err) {
    await putMetric("daily-cleanup", "failed");
    await sendAlert(err);
  }
});

7. Document All Cron Jobs

Maintain a registry of all cron jobs:

// src/cron/registry.js
const cronJobs = [
  {
    name: "daily-cleanup",
    schedule: "0 0 * * *",
    description: "Delete expired tokens and old records",
    timezone: "UTC",
    timeout: 300000,
    retries: 3
  },
  {
    name: "daily-digest",
    schedule: "0 9 * * *",
    description: "Send daily digest emails to users",
    timezone: "Asia/Kolkata",
    timeout: 600000,
    retries: 2
  },
  {
    name: "weekly-report",
    schedule: "0 0 * * 1",
    description: "Generate weekly analytics report",
    timezone: "UTC",
    timeout: 900000,
    retries: 1
  }
];

module.exports = cronJobs;

The Takeaway

Production cron best practices include: making jobs idempotent (track execution to prevent duplicates), handling timezones (specify timezone in cron.schedule or check per-user), setting timeouts (prevent hung jobs), using distributed locks (prevent duplicate execution across instances), gradual rollout (test with 10% first), monitoring with CloudWatch metrics, alerting on failures, and documenting all jobs in a registry with name, schedule, description, timeout, and retries.