Scheduled jobs gets skipped on the startup of new (additional) pulse instance

[het-delwadiya]

Description

I have scheduled jobs that gets executed by multiple containers. When I trigger a deployment, It creates new containers. At startup of new container, somehow it updates lastRunAt date and nextRunAt date without running the queued recurring jobs (jobs with nextRunAt passed current date). lockedAt will be still null and lastFinishedAt date doesn't get updated. I was able to reproduce it on my local by having 20-25 recurring jobs queued and starting an instance. Btw I have a job that runs after every 10 seconds and prints the number of jobs that are in queue. That number gets reduced very quickly without actually running jobs.

Node version - 18.12.1.
Pulse version - 1.5.1

Pulse options -

new Pulse({ mongo: mongoose.connection.db, defaultLockLifetime: 20 * 60 * 60 * 1000, // 20 hours resumeOnRestart: false, db: { address: null, collection: "agendaJobs" } })

Example job document -

{ "_id": { "$oid": "6788ee553fad7cfdfd8e084d" }, "name": "API_SYNC", "data": { "somedata": "jobdata" }, "priority": 0, "shouldSaveResult": false, "attempts": 0, "backoff": null, "type": "normal", "nextRunAt": { "$date": "2025-01-16T11:52:00.000Z" }, "lastModifiedBy": null, "lockedAt": null, "lastRunAt": { "$date": "2025-01-16T11:51:38.497Z" }, "finishedCount": 7, "lastFinishedAt": { "$date": "2025-01-16T11:46:29.935Z" }, "runCount": 7, "disabled": false, "endDate": null, "repeatInterval": "*/2 * * * *", "repeatTimezone": "America/New_York", "skipDays": null, "startDate": { "$date": "2025-01-16T00:30:00.000Z" } }

Code example

No response

Additional context

No response

[het-delwadiya]

Tried the same scenario on pulse v1.6.7 (latest) and still able to reproduce it

[het-delwadiya]

I have tested the same with non repetitive jobs that are in queue (nextRunAt date is past current date) and it works as expected. Something wrong with the repetitive jobs only.

[het-delwadiya]

I was trying to debug this issue and came across a functionality to skip job on restart in run() (in run.ts). It has a if block with a comment 'on restart, skip the job if it's not time to run' and i didn't understand its purpose. In my scenario where resumeOnRestart is turned off, previousRunAt is older than pulse's readyAt date as it is scheduled daily and was last ran yesterday and it does have nextRunAt defined. Because of all above conditions, it always go inside that if where it just skips the job run on the startup of an instance for repetitive queued jobs. It would be hard for me to contribute in fixing the issue since I'm not aware of that restart functionality and especially the motivation behind that if block.

[het-delwadiya]

By looking at the open bugs, it looks like resume on restart functionality is not matured enough to be pushed into LTS version. Can we have a version which doesn't have it at all ?

[het-delwadiya]

I have raised a PR to remove that logic since we should not skip execution of queued recurring job on startup of new pulse instance in case if resumeOnRestart is disabled unless I'm missing something. - #72

[code-xhyun]

Due to personal circumstances, I had difficulty managing the project. Starting this month, I plan to gradually resume overseeing it. I have also applied to renew the documentation site domain, and it is expected to be restored soon

[het-delwadiya]

Great to have you back! Can you take a look at it when you get some time ?

[code-xhyun]

@het-delwadiya What is the reason for not enabling the resumeOnRestart option?

[het-delwadiya]

@code-xhyun I have a setup of multiple containers. All of them uses a single collection to process jobs. Now when I do the deployment, new containers that spins up, picks the job that are already running in existing containers as it clears lockedAt date. It results into multiple executions of same job at deployment. This is the reason to avoid using resumeOnRestart option.