We've got a job queue system that'll cheerfully process any kind of job given to it. We intend to use it to process jobs that each contain 2 tasks:

• Job (Pass information from one server to another)
  • Fetch task (get the data, slowly)
  • Send task (send the data, comparatively quickly)

The difficulty we're having is that we don't know whether to break the tasks into separate jobs, or process the job in one go.

Are there any best practices or useful references on this subject? Is there some obvious benefit to a method that we're missing?

So far we can see these benefits for each method:

Split

• Job lease length reflects job length: Rather than total of two
• Finer granularity on recovery: If we lose outgoing connectivity we can tell them all to retry
• The starting state of the second task is saved to job history: Helps with debugging (although similar logging could be added in single task method)

Single

• Single job to be scheduled: Less processing overhead
• Data not stale on recovery: If the outgoing downtime is quite long, the pending Send jobs could be outdated