Sun Grid Engine (SGE) / limiting simultaneous array job sub-tasks

Posted by wfaulk on Server Fault See other posts from Server Fault or by wfaulk
Published on 2009-11-21T00:07:45Z Indexed on 2010/04/26 23:03 UTC
Read the original article Hit count: 326

Filed under:
|
|
|

I am installing a Sun Grid Engine environment and I have a scheduler limit that I can't quite figure out how to implement.

My users will create array jobs that have hundreds of sub-tasks. I would like to be able to limit those jobs to only running a set number of tasks at the same time, independent of other jobs. Like I might have one array job that I want to run 20 tasks at a time, and another I want to run 50 tasks at a time, and yet another that I'm fine running without limit. It seems like this ought to be doable, but I can't figure it out.

There's a max_aj_instances configuration option, but that appears to apply globally to all array jobs.

I can't see any way to use consumable resources, as I'd need a "complex attribute" that is per-job, and that feature doesn't seem to exist.

It didn't look like resource quotas would work, but now I'm not so sure of that. It says "A resource quota set defines a maximum resource quota for a particular job request", but it's unclear if an array job's sub-tasks' resource requests will be aggregated for the purposes of the resource quota. I'm going to play with this, but hopefully someone already knows outright.

© Server Fault or respective owner

Related posts about sge

Related posts about array-job