Skip to content
GitLab
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Sign in
Toggle navigation
Menu
Open sidebar
machinejobfeatures
mjf-scripts
Commits
b186f8bb
Commit
b186f8bb
authored
Oct 06, 2016
by
Andrew McNab
Browse files
_per_cpu values to accomodate mixed job sizes
parent
85110963
Changes
3
Hide whitespace changes
Inline
Side-by-side
README
View file @
b186f8bb
...
...
@@ -33,6 +33,12 @@ total number of logical processors), but are overridden in turn when
per-job values can be determined from the batch system (eg the number
of logical processors allocated to this job.)
The values cpu_limit_secs_per_cpu, max_rss_bytes_per_cpu,
max_swap_bytes_per_cpu, and scratch_limit_bytes_per_cpu can be set in
either mjf file to cause the scripts to calculate the corresponding
per-job value (eg cpu_limit_secs) using $JOBFEATURES/allocated_cpu (which
will be determined from the batch system if available, otherwise 1.)
If you know the HS06 of the worker node, you can also include a line like
hs06=99.99 which will be picked up when populating /etc/machinefeatures/
(you can force updates after changing that file with service mjf start
...
...
make-jobfeatures.htcondor
View file @
b186f8bb
...
...
@@ -52,10 +52,19 @@
# - hs06_job
# - db12_job
#
# Values in /var/run/mjf are preferred over /etc/sysconfig/mjf.
#
Values in /var/run/mjf are preferred over /etc/sysconfig/mjf.
#
# Additionally you can set mjf_tmp_dir in those files to use a
# directory other than /tmp for the mjf-$USER directories.
# The following per-cpu values in either mjf file will be used
# to calculate the corresponding per-job values if not given in
# either file or obtainable from PBS/Torque:
#
# - cpu_limit_secs_per_cpu
# - max_rss_bytes_per_cpu
# - max_swap_bytes_per_cpu
# - scratch_limit_bytes_per_cpu
#
# Additionally you can set mjf_tmp_dir in those files to use a
# directory other than /tmp for the mjf-$USER directories.
#
import
os
...
...
@@ -162,13 +171,27 @@ for key in ['allocated_cpu', 'wall_limit_secs', 'cpu_limit_secs',
except
:
pass
# Examine the Job Ad, for
resource limits
# Examine the Job Ad, for
number of CPUs allocated
try
:
cpusMatchObject
=
re
.
search
(
'CpusProvisioned *= *([0-9 +*/()-]*)'
,
fromJobAd
)
jobfeatures
[
'allocated_cpu'
]
=
int
(
eval
(
cpusMatchObject
.
group
(
1
)))
except
:
pass
# Look for any per-cpu values given in either file
for
key
in
[
'cpu_limit_secs'
,
'max_rss_bytes'
,
'max_swap_bytes'
,
'scratch_limit'
]:
if
key
+
'_per_cpu='
in
fromRun
+
fromSysconfig
:
try
:
matchObject
=
re
.
search
(
key
+
"_per_cpu=([0-9]*)"
,
fromRun
+
'
\n
'
+
fromSysconfig
)
if
key
not
in
jobfeatures
:
jobfeatures
[
key
]
=
int
(
matchObject
.
group
(
1
))
*
jobfeatures
[
'allocated_cpu'
]
except
:
pass
# Examine the Job Ad, for resource limits
try
:
wallMatchObject
=
re
.
search
(
'MaxJobRetirementTime *= *([0-9 +*/()-]*)'
,
fromMachineAd
)
jobfeatures
[
'wall_limit_secs'
]
=
int
(
eval
(
wallMatchObject
.
group
(
1
)))
...
...
prologue.user
View file @
b186f8bb
...
...
@@ -52,10 +52,19 @@
# - hs06_job
# - db12_job
#
# Values in /var/run/mjf are preferred over /etc/sysconfig/mjf.
#
Values in /var/run/mjf are preferred over /etc/sysconfig/mjf.
#
# Additionally you can set mjf_tmp_dir in those files to use a
# directory other than /tmp for the mjf-$USER directories.
# The following per-cpu values in either mjf file will be used
# to calculate the corresponding per-job values if not given in
# either file or obtainable from PBS/Torque:
#
# - cpu_limit_secs_per_cpu
# - max_rss_bytes_per_cpu
# - max_swap_bytes_per_cpu
# - scratch_limit_bytes_per_cpu
#
# Additionally you can set mjf_tmp_dir in those files to use a
# directory other than /tmp for the mjf-$USER directories.
#
import
os
...
...
@@ -153,6 +162,19 @@ try:
except
:
pass
# Look for any per-cpu values given in either file
for
key
in
[
'cpu_limit_secs'
,
'max_rss_bytes'
,
'max_swap_bytes'
,
'scratch_limit'
]:
if
key
+
'_per_cpu='
in
fromRun
+
fromSysconfig
:
try
:
matchObject
=
re
.
search
(
key
+
"_per_cpu=([0-9]*)"
,
fromRun
+
'
\n
'
+
fromSysconfig
)
if
key
not
in
jobfeatures
:
jobfeatures
[
key
]
=
int
(
matchObject
.
group
(
1
))
*
jobfeatures
[
'allocated_cpu'
]
except
:
pass
try
:
wallMatchObject
=
re
.
search
(
'walltime=([0-9]*):([0-9]*):([0-9]*)'
,
sys
.
argv
[
5
])
jobfeatures
[
'wall_limit_secs'
]
=
int
(
wallMatchObject
.
group
(
1
))
*
3600
+
int
(
wallMatchObject
.
group
(
2
))
*
60
+
int
(
wallMatchObject
.
group
(
3
))
...
...
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment