| | |
- __builtin__.object
-
- ScriptMixin
-
- BaseScript(ScriptMixin, mozharness.base.log.LogMixin, __builtin__.object)
class BaseScript(ScriptMixin, mozharness.base.log.LogMixin, __builtin__.object) |
| |
# BaseScript {{{1
|
| |
- Method resolution order:
- BaseScript
- ScriptMixin
- mozharness.base.log.LogMixin
- __builtin__.object
Methods defined here:
- __init__(self, config_options=None, default_log_level='info', **kwargs)
- action_message(self, message)
- add_failure(self, key, message='%(key)s failed.', level='error')
- add_summary(self, message, level='info')
- clobber(self)
- Delete the working directory
- copy_logs_to_upload_dir(self)
- Copies logs to the upload directory
- copy_to_upload_dir(self, target, dest=None, short_desc='unknown', long_desc='unknown', log_level='debug', error_level='error', max_backups=None)
- Copy target file to upload_dir/dest.
Potentially update a manifest in the future if we go that route.
Currently only copies a single file; would be nice to allow for
recursive copying; that would probably done by creating a helper
_copy_file_to_upload_dir().
short_desc and long_desc are placeholders for if/when we add
upload_dir manifests.
- dump_config(self, file_path=None, config=None)
- Dump self.config to localconfig.json
- new_log_obj(self, default_log_level='info')
- # logging {{{2
- query_abs_dirs(self)
- We want to be able to determine where all the important things
are. Absolute paths lend themselves well to this, though I wouldn't
be surprised if this causes some issues somewhere.
This should be overridden in any script that has additional dirs
to query.
The query_* methods tend to set self.VAR variables as their
runtime cache.
- query_failure(self, key)
- run(self)
- Default run method.
This is the "do everything" method, based on actions and all_actions.
First run dump_config() if it exists.
Second, go through the list of all_actions.
If they're in the list of self.actions, try to run
preflight_ACTION(), ACTION(), and postflight_ACTION().
Preflight is sanity checking before doing anything time consuming or
destructive.
Postflight is quick testing for success after an action.
- summarize_success_count(self, success_count, total_count, message='%d of %d successful.', level=None)
- summary(self)
- Print out all the summary lines added via add_summary()
throughout the script.
I'd like to revisit how to do this in a prettier fashion.
Methods inherited from ScriptMixin:
- chdir(self, dir_name)
- chmod(self, path, mode)
- copyfile(self, src, dest, log_level='info', error_level='error', copystat=False)
- copytree(self, src, dest, overwrite='no_overwrite', log_level='info', error_level='error')
- an implementation of shutil.copytree however it allows for
dest to exist and implements different overwrite levels.
overwrite uses:
'no_overwrite' will keep all(any) existing files in destination tree
'overwrite_if_exists' will only overwrite destination paths that have
the same path names relative to the root of the src and
destination tree
'clobber' will replace the whole destination tree(clobber) if it exists
- download_file(self, url, file_name=None, parent_dir=None, create_parent_dir=True, error_level='error', exit_code=-1)
- Python wget.
- get_filename_from_url(self, url)
- get_output_from_command(self, command, cwd=None, halt_on_failure=False, env=None, silent=False, log_level='info', tmpfile_base_path='tmpfile', return_type='output', save_tmpfiles=False, throw_exception=False)
- Similar to run_command, but where run_command is an
os.system(command) analog, get_output_from_command is a `command`
analog.
Less error checking by design, though if we figure out how to
do it without borking the output, great.
TODO: binary mode? silent is kinda like that.
TODO: since p.wait() can take a long time, optionally log something
every N seconds?
TODO: optionally only keep the first or last (N) line(s) of output?
TODO: optionally only return the tmp_stdout_filename?
- mkdir_p(self, path, error_level='error')
- Returns None for success, not None for failure
- move(self, src, dest, log_level='info', error_level='error', exit_code=-1)
- query_env(self, partial_env=None, replace_dict=None, set_self_env=None, log_level='debug')
- Environment query/generation method.
The default, query_env(), will look for self.config['env']
and replace any special strings in there ( %(PATH)s ).
It will then store it as self.env for speeding things up later.
If you specify partial_env, partial_env will be used instead of
self.config['env'], and we don't save self.env as it's a one-off.
- query_exe(self, exe_name, exe_dict='exes', default=None, return_type=None, error_level='fatal')
- One way to work around PATH rewrites.
By default, return exe_name, and we'll fall through to searching
os.environ["PATH"].
However, if self.config[exe_dict][exe_name] exists, return that.
This lets us override exe paths via config file.
'return_type' can be None (don't do anything to the value),
'list' (return a list), or 'string' (return a string).
If we need runtime setting, we can build in self.exes support later.
- query_msys_path(self, path)
- read_from_file(self, file_path, verbose=True, open_mode='r', error_level='error')
- Reads from file_path.
Returns contents if successful, None if not.
- retry(self, action, attempts=None, sleeptime=60, max_sleeptime=300, retry_exceptions=(<type 'exceptions.Exception'>,), good_statuses=None, cleanup=None, error_level='error', error_message='%(action)s failed after %(attempts)d tries!', failure_status=-1, args=(), kwargs={})
- Generic retry command.
Ported from tools util.retry.
Call `action' a maximum of `attempts' times until it succeeds,
defaulting to self.config.get('global_retries', 5).
`sleeptime' is the number of seconds to wait between attempts,
defaulting to 60 and doubling each retry attempt, to a maximum of
`max_sleeptime'.
`retry_exceptions' is a tuple of Exceptions that should be caught.
If exceptions other than those listed in `retry_exceptions' are
raised from `action', they will be raised immediately.
`good_statuses' is a tuple of return values which, if specified,
will result in retrying if the return value isn't listed.
If `cleanup' is provided and callable it will be called immediately
after an Exception is caught. No arguments will be passed to it.
If your cleanup function requires arguments it is recommended that
you wrap it in an argumentless function.
`args' and `kwargs' are a tuple and dict of arguments to pass onto
to `callable'.
- rmtree(self, path, log_level='info', error_level='error', exit_code=-1)
- Returns None for success, not None for failure
- run_command(self, command, cwd=None, error_list=None, parse_at_end=False, halt_on_failure=False, success_codes=None, env=None, return_type='status', throw_exception=False, output_parser=None)
- Run a command, with logging and error parsing.
TODO: parse_at_end, context_lines
TODO: retry_interval?
TODO: error_level_override?
TODO: Add a copy-pastable version of |command| if it's a list.
TODO: print env if set
output_parser lets you provide an instance of your own OutputParser
subclass, or pass None to use OutputParser.
error_list example:
[{'regex': re.compile('^Error: LOL J/K'), level=IGNORE},
{'regex': re.compile('^Error:'), level=ERROR, contextLines='5:5'},
{'substr': 'THE WORLD IS ENDING', level=FATAL, contextLines='20:'}
]
(context_lines isn't written yet)
- which(self, program)
- OS independent implementation of Unix's which command
Takes in a program name
Returns path to executable or None
- write_to_file(self, file_path, contents, verbose=True, open_mode='w', create_parent_dir=False, error_level='error')
- Write contents to file_path.
This doesn't currently create the parent_dir or translate into
abs_path; that needs to be done beforehand, since ScriptMixin doesn't
necessarily have access to query_abs_dirs().
Returns file_path if successful, None if not.
Data descriptors inherited from ScriptMixin:
- __dict__
- dictionary for instance variables (if defined)
- __weakref__
- list of weak references to the object (if defined)
Data and other attributes inherited from ScriptMixin:
- env = None
Methods inherited from mozharness.base.log.LogMixin:
- critical(self, message)
- debug(self, message)
- error(self, message)
- exception(self, message=None, level='error')
- # Copying Bear's dumpException():
# http://hg.mozilla.org/build/tools/annotate/1485f23c38e0/sut_tools/sut_lib.py#l23
- fatal(self, message, exit_code=-1)
- info(self, message)
- log(self, message, level='info', exit_code=-1)
- warning(self, message)
- worst_level(self, target_level, existing_level, levels=None)
- returns either existing_level or target level.
This depends on which is closest to levels[0]
By default, levels is the list of log levels
|
class ScriptMixin(__builtin__.object) |
| |
This mixin contains simple filesystem commands and the like.
It also contains some very special but very complex methods that,
together with logging and config, provide the base for all scripts
in this harness.
Depends on LogMixin and a self.config of some sort.
|
| |
Methods defined here:
- chdir(self, dir_name)
- chmod(self, path, mode)
- copyfile(self, src, dest, log_level='info', error_level='error', copystat=False)
- copytree(self, src, dest, overwrite='no_overwrite', log_level='info', error_level='error')
- an implementation of shutil.copytree however it allows for
dest to exist and implements different overwrite levels.
overwrite uses:
'no_overwrite' will keep all(any) existing files in destination tree
'overwrite_if_exists' will only overwrite destination paths that have
the same path names relative to the root of the src and
destination tree
'clobber' will replace the whole destination tree(clobber) if it exists
- download_file(self, url, file_name=None, parent_dir=None, create_parent_dir=True, error_level='error', exit_code=-1)
- Python wget.
- get_filename_from_url(self, url)
- get_output_from_command(self, command, cwd=None, halt_on_failure=False, env=None, silent=False, log_level='info', tmpfile_base_path='tmpfile', return_type='output', save_tmpfiles=False, throw_exception=False)
- Similar to run_command, but where run_command is an
os.system(command) analog, get_output_from_command is a `command`
analog.
Less error checking by design, though if we figure out how to
do it without borking the output, great.
TODO: binary mode? silent is kinda like that.
TODO: since p.wait() can take a long time, optionally log something
every N seconds?
TODO: optionally only keep the first or last (N) line(s) of output?
TODO: optionally only return the tmp_stdout_filename?
- mkdir_p(self, path, error_level='error')
- Returns None for success, not None for failure
- move(self, src, dest, log_level='info', error_level='error', exit_code=-1)
- query_env(self, partial_env=None, replace_dict=None, set_self_env=None, log_level='debug')
- Environment query/generation method.
The default, query_env(), will look for self.config['env']
and replace any special strings in there ( %(PATH)s ).
It will then store it as self.env for speeding things up later.
If you specify partial_env, partial_env will be used instead of
self.config['env'], and we don't save self.env as it's a one-off.
- query_exe(self, exe_name, exe_dict='exes', default=None, return_type=None, error_level='fatal')
- One way to work around PATH rewrites.
By default, return exe_name, and we'll fall through to searching
os.environ["PATH"].
However, if self.config[exe_dict][exe_name] exists, return that.
This lets us override exe paths via config file.
'return_type' can be None (don't do anything to the value),
'list' (return a list), or 'string' (return a string).
If we need runtime setting, we can build in self.exes support later.
- query_msys_path(self, path)
- read_from_file(self, file_path, verbose=True, open_mode='r', error_level='error')
- Reads from file_path.
Returns contents if successful, None if not.
- retry(self, action, attempts=None, sleeptime=60, max_sleeptime=300, retry_exceptions=(<type 'exceptions.Exception'>,), good_statuses=None, cleanup=None, error_level='error', error_message='%(action)s failed after %(attempts)d tries!', failure_status=-1, args=(), kwargs={})
- Generic retry command.
Ported from tools util.retry.
Call `action' a maximum of `attempts' times until it succeeds,
defaulting to self.config.get('global_retries', 5).
`sleeptime' is the number of seconds to wait between attempts,
defaulting to 60 and doubling each retry attempt, to a maximum of
`max_sleeptime'.
`retry_exceptions' is a tuple of Exceptions that should be caught.
If exceptions other than those listed in `retry_exceptions' are
raised from `action', they will be raised immediately.
`good_statuses' is a tuple of return values which, if specified,
will result in retrying if the return value isn't listed.
If `cleanup' is provided and callable it will be called immediately
after an Exception is caught. No arguments will be passed to it.
If your cleanup function requires arguments it is recommended that
you wrap it in an argumentless function.
`args' and `kwargs' are a tuple and dict of arguments to pass onto
to `callable'.
- rmtree(self, path, log_level='info', error_level='error', exit_code=-1)
- Returns None for success, not None for failure
- run_command(self, command, cwd=None, error_list=None, parse_at_end=False, halt_on_failure=False, success_codes=None, env=None, return_type='status', throw_exception=False, output_parser=None)
- Run a command, with logging and error parsing.
TODO: parse_at_end, context_lines
TODO: retry_interval?
TODO: error_level_override?
TODO: Add a copy-pastable version of |command| if it's a list.
TODO: print env if set
output_parser lets you provide an instance of your own OutputParser
subclass, or pass None to use OutputParser.
error_list example:
[{'regex': re.compile('^Error: LOL J/K'), level=IGNORE},
{'regex': re.compile('^Error:'), level=ERROR, contextLines='5:5'},
{'substr': 'THE WORLD IS ENDING', level=FATAL, contextLines='20:'}
]
(context_lines isn't written yet)
- which(self, program)
- OS independent implementation of Unix's which command
Takes in a program name
Returns path to executable or None
- write_to_file(self, file_path, contents, verbose=True, open_mode='w', create_parent_dir=False, error_level='error')
- Write contents to file_path.
This doesn't currently create the parent_dir or translate into
abs_path; that needs to be done beforehand, since ScriptMixin doesn't
necessarily have access to query_abs_dirs().
Returns file_path if successful, None if not.
Data descriptors defined here:
- __dict__
- dictionary for instance variables (if defined)
- __weakref__
- list of weak references to the object (if defined)
Data and other attributes defined here:
- env = None
| |