| | |
- __builtin__.object
-
- OSMixin
- ShellMixin
-
- BaseScript(ShellMixin, OSMixin, mozharness.base.log.LogMixin, __builtin__.object)
class BaseScript(ShellMixin, OSMixin, mozharness.base.log.LogMixin, __builtin__.object) |
| |
# BaseScript {{{1
|
| |
- Method resolution order:
- BaseScript
- ShellMixin
- OSMixin
- mozharness.base.log.LogMixin
- __builtin__.object
Methods defined here:
- __init__(self, config_options=None, default_log_level='info', **kwargs)
- action_message(self, message)
- add_failure(self, key, message='%(key)s failed.', level='error')
- add_summary(self, message, level='info')
- clobber(self)
- Delete the working directory
- copy_to_upload_dir(self, target, dest=None, short_desc='unknown', long_desc='unknown', log_level='debug', error_level='error', rotate=False, max_backups=None)
- Copy target file to upload_dir/dest.
Potentially update a manifest in the future if we go that route.
Currently only copies a single file; would be nice to allow for
recursive copying; that would probably done by creating a helper
_copy_file_to_upload_dir().
short_desc and long_desc are placeholders for if/when we add
upload_dir manifests.
- dump_config(self, file_path=None)
- Dump self.config to localconfig.json
- new_log_obj(self, default_log_level='info')
- # logging {{{2
- query_abs_dirs(self)
- We want to be able to determine where all the important things
are. Absolute paths lend themselves well to this, though I wouldn't
be surprised if this causes some issues somewhere.
This should be overridden in any script that has additional dirs
to query.
The query_* methods tend to set self.VAR variables as their
runtime cache.
- query_failure(self, key)
- run(self)
- Default run method.
This is the "do everything" method, based on actions and all_actions.
First run dump_config() if it exists.
Second, go through the list of all_actions.
If they're in the list of self.actions, try to run
preflight_ACTION(), ACTION(), and postflight_ACTION().
Preflight is sanity checking before doing anything time consuming or
destructive.
Postflight is quick testing for success after an action.
Run summary() at the end.
- summarize_success_count(self, success_count, total_count, message='%d of %d successful.')
- summary(self)
- Print out all the summary lines added via add_summary()
throughout the script.
I'd like to revisit how to do this in a prettier fashion.
Methods inherited from ShellMixin:
- get_output_from_command(self, command, cwd=None, halt_on_failure=False, env=None, silent=False, log_level='info', tmpfile_base_path='tmpfile', return_type='output', save_tmpfiles=False, throw_exception=False)
- Similar to run_command, but where run_command is an
os.system(command) analog, get_output_from_command is a `command`
analog.
Less error checking by design, though if we figure out how to
do it without borking the output, great.
TODO: binary mode? silent is kinda like that.
TODO: since p.wait() can take a long time, optionally log something
every N seconds?
TODO: optionally only keep the first or last (N) line(s) of output?
TODO: optionally only return the tmp_stdout_filename?
- query_env(self, partial_env=None, replace_dict=None, set_self_env=None)
- Environment query/generation method.
The default, query_env(), will look for self.config['env']
and replace any special strings in there ( %(PATH)s ).
It will then store it as self.env for speeding things up later.
If you specify partial_env, partial_env will be used instead of
self.config['env'], and we don't save self.env as it's a one-off.
- query_exe(self, exe_name, exe_dict='exes', default=None, return_type=None, error_level='fatal')
- One way to work around PATH rewrites.
By default, return exe_name, and we'll fall through to searching
os.environ["PATH"].
However, if self.config[exe_dict][exe_name] exists, return that.
This lets us override exe paths via config file.
'return_type' can be None (don't do anything to the value),
'list' (return a list), or 'string' (return a string).
If we need runtime setting, we can build in self.exes support later.
- run_command(self, command, cwd=None, error_list=None, parse_at_end=False, halt_on_failure=False, success_codes=None, env=None, return_type='status', throw_exception=False)
- Run a command, with logging and error parsing.
TODO: parse_at_end, context_lines
TODO: retry_interval?
TODO: error_level_override?
TODO: Add a copy-pastable version of |command| if it's a list.
TODO: print env if set
error_list example:
[{'regex': re.compile('^Error: LOL J/K'), level=IGNORE},
{'regex': re.compile('^Error:'), level=ERROR, contextLines='5:5'},
{'substr': 'THE WORLD IS ENDING', level=FATAL, contextLines='20:'}
]
(context_lines isn't written yet)
Data descriptors inherited from ShellMixin:
- __dict__
- dictionary for instance variables (if defined)
- __weakref__
- list of weak references to the object (if defined)
Methods inherited from OSMixin:
- chdir(self, dir_name, ignore_if_noop=False)
- chmod(self, path, mode)
- copyfile(self, src, dest, log_level='info', error_level='error')
- download_file(self, url, file_name=None, parent_dir=None, create_parent_dir=True, error_level='error', exit_code=-1)
- Python wget.
TODO: should noop touch the filename? seems counter-noop.
TODO: the initial log line should say "Downloading url to file_name"
- get_filename_from_url(self, url)
- mkdir_p(self, path, error_level='error')
- Returns None for success, not None for failure
- move(self, src, dest, log_level='info', error_level='error', exit_code=-1)
- read_from_file(self, file_path, verbose=True, open_mode='r', error_level='error')
- Reads from file_path.
Returns contents if successful, None if not.
- rmtree(self, path, log_level='info', error_level='error', exit_code=-1)
- Returns None for success, not None for failure
- which(self, program)
- OS independent implementation of Unix's which command
Takes in a program name
Returns path to executable or None
- write_to_file(self, file_path, contents, verbose=True, open_mode='w', create_parent_dir=False, error_level='error')
- Write contents to file_path.
This doesn't currently create the parent_dir or translate into
abs_path; that needs to be done beforehand, since OSMixin doesn't
necessarily have access to query_abs_dirs().
Returns file_path if successful, None if not.
Methods inherited from mozharness.base.log.LogMixin:
- critical(self, message)
- debug(self, message)
- dump_exception(self, message=None, level='error')
- # Copying Bear's dumpException():
# http://hg.mozilla.org/build/tools/annotate/1485f23c38e0/sut_tools/sut_lib.py#l23
- error(self, message)
- fatal(self, message, exit_code=-1)
- info(self, message)
- log(self, message, level='info', exit_code=-1)
- warning(self, message)
|
class OSMixin(__builtin__.object) |
| |
Filesystem commands and the like.
Depends on LogMixin, ShellMixin, and a self.config of some sort.
|
| |
Methods defined here:
- chdir(self, dir_name, ignore_if_noop=False)
- chmod(self, path, mode)
- copyfile(self, src, dest, log_level='info', error_level='error')
- download_file(self, url, file_name=None, parent_dir=None, create_parent_dir=True, error_level='error', exit_code=-1)
- Python wget.
TODO: should noop touch the filename? seems counter-noop.
TODO: the initial log line should say "Downloading url to file_name"
- get_filename_from_url(self, url)
- mkdir_p(self, path, error_level='error')
- Returns None for success, not None for failure
- move(self, src, dest, log_level='info', error_level='error', exit_code=-1)
- read_from_file(self, file_path, verbose=True, open_mode='r', error_level='error')
- Reads from file_path.
Returns contents if successful, None if not.
- rmtree(self, path, log_level='info', error_level='error', exit_code=-1)
- Returns None for success, not None for failure
- which(self, program)
- OS independent implementation of Unix's which command
Takes in a program name
Returns path to executable or None
- write_to_file(self, file_path, contents, verbose=True, open_mode='w', create_parent_dir=False, error_level='error')
- Write contents to file_path.
This doesn't currently create the parent_dir or translate into
abs_path; that needs to be done beforehand, since OSMixin doesn't
necessarily have access to query_abs_dirs().
Returns file_path if successful, None if not.
Data descriptors defined here:
- __dict__
- dictionary for instance variables (if defined)
- __weakref__
- list of weak references to the object (if defined)
|
class ShellMixin(__builtin__.object) |
| |
These are very special but very complex methods that, together with
logging and config, provide the base for all scripts in this harness.
This is currently dependent on LogMixin and OSMixin, and assumes that
there is a self.config of some sort.
|
| |
Methods defined here:
- __init__(self)
- get_output_from_command(self, command, cwd=None, halt_on_failure=False, env=None, silent=False, log_level='info', tmpfile_base_path='tmpfile', return_type='output', save_tmpfiles=False, throw_exception=False)
- Similar to run_command, but where run_command is an
os.system(command) analog, get_output_from_command is a `command`
analog.
Less error checking by design, though if we figure out how to
do it without borking the output, great.
TODO: binary mode? silent is kinda like that.
TODO: since p.wait() can take a long time, optionally log something
every N seconds?
TODO: optionally only keep the first or last (N) line(s) of output?
TODO: optionally only return the tmp_stdout_filename?
- query_env(self, partial_env=None, replace_dict=None, set_self_env=None)
- Environment query/generation method.
The default, query_env(), will look for self.config['env']
and replace any special strings in there ( %(PATH)s ).
It will then store it as self.env for speeding things up later.
If you specify partial_env, partial_env will be used instead of
self.config['env'], and we don't save self.env as it's a one-off.
- query_exe(self, exe_name, exe_dict='exes', default=None, return_type=None, error_level='fatal')
- One way to work around PATH rewrites.
By default, return exe_name, and we'll fall through to searching
os.environ["PATH"].
However, if self.config[exe_dict][exe_name] exists, return that.
This lets us override exe paths via config file.
'return_type' can be None (don't do anything to the value),
'list' (return a list), or 'string' (return a string).
If we need runtime setting, we can build in self.exes support later.
- run_command(self, command, cwd=None, error_list=None, parse_at_end=False, halt_on_failure=False, success_codes=None, env=None, return_type='status', throw_exception=False)
- Run a command, with logging and error parsing.
TODO: parse_at_end, context_lines
TODO: retry_interval?
TODO: error_level_override?
TODO: Add a copy-pastable version of |command| if it's a list.
TODO: print env if set
error_list example:
[{'regex': re.compile('^Error: LOL J/K'), level=IGNORE},
{'regex': re.compile('^Error:'), level=ERROR, contextLines='5:5'},
{'substr': 'THE WORLD IS ENDING', level=FATAL, contextLines='20:'}
]
(context_lines isn't written yet)
Data descriptors defined here:
- __dict__
- dictionary for instance variables (if defined)
- __weakref__
- list of weak references to the object (if defined)
| |