| | |
- __builtin__.object
-
- VCSMixin
-
- VCSScript(VCSMixin, mozharness.base.script.BaseScript)
-
- MercurialScript
class MercurialScript(VCSScript) |
| |
# Specific VCS stubs {{{1
# For ease of use.
# This is here instead of mercurial.py because importing MercurialVCS into
# vcsbase from mercurial, and importing VCSScript into mercurial from
# vcsbase, was giving me issues.
|
| |
- Method resolution order:
- MercurialScript
- VCSScript
- VCSMixin
- mozharness.base.script.BaseScript
- mozharness.base.script.ShellMixin
- mozharness.base.script.OSMixin
- mozharness.base.log.LogMixin
- __builtin__.object
Data and other attributes defined here:
- default_vcs = 'hg'
Methods inherited from VCSScript:
- __init__(self, **kwargs)
Methods inherited from VCSMixin:
- vcs_checkout(self, vcs=None, **kwargs)
- Check out a single repo.
- vcs_checkout_repos(self, repo_list, parent_dir=None, tag_override=None, **kwargs)
- Check out a list of repos.
Data descriptors inherited from VCSMixin:
- __dict__
- dictionary for instance variables (if defined)
- __weakref__
- list of weak references to the object (if defined)
Methods inherited from mozharness.base.script.BaseScript:
- action_message(self, message)
- add_failure(self, key, message='%(key)s failed.', level='error')
- add_summary(self, message, level='info')
- clobber(self)
- Delete the working directory
- copy_to_upload_dir(self, target, dest=None, short_desc='unknown', long_desc='unknown', log_level='debug', error_level='error', rotate=False, max_backups=None)
- Copy target file to upload_dir/dest.
Potentially update a manifest in the future if we go that route.
Currently only copies a single file; would be nice to allow for
recursive copying; that would probably done by creating a helper
_copy_file_to_upload_dir().
short_desc and long_desc are placeholders for if/when we add
upload_dir manifests.
- dump_config(self, file_path=None)
- Dump self.config to localconfig.json
- new_log_obj(self, default_log_level='info')
- # logging {{{2
- query_abs_dirs(self)
- We want to be able to determine where all the important things
are. Absolute paths lend themselves well to this, though I wouldn't
be surprised if this causes some issues somewhere.
This should be overridden in any script that has additional dirs
to query.
The query_* methods tend to set self.VAR variables as their
runtime cache.
- query_failure(self, key)
- run(self)
- Default run method.
This is the "do everything" method, based on actions and all_actions.
First run dump_config() if it exists.
Second, go through the list of all_actions.
If they're in the list of self.actions, try to run
preflight_ACTION(), ACTION(), and postflight_ACTION().
Preflight is sanity checking before doing anything time consuming or
destructive.
Postflight is quick testing for success after an action.
Run summary() at the end.
- summarize_success_count(self, success_count, total_count, message='%d of %d successful.')
- summary(self)
- Print out all the summary lines added via add_summary()
throughout the script.
I'd like to revisit how to do this in a prettier fashion.
Methods inherited from mozharness.base.script.ShellMixin:
- get_output_from_command(self, command, cwd=None, halt_on_failure=False, env=None, silent=False, log_level='info', tmpfile_base_path='tmpfile', return_type='output', save_tmpfiles=False, throw_exception=False)
- Similar to run_command, but where run_command is an
os.system(command) analog, get_output_from_command is a `command`
analog.
Less error checking by design, though if we figure out how to
do it without borking the output, great.
TODO: binary mode? silent is kinda like that.
TODO: since p.wait() can take a long time, optionally log something
every N seconds?
TODO: optionally only keep the first or last (N) line(s) of output?
TODO: optionally only return the tmp_stdout_filename?
- query_env(self, partial_env=None, replace_dict=None, set_self_env=None)
- Environment query/generation method.
The default, query_env(), will look for self.config['env']
and replace any special strings in there ( %(PATH)s ).
It will then store it as self.env for speeding things up later.
If you specify partial_env, partial_env will be used instead of
self.config['env'], and we don't save self.env as it's a one-off.
- query_exe(self, exe_name, exe_dict='exes', default=None, return_type=None, error_level='fatal')
- One way to work around PATH rewrites.
By default, return exe_name, and we'll fall through to searching
os.environ["PATH"].
However, if self.config[exe_dict][exe_name] exists, return that.
This lets us override exe paths via config file.
'return_type' can be None (don't do anything to the value),
'list' (return a list), or 'string' (return a string).
If we need runtime setting, we can build in self.exes support later.
- run_command(self, command, cwd=None, error_list=None, parse_at_end=False, halt_on_failure=False, success_codes=None, env=None, return_type='status', throw_exception=False)
- Run a command, with logging and error parsing.
TODO: parse_at_end, context_lines
TODO: retry_interval?
TODO: error_level_override?
TODO: Add a copy-pastable version of |command| if it's a list.
TODO: print env if set
error_list example:
[{'regex': re.compile('^Error: LOL J/K'), level=IGNORE},
{'regex': re.compile('^Error:'), level=ERROR, contextLines='5:5'},
{'substr': 'THE WORLD IS ENDING', level=FATAL, contextLines='20:'}
]
(context_lines isn't written yet)
Methods inherited from mozharness.base.script.OSMixin:
- chdir(self, dir_name, ignore_if_noop=False)
- chmod(self, path, mode)
- copyfile(self, src, dest, log_level='info', error_level='error')
- download_file(self, url, file_name=None, parent_dir=None, create_parent_dir=True, error_level='error', exit_code=-1)
- Python wget.
TODO: should noop touch the filename? seems counter-noop.
TODO: the initial log line should say "Downloading url to file_name"
- get_filename_from_url(self, url)
- mkdir_p(self, path, error_level='error')
- Returns None for success, not None for failure
- move(self, src, dest, log_level='info', error_level='error', exit_code=-1)
- read_from_file(self, file_path, verbose=True, open_mode='r', error_level='error')
- Reads from file_path.
Returns contents if successful, None if not.
- rmtree(self, path, log_level='info', error_level='error', exit_code=-1)
- Returns None for success, not None for failure
- which(self, program)
- OS independent implementation of Unix's which command
Takes in a program name
Returns path to executable or None
- write_to_file(self, file_path, contents, verbose=True, open_mode='w', create_parent_dir=False, error_level='error')
- Write contents to file_path.
This doesn't currently create the parent_dir or translate into
abs_path; that needs to be done beforehand, since OSMixin doesn't
necessarily have access to query_abs_dirs().
Returns file_path if successful, None if not.
Methods inherited from mozharness.base.log.LogMixin:
- critical(self, message)
- debug(self, message)
- dump_exception(self, message=None, level='error')
- # Copying Bear's dumpException():
# http://hg.mozilla.org/build/tools/annotate/1485f23c38e0/sut_tools/sut_lib.py#l23
- error(self, message)
- fatal(self, message, exit_code=-1)
- info(self, message)
- log(self, message, level='info', exit_code=-1)
- warning(self, message)
|
class VCSMixin(__builtin__.object) |
| |
Basic VCS methods that are vcs-agnostic.
The vcs_class handles all the vcs-specific tasks.
|
| |
Methods defined here:
- vcs_checkout(self, vcs=None, **kwargs)
- Check out a single repo.
- vcs_checkout_repos(self, repo_list, parent_dir=None, tag_override=None, **kwargs)
- Check out a list of repos.
Data descriptors defined here:
- __dict__
- dictionary for instance variables (if defined)
- __weakref__
- list of weak references to the object (if defined)
|
class VCSScript(VCSMixin, mozharness.base.script.BaseScript) |
| | |
- Method resolution order:
- VCSScript
- VCSMixin
- mozharness.base.script.BaseScript
- mozharness.base.script.ShellMixin
- mozharness.base.script.OSMixin
- mozharness.base.log.LogMixin
- __builtin__.object
Methods defined here:
- __init__(self, **kwargs)
Methods inherited from VCSMixin:
- vcs_checkout(self, vcs=None, **kwargs)
- Check out a single repo.
- vcs_checkout_repos(self, repo_list, parent_dir=None, tag_override=None, **kwargs)
- Check out a list of repos.
Data descriptors inherited from VCSMixin:
- __dict__
- dictionary for instance variables (if defined)
- __weakref__
- list of weak references to the object (if defined)
Methods inherited from mozharness.base.script.BaseScript:
- action_message(self, message)
- add_failure(self, key, message='%(key)s failed.', level='error')
- add_summary(self, message, level='info')
- clobber(self)
- Delete the working directory
- copy_to_upload_dir(self, target, dest=None, short_desc='unknown', long_desc='unknown', log_level='debug', error_level='error', rotate=False, max_backups=None)
- Copy target file to upload_dir/dest.
Potentially update a manifest in the future if we go that route.
Currently only copies a single file; would be nice to allow for
recursive copying; that would probably done by creating a helper
_copy_file_to_upload_dir().
short_desc and long_desc are placeholders for if/when we add
upload_dir manifests.
- dump_config(self, file_path=None)
- Dump self.config to localconfig.json
- new_log_obj(self, default_log_level='info')
- # logging {{{2
- query_abs_dirs(self)
- We want to be able to determine where all the important things
are. Absolute paths lend themselves well to this, though I wouldn't
be surprised if this causes some issues somewhere.
This should be overridden in any script that has additional dirs
to query.
The query_* methods tend to set self.VAR variables as their
runtime cache.
- query_failure(self, key)
- run(self)
- Default run method.
This is the "do everything" method, based on actions and all_actions.
First run dump_config() if it exists.
Second, go through the list of all_actions.
If they're in the list of self.actions, try to run
preflight_ACTION(), ACTION(), and postflight_ACTION().
Preflight is sanity checking before doing anything time consuming or
destructive.
Postflight is quick testing for success after an action.
Run summary() at the end.
- summarize_success_count(self, success_count, total_count, message='%d of %d successful.')
- summary(self)
- Print out all the summary lines added via add_summary()
throughout the script.
I'd like to revisit how to do this in a prettier fashion.
Methods inherited from mozharness.base.script.ShellMixin:
- get_output_from_command(self, command, cwd=None, halt_on_failure=False, env=None, silent=False, log_level='info', tmpfile_base_path='tmpfile', return_type='output', save_tmpfiles=False, throw_exception=False)
- Similar to run_command, but where run_command is an
os.system(command) analog, get_output_from_command is a `command`
analog.
Less error checking by design, though if we figure out how to
do it without borking the output, great.
TODO: binary mode? silent is kinda like that.
TODO: since p.wait() can take a long time, optionally log something
every N seconds?
TODO: optionally only keep the first or last (N) line(s) of output?
TODO: optionally only return the tmp_stdout_filename?
- query_env(self, partial_env=None, replace_dict=None, set_self_env=None)
- Environment query/generation method.
The default, query_env(), will look for self.config['env']
and replace any special strings in there ( %(PATH)s ).
It will then store it as self.env for speeding things up later.
If you specify partial_env, partial_env will be used instead of
self.config['env'], and we don't save self.env as it's a one-off.
- query_exe(self, exe_name, exe_dict='exes', default=None, return_type=None, error_level='fatal')
- One way to work around PATH rewrites.
By default, return exe_name, and we'll fall through to searching
os.environ["PATH"].
However, if self.config[exe_dict][exe_name] exists, return that.
This lets us override exe paths via config file.
'return_type' can be None (don't do anything to the value),
'list' (return a list), or 'string' (return a string).
If we need runtime setting, we can build in self.exes support later.
- run_command(self, command, cwd=None, error_list=None, parse_at_end=False, halt_on_failure=False, success_codes=None, env=None, return_type='status', throw_exception=False)
- Run a command, with logging and error parsing.
TODO: parse_at_end, context_lines
TODO: retry_interval?
TODO: error_level_override?
TODO: Add a copy-pastable version of |command| if it's a list.
TODO: print env if set
error_list example:
[{'regex': re.compile('^Error: LOL J/K'), level=IGNORE},
{'regex': re.compile('^Error:'), level=ERROR, contextLines='5:5'},
{'substr': 'THE WORLD IS ENDING', level=FATAL, contextLines='20:'}
]
(context_lines isn't written yet)
Methods inherited from mozharness.base.script.OSMixin:
- chdir(self, dir_name, ignore_if_noop=False)
- chmod(self, path, mode)
- copyfile(self, src, dest, log_level='info', error_level='error')
- download_file(self, url, file_name=None, parent_dir=None, create_parent_dir=True, error_level='error', exit_code=-1)
- Python wget.
TODO: should noop touch the filename? seems counter-noop.
TODO: the initial log line should say "Downloading url to file_name"
- get_filename_from_url(self, url)
- mkdir_p(self, path, error_level='error')
- Returns None for success, not None for failure
- move(self, src, dest, log_level='info', error_level='error', exit_code=-1)
- read_from_file(self, file_path, verbose=True, open_mode='r', error_level='error')
- Reads from file_path.
Returns contents if successful, None if not.
- rmtree(self, path, log_level='info', error_level='error', exit_code=-1)
- Returns None for success, not None for failure
- which(self, program)
- OS independent implementation of Unix's which command
Takes in a program name
Returns path to executable or None
- write_to_file(self, file_path, contents, verbose=True, open_mode='w', create_parent_dir=False, error_level='error')
- Write contents to file_path.
This doesn't currently create the parent_dir or translate into
abs_path; that needs to be done beforehand, since OSMixin doesn't
necessarily have access to query_abs_dirs().
Returns file_path if successful, None if not.
Methods inherited from mozharness.base.log.LogMixin:
- critical(self, message)
- debug(self, message)
- dump_exception(self, message=None, level='error')
- # Copying Bear's dumpException():
# http://hg.mozilla.org/build/tools/annotate/1485f23c38e0/sut_tools/sut_lib.py#l23
- error(self, message)
- fatal(self, message, exit_code=-1)
- info(self, message)
- log(self, message, level='info', exit_code=-1)
- warning(self, message)
| |