multi_locale_build
index
/home/aki/wrk/git-mozharness-ro/mozharness/mozilla/l10n/multi_locale_build.py

multi_locale_build.py
 
This should be a mostly generic multilocale build script.

 
Modules
       
os
sys

 
Classes
       
mozharness.base.vcs.vcsbase.MercurialScript(mozharness.base.vcs.vcsbase.VCSScript)
MultiLocaleBuild(mozharness.mozilla.l10n.locales.LocalesMixin, mozharness.base.vcs.vcsbase.MercurialScript)
mozharness.mozilla.l10n.locales.LocalesMixin(mozharness.base.parallel.ChunkingMixin)
MultiLocaleBuild(mozharness.mozilla.l10n.locales.LocalesMixin, mozharness.base.vcs.vcsbase.MercurialScript)

 
class MultiLocaleBuild(mozharness.mozilla.l10n.locales.LocalesMixin, mozharness.base.vcs.vcsbase.MercurialScript)
    MultiLocaleBuild {{{1
 
 
Method resolution order:
MultiLocaleBuild
mozharness.mozilla.l10n.locales.LocalesMixin
mozharness.base.parallel.ChunkingMixin
mozharness.base.vcs.vcsbase.MercurialScript
mozharness.base.vcs.vcsbase.VCSScript
mozharness.base.vcs.vcsbase.VCSMixin
mozharness.base.script.BaseScript
mozharness.base.script.ShellMixin
mozharness.base.script.OSMixin
mozharness.base.log.LogMixin
__builtin__.object

Methods defined here:
__init__(self, require_config_file=True)
add_locales(self)
additional_packaging(self, package_type='en-US', env=None)
backup_objdir(self)
build(self)
clobber(self)
package(self, package_type='en-US')
package_en_US(self)
package_multi(self)
preflight_package_multi(self)
pull_build_source(self)
restore_objdir(self)
upload_en_US(self)
upload_multi(self)

Data and other attributes defined here:
config_options = [[['--locale'], {'action': 'extend', 'dest': 'locales', 'help': 'Specify the locale(s) to repack', 'type': 'string'}], [['--merge-locales'], {'action': 'store_true', 'default': False, 'dest': 'merge_locales', 'help': 'Use default [en-US] if there are missing strings'}], [['--no-merge-locales'], {'action': 'store_false', 'dest': 'merge_locales', 'help': 'Do not allow missing strings'}], [['--objdir'], {'action': 'store', 'default': 'objdir', 'dest': 'objdir', 'help': 'Specify the objdir', 'type': 'string'}], [['--l10n-base'], {'action': 'store', 'dest': 'hg_l10n_base', 'help': 'Specify the L10n repo base directory', 'type': 'string'}], [['--l10n-tag'], {'action': 'store', 'dest': 'hg_l10n_tag', 'help': 'Specify the L10n tag', 'type': 'string'}], [['--tag-override'], {'action': 'store', 'dest': 'tag_override', 'help': 'Override the tags set for all repos', 'type': 'string'}], [['--user-repo-override'], {'action': 'store', 'dest': 'user_repo_override', 'help': 'Override the user repo path for all repos', 'type': 'string'}], [['--l10n-dir'], {'action': 'store', 'default': 'l10n', 'dest': 'l10n_dir', 'help': 'Specify the l10n dir name', 'type': 'string'}]]

Methods inherited from mozharness.mozilla.l10n.locales.LocalesMixin:
list_locales(self)
Stub action method.
parse_locales_file(self, locales_file)
pull_locale_source(self)
# This requires self to inherit a VCSMixin.
query_abs_dirs(self)
query_locales(self)
run_compare_locales(self, locale, halt_on_failure=False)

Methods inherited from mozharness.base.parallel.ChunkingMixin:
query_chunked_list(self, possible_list, this_chunk, total_chunks, sort=False)
Split a list of items into a certain number of chunks and
return the subset of that will occur in this chunk.
 
Ported from build.l10n.getLocalesForChunk in build/tools.

Data descriptors inherited from mozharness.base.parallel.ChunkingMixin:
__dict__
dictionary for instance variables (if defined)
__weakref__
list of weak references to the object (if defined)

Data and other attributes inherited from mozharness.base.vcs.vcsbase.MercurialScript:
default_vcs = 'hg'

Methods inherited from mozharness.base.vcs.vcsbase.VCSMixin:
vcs_checkout(self, vcs=None, **kwargs)
Check out a single repo.
vcs_checkout_repos(self, repo_list, parent_dir=None, tag_override=None, **kwargs)
Check out a list of repos.

Methods inherited from mozharness.base.script.BaseScript:
action_message(self, message)
add_failure(self, key, message='%(key)s failed.', level='error')
add_summary(self, message, level='info')
copy_to_upload_dir(self, target, dest=None, short_desc='unknown', long_desc='unknown', log_level='debug', error_level='error', rotate=False, max_backups=None)
Copy target file to upload_dir/dest.
 
Potentially update a manifest in the future if we go that route.
 
Currently only copies a single file; would be nice to allow for
recursive copying; that would probably done by creating a helper
_copy_file_to_upload_dir().
 
short_desc and long_desc are placeholders for if/when we add
upload_dir manifests.
dump_config(self, file_path=None)
Dump self.config to localconfig.json
new_log_obj(self, default_log_level='info')
# logging {{{2
query_failure(self, key)
run(self)
Default run method.
This is the "do everything" method, based on actions and all_actions.
 
First run dump_config() if it exists.
Second, go through the list of all_actions.
If they're in the list of self.actions, try to run
preflight_ACTION(), ACTION(), and postflight_ACTION().
 
Preflight is sanity checking before doing anything time consuming or
destructive.
 
Postflight is quick testing for success after an action.
 
Run summary() at the end.
summarize_success_count(self, success_count, total_count, message='%d of %d successful.')
summary(self)
Print out all the summary lines added via add_summary()
throughout the script.
 
I'd like to revisit how to do this in a prettier fashion.

Methods inherited from mozharness.base.script.ShellMixin:
get_output_from_command(self, command, cwd=None, halt_on_failure=False, env=None, silent=False, log_level='info', tmpfile_base_path='tmpfile', return_type='output', save_tmpfiles=False, throw_exception=False)
Similar to run_command, but where run_command is an
os.system(command) analog, get_output_from_command is a `command`
analog.
 
Less error checking by design, though if we figure out how to
do it without borking the output, great.
 
TODO: binary mode? silent is kinda like that.
TODO: since p.wait() can take a long time, optionally log something
every N seconds?
TODO: optionally only keep the first or last (N) line(s) of output?
TODO: optionally only return the tmp_stdout_filename?
query_env(self, partial_env=None, replace_dict=None, set_self_env=None)
Environment query/generation method.
 
The default, query_env(), will look for self.config['env']
and replace any special strings in there ( %(PATH)s ).
It will then store it as self.env for speeding things up later.
 
If you specify partial_env, partial_env will be used instead of
self.config['env'], and we don't save self.env as it's a one-off.
query_exe(self, exe_name, exe_dict='exes', default=None, return_type=None, error_level='fatal')
One way to work around PATH rewrites.
 
By default, return exe_name, and we'll fall through to searching
os.environ["PATH"].
However, if self.config[exe_dict][exe_name] exists, return that.
This lets us override exe paths via config file.
 
'return_type' can be None (don't do anything to the value),
'list' (return a list), or 'string' (return a string).
 
If we need runtime setting, we can build in self.exes support later.
run_command(self, command, cwd=None, error_list=None, parse_at_end=False, halt_on_failure=False, success_codes=None, env=None, return_type='status', throw_exception=False)
Run a command, with logging and error parsing.
 
TODO: parse_at_end, context_lines
TODO: retry_interval?
TODO: error_level_override?
TODO: Add a copy-pastable version of |command| if it's a list.
TODO: print env if set
 
error_list example:
[{'regex': re.compile('^Error: LOL J/K'), level=IGNORE},
 {'regex': re.compile('^Error:'), level=ERROR, contextLines='5:5'},
 {'substr': 'THE WORLD IS ENDING', level=FATAL, contextLines='20:'}
]
(context_lines isn't written yet)

Methods inherited from mozharness.base.script.OSMixin:
chdir(self, dir_name, ignore_if_noop=False)
chmod(self, path, mode)
copyfile(self, src, dest, log_level='info', error_level='error')
download_file(self, url, file_name=None, parent_dir=None, create_parent_dir=True, error_level='error', exit_code=-1)
Python wget.
TODO: should noop touch the filename? seems counter-noop.
TODO: the initial log line should say "Downloading url to file_name"
get_filename_from_url(self, url)
mkdir_p(self, path, error_level='error')
Returns None for success, not None for failure
move(self, src, dest, log_level='info', error_level='error', exit_code=-1)
read_from_file(self, file_path, verbose=True, open_mode='r', error_level='error')
Reads from file_path.
 
Returns contents if successful, None if not.
rmtree(self, path, log_level='info', error_level='error', exit_code=-1)
Returns None for success, not None for failure
which(self, program)
OS independent implementation of Unix's which command
Takes in a program name
Returns path to executable or None
write_to_file(self, file_path, contents, verbose=True, open_mode='w', create_parent_dir=False, error_level='error')
Write contents to file_path.
 
This doesn't currently create the parent_dir or translate into
abs_path; that needs to be done beforehand, since OSMixin doesn't
necessarily have access to query_abs_dirs().
 
Returns file_path if successful, None if not.

Methods inherited from mozharness.base.log.LogMixin:
critical(self, message)
debug(self, message)
dump_exception(self, message=None, level='error')
# Copying Bear's dumpException():
http://hg.mozilla.org/build/tools/annotate/1485f23c38e0/sut_tools/sut_lib.py#l23
error(self, message)
fatal(self, message, exit_code=-1)
info(self, message)
log(self, message, level='info', exit_code=-1)
warning(self, message)

 
Data
        FATAL = 'fatal'
MakefileErrorList = [{'level': 'error', 'substr': 'command not found'}, {'level': 'error', 'substr': 'command not found'}, {'level': 'error', 'substr': 'Traceback (most recent call last)'}, {'level': 'error', 'substr': 'SyntaxError: '}, {'level': 'error', 'substr': 'TypeError: '}, {'level': 'error', 'substr': 'NameError: '}, {'level': 'error', 'substr': 'ZeroDivisionError: '}, {'level': 'critical', 'regex': <_sre.SRE_Pattern object at 0xb7ef8180>}, {'level': 'critical', 'regex': <_sre.SRE_Pattern object at 0xb7f26980>}, {'level': 'error', 'substr': 'No rule to make target '}, {'level': 'error', 'regex': <_sre.SRE_Pattern object at 0xb7eeb3b8>}, {'level': 'error', 'regex': <_sre.SRE_Pattern object at 0xb7ee2158>}, {'level': 'error', 'regex': <_sre.SRE_Pattern object at 0xb7ed84d0>}, {'level': 'error', 'regex': <_sre.SRE_Pattern object at 0xb7ee0458>}, {'level': 'warning', 'regex': <_sre.SRE_Pattern object at 0xb7f0b920>}, {'level': 'warning', 'substr': 'Warning: '}]
SSHErrorList = [{'level': 'error', 'substr': 'command not found'}, {'level': 'error', 'substr': 'Name or service not known'}, {'level': 'error', 'substr': 'Could not resolve hostname'}, {'level': 'warning', 'substr': 'POSSIBLE BREAK-IN ATTEMPT'}, {'level': 'error', 'substr': 'Network error:'}, {'level': 'error', 'substr': 'Access denied'}, {'level': 'error', 'substr': 'Authentication refused'}, {'level': 'error', 'substr': 'Out of memory'}, {'level': 'warning', 'substr': 'Connection reset by peer'}, {'level': 'error', 'substr': 'Host key verification failed'}, {'level': 'warning', 'substr': 'WARNING:'}, {'level': 'error', 'substr': 'rsync error:'}, {'level': 'error', 'substr': 'Broken pipe:'}, {'level': 'error', 'substr': 'Permission denied:'}, {'level': 'error', 'substr': 'connection unexpectedly closed'}, {'level': 'error', 'substr': 'Warning: Identity file'}, {'level': 'error', 'substr': 'command-line line 0: Missing argument'}]