domainlab.exp_protocol package

Submodules

domainlab.exp_protocol.aggregate_results module

Functions to join the csv result files generate by different jobs of the benchmarks into a single csv file.

domainlab.exp_protocol.aggregate_results.agg_from_directory(input_dir: str, output_file: str)

Aggregates all results from a directory. Used to aggregate partial results.

domainlab.exp_protocol.aggregate_results.agg_main(bm_dir: str, skip_plotting: bool = False)

Aggregates partial results and generate plots.

domainlab.exp_protocol.aggregate_results.agg_results(input_files: List[str], output_file: str)

Aggregrates the results of the snakemake benchmark.

Combines csv files with identical columns into a single csv file.

Parameters:

• input_files – List of csv files with identical header.

• output_file – Output csv file.

domainlab.exp_protocol.run_experiment module

Runs one task for a single hyperparameter sample for each leave-out-domain and each random seed.

domainlab.exp_protocol.run_experiment.convert_dict2float(dict_in)

convert scientific notation from 1e5 to 10000

domainlab.exp_protocol.run_experiment.load_parameters(file: str, index: int) → tuple

Loads a single parameter sample @param file: csv file @param index: index of hyper-parameter

domainlab.exp_protocol.run_experiment.run_experiment(config: dict, param_file: str, param_index: int, out_file: str, start_seed=None, misc=None, num_gpus=1)

Runs the experiment several times:

for test_domain in test_domains:

for seed from startseed to endseed:

evaluate the algorithm with test_domain, initialization with seed

Parameters:

• config – dictionary from the benchmark yaml

• param_file – path to the csv with the parameter samples

• param_index – parameter index that should be covered by this task,

currently this correspond to the line number in the csv file, or row number in the resulting pandas dataframe :param out_file: path to the output csv :param start_seed: random seed to start for stochastic variations of pytorch :param misc: optional dictionary of additional parameters, if any.

# FIXME: we might want to run the experiment using commandline arguments

Module contents