trixi.util

class trixi.util.util.CustomJSONDecoder(*, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, strict=True, object_pairs_hook=None)

Bases: json.decoder.JSONDecoder

decode(obj)

Return the Python representation of s (a str instance containing a JSON document).

class trixi.util.util.CustomJSONEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: json.encoder.JSONEncoder

encode(obj)

Return a JSON string representation of a Python data structure.

>>> from json.encoder import JSONEncoder
>>> JSONEncoder().encode({"foo": ["bar", "baz"]})
'{"foo": ["bar", "baz"]}'

iterencode(obj, *args, **kwargs)

Encode the given object and yield each string representation as available.

For example:

for chunk in JSONEncoder().iterencode(bigobject):
    mysocket.write(chunk)

class trixi.util.util.LogDict(file_name, base_dir=None, to_console=False, mode='a')

Bases: dict

log_complete_content()

Logs the current content of the dict to the output file as a whole.

class trixi.util.util.ModuleMultiTypeDecoder(*, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, strict=True, object_pairs_hook=None)

Bases: trixi.util.util.MultiTypeDecoder

class trixi.util.util.ModuleMultiTypeEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: trixi.util.util.MultiTypeEncoder

class trixi.util.util.MultiTypeDecoder(*, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, strict=True, object_pairs_hook=None)

Bases: trixi.util.util.CustomJSONDecoder

class trixi.util.util.MultiTypeEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: trixi.util.util.CustomJSONEncoder

class trixi.util.util.PyLock(name, timeout, check_interval=0.25)

Bases: object

class trixi.util.util.ResultElement(data=None, label=None, epoch=None, counter=None)

Bases: dict

class trixi.util.util.ResultLogDict(file_name, base_dir=None, running_mean_length=10, **kwargs)

Bases: trixi.util.util.LogDict

close()

load(reload_dict)

print_to_file(text)

class trixi.util.util.SafeDict

Bases: dict

class trixi.util.util.Singleton(decorated)

Bases: object

A non-thread-safe helper class to ease implementing singletons. This should be used as a decorator – not a metaclass – to the class that should be a singleton.

The decorated class can define one __init__ function that takes only the self argument. Also, the decorated class cannot be inherited from. Other than that, there are no restrictions that apply to the decorated class.

To get the singleton instance, use the Instance method. Trying to use __call__ will result in a TypeError being raised.

get_instance(**kwargs)

Returns the singleton instance. Upon its first call, it creates a new instance of the decorated class and calls its __init__ method. On all subsequent calls, the already created instance is returned.

class trixi.util.util.StringMultiTypeDecoder(*, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, strict=True, object_pairs_hook=None)

Bases: trixi.util.util.CustomJSONDecoder

trixi.util.util.chw_to_hwc(np_array)

trixi.util.util.create_folder(path)

Creates a folder if not already exists :param : param path: The folder to be created

Returns

return:True if folder was newly created, false if folder already exists

trixi.util.util.figure_to_image(figures, close=True)

Render matplotlib figure to numpy format.

Note that this requires the matplotlib package. (https://tensorboardx.readthedocs.io/en/latest/_modules/tensorboardX/utils.html#figure_to_image)

Parameters:

• figure (matplotlib.pyplot.figure) – figure or a list of figures
• close (bool) – Flag to automatically close the figure

Returns:

image in [CHW] order

Return type:

numpy.array

trixi.util.util.get_image_as_buffered_file(image_array)

Returns a images as file pointer in a buffer

Parameters:image_array – (C,W,H) To be returned as a file pointer Returns:Buffer file-pointer object containing the image file

trixi.util.util.get_tensor_embedding(tensor, method='tsne', n_dims=2, n_neigh=30, **meth_args)

Return a embedding of a tensor (in a lower dimensional space, e.g. t-SNE)

Parameters:

• tensor – Tensor to be embedded
• method – Method used for embedding, options are: tsne, standard, ltsa, hessian, modified, isomap, mds,
• umap (spectral,) –
• n_dims – dimensions to embed the data into
• n_neigh – Neighbour parameter to kind of determin the embedding (see t-SNE for more information)
• **meth_args – Further arguments which can be passed to the embedding method

Returns:

The embedded tensor

trixi.util.util.is_picklable(obj)

trixi.util.util.name_and_iter_to_filename(name, n_iter, ending, iter_format='{:05d}', prefix=False)

trixi.util.util.np_make_grid(np_array, nrow=8, padding=2, normalize=False, range=None, scale_each=False, pad_value=0, to_int=False, standardize=False)

Make a grid of images.

Parameters:

• np_array (numpy array) – 4D mini-batch Tensor of shape (B x C x H x W) or a list of images all of the same size.
• nrow (int, optional) – Number of images displayed in each row of the grid. The Final grid size is (B / nrow, nrow). Default is 8.
• padding (int, optional) – amount of padding. Default is 2.
• normalize (bool, optional) – If True, shift the image to the range (0, 1), by subtracting the minimum and dividing by the maximum pixel value.
• range (tuple, optional) – tuple (min, max) where min and max are numbers, then these numbers are used to normalize the image. By default, min and max are computed from the tensor.
• scale_each (bool, optional) – If True, scale each image in the batch of images separately rather than the (min, max) over all images.
• pad_value (float, optional) – Value for the padded pixels.
• to_int (bool) – Transforms the np array to a unit8 array with min 0 and max 255

Example

See this notebook here

trixi.util.util.random_string(length)

trixi.util.util.savefig_and_close(figure, filename, close=True)

Config

class trixi.util.config.Config(file_=None, config=None, update_from_argv=False, deep=False, **kwargs)

Bases: dict

Config is the main object used to store configurations. As a rule of thumb, anything you might want to change in your experiment should go into the Config. It’s basically a dict, but vastly more powerful. Key features are

• Access keys as attributes

  Config[“a”][“b”][“c”] is the same as Config.a.b.c. Can also be used for setting if the second to last key exists. Only works for keys that conform with Python syntax (Config.myattr-1 is not allowed).

• Advanced de-/serialization

  Using specialized JSON encoders and decoders, almost anything can be serialized and deserialized. This includes types, functions (except lambdas) and modules. For example, you could have something like:

  c = Config(model=MyModel)
  c.dump("somewhere")
  

  and end up with a JSON file that looks like this:

  {
      "model": "__type__(your.model.module.MyModel)"
  }
  

  and vice versa. We use double underscores and parentheses for serialization, so it’s probably a good idea to not use this pattern for other stuff!

• Automatic CLI exposure

  If desired, the Config will create an ArgumentParser that contains all keys in the Config as arguments in the form “- - key”, so you can run your experiment from the command line and manually overwrite certain values. Deeper levels are also accessible via dot notation “- - key_with_dict_value.inner_key”.

• Comparison

  Compare any number of Configs and get a new Config containing only the values that differ among input Configs.

Parameters:

• file (str) – Load Config from this file.
• config (Config) – Update with values from this Config (can be combined with file_). Will by default only make shallow copies, see deep.
• update_from_argv (bool) – Update values from argv. Will automatically expose keys to the CLI as ‘- - key’.
• deep (bool) – Make deep copies if config is given.

contains(dict_like)

Check whether all items in a dictionary-like object match the ones in this Config.

Parameters:dict_like (dict or derivative thereof) – Returns True if this is contained in this Config. Returns:True if dict_like is contained in self, otherwise False. Return type:bool

deepcopy()

Get a deep copy of this Config.

Returns:A deep copy of self. Return type:Config

deepupdate(dict_like, ignore=None, allow_dict_overwrite=True)

Identical to update() with deep=True.

Parameters:

• dict_like (dict or derivative thereof) – Update source.
• ignore (iterable) – Iterable of keys to ignore in update.
• allow_dict_overwrite (bool) – Allow overwriting with dict. Regular dicts only update on the highest level while we recurse and merge Configs. This flag decides whether it is possible to overwrite a ‘regular’ value with a dict/Config at lower levels. See examples for an illustration of the difference

difference_config(*other_configs)

Get the difference of this and any number of other configs. See difference_config_static() for more information.

Parameters:*other_configs (Config) – Compare these configs and self. Returns:Difference of self and the other configs. Return type:Config

static difference_config_static(*configs, only_set=False, encode=True)

Make a Config of all elements that differ between N configs.

The resulting Config looks like this:

{
    key: (config1[key], config2[key], ...)
}

If the key is missing, None will be inserted. The inputs will not be modified.

Parameters:

• configs (Config) – Any number of Configs
• only_set (bool) – If only the set of different values hould be returned or for each config the
• one (corresponding) –
• encode (bool) – If True, values will be encoded the same way as they are when exported to disk (e.g.”__type__(MyClass)”)

Returns:

Possibly empty Config

Return type:

Config

dump(file_, indent=4, separators=(', ', ': '), **kwargs)

Write config to file using json.dump().

Parameters:

• file (str or File) – Write to this location.
• indent (int) – Formatting option.
• separators (iterable) – Formatting option.
• **kwargs – Will be passed to json.dump().

dumps(indent=4, separators=(', ', ': '), **kwargs)

Get string representation using json.dumps().

Parameters:

• indent (int) – Formatting option.
• separators (iterable) – Formatting option.
• **kwargs – Will be passed to json.dumps().

flat(keep_lists=True, max_split_size=10, flatten_int=False)

Returns a flattened version of the Config as dict.

Nested Configs and lists will be replaced by concatenated keys like so:

{
    "a": 1,
    "b": [2, 3],
    "c": {
        "x": 4,
        "y": {
            "z": 5
        }
    },
    "d": (6, 7)
}

Becomes:

{
    "a": 1,
    "b": [2, 3], # if keep_lists is True
    "b.0": 2,
    "b.1": 3,
    "c.x": 4,
    "c.y.z": 5,
    "d": (6, 7)
}

We return a dict because dots are disallowed within Config keys.

Parameters:

• keep_lists – Keeps list along with unpacked values
• max_split_size – List longer than this will not be unpacked
• flatten_int – Integer keys will be treated as strings

Returns:

A flattened version of self

Return type:

dict

hasattr_not_none(key)

static init_objects(config)

Returns a new Config with types converted to instances.

Any value that is a Config and contains a type key will be converted to an instance of that type:

{
    "stuff": "also_stuff",
    "convert_me": {
        type: {
            "param": 1,
            "other_param": 2
        },
        "something_else": "hopefully_useless"
    }
}

becomes:

{
    "stuff": "also_stuff",
    "convert_me": type(param=1, other_param=2)
}

Note that additional entries can be lost as shown above.

Parameters:config (Config) – New Config will be built from this one Returns:A new config with instances made from type entries. Return type:Config

load(file_, raise_=True, decoder_cls_=<class 'trixi.util.util.ModuleMultiTypeDecoder'>, **kwargs)

Load config from file using json.load().

Parameters:

• file (str or File) – Read from this location.
• raise (bool) – Raise errors.
• decoder_cls (type) – Class that is used to decode JSON string.
• **kwargs – Will be passed to json.load().

loads(json_str, decoder_cls_=<class 'trixi.util.util.ModuleMultiTypeDecoder'>, **kwargs)

Load config from JSON string using json.loads().

Parameters:

• json_str (str) – Interpret this string.
• decoder_cls (type) – Class that is used to decode JSON string.
• **kwargs – Will be passed to json.loads().

set_from_string(str_, stringify_value=False)

Set a value from a single string, separated with “=”. Uses :meth:´set_with_decode´.

Parameters:str (str) – String that looks like “key=value”.

set_with_decode(key, value, stringify_value=False)

Set single value, using ModuleMultiTypeDecoder to interpret key and value strings by creating a temporary JSON string.

Parameters:

• key (str) – Config key.
• value (str) – New value key will map to.
• stringify_value (bool) – If True, will insert the value into the temporary JSON as a real string. See examples!

Examples

Example for when you need to set stringify_value=True:

config.set_with_decode("key", "__type__(trixi.util.config.Config)", stringify_value=True)

Example for when you need to set stringify_value=False:

config.set_with_decode("key", "[1, 2, 3]")

to_cmd_args_str()

Create a string representing what one would need to pass to the command line. Does not yet use JSON encoding!

Returns:Command line string Return type:str

update(dict_like, deep=False, ignore=None, allow_dict_overwrite=True)

Update entries in the Config.

Parameters:

• dict_like (dict or derivative thereof) – Update source.
• deep (bool) – Make deep copies of all references in the source.
• ignore (iterable) – Iterable of keys to ignore in update.
• allow_dict_overwrite (bool) – Allow overwriting with dict. Regular dicts only update on the highest level while we recurse and merge Configs. This flag decides whether it is possible to overwrite a ‘regular’ value with a dict/Config at lower levels. See examples for an illustration of the difference

Examples

The following illustrates the update behaviour if :obj:allow_dict_overwrite is active. If it isn’t, an AttributeError would be raised, originating from trying to update “string”:

config1 = Config(config={
    "lvl0": {
        "lvl1": "string",
        "something": "else"
    }
})

config2 = Config(config={
    "lvl0": {
        "lvl1": {
            "lvl2": "string"
        }
    }
})

config1.update(config2, allow_dict_overwrite=True)

>>>config1
{
    "lvl0": {
        "lvl1": {
            "lvl2": "string"
        },
        "something": "else"
    }
}

update_missing(dict_like, deep=False, ignore=None)

Recursively insert values that do not yet exist.

Parameters:

• dict_like (dict or derivative thereof) – Update source.
• deep (bool) – Make deep copies of all references in the source.
• ignore (iterable) – Iterable of keys to ignore in update.

trixi.util.config.monkey_patch_fn_args_as_config(f)

Decorator: Monkey patches, aka addes a variable ‘fn_args_as_config’ to globals, so that it can be accessed by the decorated function. Adds all function parameters to a dict ‘fn_args_as_config’, which can be accessed by the method. Be careful using it!

trixi.util.config.update_from_sys_argv(config, warn=False)

Updates Config with the arguments passed as args when running the program. Keys will be converted to command line options, then matching options in sys.argv will be used to update the Config.

Parameters:

• config (Config) – Update this Config.
• warn (bool) – Raise warnings if there are unknown options. Turn this on if you don’t use any argparse.ArgumentParser after to check for possible errors.

ExtraVisdom

class trixi.util.extravisdom.ExtraVisdom(*args, **kwargs)

Bases: sphinx.ext.autodoc.importer._MockObject

histogram_3d(X, win=None, env=None, opts=None)

Given an array it plots the histrograms of the entries.

Parameters:

• X – An array of at least 2 dimensions, where the first dimensions gives the number of histograms.
• win – Window name.
• env – Env name.
• opts – dict with options, especially opts[‘numbins’] (number of histogram bins) and opts[‘mutiplier’]

( factor to stretch / queeze the values on the x axis) should be considered.

Returns:The send result.

GridSearch

class trixi.util.gridsearch.GridSearch

Bases: dict

all_combinations()

read(file_, raise_=True, decoder_cls_=<class 'trixi.util.util.ModuleMultiTypeDecoder'>, **kwargs)

pytorchutils

trixi.util.pytorchutils.get_guided_image_gradient(model: <sphinx.ext.autodoc.importer._MockObject object at 0x7f8886120450>, inpt, err_fn, abs=False)

trixi.util.pytorchutils.get_input_gradient(model, inpt, err_fn, grad_type='vanilla', n_runs=20, eps=0.1, abs=False, results_fn=<function <lambda>>)

Given a model creates calculates the error and backpropagates it to the image and saves it (saliency map).

Parameters:

• model – The model to be evaluated
• inpt – Input to the model
• err_fn – The error function the evaluate the output of the model on
• grad_type – Gradient calculation method, currently supports (vanilla, vanilla-smooth, guided,
• guided-smooth) (the guided backprob can lead to segfaults -.-) –
• n_runs – Number of runs for the smooth variants
• eps – noise scaling to be applied on the input image (noise is drawn from N(0,1))
• abs (bool) – Flag, if the gradient should be a absolute value
• results_fn – function which is called with the results/ return values. Expected f(grads)

trixi.util.pytorchutils.get_smooth_image_gradient(model, inpt, err_fn, abs=True, n_runs=20, eps=0.1, grad_type='vanilla')

trixi.util.pytorchutils.get_vanilla_image_gradient(model, inpt, err_fn, abs=False)

trixi.util.pytorchutils.set_seed(seed)

Sets the seed

trixi.util.pytorchutils.update_model(original_model, update_dict, exclude_layers=(), do_warnings=True)

SourcePacker

class trixi.util.sourcepacker.SourcePacker

Bases: object

Inspired by https://github.com/IDSIA/sacred

static create_source_or_dep(mod, sources)

static gather_sources_and_dependencies(globs)

static git_info(file_)

static is_source(filename)

static iter_prefixes(path)

Iterate through all (non-empty) prefixes of a dotted path. Example: >>> list(iter_prefixes(‘foo.bar.baz’)) [‘foo’, ‘foo.bar’, ‘foo.bar.baz’]

static join_paths(*parts)

Join different parts together to a valid dotted path.

static zip_sources(globs, filename)