attribench.result.GroupedMetricResult

class attribench.result.GroupedMetricResult(method_names, shape, levels, level_order)[source]

Bases: MetricResult

Abstract class to represent grouped results of distributed metrics. These are results of metrics where the attributions dataset is grouped, i.e. the metric is computed for all attribution methods at a time. This is used for metrics which have a shared computation for all attribution methods to save computation time, e.g. Infidelity.

Parameters:

method_namesTuple[str, …] | List[str]

Names of attribution methods tested by the metric.

shapeTuple[int, …] | List[int]

Shape of numpy arrays that contain the results. Note that this is the result on the full dataset, not a single sample. For example, if the metric is computed on 100 samples and the metric returns 10 values per sample, then the shape should be (100, 10).

levelsDict[str, Tuple[str, …] | List[str]]

Dictionary mapping level names to level values. For example:

{
    "method": ("a", "b"),
    "activation_fn": ("relu", "sigmoid")
}

level_orderTuple[str, …] | List[str]

Order of the levels in the result tree. This should contain all the keys in levels.

Methods

`add`	Add a batch to the result (abstract method).
`get_df`	Retrieve a dataframe from the result object for some given arguments, along with a boolean indicating if higher is better.
`load`	Load a result from an HDF5 file or a directory of CSV files.
`save`	Save the result to an HDF5 file or a nested directory of CSV files.

abstract add(batch_result)[source]: Add a batch to the result (abstract method).

abstract get_df(*args, **kwargs)

Retrieve a dataframe from the result object for some given arguments, along with a boolean indicating if higher is better. These arguments depend on the specific metric.

Return type:: Tuple[DataFrame, bool]

classmethod load(path)

Load a result from an HDF5 file or a directory of CSV files. The format is inferred from the path: if the path is a directory, the result is loaded from a directory of CSV files, otherwise the result is loaded from an HDF5 file.

The specific subclass of MetricResult is inferred from the metadata stored in the file or directory, and the appropriate load method is called.

Parameters:

pathstr: Path to the file or directory.

Returns:

MetricResult: The loaded result.

Return type:

MetricResult

save(path, format)

Save the result to an HDF5 file or a nested directory of CSV files.

Parameters:

pathstr: Path to the file.
formatstr: Format to save the result in. Options: hdf5, dir. If hdf5, the full result is stored in a single HDF5 file. If csv, the result is stored in a nested directory of CSV files.

Return type:

None