attribench.distributed.metrics.MaxSensitivity

class attribench.distributed.metrics.MaxSensitivity(model_factory, attributions_dataset, batch_size, method_factory, num_perturbations, radius, address='localhost', port='12355', devices=None)

Bases: Metric

Compute the Max-Sensitivity metric for a given Dataset and attribution methods using multiple processes.

Max-Sensitivity is computed by adding a small amount of uniform noise to the input samples and computing the norm of the difference in attributions between the original samples and the noisy samples. The maximum norm of difference is then taken as the Max-Sensitivity.

The idea is that a small amount of noise should not change the attributions significantly, so the norm of the difference should be small. If the norm is large, then the attributions are not robust to small perturbations in the input.

The number of processes is determined by the number of devices. If devices is None, then all available devices are used. Samples are distributed evenly across the processes. Each subprocess computes the Max-Sensitivity for a all attribution methods on a subset of the samples.

Parameters:

model_factoryModelFactory

ModelFactory instance or callable that returns a model. Used to create a model for each subprocess.

attributions_datasetDataset

Torch Dataset to use for computing the Max-Sensitivity.

batch_sizeint

The batch size per subprocess to use for computing the Max-Sensitivity.

method_factoryMethodFactory

MethodFactory instance or callable that returns a dictionary of attribution methods, given a model.

num_perturbationsint

The number of perturbations to use for computing the Max-Sensitivity.

radiusfloat

The radius of the uniform noise to add to the input samples.

addressstr, optional

Address to use for the multiprocessing connection. Default: “localhost”

portstr, optional

Port to use for the multiprocessing connection. Default: “12355”

devicesOptional[Tuple], optional

Tuple of devices to use for multiprocessing. If None, all available devices are used.

Methods

run	Runs the metric computation and optionally saves the result.
save_result	Save the result to disk.

Attributes

result

run(result_path=None, progress=True)

Runs the metric computation and optionally saves the result. If no result path is given, the result will not be saved to disk. It can still be accessed via the result property.

Parameters:

result_pathstr, optional

Path to save the result to. If None, the result is not saved to disk.

progressbool, optional

Whether to show a progress bar. Defaults to True.

save_result(path, format='hdf5')

Save the result to disk.

Parameters:

pathstr

Path to save the result to.

formatstr, optional

Format to save the result in. If "hdf5", the result is saved as an HDF5 file. If "csv", the result is saved as a directory structure containing CSV files. Default: "hdf5".

Raises:

ValueError

If the result is None.