Module: standardizer

Standardizer converts data from a given data source into a LayeredImage object, applying data-source specific, transformations in the process. A layered image is a collection containing:

• science image

• variance image

• mask image

along with the:

• on-sky coordinates of the pixel in the center of the science image

• observation timestamp

• location of the data source

When possible, standardizers should attempt to extract a valid WCS and/or bounding box from the data source.

exception kbmod.standardizers.standardizer.ConfigurationError

Error that is raised when configuration parameters contain a logical error.

class kbmod.standardizers.standardizer.Standardizer(location, config=None, **kwargs)

Transforms given data into the format required by KBMOD.

The given data are FITS images, Vera C. Rubin Science Pipeline objects and other data that provides access to the images and observation metadata. Standardizers are required to extract certain information from this data.

The required metadata are the location of the data, timestamp and the on-sky coordinates of the observation. The required image data is the science exposure, variance and mask.

The location of the data is a local filesystem path, URI or an URL. Timestamp is the decimal MJD of the start of the exposure. The on-sky coordinates are a pair of (ra, dec) values in decimal degrees of the pointing information extracted from the header or the central pixel on-sky coordinates calculated via WCS.

The science exposure and variance are self-explanatory and the mask is a simple bitmask in which pixels with value 1 will not be included in KBMOD search.

The standardizers are allowed to extract any additional information desired, f.e. such as filter, but there are no guarantess that this information is present for all availible standardizers.

Of the optional metadata two properties are of particular interest - the WCS and the bounding box of each science exposure. They underpin the way the exposures of interest are grouped together. KBMOD does not insist on providing them. however, requiring instead just a more general “pointing” information. For these reason the WCS and BBOX have their own standardize methods, and should be provided whenever possible.

Data is expected to be unravelled on a per-science exposure level. Data that shares some metadata, such as timestamp or filter for example, for multiple science exposures is expected to unravel that metadata and associate it with each of the science exposures individually.

The data for which this standardization and unravelling will occur needs to be appended to the items in processable attribute of the class.

This class is an abstract base class to serve as a recipe for implementing a Standardizer specialized for processing an particular dataset. Do not attempt to instantiate this class directly.

Parameters:

locationstr

Location of the file to standardize, a filepath or URI.

Attributes:

locationstr

Location of the file being standardized.

processablelist

List of standardizers internal units that can be processed by KBMOD. For example, for FITS files on local filesystem these could be the AstroPy header units (~astropy.io.fits.HDUList elements), for Vera C. Rubin their internal Exposure* objects etc. The number of processable units will match the number of returned standardized images, variances, masks and PSFs.

wcslist

A list of WCS’s or None for each entry marked as processable.

bboxlist

A list of bounding boxes or None for each entry marked as processable.

Methods

canStandardize(tgt)	Returns True when the standardizer knows how to process the given upload (file) type.
configClass	Standardizer's configuration.
get(tgt[, force, config])	Get the standardizer class that can handle given file.
resolveTarget(tgt)	Returns a pair of values (canStandardize, resources).
standardize()	Invokes standardize metadata, image, variance, mask and PSF and returns the results as a dictionary.
standardizeBBox()	Calculate the standardized bounding box, the world coordinates at image corner and image center.
standardizeMaskImage()	Standardizes the mask data as an simple 0 (not masked) and 1 (masked) bitmap.
standardizeMetadata()	Standardizes required and optional metadata from the given data.
standardizePSF()	Returns PSF for each extension marked as processable.
standardizeScienceImage()	Standardizes the science image data.
standardizeVarianceImage()	Standardizes the variance data.
standardizeWCS()	Creates an WCS for every processable science image.
toLayeredImage()	Run metadata standardization methods.
wcs()	A list of WCS's or None for each entry marked as processable.

abstract property bbox

A list of bounding boxes or None for each entry marked as processable.

Key	Description
center_ra	On-sky (ICRS) right ascension coordinate, expressed in decimal degrees, of the center of an rectangular bounding box.
center_dec	On-sky (ICRS) declination coordinate, expressed in decimal degrees, of the center of an rectangular bounding box.
corner_ra	On-sky (ICRS) right ascension coordinate, expressed in decimal degrees, of an corner of a rectangular bounding box.
corner_dec	On-sky (ICRS) declination coordinate, expressed in decimal degrees, of an corner of a rectangular bounding box.

Often, when WCS can not be constructed, the BBOX can not be constructed. Then None is expected to be returned in it’s place.

classmethod canStandardize(tgt)

Returns True when the standardizer knows how to process the given upload (file) type.

Parameters:

tgtdata-source

Some data source, URL, URI, POSIX filepath, Rubin Data Butler Data Repository, an Python object, callable, etc.

Returns:

canStandardizebool

True when the processor knows how to handle uploaded file and False otherwise.

Notes

Implementation is instrument-specific.

can_volunteer = True

This standardizer can be automatically detected and used. If set to False the standardizer can only be used with manual specification.

configClass

Standardizer’s configuration. A class whose attributes set the behavior of the standardization.

Methods

alias of StandardizerConfig

classmethod get(tgt, force=None, config=None, **kwargs)

Get the standardizer class that can handle given file.

When the standardizer is registered, it can be requested by its name. See self.registry for a list of registered Standardizers.

When the correct standardizer is not known, the the target can be provided. The Standardizer with the highest priority that marks the target as processable, will be returned.

At least one of either the target or the standardizer parameters have to be given.

Parameters:

tgtany

The target to be standardized.

forcestr or cls, optional

Force the use of the given Standardizer. The given name must be a part of the registered Standardizers or a callable. When no Standardizer is forced, all registered standardizers are tested and the highest priority Standardizer is selected.

config~StandardizerConfig, dict or None, optional

Standardizer configuration or dictionary containing the config parameters for standardization. When None default values for the appropriate Standardizer will be used.

**kwargsdict, optional

Any additional, optional, keyword arguments are passed into the Standardizer. See relevant Standardizer documentation for details.

Returns:

standardizerobject

Standardizer instance forced, or selected as the most appropriate one, to process the given target.

Raises:

ValueError

When neither target or standardizer are given.

ValueError

When no standardizer that marks the target as processable is found.

TypeError

When no standardizer that marks the target as processable is found.

KeyError

When the given standardizer name is not a part of the standardizer registry.

name = None

Processor’s name. Only named standardizers will be registered.

priority = 0

Priority. Standardizers with high priority are prefered over standardizers with low priority when processing an upload.

registry = {'ButlerStandardizer': <class 'kbmod.standardizers.butler_standardizer.ButlerStandardizer'>, 'KBMODV0_5': <class 'kbmod.standardizers.fits_standardizers.kbmodv05.KBMODV0_5'>, 'KBMODV1': <class 'kbmod.standardizers.fits_standardizers.kbmodv1.KBMODV1'>}

All registered upload standardizer classes.

abstract classmethod resolveTarget(tgt)

Returns a pair of values (canStandardize, resources).

The first value, canStandardize, indicates that the standardizer is able to standardize the given target. The second value, resources, is an optional returned value, a dictionary containing any constructed or resolved resources in the process.

This method is called during automatic resolution of standardizers. In that process, each registered Standardizer is asked to resolve the target. The standardizers, and their resources if any, that successfully resolved the target are sorted based on the Standardizer priority and used to standardize the target.

Because each Standardizer is asked to resolve each target, of which there are potentially many, it is encouraged that this method is implemented with performance in mind.

Sometimes, however, it may not be possible to avoid acquiring or constructing a resource that, in the given context, could be considered expensive. Returning the resource(s) allows that cost to be optimized away, by avoiding acquiring or constructing the resource again during initialization.

On a practical example: FitsStandardizer standardize FITS files pointed to by a local filesystem path (i.e. the target). To confirm their ability to standardize the target, they have to make sure they can open and read the file. They use AstroPy to construct an ~astropy.io.fits.HDUList, i.e. the resource. Returning the already constructed ~astropy.io.fits.HDUList allows FitsStandardizer to skip constructing a new one via astropy.io.fits.open.

Parameters:

tgtdata-source

Some data source, URL, URI, POSIX filepath, Rubin Data Butler Data Repository, an Python object, callable, etc.

Returns:

canStandardizebool

True when the processor knows how to handle uploaded file and False otherwise.

resourcesdict, optional

Any optional resources constructed during target resolution. An empty dictionary if there are none.

standardize()

Invokes standardize metadata, image, variance, mask and PSF and returns the results as a dictionary.

Returns:

standardizedDatadict

Dictionary with standardized meta, science, variance, mask and psf values.

abstract standardizeBBox()

Calculate the standardized bounding box, the world coordinates at image corner and image center.

Returns:

standardizedBBoxdict

Calculated coordinate values, a dict with, wcs_center_[ra, dec] and wcs_corner_[ra, dec] keys.

Notes

Center can be assumed to be at the (dimX/2, dimY/2) pixel coordinates, rounded down. Corner is taken to be the (0,0)-th pixel.

abstract standardizeMaskImage()

Standardizes the mask data as an simple 0 (not masked) and 1 (masked) bitmap.

For FITS files, this is sometimes a trivial no-op operation returning the correct FITS extension. In other cases, the mask has to be constructed from external data, such as pixel masks and catalogs, or downloaded/read from a different file at the data source.

Returns:

masklist[~np.array]

Mask images.

abstract standardizeMetadata()

Standardizes required and optional metadata from the given data.

The standardized metadata is expected to be a dict containing the following keys and values:

Key	Description
location	Path, URL or URI to the data.
mjd	Decimal MJD timestamp of the start of the observation
ra	Right Ascension in ICRS coordinate system of the extracted, or calculated on-sky poisiton of the central pixel, pointing data.
dec	Declination in ICRS coordinate system, expressed in decimal degrees, of the extracted, or calculated, on-sky poisiton of the data.
wcs	Result of ~standardizeWCS, a list of ~astropy.wcs.WCS objects or a list of `None`s if they can not be constructed.
bbox	Result of standardizeBBox, a list of standardized bounding boxes or a list of `None`s if they can not be calculated.

Returns:

metadatadict

Standardized metadata.

Notes

If a target being standardized contains only 1 image that can be standardized and processed by KBMOD, the values in the dictionary can be strings, integers, floats, and other such single-values non-iterable types, or an iterable type containing only 1 element. When the target being standardized contains multiple images/extensions that are processable by KBMOD the single-valued dict entries will be considered as shared by all the processable images/extension/units of the target. The iterable values in the dictionary must match the number and order of the units marked as processable.

abstract standardizePSF()

Returns PSF for each extension marked as processable.

Returns:

psfslist[numpy.ndarray]

List of PSF kernels as numpy matrices.

abstract standardizeScienceImage()

Standardizes the science image data.

For FITS files, this is usually a trivial no-op operation that returns the data attribute of the correct HDU. For different data sources this is a type casting, or a download operation.

Returns:

imagelist[~np.array]

Science images.

abstract standardizeVarianceImage()

Standardizes the variance data.

For FITS files, this is sometimes a trivial no-op operation returning the correct FITS extension. In other cases, this has to be calculated if sufficient information is provided in the header or a different file needs to be downloaded/read.

Note

Refer to the manual for the originating dataset, whether the instrument or processing pipeline reference, as some variance planes store the inverse of variance.

Returns:

variancelist[`np.array]

Variance images.

abstract standardizeWCS()

Creates an WCS for every processable science image.

Returns:

wcslist

List of ~astropy.wcs.WCS objects.

abstract toLayeredImage()

Run metadata standardization methods. These include header and bounding box standardization.

Notes

Implementation is standardizer-specific.

wcs()

A list of WCS’s or None for each entry marked as processable.

Expected to be an WCS object, but when it is not possible to construct one None will be returned in its place.

class kbmod.standardizers.standardizer.StandardizerConfig(config=None, **kwargs)

Base class for Standardizer configuration.

Not all standardizers will (can) use the same parameters so refer to their respective documentation for a more complete list.

Parameters:

configdict, StandardizerConfig or None, optional

Collection of configuration key-value pairs.

kwargsoptional

Keyword arguments, assigned as configuration key-values.

Methods

items()	A set-like object providing a view on config's items.
keys()	A set-like object providing a view on config's keys.
toDict()	Return this config as a dict.
update([conf])	Update this config from dict/other config/iterable.
values()	A set-like object providing a view on config's values.

items()

A set-like object providing a view on config’s items.

keys()

A set-like object providing a view on config’s keys.

toDict()

Return this config as a dict.

update(conf=None, **kwargs)

Update this config from dict/other config/iterable.

A dict-like update. If conf is present and has a .keys() method, then does: for k in conf: this[k] = conf[k]. If conf is present but lacks a .keys() method, then does: for k, v in conf: this[k] = v.

In either case, this is followed by: for k in kwargs:  this[k] = kwargs[k]

values()

A set-like object providing a view on config’s values.