Module: butler_standardizer

Class for standardizing Data Products of Vera C. Rubin Science Pipelines via the Rubin Data Butler.

class kbmod.standardizers.butler_standardizer.ButlerStandardizer(dataId, butler, config=None, **kwargs)

Standardizer for Vera C. Rubin Data Products, namely the underlying Exposure objects of calexp, difim and various *coadd dataset types.

This standardizer will volunteer to process DatasetRef or DatasetId objects. The Rubin Data Butler is expected to be provided at instantiation time, in addition to the target we want to standardize.

Parameters:

tgtlsst.daf.butler.core.DatasetId, lsst.daf.butler.core.DatasetRef or int

Target to standardize.

butlerlsst.daf.butler.Butler

Vera C. Rubin Data Butler.

configStandardizerConfig, dict or None, optional

Configuration key-values used when standardizing the file.

Attributes:

butlerlsst.daf.butler.Butler

Vera C. Rubin Data Butler.

reflsst.daf.butler.core.DatasetRef

Dataset reference to the given target

explsst.afw.image.exposure.Exposure

The Exposure object targeted by the ref

processablelist[lsst.afw.image.exposure.Exposure]

Items marked as processable by the standardizer. See Standardizer.

Methods

canStandardize(tgt)	Returns True when the standardizer knows how to process the given upload (file) type.
get(tgt[, force, config])	Get the standardizer class that can handle given file.
resolveTarget(tgt)	Returns a pair of values (canStandardize, resources).
standardize()	Invokes standardize metadata, image, variance, mask and PSF and returns the results as a dictionary.
standardizeBBox()	Calculate the standardized bounding box, the world coordinates at image corner and image center.
standardizeMaskImage()	Standardizes the mask data as an simple 0 (not masked) and 1 (masked) bitmap.
standardizeMetadata()	Standardizes required and optional metadata from the given data.
standardizePSF()	Returns PSF for each extension marked as processable.
standardizeScienceImage()	Standardizes the science image data.
standardizeVarianceImage()	Standardizes the variance data.
standardizeWCS()	Creates an WCS for every processable science image.
toLayeredImage()	Run metadata standardization methods.

configClass

property bbox

A list of bounding boxes or None for each entry marked as processable.

Key	Description
center_ra	On-sky (ICRS) right ascension coordinate, expressed in decimal degrees, of the center of an rectangular bounding box.
center_dec	On-sky (ICRS) declination coordinate, expressed in decimal degrees, of the center of an rectangular bounding box.
corner_ra	On-sky (ICRS) right ascension coordinate, expressed in decimal degrees, of an corner of a rectangular bounding box.
corner_dec	On-sky (ICRS) declination coordinate, expressed in decimal degrees, of an corner of a rectangular bounding box.

Often, when WCS can not be constructed, the BBOX can not be constructed. Then None is expected to be returned in it’s place.

configClass

alias of ButlerStandardizerConfig

name = 'ButlerStandardizer'

Processor’s name. Only named standardizers will be registered.

priority = 2

Priority. Standardizers with high priority are prefered over standardizers with low priority when processing an upload.

classmethod resolveTarget(tgt)

Returns a pair of values (canStandardize, resources).

The first value, canStandardize, indicates that the standardizer is able to standardize the given target. The second value, resources, is an optional returned value, a dictionary containing any constructed or resolved resources in the process.

This method is called during automatic resolution of standardizers. In that process, each registered Standardizer is asked to resolve the target. The standardizers, and their resources if any, that successfully resolved the target are sorted based on the Standardizer priority and used to standardize the target.

Because each Standardizer is asked to resolve each target, of which there are potentially many, it is encouraged that this method is implemented with performance in mind.

Sometimes, however, it may not be possible to avoid acquiring or constructing a resource that, in the given context, could be considered expensive. Returning the resource(s) allows that cost to be optimized away, by avoiding acquiring or constructing the resource again during initialization.

On a practical example: FitsStandardizer standardize FITS files pointed to by a local filesystem path (i.e. the target). To confirm their ability to standardize the target, they have to make sure they can open and read the file. They use AstroPy to construct an ~astropy.io.fits.HDUList, i.e. the resource. Returning the already constructed ~astropy.io.fits.HDUList allows FitsStandardizer to skip constructing a new one via astropy.io.fits.open.

Parameters:

tgtdata-source

Some data source, URL, URI, POSIX filepath, Rubin Data Butler Data Repository, an Python object, callable, etc.

Returns:

canStandardizebool

True when the processor knows how to handle uploaded file and False otherwise.

resourcesdict, optional

Any optional resources constructed during target resolution. An empty dictionary if there are none.

standardizeBBox()

Calculate the standardized bounding box, the world coordinates at image corner and image center.

Returns:

standardizedBBoxdict

Calculated coordinate values, a dict with, wcs_center_[ra, dec] and wcs_corner_[ra, dec] keys.

Notes

Center can be assumed to be at the (dimX/2, dimY/2) pixel coordinates, rounded down. Corner is taken to be the (0,0)-th pixel.

standardizeMaskImage()

Standardizes the mask data as an simple 0 (not masked) and 1 (masked) bitmap.

For FITS files, this is sometimes a trivial no-op operation returning the correct FITS extension. In other cases, the mask has to be constructed from external data, such as pixel masks and catalogs, or downloaded/read from a different file at the data source.

Returns:

masklist[~np.array]

Mask images.

standardizeMetadata()

Standardizes required and optional metadata from the given data.

The standardized metadata is expected to be a dict containing the following keys and values:

Key	Description
location	Path, URL or URI to the data.
mjd	Decimal MJD timestamp of the start of the observation
ra	Right Ascension in ICRS coordinate system of the extracted, or calculated on-sky poisiton of the central pixel, pointing data.
dec	Declination in ICRS coordinate system, expressed in decimal degrees, of the extracted, or calculated, on-sky poisiton of the data.
wcs	Result of ~standardizeWCS, a list of ~astropy.wcs.WCS objects or a list of `None`s if they can not be constructed.
bbox	Result of standardizeBBox, a list of standardized bounding boxes or a list of `None`s if they can not be calculated.

Returns:

metadatadict

Standardized metadata.

Notes

If a target being standardized contains only 1 image that can be standardized and processed by KBMOD, the values in the dictionary can be strings, integers, floats, and other such single-values non-iterable types, or an iterable type containing only 1 element. When the target being standardized contains multiple images/extensions that are processable by KBMOD the single-valued dict entries will be considered as shared by all the processable images/extension/units of the target. The iterable values in the dictionary must match the number and order of the units marked as processable.

standardizePSF()

Returns PSF for each extension marked as processable.

Returns:

psfslist[numpy.ndarray]

List of PSF kernels as numpy matrices.

standardizeScienceImage()

Standardizes the science image data.

For FITS files, this is usually a trivial no-op operation that returns the data attribute of the correct HDU. For different data sources this is a type casting, or a download operation.

Returns:

imagelist[~np.array]

Science images.

standardizeVarianceImage()

Standardizes the variance data.

For FITS files, this is sometimes a trivial no-op operation returning the correct FITS extension. In other cases, this has to be calculated if sufficient information is provided in the header or a different file needs to be downloaded/read.

Note

Refer to the manual for the originating dataset, whether the instrument or processing pipeline reference, as some variance planes store the inverse of variance.

Returns:

variancelist[`np.array]

Variance images.

standardizeWCS()

Creates an WCS for every processable science image.

Returns:

wcslist

List of ~astropy.wcs.WCS objects.

toLayeredImage()

Run metadata standardization methods. These include header and bounding box standardization.

Notes

Implementation is standardizer-specific.

property wcs

A list of WCS’s or None for each entry marked as processable.

Expected to be an WCS object, but when it is not possible to construct one None will be returned in its place.

class kbmod.standardizers.butler_standardizer.ButlerStandardizerConfig(config=None, **kwargs)

Methods

items()	A set-like object providing a view on config's items.
keys()	A set-like object providing a view on config's keys.
toDict()	Return this config as a dict.
update([conf])	Update this config from dict/other config/iterable.
values()	A set-like object providing a view on config's values.

brightness_treshold = 10

Pixels with value greater than this threshold will be masked.

do_bitmask = True

Mask mask_flags from the mask plane in the FITS file.

do_mask = True

Perform masking if True, otherwise return an empty mask.

do_threshold = False

Mask all pixels above the given count threshold.

greedy_export = False

If True, the standardizer will keep the Exposure object in memory after the LayeredImage is created. This is useful for large datasets.

grow_kernel_shape = (10, 10)

Size of the symmetric square kernel by which mask footprints will be increased by.

grow_mask = True

Grow mask footprint by grow_kernel_shape

mask_flags = ['BAD', 'CLIPPED', 'CR', 'CROSSTALK', 'EDGE', 'NO_DATA', 'SAT', 'SENSOR_EDGE', 'SUSPECT']

List of flags that will be masked.

psf_std = 1

Standard deviation of the Point Spread Function.

standardize_effective_summary_stats = False

Include the “effective” fit metrics from SummaryStats

standardize_metadata = True

Fetch and include values from Rubin’s Exposure.metadata PropertyList such as ‘OBSID’, ‘AIRMASS’, ‘DIMM2SEE’ ec. Typically corresponds to ingested header values, ergo the capitalized column names.

standardize_uri = False

Include an URL-like path to to the file

zero_point = 31

Photometric zero point to which all the science and variance will be scaled to.