Module:image_collection

Classes for working with the input files for KBMOD.

The ImageCollection class stores additional information for the input FITS files that is used during a variety of analysis.

class kbmod.image_collection.ImageCollection(metadata, standardizers=None, enable_std_caching=True, validate=True)

A collection of metadata extracted by standardizers.

Columns listed in the ImageCollection.required_metadata are required to exist and be non-zero. Additional columns may or may not exist. The names of the standardizers used to extract the metadata are also guaranteed to exist.

Generating the exact Standardizer objects that created the row may or may not be possible, depending on whether the data is accessible to the user.

Avoid constructing this object directly. Use one of the provided factory methods instead. Their behaviour can be modified by supplying a callable as the forceStandardizer argument in order to change the mechanism or modify the extracted data. The provided callable has to be an instance of Standardizer class, see the factory method’s documentation and Standardizer documentation for more information.

Parameters:

metadata~astropy.table.Table

A table of exposure metadata properties

serializerslist

A list of valid serializer names, used to extract the metadata in its row.

enable_lazy_loadingbool

Enable lazy loading of the standardizers, True by default. When enabled, and if possible, a reference to the constructed Standardizer objects that built the table are kept. Further calls to get_standardizer or get_standardizers methods then will not pay the price of the resource acquisition when building the metadata table. When that price is low, f.e. FITS files on a local SSD drive, turning lazy loading off will reduce memory footprint. When data acquisition is large, f.e. a non-local Butler, that space is traded in favor of avoiding the cost of accessing the file.

validatebool

Validate the given metadata during initialization.

Raises:

ValueError

when instantiated from a Table which does not have the required columns, or has null-values in the required columns.

Attributes:

data~astropy.table.Table

Table of exposure metadata and internal metadata book-keeping properties. Should not be directly modified.

_standardizerslist or None

The current list of loaded Standardizer`s. Entry will be none if the standardizer was not previously loaded and lazy loading is enabled. When `None, lazy loading is disabled. Should not be directly modified.

Methods

copy([copy_data])	Return a copy of ImageCollection.
filter_by_mjds(mjds[, time_sep_s])	Filter the visits in the ImageCollection by the given MJDs.
filter_by_time_range([start_mjd, end_mjd])	Filter the ImageCollection by the given time range.
fromBinTableHDU(hdu)	Create an image collection out of a BinTableHDU object.
fromDir(dirpath[, recursive, force, config])	Instantiate ImageInfoSet from a path to a directory containing FITS files.
fromStandardizers(standardizers[, meta])	Create ImageCollection from a collection Standardizers.
fromTargets(tgts[, force, config])	Instantiate a ImageCollection class from a collection of targets recognized by at least one of the standardizers.
get_bbox(idxs)	Get a list of BBOX objects for selected rows.
get_standardizer(index, **kwargs)	Get the standardizer and extension index for the selected row of the unravelled metadata table.
get_standardizers([idxs])	Get the standardizers used to extract metadata of the selected rows.
get_wcs(idxs)	Get a list of WCS objects for selected rows.
get_zero_shifted_times()	Returns a list of timestamps such that the earliest time is treated as 0.
pack()	Identifies columns containing the same repeated value and stores it as a key in the meta attribute.
read(*args[, format, units, descriptions, ...])	Create ImageCollection from a file containing serialized image collection.
reflex_correct(guess_distance, earth_loc)	Generate reflex-corrected coordinates for each image in the ImageCollection.
reflex_corrected_col(col_name, guess_dist)	Get the name of the reflex-corrected column for a given guess distance.
reset_lazy_loading_indices()	Resets the internal index lookup table and standardizer list used for lazy loading to a contiguous array starting at 0.
toBinTableHDU()	Writes the image collection as a BinTableHDU object.
toImageStack()	Return an ~kbmod.search.image_stack object for processing with KBMOD. Returns ------- imageStack : ~kbmod.search.image_stack Image stack for processing with KBMOD.
toWorkUnit([search_config])	Return an ~kbmod.WorkUnit object for processing with KBMOD.
unpack([data])	Unpacks the shared data from meta into columns.
validate()	Validate the metadata table has all the required values and that none of them are false.
vstack(ics)	Stack multiple image collections vertically (along rows) into a new, larger, image collection.
write(*args[, format, serialize_method, ...])	Write the ImageCollection to a file or file-like object.

property bbox

Iterate through BBox of each row.

property columns

Return metadata columns.

copy(copy_data=True)

Return a copy of ImageCollection.

Parameters:

copy_data = True

If True (default) copies the underlying Table data and creates a deep copy of meta attribute.

filter_by_mjds(mjds, time_sep_s=0.001)

Filter the visits in the ImageCollection by the given MJDs. Is performed in-place.

Note that the comparison is made against “mjd_mid”

Parameters:

icImageCollection

The ImageCollection to filter.

timestampslist of floats

List of timestamps to keep.

time_sep_sfloat, optional

The maximum separation in seconds between the timestamps in the ImageCollection and the timestamps to keep.

Returns:

None

filter_by_time_range(start_mjd=None, end_mjd=None)

Filter the ImageCollection by the given time range. Is performed in-place.

Note that it uses the “mjd_mid” column to filter.

Parameters:

start_mjdfloat, optional

The start of the time range in MJD. Optional if end_mjd is provided.

end_mjdfloat, optional

The end of the time range in MJD. Optional if start_mjd is provided.

classmethod fromBinTableHDU(hdu)

Create an image collection out of a BinTableHDU object.

Parameters:

hduBinTableHDU

A fits table object containing the metadata required to make an image collection.

Returns:

icImageCollection

Image collection

classmethod fromDir(dirpath, recursive=False, force=None, config=None, **kwargs)

Instantiate ImageInfoSet from a path to a directory containing FITS files.

Parameters:

dirpathpath-like

Path to a directory containing FITS files.

recursivebool

If the location is a local filesystem directory, scan it recursively including all sub-directories.

forceStandardizer or None

If None, when applicable, determine the correct Standardizer to use automatically. Otherwise force the use of the given Standardizer.

config~StandardizerConfig, dict or None, optional

Standardizer configuration or dictionary containing the config parameters for standardization. When None default values for the appropriate Standardizer will be used.

**kwargsdict

Remaining kwargs, not listed here, are passed onwards to the underlying Standardizer.

classmethod fromStandardizers(standardizers, meta=None)

Create ImageCollection from a collection Standardizers.

The Standardizer is “unravelled”, i.e. the shared metadata is duplicated for each entry marked as processable by the standardizer. On an practical example - MJD timestamps are shared by all 62 science exposures created by DECam in a single exposure, but pointing of each one isn’t. So each pointing is a new row in the metadata table for each of the individual pointings.

Parameters:

standardizersiterable

Collection of Standardizer objects.

Returns:

icImageCollection

Image Collection

classmethod fromTargets(tgts, force=None, config=None, **kwargs)

Instantiate a ImageCollection class from a collection of targets recognized by at least one of the standardizers.

Parameters:

tgtsiterable

Collection of file-paths, a path to a directory, URIs, a butler and dataset ids or reference objects, or any other data targets that are supported by the standardizers.

forceStandardizer, str or None

If None, all available Standardizer`s are tested to find the appropriate one. When multiple `Standardizer`s are found, the one with the highest priority is selected. A name of one of the registered standardizers can be provided. Optionally, provide the `Standardizer class itself in which case it will be called for each target in the iterable.

config~StandardizerConfig, dict or None, optional

Standardizer configuration or dictionary containing the config parameters for standardization. When None default values for the appropriate Standardizer will be used.

**kwargsdict

Remaining keyword arguments are passed to the Standardizer.

Raises:

KeyError:

When a name of a non-registered standardizer is given.

ValueError:

When none of the registered standardizers volunteer to process the given target.

get_bbox(idxs)

Get a list of BBOX objects for selected rows.

Parameters:

idxsint, slice, list[int]

Indices of rows for which to get WCS objects.

Returns:

bboxeslist[BBox]

BBox object for the selected rows.

get_standardizer(index, **kwargs)

Get the standardizer and extension index for the selected row of the unravelled metadata table.

A helper function that allows for access to non-required common properties such as standardizer object, WCS, bounding boxes etc.

Parameters:

indexint

Index, as it appears in the unravelled table of metadata properties.

**kwargsdict

Keyword arguments are passed onto the Standardizer constructor.

Returns:

stddict

A dictionary containing the standardizer (std) and the extension (ext) that maps to the given metadata row index.

get_standardizers(idxs=None, **kwargs)

Get the standardizers used to extract metadata of the selected rows.

Parameters:

idxsint or iterable

Index of the row for which to retrieve the Standardizer.

**kwargsdict

Keyword arguments are passed onto the constructors of the retrieved Standardizer.

Returns:

stdlist`

A list of dictionaries containing the standardizer (std) and the extension (ext) that maps to the given metadata row index.

get_wcs(idxs)

Get a list of WCS objects for selected rows.

Parameters:

idxsint, slice, list[int]

Indices of rows for which to get WCS objects.

Returns:

wcsslist[WCS]

WCS object for the selected rows.

get_zero_shifted_times()

Returns a list of timestamps such that the earliest time is treated as 0.

Returns:

List of floats

A list of zero-shifted times (JD or MJD).

property is_packed

Values shared by all rows are packed as table metadata to save space.

property meta

Image collection metadata.

Contains shared_cols and values when collection is in packed state.

pack()

Identifies columns containing the same repeated value and stores it as a key in the meta attribute.

Lists all the stored keys under the shared_cols value of meta. Reduce the size of the final serialized Table on disk.

classmethod read(*args, format='ascii.ecsv', units=None, descriptions=None, unpack=True, validate=True, **kwargs)

Create ImageCollection from a file containing serialized image collection.

Parameters:

*argstuple, optional

Positional arguments passed through to data reader. If supplied the first argument is the input filename.

formatstr

File format specified, one of AstroPy IO formats that must support comments. Default: ascii.ecsv

unitslist

List or dict of units to apply to columns.

descriptionslist

List or dict of descriptions to apply to columns

unpackbool

If reading a packed image collection, unpack the shared values.

validatebool

Validate that all required metadata exists.

kwargs: `dict`

Other keyword arguments passed onwards to AstroPy’s Table.read.

Returns:

icImageCollection

Image Collection

reflex_correct(guess_distance, earth_loc)

Generate reflex-corrected coordinates for each image in the ImageCollection.

This adds inplace new columns in the ImageCollection with the suffix _{guess_distance} for each coordinate column. If a list of distances is provided, the correction will be applied for each distance in the list.

The helper function reflex_corrected_col is used to generate the new column names.

Parameters:

guess_distancefloat or list

The guess distance in au. If a list is provided, the correction will be applied for each distance in the list.

earth_locEarthLocation

The location of the Earth in which the parallax correction is calculated.

reflex_corrected_col(col_name, guess_dist)

Get the name of the reflex-corrected column for a given guess distance.

These columns may be added by calling ImageCollection.reflex_correct.

Parameters:

guess_distfloat

The guess distance in parsecs.

Returns:

col_namestr

The name of the reflex-corrected column.

reset_lazy_loading_indices()

Resets the internal index lookup table and standardizer list used for lazy loading to a contiguous array starting at 0.

Image collection tracks Standardizers that were used to create the metadata table rows on a per-row basis. Selecting rows or columns from the image collection does not sub-select the requested standardizers or reset these internal counters as often this can rather time-consuming.

Instead, the full list of already loaded standardizers is carried over and original lookup indices remain unchanged. While faster than recalculating the indices at every selection, this can leave a fragmented index lookup table and a longer list of standardizers compared to the number of rows in the table.

Calling this method will reset the index lookup table to a new zero-based contiguous state while trimming all unused lazy-loaded standardizers from the list. Loaded standardizers will not be un-loaded.

In practical use-case the standardizer indices rarely have to be reset because the cost of carrying even few-thousand item long list of None entries carries an insignificant memory footprint. Nominally, the use-case is the situation when creating small, few hundreds rows, image collections from a very large image collection containing >10 000+ rows.

toBinTableHDU()

Writes the image collection as a BinTableHDU object.

If image collection was packed, it is unpacked before the table is created.

Returns:

bintblastropy.io.fits.BinTableHDU

Image collection as a flattened table HDU.

toImageStack()

Return an ~kbmod.search.image_stack object for processing with KBMOD. Returns ——- imageStack : ~kbmod.search.image_stack

Image stack for processing with KBMOD.

toWorkUnit(search_config=None, **kwargs)

Return an ~kbmod.WorkUnit object for processing with KBMOD.

Parameters:

search_config~kbmod.SearchConfiguration or None, optional

Search configuration. Default None.

Returns:

work_unit~kbmod.WorkUnit

A ~kbmod.WorkUnit object for processing with KBMOD.

unpack(data=None)

Unpacks the shared data from meta into columns.

validate()

Validate the metadata table has all the required values and that none of them are false.

Requires all columns in required_cols and _supporting_cols attributes exist.

Returns:

validbool

True if valid

Raises:

ValueError:

When not valid, raises a value error with explanation of condition that wasn’t satisfied.

vstack(ics)

Stack multiple image collections vertically (along rows) into a new, larger, image collection.

Note

Modifies the ImageCollection in place.

Parameters:

icslist[ImageCollection]

List of image collections that will be stacked.

Returns:

icImageCollection

Extended image collection.

property wcs

Iterate through WCS of each row.

write(*args, format='ascii.ecsv', serialize_method=None, pack=True, validate=True, **kwargs)

Write the ImageCollection to a file or file-like object.

A light wrapper around the underlying AstroPy’s Table write functionality. See astropy/io.ascii.write documentation

Parameters:

*argstuple, optional

Positional arguments passed through to data writer. If supplied the first argument is the output filename.

formatstr

File format specified, one of AstroPy IO formats that must support comments. Default: ascii.ecsv

serialize_methodstr, dict, optional

Serialization method specifier for columns.

packbool

Pack the values shared by all rows into the table metadata.

validatebool

Validate that all required metadata exists before writing it.

kwargs: `dict`

Other keyword arguments passed onwards to AstroPy’s Table.write.