lsst.sims.maf.db package

Submodules

lsst.sims.maf.db.database module

class lsst.sims.maf.db.database.Database(database, driver='sqlite', host=None, port=None, defaultTable=None, longstrings=False, verbose=False)

Bases: lsst.sims.maf.db.dbObj.DBObject

Base class for database access. Implements some basic query functionality and demonstrates API.

Parameters

• database (str) – Name of the database (or full path + filename for sqlite db).

• driver (str, opt) – Dialect+driver for sqlalchemy. Default ‘sqlite’. (other examples, ‘pymssql+mssql’).

• host (str, opt) – Hostname for database. Default None (for sqlite).

• port (int, opt) – Port for database. Default None.

• defaultTable (str, opt) – Default table in the database to query for metric data.

• longstrings (bool, opt) – Flag to convert strings in database to long (1024) or short (256) characters in numpy recarray. Default False (convert to 256 character strings).

• verbose (bool, opt) – Flag for additional output. Default False.

close()

fetchConfig(*args, **kwargs)

Get config (metadata) info on source of data for metric calculation.

fetchMetricData(colnames, sqlconstraint=None, groupBy=None, tableName=None)

Fetch ‘colnames’ from ‘tableName’.

This is basically a thin wrapper around query_columns, but uses the default table. It’s mostly still here for backward compatibility.

Parameters

• colnames (list) – The columns to fetch from the table.

• sqlconstraint (str or None, opt) – The sql constraint to apply to the data (minus “WHERE”). Default None. Examples: to fetch data for the r band filter only, set sqlconstraint to ‘filter = “r”’.

• groupBy (str or None, opt) – The column to group the returned data by. Default (when using summaryTable) is the MJD, otherwise will be None.

• tableName (str or None, opt) – The table to query. The default (None) will use the summary table, set by self.defaultTable.

Returns

A structured array containing the data queried from the database.

Return type

np.recarray

query_arbitrary(sqlQuery, dtype=None)

Simple wrapper around execute_arbitrary for backwards compatibility.

Parameters

• sqlQuery (str) – SQL query.

• dtype (opt, numpy dtype.) – Numpy recarray dtype. If None, then an attempt to determine the dtype will be made. This attempt will fail if there are commas in the data you query.

Returns

Return type

numpy.recarray

query_columns(tablename, colnames=None, sqlconstraint=None, groupBy=None, numLimit=None, chunksize=1000000)

Query a table in the database and return data from colnames in recarray.

Parameters

• tablename (str) – Name of table to query.

• colnames (list of str or None, opt) – Columns from the table to query for. If None, all columns are selected.

• sqlconstraint (str or None, opt) – Constraint to apply to to the query. Default None.

• groupBy (str or None, opt) – Name of column to group by. Default None.

• numLimit (int or None, opt) – Number of records to return. Default no limit.

• chunksize (int, opt) – Query database and convert to recarray in series of chunks of chunksize.

Returns

Return type

numpy.recarray

registry

class lsst.sims.maf.db.database.DatabaseRegistry(name, bases, dict)

Bases: type

Meta class for databases, to build a registry of database classes.

getClass(databasename)

help(doc=False)

lsst.sims.maf.db.dbObj module

class lsst.sims.maf.db.dbObj.DBObject(database=None, driver=None, host=None, port=None, verbose=False, connection=None, cache_connection=True)

Bases: object

execute_arbitrary(query, dtype=None)

Executes an arbitrary query. Returns a recarray of the results.

dtype will be the dtype of the output recarray. If it is None, then the code will guess the datatype and assign generic names to the columns

get_arbitrary_chunk_iterator(query, chunk_size=None, dtype=None)

This wrapper exists so that CatalogDBObjects can refer to get_arbitrary_chunk_iterator and DBObjects can refer to get_chunk_iterator

get_chunk_iterator(query, chunk_size=None, dtype=None)

Take an arbitrary, user-specified query and return a ChunkIterator that executes that query

dtype will tell the ChunkIterator what datatype to expect for this query. This information gets passed to _postprocess_results.

If ‘None’, then _postprocess_results will just guess the datatype and return generic names for the columns.

get_column_names(tableName=None)

Return a list of the names of the columns in the specified table. If no table is specified, return a dict of lists. The dict will be keyed to the table names. The lists will be of the column names in that table

get_table_names()

Return a list of the names of the tables (and views) in the database

lsst.sims.maf.db.opsimDatabase module

lsst.sims.maf.db.opsimDatabase.OpsimDatabase(database, driver='sqlite', host=None, port=None, longstrings=False, verbose=False)

Convenience method to return an appropriate OpsimDatabaseV3/V4 version.

This is here for backwards compatibility, as ‘opsdb = db.OpsimDatabase(dbFile)’ will work as naively expected. However note that OpsimDatabase itself is no longer a class, but a simple method that will attempt to instantiate the correct type of OpsimDatabaseV3 or OpsimDatabaseV4.

class lsst.sims.maf.db.opsimDatabase.OpsimDatabaseFBS(database, driver='sqlite', host=None, port=None, defaultTable='SummaryAllProps', longstrings=False, verbose=False)

Bases: lsst.sims.maf.db.opsimDatabase.BaseOpsimDatabase

Database to class to interact with FBS versions of the opsim outputs.

Parameters

• database (str) – Name of the database or sqlite filename.

• driver (str, opt) – Name of the dialect + driver for sqlalchemy. Default ‘sqlite’.

• host (str, opt) – Name of the database host. Default None (appropriate for sqlite files).

• port (str, opt) – String port number for the database. Default None (appropriate for sqlite files).

• dbTables (dict, opt) – Dictionary of the names of the tables in the database. The dict should be key = table name, value = [table name, primary key].

createSQLWhere(tag, propTags)

Create a SQL constraint to identify observations taken for a particular proposal, using the information in the propTags dictionary.

Parameters

• tag (str) – The name of the proposal for which to create a SQLwhere clause (WFD or DD).

• propTags (dict) – A dictionary of {proposal name : [proposal ids]} This can be created using OpsimDatabase.fetchPropInfo()

Returns

The SQL constraint, such as ‘(proposalID = 1) or (proposalID = 2)’

Return type

str

fetchConfig()

Fetch config data from configTable, match proposal IDs with proposal names and some field data, and do a little manipulation of the data to make it easier to add to the presentation layer.

fetchPropInfo()

There is no inherent proposal information or mapping in the FBS output. An afterburner script does identify which visits may be counted as contributing toward WFD and identifies these as such in the proposalID column (0 = general, 1 = WFD, 2+ = DD).

Returns dictionary of propID / propname, and dictionary of propTag / propID.

class lsst.sims.maf.db.opsimDatabase.OpsimDatabaseV3(database, driver='sqlite', host=None, port=None, defaultTable='Summary', longstrings=False, verbose=False)

Bases: lsst.sims.maf.db.opsimDatabase.BaseOpsimDatabase

createSQLWhere(tag, propTags)

Create a SQL constraint to identify observations taken for a particular proposal, using the information in the propTags dictionary.

Parameters

• tag (str) – The name of the proposal for which to create a SQLwhere clause (WFD or DD).

• propTags (dict) – A dictionary of {proposal name : [proposal ids]} This can be created using OpsimDatabase.fetchPropInfo()

Returns

The SQL constraint, such as ‘(propID = 365) or (propID = 366)’

Return type

str

createSlewConstraint(startTime=None, endTime=None)

Create a SQL constraint for the slew tables (slew activities, slew speeds, slew states) to select slews between startTime and endTime (MJD).

Parameters

• startTime (float or None, opt) – Time of first slew. Default (None) means no constraint on time of first slew.

• endTime (float or None, opt) – Time of last slew. Default (None) means no constraint on time of last slew.

Returns

The SQL constraint, like ‘slewHistory_slewID > XXX and slewHistory_slewID < XXXX’

Return type

str

fetchConfig()

Fetch config data from configTable, match proposal IDs with proposal names and some field data, and do a little manipulation of the data to make it easier to add to the presentation layer.

fetchFieldsFromFieldTable(propId=None, degreesToRadians=True)

Fetch field information (fieldID/RA/Dec) from Field (+Proposal_Field) tables.

propID = the proposal ID (default None), if selecting particular proposal - can be a list degreesToRadians = RA/Dec values are in degrees in the Field table (so convert to radians).

fetchPropInfo()

Fetch the proposal IDs as well as their (short) proposal names and science type tags from the full opsim database. Returns dictionary of propID / propname, and dictionary of propTag / propID. If not using a full database, will return dict of propIDs with empty propnames + empty propTag dict.

fetchRequestedNvisits(propId=None)

Find the requested number of visits for proposals in propId. Returns a dictionary - Nvisits{u/g/r/i/z/y}

class lsst.sims.maf.db.opsimDatabase.OpsimDatabaseV4(database, driver='sqlite', host=None, port=None, defaultTable='SummaryAllProps', longstrings=False, verbose=False)

Bases: lsst.sims.maf.db.opsimDatabase.BaseOpsimDatabase

Database to class to interact with v4 versions of the opsim outputs.

Parameters

• database (str) – Name of the database or sqlite filename.

• driver (str, opt) – Name of the dialect + driver for sqlalchemy. Default ‘sqlite’.

• host (str, opt) – Name of the database host. Default None (appropriate for sqlite files).

• port (str, opt) – String port number for the database. Default None (appropriate for sqlite files).

• dbTables (dict, opt) – Dictionary of the names of the tables in the database. The dict should be key = table name, value = [table name, primary key].

createSQLWhere(tag, propTags)

Create a SQL constraint to identify observations taken for a particular proposal, using the information in the propTags dictionary.

Parameters

• tag (str) – The name of the proposal for which to create a SQLwhere clause (WFD or DD).

• propTags (dict) – A dictionary of {proposal name : [proposal ids]} This can be created using OpsimDatabase.fetchPropInfo()

Returns

The SQL constraint, such as ‘(propID = 365) or (propID = 366)’

Return type

str

createSlewConstraint(startTime=None, endTime=None)

Create a SQL constraint for the slew tables (slew activities, slew speeds, slew states) to select slews between startTime and endTime (MJD).

Parameters

• startTime (float or None, opt) – Time of first slew. Default (None) means no constraint on time of first slew.

• endTime (float or None, opt) – Time of last slew. Default (None) means no constraint on time of last slew.

Returns

The SQL constraint, like ‘slewHistory_slewID between XXX and XXXX’

Return type

str

fetchConfig()

Fetch config data from configTable, match proposal IDs with proposal names and some field data, and do a little manipulation of the data to make it easier to add to the presentation layer.

fetchFieldsFromFieldTable(propId=None, degreesToRadians=True)

Fetch field information (fieldID/RA/Dec) from the Field table.

This will select fields which were requested by a particular proposal or proposals, even if they did not receive any observations.

Parameters

• propId (int or list of ints) – Proposal ID or list of proposal IDs to use to select fields.

• degreesToRadians (bool, opt) – If True, convert degrees in Field table into radians.

Returns

Structured array containing the field data (fieldID, fieldRA, fieldDec).

Return type

np.recarray

fetchPropInfo()

Fetch the proposal IDs as well as their (short) proposal names and science type tags from the full opsim database. Returns dictionary of propID / propname, and dictionary of propTag / propID. If not using a full database, will return dict of propIDs with empty propnames + empty propTag dict.

fetchRequestedNvisits(propId=None)

Find the requested number of visits for the simulation or proposal(s).

Parameters

propId (int or list of ints, opt) –

Returns

Number of visits in u/g/r/i/z/y.

Return type

dict

lsst.sims.maf.db.opsimDatabase.testOpsimVersion(database, driver='sqlite', host=None, port=None)

lsst.sims.maf.db.resultsDb module

class lsst.sims.maf.db.resultsDb.DisplayRow(**kwargs)

Bases: sqlalchemy.ext.declarative.api.Base

Define contents and format of the displays table.

(Table to list the display properties for each metric.)

displayCaption

displayGroup

displayId

displayOrder

displaySubgroup

metric

metricId

class lsst.sims.maf.db.resultsDb.MetricRow(**kwargs)

Bases: sqlalchemy.ext.declarative.api.Base

Define contents and format of metric list table.

(Table to list all metrics, their metadata, and their output data files).

metricDataFile

metricId

metricMetadata

metricName

simDataName

slicerName

sqlConstraint

class lsst.sims.maf.db.resultsDb.PlotRow(**kwargs)

Bases: sqlalchemy.ext.declarative.api.Base

Define contents and format of plot list table.

(Table to list all plots, link them to relevant metrics in MetricList, and provide info on filename).

metric

metricId

plotFile

plotId

plotType

class lsst.sims.maf.db.resultsDb.ResultsDb(outDir=None, database=None, verbose=False)

Bases: object

The ResultsDb is a sqlite database containing information on the metrics run via MAF, the plots created, the display information (such as captions), and any summary statistics output.

close()

Close connection to database.

getAllMetricIds()

Return a list of all metricIds.

getMetricDataFiles(metricId=None)

Get the metric data filenames for all or a single metric. Returns a list.

getMetricDisplayInfo(metricId=None)

Get the contents of the metrics and displays table, together with the ‘basemetricname’ (optionally, for metricId list). Returns a numpy array of the metric information + display information.

One underlying assumption here is that all metrics have some display info. In newer batches, this may not be the case, as the display info gets auto-generated when the metric is plotted.

getMetricId(metricName, slicerName=None, metricMetadata=None, simDataName=None)

Given a metric name and optional slicerName/metricMetadata/simData information, Return a list of the matching metricIds.

getMetricIdLike(metricNameLike=None, slicerNameLike=None, metricMetadataLike=None, simDataName=None)

getMetricInfo(metricId=None)

Get the simple metric info, without display information.

getPlotFiles(metricId=None)

Return the metricId, name, metadata, and all plot info (optionally for metricId list). Returns a numpy array of the metric information + plot file names.

getSummaryStats(metricId=None, summaryName=None, withSimName=False)

Get the summary stats (optionally for metricId list). Optionally, also specify the summary metric name. Returns a numpy array of the metric information + summary statistic information.

updateDisplay(metricId, displayDict, overwrite=True)

Add a row to or update a row in the displays table.

• metricID: the metric Id of this metric in the metrics table

• displayDict: dictionary containing the display info

Replaces existing row with same metricId.

updateMetric(metricName, slicerName, simDataName, sqlConstraint, metricMetadata, metricDataFile)

Add a row to or update a row in the metrics table.

• metricName: the name of the metric

• sliceName: the name of the slicer

• simDataName: the name used to identify the simData

• sqlConstraint: the sql constraint used to select data from the simData

• metricMetadata: the metadata associated with the metric

• metricDatafile: the data file the metric data is stored in

If same metric (same metricName, slicerName, simDataName, sqlConstraint, metadata) already exists, it does nothing.

Returns metricId: the Id number of this metric in the metrics table.

updatePlot(metricId, plotType, plotFile)

Add a row to or update a row in the plot table.

• metricId: the metric Id of this metric in the metrics table

• plotType: the ‘type’ of this plot

• plotFile: the filename of this plot

Remove older rows with the same metricId, plotType and plotFile.

updateSummaryStat(metricId, summaryName, summaryValue)

Add a row to or update a row in the summary statistic table.

• metricId: the metric ID of this metric in the metrics table

• summaryName: the name of this summary statistic

• summaryValue: the value for this summary statistic

Most summary statistics will be a simple name (string) + value (float) pair. For special summary statistics which must return multiple values, the base name can be provided as ‘name’, together with a np recarray as ‘value’, where the recarray also has ‘name’ and ‘value’ columns (and each name/value pair is then saved as a summary statistic associated with this same metricId).

class lsst.sims.maf.db.resultsDb.SummaryStatRow(**kwargs)

Bases: sqlalchemy.ext.declarative.api.Base

Define contents and format of the summary statistics table.

(Table to list and link summary stats to relevant metrics in MetricList, and provide summary stat name, value and potentially a comment).

metric

metricId

statId

summaryName

summaryValue

lsst.sims.maf.db.sdssDatabase module

class lsst.sims.maf.db.sdssDatabase.SdssDatabase(database=None, driver='sqlite', host=None, port=None, dbTables={'clue.dbo.viewStripe82JoinAll': ['viewStripe82JoinAll', 'id']}, defaultdbTables=None, chunksize=1000000, **kwargs)

Bases: lsst.sims.maf.db.database.Database

Connect to the stripe 82 database

fetchMetricData(colnames, sqlconstraint, groupBy=None, cleanNaNs=True, **kwargs)

Get data for metric

lsst.sims.maf.db.trackingDb module

class lsst.sims.maf.db.trackingDb.TrackingDb(database=None, trackingDbverbose=False)

Bases: object

Sqlite database to track MAF output runs and their locations, for showMaf.py

addRun(opsimGroup=None, opsimRun=None, opsimComment=None, opsimVersion=None, opsimDate=None, mafComment=None, mafVersion=None, mafDate=None, mafDir=None, dbFile=None, mafRunId=None)

Add a run to the tracking database.

Parameters

• opsimGroup (str, opt) – Set a name to group this run with (eg. “Tier 1, 2016”).

• opsimRun (str, opt) – Set a name for the opsim run.

• opsimComment (str, opt) – Set a comment describing the opsim run.

• opsimVersion (str, opt) – Set the version of opsim.

• opsimDate (str, opt) – Set the date the opsim run was created.

• mafComment (str, opt) – Set a comment to describe the MAF analysis.

• mafVersion (str, opt) – Set the version of MAF used for analysis.

• mafDate (str, opt) – Set the date the MAF analysis was run.

• mafDir (str, opt) – The relative path to the MAF directory.

• dbFile (str, opt) – The relative path to the Opsim SQLite database file.

• mafRunId (int, opt) – The MafRunID to assign to this record in the database (note this is a primary key!). If this run (ie the mafDir) exists in the database already, this will be ignored.

Returns

The mafRunID stored in the database.

Return type

int

close()

delRun(runId)

Remove a run from the tracking database.

lsst.sims.maf.db.trackingDb.addRunToDatabase(mafDir, trackingDbFile, opsimGroup=None, opsimRun=None, opsimComment=None, mafComment=None, dbFile=None)

Adds information about a MAF analysis run to a MAF tracking database.

Parameters

• mafDir (str) – Path to the directory where the MAF results are located.

• trackingDb (str or lsst.sims.maf.TrackingDb) – Full filename (+path) to the tracking database storing the MAF run information or a TrackingDb object.

• opsimGroup (str, opt) – Name to use to group this run with other opsim runs. Default None.

• opsimRun (str, opt) – Name of the opsim run. If not provided, will attempt to use runName from confSummary.txt.

• opsimComment (str, opt) – Comment about the opsim run. If not provided, will attempt to use runComment from confSummary.txt.

• mafComment (str, opt) – Comment about the MAF analysis. If not provided, no comment will be recorded.

• dbFile (str, opt) – Relative path + name of the opsim database file. If not provided, no location will be recorded.

Module contents